# Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model
## 5 Results and Discussion
### Performance Comparison for All Models  
As shown in Figure 10, we presented the training and validation accuracy and the loss values for all models during the training phase. The best validation accuracy was \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Architecture**} & **LR** & **Accuracy** & **F1 Score** \\ \hline \multirow{3}{*}{**1 LSTM**} & 1 Dense & \(1\times 10^{-4}\) & 0.873 & 0.864 \\  & **2 Dense** & \(4\times 10^{-5}\) & **0.867** & **0.830** \\  & 3 Dense & \(5\times 10^{-5}\) & 0.871 & 0.859 \\ \hline \multirow{3}{*}{2 LSTM} & 1 Dense & \(2\times 10^{-4}\) & 0.871 & 0.860 \\  & 2 Dense & \(5\times 10^{-5}\) & 0.872 & 0.862 \\  & 3 Dense & \(4\times 10^{-5}\) & 0.866 & 0.855 \\ \hline \multirow{3}{*}{3 LSTM} & 1 Dense & \(2\times 10^{-4}\) & 0.871 & 0.862 \\  & 2 Dense & \(2\times 10^{-3}\) & 0.865 & 0.853 \\ \cline{1-1}  & 3 Dense & \(5\times 10^{-3}\) & 0.861 & 0.851 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Comparing LSTM-based classifier results using the BERT tokenizer. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Architecture**} & **LR** & **Accuracy** & **F1 Score** \\ \hline \multirow{3}{*}{**1 BiLSTM**} & **1 Dense** & \(1\times 10^{-5}\) & **0.88** & **0.87** \\  & 2 Dense & \(5\times 10^{-5}\) & 0.87 & 0.86 \\  & 3 Dense & \(1\times 10^{-4}\) & 0.88 & 0.87 \\ \hline \multirow{3}{*}{2 BiLSTM} & 1 Dense & \(5\times 10^{-4}\) & 0.87 & 0.86 \\  & 2 Dense & \(4\times 10^{-5}\) & 0.87 & 0.86 \\  & 3 Dense & \(2\times 10^{-5}\) & 0.87 & 0.85 \\ \hline \multirow{3}{*}{3 BiLSTM} & 1 Dense & \(2\times 10^{-5}\) & 0.87 & 0.86 \\  & 2 Dense & \(4\times 10^{-6}\) & 0.86 & 0.85 \\ \cline{1-1}  & 3 Dense & \(4\times 10^{-5}\) & 0.87 & 0.86 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Comparing BiLSTM-based classifier results using the BERT tokenizer. achieved by CyBERT, proposed by our NLP model, with a 95.4% accuracy. The loss plots indicate the effectiveness of the learning rate function we used for initializing all models. Additionally, the loss plots show the effectiveness of the early-stopping method to avoid overfitting in cases where the validation loss increases during the training phase. Table 12 compares the accuracy for each model from our test set. The learning rates we report in this table indicate the best LR we found for each model based on the architecture and dataset we provided earlier. The learning rate was found via the LrFinder method and fine-tuned during the training phase. Binary classification algorithms are often evaluated using receiver operating characteristic (ROC) curves. Instead of a single value, it provides a two-dimensional illustration of classifier performance. ROC plots the false-positive rates against the true-positive rates in classification. For the ideal classifier scenario, we want to observe a high true-positive rate and a low false-positive rate [49]. Figure 11 compares the ROC curve for all the models we tested in this study. The classifier was desired to be closer the upper-left corner of the ROC curve plot. Figure 11 illustrates that our CyBERT classifier performed better in all areas compared to all other models. Figure 10: Comparing training accuracy (**left top**), validation accuracy (**right top**), training Loss (**bottom left**), and validation loss (**bottom right**) for all models. The other important parameter for a classifier is the \"area under the ROC curve\" (AUC). Generally, the higher the AUC score, the better a classifier performs for the given task [49]. The AUC is reported for all models in Table 12 and Figure 11, which indicate that CyBERT had the highest AUC value among all tested models. Figure 10 shows that the best value for AUC belongs to CyBERT, whichwasis 0.948. This indicates that CyBERT is better at distinguishing classes than any other models we compared it against in this study. The next-highest AUC values belonged to the ELMo models, which were all at least three percentage points lower than CyBERT's AUC.