# Explainable Security Requirements Classification Through Transformer Models
## 4 Results
### Fine-Tuning Results  
Table 4 summarizes the hyperparameters used to fine-tune the models across the four datasets employed and based on the different augmentation techniques. We use the same hyperparameters to ensure that any differences in the model's performance can be attributed to the data rather than variations in the training process. Learning rate, batch size, and number of epochs can significantly affect model performance, and then using the same can also make it easier to reproduce results. For this reason, we fine-tuned the models in all the experiments using ten epochs with an initial learning rate of 2e\(-\)5. Starting with the normal dataset, it can be seen that both BERT and DistilBERT attain high accuracy scores of 0.89, with XLNet and RoBERTa coming in second and third, respectively, at 0.90. This consistency shows that all models perform well even when data augmentation is not used, indicating a solid baseline performance. BERT and DistilBERT balance precision and recall, but DistilBERT performs marginally better in recall (0.86 vs. 0.76), indicating that it could catch more pertinent examples. However, the RoBERTa model achieves higher scores in all the calculated metrics, suggesting it effectively captures the correct classifications when working with the base dataset. It is possible thanks to RoBERTa's more modern architecture, which improves attention mechanisms over classic BERT. Regarding the dataset obtained using contextual word augmentation, BERT retains great precision and recall while achieving a higher accuracy (0.94) than the original. This improvement shows how flexible BERT is to small contextual and lexical changes brought about by embedding-based augmentation. However, RoBERTa drops significantly \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Learning Rate** & **Weight Decay** & **Batch Size** & **Epochs** \\ \hline BERT-base-uncased & 2e\(-\)5 & 0.01 & 8 & 10 \\ DistilBERT-base-uncased & 2e\(-\)5 & 0.01 & 8 & 10 \\ RoBERTa-base & 2e\(-\)5 & 0.01 & 8 & 10 \\ xnet-base-cased & 2e\(-\)5 & 0.01 & 8 & 10 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Summary of the hyperparameters used to train the presented models. These parameters have been used for all the four versions of the dataset. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \multicolumn{5}{c}{**Normal dataset**} \\ \hline BERT-based-uncased & 0.89 & 0.89 & 0.76 & 0.82 \\ DistilBERT-base-uncased & 0.89 & 0.89 & 0.86 & 0.82 \\ RoBERTa-base & 0.9 & 0.86 & 0.86 & 0.86 \\ xnet-base-uncased & 0.9 & 0.89 & 0.81 & 0.85 \\ \hline \multicolumn{5}{c}{**Contextual word augmented dataset**} \\ \hline BERT-based-uncased & 0.94 & 0.9 & 0.9 & 0.9 \\ DistilBERT-base-uncased & 0.89 & 0.85 & 0.81 & 0.83 \\ RoBERTa-base & 0.84 & 0.92 & 0.57 & 0.71 \\ xnet-base-uncased & 0.92 & 0.94 & 0.81 & 0.87 \\ \hline \multicolumn{5}{c}{**Back translation augmented dataset**} \\ \hline BERT-based-uncased & 0.92 & 0.9 & 0.96 & 0.88 \\ DistilBERT-base-uncased & 0.92 & 0.9 & 0.96 & 0.88 \\ RoBERTa-base & 0.84 & 0.87 & 0.62 & 0.71 \\ xnet-base-uncased & 0.9 & 0.83 & 0.9 & 0.86 \\ \hline \multicolumn{5}{c}{**Paraphrasing augmented dataset**} \\ \hline BERT-based-uncased & 0.9 & 0.86 & 0.86 & 0.86 \\ DistilBERT-base-uncased & 0.94 & 0.9 & 0.9 & 0.9 \\ RoBERTa-base & 0.87 & 0.88 & 0.71 & 0.79 \\ xnet-base-uncased & 0.89 & 0.82 & 0.86 & 0.84 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results of the fine-tuned models in terms of accuracy, precision, recall, and F1-score on the four datasets. regarding the recall (0.57). It could suggest that the model may be sensitive to embedding augmentation, which could interfere with its internal representations to classify data. XLNet, in contrast, continues to perform well, suggesting that its bidirectionality may enable it to incorporate enriched settings more effectively than RoBERTa.