# Explainable Security Requirements Classification Through Transformer Models
## 4 Results
### Fine-Tuning Results
In the context of the dataset obtained through the back translation process, again, BERT and DistilBERT exhibit great precision and recall (scoring 0.96 in recall), and both achieve an accuracy of 0.92. This implies that the dataset is successfully diversified by back translation while retaining the essential semantic information required for these models to function correctly. Nevertheless, XLNet maintains its stability with a strong recall of 0.90 and an F1-score of 0.86, whereas RoBERTa has another performance decline, obtaining a lower recall (0.62). Finally, BERT and DistilBERT continue to perform well on the dataset that contains paraphrased sentences. This implies that both BERT-based models successfully reflect the variety in phrasing brought forth by paraphrasing techniques. RoBERTa may have trouble with the increased lexical diversity that paraphrasing introduces, as evidenced by its lower recall (0.71). This can be because while RoBERTa excels at token-level contextual relationships, paraphrasing disrupts sentence structures and alters key terms, introducing linguistic variability that the model struggles to generalize. It performs strongly on datasets that maintain consistent syntax and vocabulary, as its masked language modeling objective allows it to capture relationships between tokens in highly contextualized settings. However, paraphrasing introduces variations that RoBERTa may not generalize well if its attention has been overfitted to specific patterns during fine-tuning. With an accuracy of 0.89 and a balance between precision and recall, XLNet does rather well, showing that it can handle paraphrasing more effectively than RoBERTa but with more variability than BERT. Overall, BERT and DistilBERT exhibit steady flexibility across a range of datasets, with DistilBERT proving effective even in more challenging situations brought about by augmented data. Even while RoBERTa is effective in the standard dataset, it exhibits considerable sensitivity to augmentation, which may indicate that more fine-tuning is required to maximize its utility with extremely diverse datasets. XLNet's performance is competitive and stable but shows a different augmentation resilience than BERT, mainly when dealing with higher lexical diversity. To evaluate the impact of dataset augmentation on the performance, we created the confusion matrix of the BERT model in the standard dataset and the contextual word augmentation and compared them. As can be seen in Figure 5(a), in this matrix, the model correctly identified 39 non-security requirements (True Negatives) and 16 security requirements (True Positives). However, it misclassified two non-security requirements as security (False Positives) and failed to identify five security requirements (False Negatives). In contrast, the confusion matrix (Figure 5(b)) for the model trained on the augmented dataset maintained the same True Negatives (39) and False Positives (2) as the original model, indicating consistent performance in identifying non-security requirements. The augmented model significantly reduced the False Negatives to 2 and increased the True Positives to 19. This improvement suggests that the augmentation process enhanced the model's ability to classify security requirements accurately. Overall, the results indicate that contextual word embeddings for dataset augmentation positively influenced the model's performance, leading to a more effective classification of security-related requirements.