# Explainable Security Requirements Classification Through Transformer Models
## 5 Discussion  
Information concerning model choice and augmentation techniques and overall classification effectiveness in security and non-security requirements were gained from the experiments conducted in this work, which involved the full fine-tuning and the SetFit optimization approaches. Augmentation techniques were substantial in improving the performance of the models. The baseline of the original dataset confirmed that RoBERTa and BERT models could achieve high accuracy without augmentation. One appealing feature of RoBERTa was that it outperformed its counterpart, meaning the model correctly understands the context in most raw data. On the other hand, the benefits of augmentation become more pronounced when examining the modified datasets. Word replacement augmentation altered meanings lexically at a quite minimal level by substituting, adding, or eliminating words using contextual embeddings. Following and implementing this technique improved BERT's performance drastically, mainly after SetFit had been applied, where high accuracy, precision, and recall had been obtained. With its capacity for strong token-level representations, it seems to hint that BERT can cope with variations at the word level and thus can take advantage of this augmentation's robustness. Word level augmentation seems to improve the ability of the model to generalize by providing it with minor language alterations that are crucial, allowing the model to recognize requirements across slightly modified scenarios. On the contrary, back translation involves translating sentences to another language and back, which tends to introduce syntactic changes without altering the original meaning. The slightly lower scores mean that although [PERSON] can handle syntactic shifts in a different language or structure, this type of alteration is more difficult for him than word-level changes. This indicates that sentence-level alterations may add another complexity layer, necessitating more standardized representations. Nonetheless,  
Figure 15: Heatmap for the third non-security requirement, showing the attention visualization of the last layer (sixth layer) of the DistilBERT-base-uncased model fine-tuned in the dataset obtained by the paraphrase augmentation. BERT's solid performance with back-translation highlights its adaptability to sentence rephrasings that do not fundamentally alter meaning. The dataset obtained through paraphrasing using pre-trained LLMs has posed a different difficulty as it requires restructuring a sentence without changing its meaning. In this particular instance, DistilBERT was accurate, although its recall was inferior, which suggested that it had greater difficulty comprehending all pertinent security requirements under strong restatement. This pattern suggests that while DistilBERT can deal with a certain level of semantic change, it is possible that heavy changes are not best suited for it due to its architecture size. Using SetFit optimization, DistilBERT performed relatively high in accuracy and precision, confirming that the combination of fine-tuning and SetFit can help smaller models generalize across varied paraphrases, even though recall remains a slight concern. Regarding explainability, the attention heatmaps of the final layer for each model and dataset show the explainability findings, which provide light on how these models prioritize and interpret data from various augmentation methods. Thanks to each heatmap's visual breakdown of word-level attention, we can examine how each model interprets and distinguishes security-related phrases in various input settings. RoBERTa showed a focused attention pattern on important security-related phrases, such as \"security\" and \"measures\", for the normal dataset, suggesting that it correctly concentrates on the terms that best reflect security needs. RoBERTa's excellent accuracy and F1 scores on the NR dataset are consistent with the attention heatmap's robust and focused attention.