# Explainable Security Requirements Classification Through Transformer Models
## 4 Results
### Explainability Results
Figure 7: Heatmap showing the attention visualization of the last layer (12 th layer) of the Roberta-base model fine-tuned in the original dataset. the data slightly differently depending on the augmentation in the dataset. In this heatmap, BERT seeks to give equal attention to the most significant information and the context in translation that may cause a change in the structure. Most importantly, such a network can remain relevant while losing a few pertinent features, such as translation. BERT's attention dispersion, which is most likely able to alter classification and generalize, proves to be why BERT performs exceptionally well with a strong recall and F1 score on this dataset. Lastly, for the paraphrased dataset, the DistilBERT attention heatmap (Figure 10) shows important information about how the model understands security requirements. It is evident from this heatmap that DistilBERT gives specific attention to phrases such as \"applications\", \"perform\", and \"internal\". This model acknowledges the significance of these phrases in communicating the primary intent, as they are essential to the security need. In addition, since on average this model performs better than the other models, we compare the attention heatmaps of three different security and three non-security requirements to understand the choices made by this model to perform the predictions. The attention heatmap in Figure 11 highlights distinct patterns where specific key terms draw more focus. Indeed, the terms \"responsible\" and \"generating\" have high attention, while \"generating\" receives the most attention. That would mean the model focuses on action-oriented terms, which are crucial to the meaning of the security requirement. The model focuses on words that are semantically responsible for carrying the meaning of responsibility and operational tasks. Further, \"loading\" obtains an appreciable degree of attention; it is apparently because it describes the action with \"generating\". In the last security requirement, as represented in Figure 12, \"applications\", \"expose\", and \"only\" are words that attract the highest attention values. The word \"applications\" has consistent emphasis, especially at the beginning of the sentence, which will likely reflect its role as a main subject. The term \"expose\" captures significant attention, suggesting that the model correctly focuses on the key action described. The adverb \"only\" similarly attracts robust attention, showing the crucial restricting of the exposure scope regarding the meaning of the requirement. Figure 9: Heatmap showing the attention visualization of the last layer (12 th layer) of the BERT-base-uncased model fine-tuned in the dataset obtained by the back-translation augmentation process. Figure 11: Heatmap for the second security requirement showing the attention visualization of the last layer (sixth layer) of the DistilBERT-base-uncased model, fine-tuned in the dataset obtained by the paraphrase augmentation process. Figure 10: Heatmap for the first security requirement showing the attention visualization of the last layer (sixth layer) of the DistilBERT-base-uncased model, fine-tuned in the dataset obtained by the paraphrase augmentation process. Considering the non-security requirements outlined in Table 7, Figure 13 depicts that the terms \"authentication,\" \"routing,\" and \"connectivity\" are relatively more prominent in the sentence. This is expected because these terms carry technical connotations. On the other hand, no word or term seems to have the highest concentration, so the attention is still spread across the sentence. For example, terms like \"authentication\" do not receive sufficient attention or relevance to security. Such a distribution reinforces the classification as a non-security requirement as no vital security-related keywords are identified.