# Explainable Security Requirements Classification Through Transformer Models
## 3 The Method
### Data Augmentation  
As described in the previous section and as visible in Figure 4, the number of requirements for security (166) and non-security categories (331) are unbalanced. This data type could lead to a model that may perform poorly on underrepresented classes. This imbalance can result in biased predictions, as the model favors the majority class. To address this issue, data augmentation techniques can be employed to enhance the representation of minority classes. In this work, we relied on three approaches: word augmentation, back translation, and paraphrasing. Regarding word augmentation, this technique generally consists of generating new random variations of words or phrases in a text. However, in this work, we applied a strategy that relies on a pre-trained BERT model [18] to generate contextual embeddings and add for each requirement three new sentences by randomly adding, removing, or substituting words in the sentences. Using this approach, the model identifies a target word and suggests alternatives based on the surrounding context, ensuring the replacement maintains the original sentence's semantic integrity. This process enriches the training dataset and helps us mitigate issues related to class imbalance. Instead, back translation is an augmentation technique used in natural language processing (NLP) that involves translating a sentence from the source to a target language and then back to the original language. In this phase, using Google Translate ([[https://translate.google.com/](https://translate.google.com/)]([https://translate.google.com/](https://translate.google.com/)), accessed on 18 October 2024), we translate each sentence in the training dataset to a random language and translate it back to English. Finally, paraphrasing involves generating alternative versions of a given text while preserving its original meaning. In this process, we used three different Large Language Models to generate three different versions from the starting requirements. The first of these three models is based on the T5 (Text-to-Text Transfer Transformer) [19] architecture, which treats every NLP task as a text-to-text problem. The T5 model is pre-trained on a diverse range of tasks and can generate paraphrases by understanding the context and semantics of the input text. It excels in generating coherent and contextually relevant paraphrases. The second involved model is GPT-Neo [20], an open-source alternative to OpenAI's GPT-3 and, with its 2.7 billion parameter version, can generate human-like text based on the input it receives. It can perform various tasks, including text generation, completion, and paraphrasing. Its large size allows it to capture complex language patterns, making it practical for generating diverse and creative para \begin{table}
\begin{tabular}{l c c} \hline \hline
**Text** & **Label** \\ \hline The GlobalPlatform API shall provides services to the application such as card verification and security services & sec \\ Send transactions to other processors in a standard format defined in & nonsec \\ agreement with the scheme provider & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Text examples of labeled security and non-security requirements. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Type** & \
\begin{tabular}{c} **No. of Security** \\ **Requirements** \\ \end{tabular} &
\begin{tabular}{c} **No. of Non-Security** \\ **Requirements** \\ \end{tabular} & **Total** \\ \hline Train & 166 & 331 & 497 \\ Test & 21 & 41 & 62 \\ Validation & 21 & 43 & 64 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The dataset’s statistics show the number of security and non-security requirements for each set and the total number of elements. phrases. Finally, Pegasus [21] is a model specifically designed for text generation tasks, particularly summarization and paraphrasing. It uses a transformer architecture and is pre-trained on a large text corpus.