# Explainable Security Requirements Classification Through Transformer Models
## 3 The Method
### Models
#### 3.3.1 Fine-Tuning  
Transformer models are advanced artificial intelligence systems [4; 22; 23; 24] designed to understand, generate, and manipulate human language. They are built on deep learning architectures, especially transformer models, for text data collection and analysis. Transformers are particularly popular since they quickly execute various tasks, such as text classification, sentiment analysis, translation, text summarization, and many others. Transformers can be particularly powerful in text classification tasks because they model the contextual relations given words or phrases much better than traditional approaches. This is mainly because knowledge is derived from a lot of diverse text data, which makes these models adapt very quickly to new tasks with only a slight modification. Fine-tuning them means adapting them to specific domains and enhancing their performance on relevant tasks and terminology. This process improves accuracy and relevance in predictions and responses while also helping to reduce biases by exposing the model to more diverse and representative data. In this work, we fine-tuned four models: BERT, DistilBERT, RoBERTa, and XLNET.  
BERT (Bidirectional Encoder Representations from Transformers) [18] is a revolutionary model that popularized the use of the bidirectional training of transformers. Specifically, in contrast to constitutive models, which tackle the text in one direction only (either left-to-right or right-to-left), BERT takes into account both contexts at once. This leads to a more comprehensive grasp of the languages and interconnections between words. Indeed, BERT is trained on a large text corpus using two tasks, masked language modeling and next-sentence prediction. The model is now a state-of-the-art language representation model and was successfully applied to different text classification tasks.  
DistilBERT [25] is a smaller, faster, and lighter version of BERT, created via a process called knowledge distillation. It retains 97% of BERT's language understanding capabilities, yet it is 60% faster and requires less memory to run. This makes DistilBERT a very enticing option when working with applications with scarce computation resources or if a real-time  
Figure 5: Size of the three different training datasets after the word augmentation, back translation, and paraphrasing process.  
response is necessary. While smaller, DistilBERT performs competitively on many NLP tasks, including text classification.  
RoBERTa (A Robustly Optimized BERT Pretraining Approach) [26] represents an advance from BERT. Researchers at Facebook AI developed it to improve the capabilities of the BERT model by modifying its architecture and training methodology. This was conducted by removing the next sentence prediction objective and training on a larger dataset with longer sequences. It also uses dynamic masking during training, making learning contextual representation possible. These further improvements enable RoBERTa to establish state-of-the-art performance on many benchmarks, including GLUE, RACE, and SQuAD, as well as substantial improvements in text classification tasks. By better capturing context and relationships within the text, RoBERTa improves accuracy and effectiveness in categorizing and understanding complex textual data, outperforming its predecessor, BERT.  
XLNet [27] is a pre-trained model that effectively incorporates the strengths of BERT and autoregressive models like GPT by leveraging bidirectional context and permutation of input sequences. This allows XLNet to capture all possible word orders, which provides a much finer understanding of word relationships in flexible ways. This flexibility, in particular, makes the model capture the meaning and contextual dependencies much better when working with text classification tasks, providing much higher accuracy and performance in text categorization.