## 3 Results
### Classification of Generative AI Models
#### 3.2.6 Transformers  
The transformer model has revolutionized the field of natural language processing (NLP) by replacing traditional recurrent neural networks (RNNs) with a self-attention mechanism. This model has achieved state-of-the-art performance on various language tasks while being computationally efficient and highly parallelizable. The core component of the transformer model is the self-attention mechanism, which allows the model to focus on different parts of the input sequence simultaneously when making predictions. Unlike RNNs that process sequential information step by step, the transformer considers the entire input sequence at once, effectively capturing dependencies between tokens [74]. Transformer architecture consists of an encoder and a decoder, both comprising multiple layers of self-attention and feed-forward neural networks. The encoder processes the input  
Figure 9: Recurrent Neural Network Architecture. sequence, while the decoder generates the output sequence. The self-attention mechanism in the transformer enables the model to selectively attend to relevant parts of the input sequence, facilitating the capture of long-range dependencies and improving translation quality, among other tasks. The attention module in the transformer adopts a multi-head design. The self-attention is formulated as a scaled dot-product [2], where the input queries (Q), keys (K), and values (V) are combined to calculate the attention weights. The scaling factor of \(\sqrt{d_{k}}\) is applied to normalize the dot-product scores. The resulting attention weights are then multiplied with the values and summed up to produce the final output. \[\text{Attention}(\text{Q},\text{K},\text{V})= \text{softmax}(\text{QK}^{\text{T}}/\sqrt{\text{d}_{\text{k}}}) \text{V}\]  
The transformer model employs multiple layers of self-attention and fully connected point-wise layers in both the encoder and decoder components illustrated in Figure 10. This architecture allows the model to effectively capture and process the complex relationships and dependencies within the input and output sequences. Transformers vary in their architectures, specific network designs, and training objectives depending on the application and input data. _BERT (Bidirectional Encoder Representations from Transformers):_ BERT consists of a multi-layer bidirectional transformer encoder. It employs a masked language modeling (MLM) objective during pre-training. It randomly masks words in the input text and trains the model to predict the masked words based on their context [110]. BERT also uses a next sentence prediction (NSP) task, where it learns to predict if two sentences are consecutive in each document. BERT is pre-trained on a large corpus of text, such as Wikipedia and Book Corpus. It utilizes unsupervised learning and large-scale transformer architectures to capture general language representations. After pre-training, BERT can be fine-tuned on specific downstream tasks using supervised learning with task-specific datasets. _GPT (Generative Pre-trained Transformer):_ GPT employs a multi-layer transformer decoder. GPT is trained using an autoregressive language modeling objective [111]. It predicts the next word in a sequence based on the previous context, enabling the generation of fluent and contextually relevant text. GPT is pre-trained on a large corpus of text, such  
Figure 10: Transformer architecture (Figure obtained from [74]). as web pages and books. It learns to generate text by conditioning on the preceding context. Fine-tuning of GPT can be performed on specific tasks by providing task-specific prompts or additional training data. _T5 (Text-to-Text Transfer Transformer):_ T5 employs a transformer architecture like BERT but follows a text-to-text framework [112]. It can handle various NLP tasks using a unified approach. T5 is trained using a text-to-text format, where both input and output are text strings. It leverages a combination of unsupervised and supervised learning objectives for pre-training and fine-tuning.