# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 2 Literature Review
### Background of Transformer Models
OpenAI GPT is one of the autoregressive Transformer models, and its pre-training architecture consists of decoders [25]. The model training goal is to predict the following words, given all previous words in the text. The various versions of GPT differ primarily in the number of decoder layers used [26], GPT-1 uses 12 decoders, GPT-2 uses 48 decoders, and GPT-3 uses 96 decoders. These increases in decoder layers allow the model to produce more consistent and contextually rich text output. The pre-training architecture of encoder-decoder Transformers includes both encoders and decoders. This architecture allows the learning of both input data (encoder) and output sequences (decoder). The encoder analyzes the input's complex language structure and context, while the decoder uses this information to produce a new text sequence. These models are particularly good at language translation, summarization, text completion, and error correction tasks. Encoder-decoder models perform strongly in tasks requiring sequential data processing because they generate the output by considering the entire input. The Text-To-Text Transfer Transformer (T5) encoder-decoder model aims to formulate all tasks as a text-to-text transformation problem by addressing language processing tasks in a unifying framework [27]. It becomes usable in various NLP tasks by always presenting input and output as text strings. The T5 model aims to provide a flexible structure for different tasks (e.g., language modeling, translation, summarization, classification, Q&A). During T5's pre-training, some distortions are applied to the input text strings. The T5 model is trained by correcting these distortions. In this way, the model improves its understanding of the broader structure and context of the language. The T5 has been used for translation, text summarization, text classification, question-and-answer, and other language processing tasks and has demonstrated high performance in many NLP benchmarks [28]. Code-T5 is introduced as an optimized version of T5 [29]. It focuses on understanding and generating software code, specifically for programming languages' tasks such as code completion, error detection, and comment generation. Another encoder-decoder Transformer model, the Bidirectional and Autoregressive Transformer (BART), offers a more powerful and flexible structure by combining forward and backward information flow [30]. BART is mainly used in text corruption and reconstruction tasks, meaning that it learns to reconstruct the input text by corrupting it (for example, by removing words or changing the order of sentences). Therefore, BART is preferred in tasks such as text enhancement, text summarization, translation, and Q&A. Multilingual BART (mBART) is derived from the BART model and focuses on multilingual NLP tasks [31]. While the BART model is trained only in a single language, mBART offers a structure trained on parallel data in multiple lan guages and supports cross-language transfer. The critical difference between mBART and BART is that mBART can overcome language barriers and learn patterns in multilingual texts, thus enabling cross-language transfer learning. mBART combines text decomposition and reconstruction capabilities with multi-language support, demonstrating high performance in language pair conversions, multi-language summarization, and multi-language translation tasks. Programming Language BART (PLBART) is a version of the BART model adapted to programming languages and is designed to understand and generate software code [32]. While BART is optimized for general NLP tasks, PLBART is explicitly used for code completion, error detection, code commenting, and natural language generation from code. It bridges the gap between natural language and programming language. PLBART uses BART's encoder-decoder architecture to process natural and programming language text. Another encoder-decoder Transformer model, BERT2 GPT, combines BERT's powerful language understanding capabilities with GPT's language generation capabilities [33].