# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 2 Literature Review
### Background of Transformer Models  
The Transformer model was first described in 2017 as a simple and innovative network architecture that eliminates recurrent and convolutional networks and relies solely on attention mechanisms [15]. These architectures are highly effective in natural language processing (NLP) tasks because they can capture long-range dependencies and relationships between elements in sequences without needing prior domain knowledge or feature engineering. [PERSON]'s essential Transformer model includes both encoder and decoder modules. The encoder processes the input sequence to create representations, and the decoder generates the output sequence based on these representations. This dual structure effectively allows the model to handle various tasks, from translation to text generation. The outstanding success of the Transformer model in NLP tasks has led to the emergence of various versions of this modeling, which have increased its applicability to different tasks. These LLM-based models can be classified as auto-encoding, encoder-decoder, and autoregressive according to the differences in their pre-training architectures [16]. The pre-training architecture of auto-encoding Transformers consists of encoders only. These models effectively capture contextual information and learn rich representations of input data by reconstructing missing or masked portions. This allows them to excel in tasks like text classification, sentiment analysis, and other downstream tasks where understanding the full context of the input is crucial. The Bidirectional Encoding Representation Transformer (BERT) is one of the automatic encoding Transformer models [17]. BERT stands out with its ability to train language models bidirectionally. Unlike the one-way models developed in the OpenAI technical report [18], BERT extracts information from both aspects of the context around words using Masked Language Modeling (MLM). Then, it performs the sentence-matching task with Next Sentence Prediction (NSP). This bidirectional pre-training approach makes language representations richer and more meaningful because the left and right contexts are evaluated together. Thus, unlike unidirectional models such as OpenAI GPT, BERT improves the task-specific fine-tuning phase by performing bidirectional pre-training. BioBERT is the first BERT-based pre-trained model for biomedical data [19]. The pre-training phase starts with weights taken from BERT and improves performance on named entity recognition (NER), relation extraction (RE), and question-answer (QA) tasks. RoBERTa is a retrained BERT version with more data and computational power. It removes the NSP task and uses dynamic masking [20]. ALBERTuses factorial embedding and cross-layer parameter sharing to reduce the parameters in BERT [21]. This way, it has 18 times fewer parameters than BERT and trains 1.7 times faster while showing similar performance [22]. DeBERTa improves on previous models with disaggregated attention and an enhanced mask decoder [23]. Instead of combining word and location information as in BERT, decoupled attention encodes content and location separately for each input token. This way, the model can calculate attention weights based on the token's content and location. The improved mask decoder allows the use of absolute locations of markers when making MLM predictions, together with location-based attention. ELECTRA is another auto-encoding model that uses alternative tokens produced by a generator to replace tokens in a sequence [24]. This model has a discriminative model that predicts whether the generator has modified each token in the corrupted input. The pre-training architecture of autoregressive Transformers consists of decoders only. These models effectively generate data sequences by predicting the next element in a sequence based on the previously generated elements, making them particularly suitable for tasks like language modeling, text generation, and dialogue systems, where producing coherent and contextually relevant outputs is essential.