# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 2 Literature Review
### Transformer Models for Attack Detection  
With significant advancements in NLP and LLMs, Transformers have expanded their reach into many research areas due to its advanced text generation and user interaction capabilities. One area that has significantly benefited from these advancements is cybersecurity [34]. Transformer models have emerged as powerful tools in attack detection thanks to their ability to capture complex patterns and relationships in data through attention mechanisms. These models can learn benign patterns found in sequential sequences, such as network traffic, and can successfully detect changes in network traffic patterns even when the number of attacks is small [35]. In an attack detection study that utilizes the attention mechanism, CNN and GRU were combined with an attention mechanism to avoid SQL injection and cross-site scripting \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Type** & \begin{tabular}{c} **Pre-Training** \\ **Architecture** \\ \end{tabular} & **Applications** \\ \hline Code-T5 & Encoder–Decoder & Encoders and Decoders & \begin{tabular}{c} Generating software \\ code, code completion, \\ error detection, \\ comment generation \\ \end{tabular} \\ BART & Encoder–Decoder & Encoders and Decoders & \begin{tabular}{c} Text enhancement, \\ text summarization, \\ translation, Q\&A \\ Pair conversions, \\ multi-language \\ summarization, \\ multi-language \\ translation \\ \end{tabular} \\ PLBART & Encoder–Decoder & Encoders and Decoders & \
\begin{tabular}{c} Error detection, code \\ commenting, natural \\ language generation, \\ text summarization \\ \end{tabular} \\ BERT2 GPT & Encoder–Decoder & Encoders and Decoders &
\begin{tabular}{c} Text completion, Q\&A, \\ dialogue generation, \\ text summarization \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: _Cont._(XSS) attacks [36]. The proposed approach leveraged an attention mechanism to attain high accuracy, minimize the need for extensive training data, and shorten the training duration. [PERSON] et al. [37] proposed the Residual 1-D Image Transformer (R1 DIT) model to tackle challenges related to privacy and generalization in malware traffic classification. The proposed model parses network headers while preserving sensitive data and uses DL and Multi-Head Attention (MHA) mechanisms to distinguish between malware and benign traffic. [PERSON] et al. [38] introduced a Transformer-based attention network (TAN) as a multi-class intrusion detection system for the in-car Control Area Network (CAN) bus protocol. The proposed model eliminates the limitations of RNN-based methods and can classify attacks without needing message labeling using sequential CAN ID data. In another study based on the Transformer attention mechanism, the proposed Transformer-based model was compared with Tree-based models, and it was observed that the Transformer-based intrusion detection model performed better [39]. One approach to address Deep Neural Network (DNN) limitations combined feature embedding and self-attention modules of the Transformer model with fully connected layers [40]. The proposed model gave better results compared to the Decision Tree (DT) algorithm. Another study introduced an MHA-based Transformer that enhanced intrusion detection robustness compared to LSTM and RNN models [41]. [PERSON] et al. [42] introduced a Denseformer model that integrates the self-attention mechanism with the cross-attention mechanism and includes encoders and decoders. Due to the Denseformer's cross-fusion structure, feature correlations were identified more effectively. Finformer, on the other hand, identifies attacks in financial data by integrating the Transformer architecture with a cluster-attention mechanism [13]. In addition to the attention mechanism provided by Transformer models, the CNN-Transformer hybrid model has also emerged as a powerful tool in intrusion detection.