# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 2 Literature Review
### Transformer Models for Attack Detection
Leveraging CNNs' ability to extract local features enables rapid and effective pattern identification, especially in large, complex datasets like network traffic. [PERSON] et al. [43] proposed a CNN-Transformer model to capture spatial relationships and patterns in more detail, enhancing feature representation across different data domains. Similarly, a CNN-Transformer hybrid model, combining dense CNNs and Transformers, has been applied for feature fusion and time series extraction [44]. Another study utilized this model to address RNNs' challenges with information loss, computational complexity, and detection performance [45]. Observing the model's strength in capturing spatial and local features, researchers experimented with adding LSTM layers or replacing CNN with LSTM to improve temporal relationship modeling, facilitating deep analysis and sequential dependencies in time series data. [PERSON] and [PERSON] [46] introduced a hybrid DdosTC network, combining Transformer and CNN architectures to detect Distributed Denial of Service (DDoS) attacks. Similarly, the XTM model, a Transformer-based approach, was developed for real-time data breach detection, achieving high performance on the IEEE-14 bus system and advancing smart grid security [47]. [PERSON] et al. [48] proposed an attack detection model that leverages multi-frequency LSTM and Transformer modules to analyze temporal details across high- and low-frequency data, enhancing detection precision. Vision Transformer (ViT) models have significantly contributed to attack detection applications. Unlike CNNs, which analyze images hierarchically by focusing on local features, ViTs process images as patch arrays and leverage self-attention to capture global relationships. While ViTs require larger datasets and more computational resources than CNNs, they excel in capturing global context. Ref. [49] highlights the advantages of ViT models on visual data compared to traditional data. [PERSON] et al. [50] proposed an architecture integrating a feature fusion network with ViT, improving the model's capability to manage imbalanced datasets and decreasing the required training sample size. In another study,network data streams were transformed into images by mapping them to RGB values [51]. After initial analysis with a decision tree algorithm, these images were categorized using a ViT classifier. Similarly, another approach converted network traffic data from packet capture (PCAP) files into grayscale images for malware classification using ViT techniques [52]. The effectiveness of ViT was evaluated by comparing its performance with that of CNNs, highlighting the strengths of ViT in this context. Classification performance in attack detection suffers due to imbalanced training data and limited feature extraction. To increase minority class examples and address the imbalance problem in attack data, a new method for multi-class attack detection that combines a conditional generative adversarial network (CGAN) and a pre-trained language model, BERT, was presented in [53]. BERT is integrated with CGAN to improve the relationship between input and output for robust feature extraction. Similarly, [PERSON] et al. [54] integrated CGAN with ViT to enhance detection accuracy in imbalanced network flow data. In addition to the hybrid models examined, the authors have proposed new Transformer-based attack detection frameworks by changing the components of the Transformer architecture. [PERSON] et al. [55] introduced the Robust Transformer-based Intrusion Detection System (RTIDS), designed to reconstruct feature representations, achieving a balance between dimensionality reduction and feature preservation in imbalanced datasets. [PERSON] et al. [56] proposed a model combining BERT and deep learning techniques to detect HTTP attacks. [PERSON] [57] developed an attack detection model based on LightGBM and Transformers to enhance system security in the healthcare domain.