# Intrusion Detection Model Based on Improved Transformer
## 3 Materials and Methods
### Improved Transformer  
A Transformer was initially applied to natural language processing tasks, abandoning the traditional recurrent neural network (RNN) and convolutional neural network (CNN) structures and solely relying on the attention mechanism to perform machine translation tasks, achieving excellent results. Compared to RNN-based sequential neural networks, the Transformer is superior. RNN training is iterative and sequential, resulting in particularly lengthy training times. In contrast, Transformer training is parallel, allowing all features to be trained simultaneously, dramatically increasing computational efficiency and reducing model training time. Therefore, in this paper, a Transformer was used as the base model to learn and extract features, thereby accelerating the model training speed.  
The Transformer is composed of an encoder and a decoder, but for network intrusion detection tasks that do not require decoding, unlike sequence-to-sequence tasks such as machine translation, only the encoder is needed to learn and extract features, which can be combined with softmax for classification. The Transformer classification structure is shown in Figure 4. The Transformer encoder consists of two sub-layers: multi-head attention mechanism and feed-forward neural network, with residual modules and normalization modules in each sub-layer. The multi-head attention mechanism allows the model to focus on different aspects of information, producing multiple subspaces that attend to different aspects of information, thus enhancing the model's performance.  
Improved Location Encoding  
The Transformer model first encodes the input with a position encoding. The calculation formulas are shown as follows:  
\[PE_{(pos,2i)}=\sin(pos/10,000^{2i/d_{x}}) \tag{6}\]  
\[PE_{(pos,2i+1)}=\cos(pos/10,000^{2i/d_{x}}) \tag{7}\]  
In the equation, the variable \(pos\) represents the position of a word within a text sequence sample, which is a value ranging from 0 to the maximum sequence length. \(d_{x}\) denotes the dimension of the text encoding, while \(i\) is the index of a word within the encoding vector, ranging from 0 to \(d_{x}\). The location embedding function has a period that varies from 2\(\pi\) to 10,000 \(\times\) 2\(\pi\), and each location in the encoding dimension is assigned a different combination of values of the sine and cosine functions with varying periods. This method generates distinct texture location data, which allows the model to capture the relationships between positions and the temporal features of natural language. By incorporating such information into the model, it can better capture the semantic meaning and structure of the text data, ultimately enhancing its performance in various natural language processing tasks.  
In the context of natural language processing, machine translation involves encoding text-based input features. However, in intrusion detection tasks, the input features are typically numeric and do not require encoding. Figure 5 illustrates the fundamental difference between these two tasks. Encoding numeric features may alter the inherent information in the data and affect the amount of information that can be extracted by the model. Consequently, the model's ability to learn effectively from the features can be diminished, thereby reducing the classification performance.  
Figure 4: Transformer classification structure.  
datasets are independent, their features are often correlated. By enhancing the position encoding to include feature position information, the Transformer model can capture the positional dependencies between sample features, which increases the information available to the model and enhances its classification performance. The proposed method is therefore more appropriate for intrusion detection tasks, where feature correlations play an important role in identifying anomalies.