# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 3 Materials and Methods
### Attack Detection Framework
The scale dot product is calculated for each head as in Equation (6). \[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{model}}} \right)V \tag{6}\]  
Here, \(Q\), \(K\), and \(V\) are vectors that calculate the attention weights obtained for each feature. \(Q\) (Query) represents the query vector for the feature the relevant head represents. This is used to emphasize which information is important. Key (\(K\)) represents the fundamental keys related to the feature the relevant head represents. Value (\(V\)) represents the actual information values carried by the feature. A multiplication operation is performed between the vectors \(Q\) and \(K\), and each query vector's relationship to the corresponding key vectors is calculated. The multiplication result is divided by \(\sqrt{d_{model}}\) to scale the large values that can result from large matrix multiplications. Softmax ensures each feature focuses on a specific value, ensuring their sum is one and obtaining attention weights. The resulting attention weights are multiplied by the \(V\) (Value) vectors. This determines  
Figure 4: Feature positional encoding vs. uniform positional encoding. which information the model should pay more attention to. The results obtained from each head are combined as in Equation (7) and then passed through a linear layer. The MHA mechanism is visualized in Figure 5.  
\[ \begin{split}\text{Multi-head Attention}(Q,K,V)&=\text{ Concat}(\text{head}_{1},\dots,\text{head}_{n})W^{Q}\\ \text{head}_{i}&=\text{Attention}(QW_{i}^{Q},KW_{i}^ {K},VW_{i}^{V})\end{split}  \tag{7}\]  
The MHA mechanism identifies important features in attack detection tasks and learns the interactions between these features. Therefore, it enables the simultaneous capture of various input elements representing different types of cyberattacks. For example, although the number of bytes is critical in a DoS attack, key certificate details can be more decisive in ransomware attacks. By learning these contextual differences, MHA increases its capacity to identify and classify different types of attacks. These features make the MHA mechanism an effective tool for attack detection. Therefore, various deep learning layers have been combined with the MHA mechanism, and powerful hybrid models have been created for attack detection [38; 40; 41]. In addition, the experimental results obtained with our FPE-Transformer model incorporating the MHA mechanism on the CIC-IDS2017 and NSL-KDD datasets demonstrate that the MHA mechanism suits the attack detection task. The Add & Norm layer performs the normalization operations by adding the attention weights resulting from MHA to the input data. Normalization is calculated as in Equation (8). \[\hat{x}_{i}=\frac{x_{i}-\mu}{\sqrt{\sigma^{2}+\epsilon}} \tag{8}\]  
Here, \(x_{i}\) represents the \(i_{th}\) input, \(\mu\) represent the mean value, and \(\sigma\) represents the variance. The feed-forward function is used for forward passing of the encoder layer. _Classification head_ contains one or more fully connected layers and uses learnable weights to summarize better and classify the information in the encoder's final output. The encoder output is passed through one or more linear layers to extract higher-level features. Finally, softmax is applied to these outputs to perform the final classification. This approach allows the encoder output to gain deeper meaning. The classification head allows the model to learn more complex features by adding additional structures (e.g., linear layers and activation functions). Using activation functions such as ReLu with linear layers supports the model's learning of non-linear relationships. Figure 5: Multi-head attention mechanism. Applying softmax directly to the encoder output instead of the classification head is a sufficient approach for simple classification. However, this method may limit the model's ability to learn complex features.