# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 4 Results  
This section evaluates the FPE-Transformer model using the CIC-IDS2017 and NSL-KDD datasets. The experiments were conducted in the Google Colab environment, equipped with a 2.9 GHz CPU, 12.68 GB of RAM, and 107.72 GB of disk space. Python version 3.10.12 and the PyTorch framework were utilized throughout the experimentation process. Accuracy and loss were employed as the primary evaluation metrics, with accuracy calculated as described in Equation (9). \[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} \tag{9}\]  
True Positive (_TP_) refers to attack data the model predicts as an attack. True Negative (_TN_) refers to the benign data the model predicts as benign. False Positive (_FP_) refers to benign data that are predicted as an attack. False Negative (_FN_) refers to the benign prediction of attack data. Binary cross entropy loss is used as the loss function, as in Equation (10). \[\text{Loss}=-\frac{1}{N}\sum_{i=1}^{N}\Bigl{[}y_{i}\log(\hat{y_{i}})+(1-y_{i}) \log(1-\hat{y_{i}})\Bigr{]} \tag{10}\]  
N refers total sample numbers, \(y_{i}\) represents the true label of the \(i_{th}\) sample, and \(\hat{y_{i}}\) represents the probability that the model predicts for the \(i_{th}\) sample. The formula calculates the logarithmic loss between the true label and the predicted probability for each example and then averages these losses across all examples. This measures how close the model's predictions are to the true values, allowing the model to be guided to improve accuracy. The experiments were conducted systematically to evaluate the effectiveness of different positional encoding methods on the CIC-IDS2017 and NSL-KDD datasets. For the NSL-KDD dataset, three positional encoding techniques--original positional encoding (PE) [15], uniform positional encoding (UPE) [58], and the proposed feature positional encoding (FPE)--were compared under two classification tasks: binary classification and multi-class classification. Similarly, for the CIC-IDS2017 dataset, the same three positional encoding methods were tested across binary and multi-class classification scenarios. The hyperparameters of all Transformer models with FPE, UPE, and PE were set in the same way to produce comparable results. The models had one encoder layer (n_layer = 1) and two attention heads (n_head = 2). To prevent over-learning, the dropout rate was set to 10% (drop_prob = 0.1), and training was performed with a batch size of 100 samples each time (batch_size = 100). The number of epochs was set to 20. The Adam optimizer was employed as the optimization function, configuring the learning rate to 0.001. The comparison of NSL-KDD binary classification test accuracy and loss values for all three positional encodings is shown in Figure 6. The red line represents FPE, the blue line represents UPE, and the green line represents PE. FPE performed better than other positional encodings in the binary classification task performed on the CIC-IDS2017 dataset. The comparison of NSL-KDD multi-class classification test accuracy and loss values for all three positional encodings is shown in Figure 8. The red line represents FPE, the blue line represents UPE, and the green line represents PE. The comparison of NSL-KDD multi-class classification test accuracy and loss values for all three positional encodings is shown in Table 9.  
FPE performed better than other positional encodings in the multi-class classification task performed on the NSL-KDD dataset. The comparison of multi-class classification on CIC-IDS2017 dataset for all three positional encodings is shown in Figure 9. The red line represents FPE, the blue line represents UPE, and the green line represents PE. The comparison of CIC-IDS2017 multi-class classification test accuracy and loss values for all three positional encodings is shown in Table 10.  
FPE performed better than other positional encodings in the multi-class classification task performed on the NSL-KDD dataset.