# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 3 Materials and Methods
### Attack Detection Framework
Figure 2 analyzes a subset of the NSL-KDD dataset, revealing that the src_bytes feature exhibits rapid changes, while the same_srv_rate feature displays periodic behavior from 0 to 1. Similarly, Figure 3 analyzes a subset of the CIC-IDS2017 dataset, revealing that the Fwd Packet Length Max feature exhibits rapid changes, while the Flow AIT Std feature exhibits periodic behavior from 0 to 4. Encoding these differences allows the model to correctly capture the context of each feature. Without considering timing patterns, applying the same positional encoding to all features can lead to the loss of critical temporal relationships within the data. Figure 2: Feature analysis of a subset of the NSL-KDD dataset, showing rapid changes in src_bytes and periodic behavior in same_srv_rate. Our _feature positional encoding_ removes the temporal pattern problem by independently computing positional encodings for each feature and capturing its temporal dynamics. To capture the temporal dynamics of each feature \(f\), \(div\_term_{f}\) is used and calculated as in Equation (4). \[div\_term_{f}=\exp\biggl{(}-\frac{\log_{10}10000}{d_{model}}\times 2i\biggr{)} \tag{4}\]  
The \(d_{model}\) feature determines the size of positional encoding matrices and represents the matrix index \(i\). The number of \(div\_term_{f}\) is set to be the same as the number of features. \(\log_{10}10000\) is used for scaling purposes in feature positional encoding. The natural exponential function (exp) converts a negative value to a positive one, ensuring the resulting value is always positive. Negative values can negatively affect the learning process of the model in deep learning and optimization processes. Using positive values allows the model to learn faster and more stably. Linear functions preserve the linear effects of the data and are often suitable for simpler problems. However, changes in time series data are often non-linear. Linear functions do not always provide accurate modeling of continuous changes, and transitions are often irregular and sharp. Ref. [65] explains different transformation methods in time series analysis and shows how exponential functions provide more effective results. The study provides experimental results supporting the effectiveness of exponential functions in better modeling continuously changing data compared to other  
Figure 3: Feature analysis of a subset of the CIC-IDS2017 dataset, showing rapid changes in Fwd Packet Length Max and periodic behavior in Flow AIT Std. methods, such as linear functions. In time series data, the exponential function emphasizes the effect of time-varying data. Exponential functions provide smoother transitions between data because they change continuously and smoothly, which can help the model generalize better. Positional encoding \(PE_{f}\) in the \(div\_term_{f}\) determined for each feature \(f\) is calculated as in Equation (5). \[ \begin{split} PE_{f}(pos,2i)&=\sin(pos\times div \_term_{f})\\ PE_{f}(pos,2i+1)&=\cos(pos\times div\_term_{f}) \end{split}  \tag{5}\]  
Here, a separate positional encoding is calculated for each feature. Pos represents the time steps within each feature. Feature positional encoding calculates positional information for each feature independently, allowing the model to learn the order and context of each feature over time separately. The proposed method provides a significant advantage when working with multi-feature data, where different features may exhibit different timing patterns. Figure 4 visualizes the difference between feature positional encoding and positional encoding methods customized for numeric data. An _encoder_ is a critical component for the model to process and encode the input data meaningfully. Input data enriched with feature positional encoding are processed in the encoder using three essential substructures: MHA, feed-forward network, and add&norm. MHA divides an input feature into parts called heads and performs attention calculations for each head separately. The outputs of these heads are combined to obtain a final output.