# FPE-Transformer: A Feature Positional Encoding-Based Transformer Model for Attack Detection
## 3 Materials and Methods
### Attack Detection Framework  
Transformer-based attack detection models proposed in previous studies have focused on attention mechanisms and hybrid models [34]. FPE-Transformer takes a different path from these approaches by offering an innovative feature position encoder module. As seen in Figure 1, the FPE-Transformer model consists of feature positional encoding, encoder, pooling, and classification head layers. Figure 1: FPE–Transformer. Positional encoding adds the positional information of the input data as a fixed vector of size \(d_{model}\) through the \(\sin\) and \(\cos\) functions and is calculated as in Equation (2) [15]. \[ \begin{split} PE_{(pos,2i)}&=\sin\left(\frac{pos}{10000 ^{\frac{2i}{d_{model}}}}\right)\\ PE_{(pos,2i+1)}&=\cos\left(\frac{pos}{10000^{ \frac{2i}{d_{model}}}}\right)\end{split}  \tag{2}\]  
The variable \(pos\) represents a word's position within the text string, which varies between 0 and the maximum string length. \(d_{model}\) is the vector dimension in which each word is expressed. \(i\) represents the index in the \(d_{model}\) vector. Sin and \(\cos\) functions have complementary properties, allowing the model to distinguish between words in different positions. The vector matrices' indexes are expressed as \(2i\) to clarify this distinction. Thus, \(\sin\) and \(\cos\) waves become more distinguishable. The \(\sin\) and \(\cos\) functions have a phase difference. The \(\sin\) function starts from zero and continues increasing. The \(\cos\) function starts from a maximum and decreases. This phase difference differentiates the \(i\) in even and odd positions. Periodicity in the \(\sin\) and \(\cos\) functions helps the model learn positional information even in long sequences. \(10000^{\frac{2i}{d_{model}}}\) controls the frequency of positional encoding and creates a different frequency scale for each dimension. Notations of Equation (2) are as in Table 6.  
In summary, positional encoding represents the sequential relationships between words in text data with \(\sin\) and \(\cos\) functions. Thus, it better captures semantic structures and grammatical relationships. [PERSON] and [PERSON] [58] suggested that in tasks based on numerical features, such as attack detection, using positional information could disrupt the intrinsic structure of the data due to the independence of the features, potentially negatively impacting the model's performance. For this reason, the authors updated the positional encoding method used in text data as in Equation (3) for numerical data. \[ \begin{split} PE_{(pos,2i)}&=\sin\left(\frac{1}{1000 0^{\frac{2i}{d_{model}}}}\right)\\ PE_{(pos,2i+1)}&=\cos\left(\frac{1}{10000^{ \frac{2i}{d_{model}}}}\right)\end{split}  \tag{3}\]  
This method aims to capture sequential information in numerical data more accurately using only the features' positional indices (\(i\)). However, a fixed positional encoding is applied to each feature and does not consider the unique timing patterns that different features may exhibit. Temporal pattern refers to the changes or frequency of information that features in a dataset show over time. \begin{table}
\begin{tabular}{l l} \hline \hline
**Notations** & **Description** \\ \hline \(pos\) & Position index in the sequence. \\ \(2i\) & Index used for positional encoding matrix. \\ \(d_{model}\) & Dimensionality of the positional encoding matrix. \\  & It refers to the even-numbered indices of the positional encoding matrices \\ \(PE_{(pos,2i)}\) & in the dimension \(d_{model}\). \\  & It refers to the odd-numbered indices of the positional encoding matrices \\ \(PE_{(pos,2i+1)}\) & in the dimension \(d_{model}\). \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Notations of Equation (2). In multi-feature datasets, the timing of each feature and the relationships between these features play a critical role in the model's learning process [64]. For example, one feature might exhibit rapid changes, while another might display periodic fluctuations over time.