## 2 Materials and Methods
### Transformer  
The transformer model is a DL architecture originally designed for sequence modeling tasks. It abandons the traditional recurrent structures and instead leverages a self-attention mechanism that enables the model to capture complex dependencies across all elements in a sequence simultaneously. This capability is particularly beneficial in contexts such as intrusion detection for IoT environments, where network traffic exhibits temporal patterns and long-range correlations that must be effectively captured and modeled [32; 33; 34]. The standard transformer is composed of two main components: an encoder and a decoder, as illustrated in Figure 1.  
The encoder is responsible for ingesting the input sequence and producing a contextualized representation, while the decoder consumes this representation to generate the output sequence or class prediction. The interaction between the encoder and decoder is illustrated by the arrows in Figure 1. These arrows indicate the data flow and dependency between layers. For instance, the arrow connecting the encoder to the decoder represents the transmission of encoded context to inform output generation. The internal arrows within each block show the residual pathways, layer stacking, and flow of tensor transformations. The repeated stacking of encoder and decoder blocks, typically 6 to 12 layers deep, enhances the model's expressiveness and capacity to learn hierarchical representations [35; 36]. The standard transformer model offers a powerful mechanism for learning from structured sequences without relying on recurrence or convolution. Its attention-driven framework enables parallelization and effective modeling of complex temporal dependencies, making it particularly suitable for analyzing traffic data in IoT environments. On the left-hand side, the encoder begins by transforming the input tokens into dense vector embedding through an embedding layer. To encode the positional information, which is not inherently captured due to the lack of recurrence, positional encodings are  
Figure 1: The architectural workflow of the transformer. added to the embedding. These encodings are computed using fixed sinusoidal functions, where each position \(pos\) and dimension \(i\) of the embedding are defined as shown in Equations (1) and (2) [37]:  
\[PE_{(pos,2i)}=\sin\left(\frac{\mathbf{pos}}{1000^{2i/d}}\right), \tag{1}\]  
\[PE_{(pos,2i+1)}=\cos\left(\frac{\mathbf{pos}}{1000^{2i/d}}\right) \tag{2}\]  
This positional information enables the model to distinguish the order of elements within the sequence. Once positional encodings are added, the sequence enters the encoder block, which consists of a stack of identical layers, each with two core components: multi-head self-attention and a feed-forward neural network (FFNN). In the first sub-layer, the model performs self-attention, where each token attends to every other token in the sequence. This operation computes a weighted sum of value vectors based on the similarity between query and key vectors, as formalized in Equation (3) [38]:  
\[Attention\ (\mathbf{Q},\ \mathbf{K},\ \mathbf{V})=softmax\left(\frac{\mathbf{Q}K^{T}}{\sqrt{d_{ K}}}\right)\mathbf{V} \tag{3}\]  
The vectors \(\mathbf{Q},\ \mathbf{K},\ \mathbf{V}\) are linear projections of the input sequence, and the scaling factor \(\sqrt{d_{K}}\) stabilizes gradients during training. Rather than relying on a single attention operation, the transformer employs multi-head attention to allow the model to jointly attend to information from different representation subspaces, as shown in Equation (4):  
\[MultiHead\ (\mathbf{Q},\ \mathbf{K},\ \mathbf{V})=Concat(head_{1},\ head_{2},\ \ldots,\ head_{h})\mathbf{W}^{O} \tag{4}\]  
where \(h\) is the number of parallel attention heads employed in the multi-head attention mechanism and the term \(\mathbf{W}\) is a learned output projection matrix. Each head independently performs the attention operation described above with its own learned projections.