## 3 Materials and Methods
### Classifier  
The classifier of EFTransformer can be divided into three main parts: input layer, encoding layer, and classification layer. Figure 3: Word2 vec and FastText processing. The input data (\(X\)) is split into multiple heads, each of which processes a portion of the input data. Each head is calculated independently, and each head projects the input into the query, key, and value space through three weight matrices, and then calculates the attention weights. Finally, the output of each head is merged and integrated through an additional linear layer to get the output of the multi-head attention mechanism. * **Input layer**: The output after dual-channel semantic embedding processing is used as the input of Transformer. In this work, the feature pattern is the semantic information extracted based on the channel level, which is mainly used to characterize the attack intention. The model processes different features of network traffic through four parallel encoders: packet number timing pattern (pn_encoder), flow interval time distribution (iat_encoder), source port access topology (sp_encoder), destination port service feature (dp_encoder). The 8-head attention mechanism is used to capture the long-distance dependencies in the feature sequence, so the model can effectively extract the traffic features in depth and form a multi-modal embedding representation. * **Encoding layer**: Each encoder block consists of two main sub-layers: Multi-Head Attention and Feed-Forward Network. First, the importance score of the feature is calculated through a learnable weight matrix, and the normalized attention weight is generated. This mechanism can help the model adaptively focus on the feature channels that are most relevant to the current feature pattern. Secondly, the feature interaction method is used in the encoding layer to establish nonlinear associations between four-dimensional features through matrix multiplication operations, which effectively solves the problem of spatial heterogeneity. Finally, in order to suppress noise, the model introduces Dropout layer and L2 regularization in feature weighting to achieve dual noise suppression function. The self attention mechanism is a special attention mechanism in Formula (5), Where \(Q\) is the query, \(K\) is the key, and \(V\) is the value matrix, which are calculated from the input matrix \(X\) through the weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\). \[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V\] (5) The core of multi-head attention is to calculate multiple attention heads in parallel as shown in Formula (6). Each head focuses on different semantic information of the input feature sequence, and finally all the heads are fused. Different attention heads can focus on different position information and semantics, which can also enhance the robustness of the model as shown in Formula (7) and avoid overfitting or information bias caused by a single attention head. In order to improve the expression ability, multi-perspective modeling can be used to enrich feature expression. \[\text{head}_{i}=\text{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})\] (6) \[\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h}) W^{O}\] (7)
* **Output layer**: The classification layer will process the output of the encoder and give a dynamic weighted summation mechanism of attention pooling. The model trains the optimal feature combination in different scenarios end-to-end, and uses the pooled feature vector to map into a two-dimensional vector through nonlinear changes to capture deeper nonlinear relationships between semantic features. Finally, the model sums the vectors from four different feature modes to generate a comprehensive representation. The Softmax function is used to process the vector and calculate the final detection probability of whether the traffic is malicious or benign, thereby completing the classification task.