## 2 Methodologies
### Feature Extraction Process
#### 2.3.2 Attention Mechanism Layer  
In text classification tasks, each word contributes differently to the textual features. Therefore, the introduction of attention mechanisms can adjust the weight coefficients (i.e., the values of attention distribution) of key words in self-attention computation, allowing the model to focus more on important information for better results. The incorporation of attention mechanisms enables the model to globally reprocess the locally extracted features obtained through convolution. The attention calculation layer consists of two main components: multi-head attention and a two-layer feed-forward neural network. Among them, multi-head attention is the most critical part, as it captures semantic information from different aspects of the input sequence through multiple attention heads. Adaptive feature weighting and aggregation are performed within each attention head, resulting in the final attention representation. The overall structure of the attention calculation process is illustrated in Figure 4.  
Figure 3: CNN performs local feature extraction and mapping of the input embedding vector.  
From the structure depicted in the diagram, it can be observed that Multi-Head Attention performs multiple self-attention computations (multiple heads) on the same input word vectors. This approach aims to capture important features from different subword spaces, thereby enhancing the model's ability to gather diverse information. Self-attention is a mechanism used to calculate the weighted sum of each element based on all elements in the input sequence, enabling the capture of relationships and importance between elements. The computation formula for self-attention is as follows:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{4}\]  
Within each output \(head_{i}\) of self-attention, important features corresponding to different subword spaces can be obtained. By concatenating these outputs and multiplying them with a mapping matrix \(W^{o}\) (aimed at compressing the output matrix), we can obtain the output of the entire Multi-Head Attention:  
\[MultiHead\ (Q,K,V)=Concat(head_{1},head_{2},\ldots,head_{h})W^{o}, \tag{5}\]  
\[head_{i}=Attention(Q_{i},K_{i},V_{i}), \tag{6}\]  
where \(Q_{i}=QW_{i}^{Q}\), \(K_{i}=KW_{i}^{K}\), \(V_{i}=VW_{i}^{V}\), \(i\in[1,h]\) and \(Q,K,V\in\mathbb{R}^{n\times d/l}\), from the input embedding \(M_{l}\), mapping matrix \(W_{i}^{Q}\in\mathbb{R}^{d/l\times d_{k}}\), \(W_{i}^{K}\in\mathbb{R}^{d/l\times d_{k}}\), \(W_{i}^{V}\in\mathbb{R}^{d/l\times d_{e}}\), \(W^{o}\in\mathbb{R}^{hd_{k}\times d/l}\). In this study, we set \(h=4\) parallel Self-Attention layers to perform global attention calculation on the local feature maps (Input Embedding) from the convolutional layer, where \(d_{k}=d_{p}=d/h=64\) in each head.  
Figure 4: Attention performs global attention calculations.  
By employing the multi-head attention mechanism, we partition the original high-dimensional feature space into multiple lower-dimensional subspaces and independently compute attention within each subspace. This approach enriches the representation of feature information while reducing the computational burden on the model. Through the attention calculation layer, we obtain the final output matrix \([O_{1},O_{2},\ldots,O_{l}]\) of the feature extraction module. This output matrix synthesizes the feature information from multiple low-dimensional subspaces, with each subspace capturing different semantic aspects of the original text. Consequently, it significantly enhances the model's expressive power and improves its robustness and performance when dealing with diverse types of textual data.