## 3 Model Design
### Multi-Scale Attention  
The essence of the attention mechanism is to assign a set of weights to different features, enabling the model to focus on the most relevant aspects of the input. In this paper, the multi-scale attention module consists of two main sub-modules: channel attention and spatial attention. In natural language processing, the channel attention mechanism enhances the model's focus on specific feature dimensions that are most significant for the task. For instance, it assigns higher weights to certain dimensions of word embedding in textual representation vectors. This approach ensures that the model pays greater attention to the most critical dimensions. Spatial attention, on the other hand, assigns different weights to words based on their positions in the text, highlighting those that are crucial for grasping the overall meaning. In summary, channel attention enhances the model's focus on specific word embedding dimensions within the text representation vector, while spatial attention emphasizes the significance of words in various positions throughout the text. The text representation vectors generated by the ERNIE and BiLSTM modules contain rich positional features and semantic relationships. To enhance the model's comprehension capabilities, we introduce channel and spatial attention mechanisms. These mechanisms automatically identify and amplify the most critical feature information for the final classification task, while reducing the influence of less relevant features. This approach enables the model to better understand both the structure and semantics of the text. Firstly, channel attention dynamically adjusts the importance of different word embedding dimensions in the text representation vector, as illustrated in Figure 4a. Figure 3: BiLSTM model structure. The channel attention mechanism begins by applying average pooling and max pooling to each word embedding dimension, capturing both the general and the most salient characteristics. Afterward, multiple fully connected layers are used to learn the complex relationships between these dimensions, generating a weight for each one and prioritizing those most relevant to the classification task. Assuming that the input text representation vector is \(F\in\mathbb{R}^{n\times h}\), the calculation formula for the channel attention mechanism is presented in Equation (13). \[ \begin{split} M_{c}(F)&=\sigma(MLP(AvgPool(F))+MLP( MaxPool(F)))\\ &=\sigma\big{(}W_{1}\big{(}W_{0}\big{(}F_{avg}\big{)}\big{)}+W_{1} (W_{0}(F_{max}))\big{)}\end{split}  \tag{13}\]  
where \(M_{c}(F)\in\mathbb{R}^{1\times h}\), AvgPool refers to average pooling, MaxPool refers to maximum pooling, and MLP stands for multilayer perceptron, \(W_{0}\in\mathbb{R}^{\frac{h}{h}\times h}\), \(r\) refers to scaling factor, \(W_{1}\in\mathbb{R}^{h\times\frac{h}{r}}\), ReLU activation function is added in front of \(W_{0}\) for nonlinear mapping, and \(\sigma\) stands for sigmoid function. The computed channel attention weights are multiplied with the input \(F\) to obtain the new feature vector \(F^{\prime}\) as shown in Equation (14):  
\[F^{\prime}=M_{c}(F)\odot F \tag{14}\]  
Spatial attention dynamically adjusts the weights of each word in the input text representation vector based on their importance, enabling the model to focus more on critical information for the current classification task and assign greater weights to important words. As shown in Figure 4b, the spatial attention mechanism starts by applying average pooling and maximum pooling to compute the average and maximum values of each word vector. It then captures the spatial relationships between words and local patterns through convolution operations. The computational formula for this process is presented in Equation (15). \[ \begin{split} M_{s}(F^{\prime})&=\sigma\Big{(}f^{ 3\times 2}\big{(}\big{[}AvgPool\big{(}F^{\prime}\big{)};MaxPool\big{(}F^{ \prime}\big{)}\big{]}\big{)}\Big{)}\\ &=\sigma\Big{(}f^{3\times 2}\big{(}\big{[}F^{\prime}_{avg};F^{ \prime}_{max}\big{]}\big{)}\Big{)}\end{split}