## 3 Model Design
### Attention Mechanisms  
The attention mechanism was first used by the Google Mind team [14] in conjunction with RNN models for image classification tasks. Later, the authors of [32] used a similar attention mechanism to accomplish the task of machine translation, enabling the model to perform translation and alignment simultaneously, which was the first application of the attention mechanism in NLP. Since then, the attention mechanism has been widely applied to various scenario tasks, and the Google machine translation team [21] has proposed the concept of a transformer whose network structure consists entirely of the attention mechanism, and no longer uses CNNs or RNNs to construct classification models, which performs very well in English-German and English-French translation tasks in the WMT2014 corpus; it also outperforms the traditional CNN and RNN models in training speed. In traditional CNNs for text classification tasks, each channel corresponds to a sentence, and the feature representation of the sentence is obtained and then input to the classifier without considering the connections between the sentence pairs before the feature representation of the text is input to the classifier.  
An attention function can be essentially considered as a mapping from a query to a set of key-value pairs, where query, key, value, and output are represented by vectors, as shown in Figure 3. The source can be regarded as the content stored in the memory, the elements of which consist of the address key and the value. We can suppose that there are L_x queries, and that the value of the corresponding value in the memory--i.e., the value  
Figure 2: Distribution of each category after data enhancement.  
of attention--is finally obtained by L_x = \(\|\)Source\(\|\), representing the length of the input text sequence.  
The functions that are often used to calculate the similarity of query and key are the inner product formula, the cosine similarity formula, and the perceptron network formula, respectively:  
\[(\text{Sim}(\text{Query},\text{Key}_{i})=\text{Query}\cdot\text{Key}_{i} \tag{1}\]  
\[\left(\text{Sim}(\text{Query},\text{Key}_{i})=\frac{\text{Query}\cdot\text{ Key}_{i}}{\|\text{Source}\|.\|\text{Key}\|}\right. \tag{2}\]  
\[(\text{Sim}(\text{Query},\text{Key}_{i})=\text{MLP}(\text{Query}\cdot\text{ Key}_{i}) \tag{3}\]  
The similarity calculation result is then normalized using softmax to obtain the probability distribution, which denotes the similarity probability distribution of the \(i\)-th query, and the formula is as follows:  
\[a_{i}=\text{Softmax}\left(\frac{e^{\text{sim}_{i}}}{\sum\limits_{i=1}^{L_{x}} e^{\text{sim}_{i}}}\right) \tag{4}\]  
Finally, the value is weighted and summed according to the weighting factor, as follows:  
\[\text{Attention}(\text{Query},\text{ Source})\ =\sum\limits_{i=1}^{L_{x}}a_{i}* \text{Value}_{i} \tag{5}\]  
The essence of the attention mechanism is actually an addressing process, which calculates the attention value by computing the attention distribution with the key and attaching it to the value. The attention mechanism also reduces the complexity of the neural network model by focusing the perspective on the information that is more relevant to the current task, and reducing or even ignoring other irrelevant information [25]. In doing so, it can effectively solve the problem of information overload of the neural network model, and at the same time improve the computational efficiency and generalization ability of the model [17,29].