## 3 Theory and Method
### Proposed Model  
The proposed model gradually extracts the semantic information of the input sequence through representation learning at multiple levels, as shown in Figure 4. First, the RoBERTa model encoded each sentence to generate sentence embedding, while also generating word embedding for each word. These embedding vectors contain the semantic information of the text. We then passed the word embedding extracted by the RoBERTa model to the bidirectional GRU model, which processes the word embedding for each word, considering the contextual information of the word in the sequence. We then used a multi-head self-attention mechanism to integrate the output of the bidirectional GRU, which helps identify essential features in the text. Finally, the output results of the multi-head attention mechanism were concatenated with the sentence embedding extracted by the RoBERTa model, which combines the global relational and context-dependent representation with the semantic information at the original sentence level to obtain a richer semantic representation. Figure 4: Schematic diagram of the structure of the proposed RB_BG_MHA model. (The solid arrow represents the forward GRU and the dashed arrow represents the backward GRU.) The model can take advantage of local and global information by concatenating the output of multi-head attention with RoBERTa's sentence embedding. This kind of model integration can provide more abundant feature representation, thus enhancing the expressiveness of the model. The process is as follows:  
Given an input sequence \(X\in\mathbb{R}^{n\times d_{x}}\), where \(d_{x}\) represents the dimensional size of the input sequence. First, we used the RoBERTa model to extract the semantic representation of the input sequence to obtain the sentence embedding \(H_{R}\in\mathbb{R}^{d_{x}}\), where \(d_{r}\) represents the sentence embedding dimension of the RoBERTa model. At the same time, the word embedding matrix \(H_{W}\in\mathbb{R}^{n\times d_{w}}\) is obtained through the RoBERTa model, where each line represents the embedding vector of a word, representing the semantic information of each word in the input sequence, and \(d_{w}\) is the dimension of word embedding. Second, we use the bidirectional GRU model to further process the word embedding, yielding the output \(H_{G}\in\mathbb{R}^{n\times d_{g}}\) of the GRU model, where \(d_{g}\) represents the output dimension of the GRU model. Then, we used the multi-head self-attention mechanism to further integrate the information from the input sequence and obtain the final representation \(H_{A}\in\mathbb{R}^{d_{a}}\), where \(d_{a}\) represents the output dimension of the attention mechanism. Specifically, we first took the output \(H_{G}\) of the GRU model as the input query, key, and value in the self-attention mechanism, respectively, and then used multiple attention heads to compute in parallel, concatenate the output of each head to obtain the feature representation of the multi-head attention computation, and, after that, we applied a linear layer to transform the output of the multi-head attention to obtain the output result \(H_{A}\in\mathbb{R}^{d_{a}}\). Finally, \(H_{A}\) and \(H_{R}\) were concatenated, and then the concatenated result was fed into the fully connected layer for classification to obtain the final result.