# SA-SGRU: Combining Improved Self-Attention and Skip-GRU for Text Classification
## 4 Experiment and Results
### Results Analysis  
#### 4.3.1 Performance  
Taking accuracy as the evaluation index, the text classification model proposed in this paper is experimentally compared with the following various baseline models:  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Window size of CNN kernel & \((2,3,4)\) \\ Hidden size of GRU & 128 \\ Dropout rate & 0.3 \\ Weight\_decay & 0.1 \\ Learning rate & 0.0001 \\ Optimizer & Adam \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental parameters setting.  
CNN-based models, including TextCNN [25] and CNN-LSTM [26].  
RNN-based models, including LSTM-CNN [27], BiLSTM [28], and LSTM [29].  
Semi-supervised models, including SALNET [30]. SALNET includes attention-based LSTM (SALNET1), attention-based LSTM+TextCNN (SALNET2), and attention-based LSTM+BERT (SALNET3).  
Adversarial training-based models, include KATG [31] and Lex-AT [32]. KATG includes KATG + TextCNN(KATG1), KATG + LSTM (KATG2), and KATG + BERT (KATG3). Lex-AT includes Lex + TextCNN (Lex-AT1), Lex + LSTM (Lex-AT2), and Lex + BERT (Lex-AT3).  
The results are shown in Table 3.  
In-depth analysis of the experimental comparison results in Table 3 reveals that the single network models, such as TextCNN, LSTM, and BiLSTM, do not perform very well because the single network model can only extract the single text feature.  
SALNET is a semi-supervised bootstrap learning framework, and it is based on semi-supervised and attention-oriented LSTM, which is insensitive to local key pattern information and performs less well than the SA-SGRU model in terms of classification accuracy. It can be seen that BERT exhibits the best performance compared to SALNet using baseline classifiers such as attention-based LSTM and TextCNN. BERT obtains text representations using a bi-directional transformer encoder and has richer text semantic understanding than attention-based LSTM and attention-based LSTM+TextCNN. Thus, attention-based LSTM+BERT performs better in text classification tasks.  
The accuracy of the models based on KATG and Lex-AT adversarial training is no less than 91.50% on AGNews and no less than 87.40% on the IMDB dataset, outperforming semi-supervised self-learning frameworks in classification results. The accuracy of the three models based on KATG adversarial training is higher than that of the three models based on Lex-AT adversarial training. KATG not only uses the previous sentence to guide the generation of adversarial sentences but also proposes a keyword bias-based sampling method to select the sentence containing the biased word as the previous sentence. Compared with the model constructed by Lex-AT, the model constructed by KATG is more capable of capturing the keyword information in the sentences, resulting in greater classification accuracy.  
Compared with the above baseline models, except KATG3 and Lex-AT3, the proposed SA-SGRU outperforms the other baseline models on all three datasets.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **AGNews** & **IMDB** & **R8** \\ \hline TextCNN & 88.08\% & 75.39\% & 90.11\% \\ \hline LSTM & 87.07\% & 84.82\% & 90.68\% \\ BiLSTM & 87.36\% & 85.78\% & 91.27\% \\ \hline CNN-LSTM & 86.79\% & 87.23\% & 90.98\% \\ LSTM-CNN & 87.89\% & 88.73\% & 92.32\% \\ \hline SALNET1 & 88.22\% & 79.00\% & - \\ SALNET2 & 89.23\% & 80.33\% & - \\ SALNET3 & 90.35\% & 84.87\% & - \\ \hline KATG1 & 92.90\% & 89.50\% & - \\ KATG2 & 92.80\% & 87.90\% & - \\ KATG3 & 95.10\% & 92.80\% & - \\ Lex-AT1 & 92.00\% & 88.70\% & - \\ Lex-AT2 & 91.90\% & 87.40\% & - \\ Lex-AT3 & 94.20\% & 92.60\% & - \\ \hline SA-SGRU (Proposed) & 93.18\% & 90.22\% & 94.51\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: The main experimental findings (Accuracy/%).