# Research on Sentiment Analysis of Online Public Opinion Based on RoBERTa-BiLSTM-Attention Model
## 4 Empirical Analysis
### RoBERTa-BiLSTM-Attention Model Application  
After counting the labelled data, it was found that the amount of data labelled \(-1\), \(0\), and \(1\) were \(13\),\(004\), \(13\),\(963\), and \(9599\), respectively. Since the data suffered from an imbalance  
\begin{table}
\begin{tabular}{c c c} \hline
**Emotion Seed Word** & **Positive Emotion Word** & **Negative Emotion Word** \\ \hline rat, school, canteen, rat’s head, director, video, pointing at a deer as a horse, pointing at a rat as a duck, duck’s head, & procedure, publication, process, sampling, solemnity, persuasion, physical, parties, extraction, public, credibility, visit, sealing of samples, interpretation, overruling & headmaster, teacher, management, part, teeth, forced, hygiene, unseen, body, sworn, catering, unclean, cost, fine, mouth, frozen goods, catering, eating, lusful \\ \hline \end{tabular}
\end{table}
Table 2: Supplementary dictionary partial display.  
in the sentiment categories, this study used SMOTE to oversample a few categories (data labelled 1) to bring their sample sizes close to the other categories. In this way, the impact of data imbalance on model performance was mitigated.  
The experimental environment was as follows: the development architectures were Tensorflow and Keras; the development frameworks were scikit-learn 1.0.2 and Pytorch 1.8.1; the programming tool was PyCharm Community Edition 2022.3.2; the programming language was Python3.9; and the operating system was Win11 64-bit.  
The dataset was divided as follows: training set, validation set, and test set according to the ratio of 8:1:1. The way to achieve this was to use the train_test_split() function to first randomly divide the corpus into the training set and validation + test set according to an 8:2 ratio and then use this method to randomly divide the validation set and test set according to a 1:1 ratio.  
The model parameter configuration mainly included RoBERTa pre-training parameter settings and BiLSTM-Attention model parameter settings, as shown in Tables 3 and 4.  
The RoBERTa model hyperparameters and baseline configuration specific information are shown in Table 3. The parameter settings were based on the best practices of the RoBERTa model in the pre-training and fine-tuning phases, as well as experimental parameters borrowed from other researchers. In the model architecture, the number of layers was 12, the hidden layer dimension was 768, and the number of attention heads was 12. In the training hyperparameters, the initial learning rate was set to \(5\times 10^{-5}\) and the cosine annealing strategy was used for decay. Using the AdamW optimiser, the weight decay coefficient was 0.01. The dropout value of all layers and attention weights in the model was 0.1. The activation function was GELU and the maximum sequence length was 512. In the pre-training phase, the batch size of RoBERTa was 4096. RoBERTa adopted a dynamic masking strategy whereby a new mask pattern was generated randomly for each round of training. RoBERTa removed the NSP task and used only the MLM (masked language model) task for pre-training. In the fine-tuning phase, this study used the RobertaForSequenceClassification model for fine-tuning, with the maximum sequence length set to 128 and the learning rate typically set to \(2\times 10^{-5}\).  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Parameter Value** \\ \hline Number of layers & 12 \\ \hline Number of attention heads & 12 \\ \hline Hidden layer dimensions & 768 \\ \hline Number of attention heads & 12 \\ \hline Dropout & 0.1 \\ \hline Maximum sequence length & 512 \\ \hline Batch size & 4096 \\ \hline Epochs & 10 \\ \hline Learning rate & \(5\times 10^{-5}\) \\ \hline Activation function & GELU \\ \hline Optimiser & Adam \\ \hline Maximum sequence length in fine-tuning & 128 \\ \hline Fine-tuning model & RobertaForSequenceClassification \\ \hline Fine-tuning learning rate & \(2\times 10^{-5}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: RoBERTa model parameter configuration.