# Research on Sentiment Analysis of Online Public Opinion Based on RoBERTa-BiLSTM-Attention Model
## 3 Model Construction
### RoBERTa-BiLSTM-Attention
The feature vectors obtained by the forwards LSTM are \(\left(\overset{\rightarrow}{h_{1}},\overset{\rightarrow}{h_{2}},\overset{ \rightarrow}{h_{3}},\ldots,\overset{\rightarrow}{h_{t}}\right)\); the feature vector obtained from the inverse LSTM is \(\left(\overset{\leftarrow}{h_{1}},\overset{\leftarrow}{h_{2}},\overset{ \leftarrow}{h_{3}},\ldots,\overset{\leftarrow}{h_{t}}\right)\); and the eigenvectors obtained from the forwards and inverse directions are combined and output as follows:  
\[h_{t}=\text{tanh}\left(w_{t}v_{t}+W_{h_{t-1}}+b_{t}\right) \tag{3}\]  
\[\overset{\rightarrow}{H}=LSTM\left(\overset{\rightarrow}{h_{1}},\overset{ \rightarrow}{h_{2}},\overset{\rightarrow}{h_{3}},\ldots,\overset{ \rightarrow}{h_{t}}\right) \tag{4}\]  
\[\overset{\leftarrow}{H}=LSTM\left(\overset{\leftarrow}{h_{1}},\overset{ \leftarrow}{h_{2}},\overset{\leftarrow}{h_{3}},\ldots,\overset{ \leftarrow}{h_{t}}\right) \tag{5}\]  
\[H=\overset{\rightarrow}{H}\oplus\overset{\leftarrow}{H} \tag{6}\]  
where \(w_{t}\) is the weight of vector \(v_{t}\), \(W_{h_{t-1}}\) is the weight matrix of \(h_{t-1}\) at moment \(t\), and \(b_{t}\) is the bias term at moment \(t\). \(\overset{\rightarrow}{H}\) is the positive LSTM output at each moment, \(\overset{\leftarrow}{H}\) is the negative LSTM output at each moment, and \(H\) is the output vector that results from combining the two. Attention mechanism assignment  
An attention mechanism can dynamically assign different weights to different parts of an input sequence to focus on the information that is more important to the current task. The basic idea is to determine the attention weight for the value vector by calculating the similarity between the query vector and the key vector and then calculate the weighted sum as the output; the specific formula is as follows:  
\[score(h_{t},h_{s})=v^{T}\text{tanh}(Wh_{t}+Uh_{s}) \tag{7}\]  
\[\alpha_{ts}\text{=}\frac{\exp(score(h_{t},h_{s}))}{\sum_{k=1}^{T_{x}}\exp(score (h_{t},h_{s}))} \tag{8}\]  
\[c_{t}=\sum_{s=1}^{T_{x}}\alpha_{ts}\;h_{s} \tag{9}\]  
where \(h_{t}\) is the current hidden state of the decoder and \(h_{s}\) is one of the encoder's hidden state sequences. \(W\) and \(U\) are weight matrices and \(v^{T}\) is a learnable weight vector. \(\alpha_{ts}\) is the attentional weight of \(h_{s}\) at time \(t\). \(T_{x}\) is the length of the encoder's sequence of hidden states. \(c_{t}\) is the context vector, i.e., the weighted sum at time step \(t\). 4. RoBERTa-BiLSTM-Attention Sentiment Classification Model  
To tackle the semantic complexity of Chinese text and solve the problem whereby the sentiment classification model does not fully consider the context and important features, this study adopted the RoBERTa-BiLSTM-Attention model for sentiment classification on the basis of the previous paper. The specific steps were as follows: firstly, we used RoBERTa to pre-train the word vectors to extract the deep-level feature representations, passed the word vector features into BiLSTM for forwards and backwards bi-directional emotion feature extraction, introduced the attention mechanism to assign weights to the different information, and, finally, spliced the vector information into the Softmax function to compute the derived probability distribution and realise the triple classification emotion computation. The RoBERTa -BiLSTM-Attention model structure is shown in Figure 2.  
As can be seen from Figure 2, the RoBERTa-BiLSTM-Attention model constructed in this study had five layers, which were the input layer, RoBERTa layer, BiLSTM layer, attention layer, and output layer. Arrows are used to show the path of information passing through the neural network. The circles in the RoBERTa and BiLSTM layers represent the output of each encoder layer. The circles of the attention layer represent the generation of keys or values and the computation of the attention scores. The CONCAT circle represents the stitching of features or information from different sources.