# Research on Sentiment Analysis of Online Public Opinion Based on RoBERTa-BiLSTM-Attention Model
## 3 Model Construction
### RoBERTa-BiLSTM-Attention  
This section introduces and explains the principles of the model construction. RoBERTa is an improved model based on the BERT architecture, which provides rich semantic features for sentiment analysis tasks by optimising the pre-training strategies (e.g., dynamic masking, eliminating the NSP task, expanding the batch size, etc. ), and the multilayered transformer encoder is used to capture bi-directional semantic information in text. It is able to provide high-quality word embeddings that capture the contextual information of text, providing powerful underlying features for subsequent sentiment analysis tasks. Although RoBERTa is able to capture contextual information, BiLSTM can further enhance its ability to model long-distance dependencies in text sequences. BiLSTM is able to capture contextual information in text more comprehensively by using LSTM networks in both forwards and reverse directions, compensating for the contextual information that may be omitted by RoBERTa when dealing with long texts and thus more accurately understanding the emotion of the text. The attention layer weighs the output of BiLSTM to help the model focus on the key information in the text, reduces the interference of redundant information, highlights the important features, filters the noisy information, and further improves the model's ability to capture the emotional features. The model combines the powerful context modelling capability of RoBERTa, the long-range dependency capturing capability of BiLSTM, and the key information focusing capability of the attention mechanism, which enables a more comprehensive understanding of the textual sentiment and thus achieves better performance in sentiment analysis tasks. 1. RoBERTa layer word vector pre-training  
Before feature extraction and sentiment classification, it is necessary to transform the participle text into recognisable word vectors. RoBERTa (Robustly Optimized BERT Pre-training Approach) is a pre-trained language model based on the Transformer architecture proposed by Facebook in 2019. RoBERTa further improves the performance and robustness of the model by optimising the pre-training process of BERT. It performs well in several natural language processing tasks, especially in tasks such as sentiment analysis, question and answer systems, and text classification. The core architecture of RoBERTa is based on Transformer, which performs deep feature extraction on input word vectors through multiple Transformer encoder layers to obtain a rich representation of text features. Similar to BERT, RoBERTa also adopts a bidirectional training mechanism that captures lexical context information. That is, the model takes into account the information of the whole input sequence, not just the local information before and after a word, when processing each word. This mechanism allows for a more accurate understanding of word polysemy and context dependency and generates more accurate word vector reproductions. In addition, RoBERTa performs several optimisations in the pre-training phase, including the use of larger training datasets, longer training sequence lengths, and dynamic masking strategies. These optimisations make RoBERTa perform better in handling complex natural language tasks. Therefore, we used RoBERTa for the deep feature extraction of input word vectors in order to take full advantage of its powerful pre-training features and context modelling capabilities and further improve the model's ability to understand and represent language. BiLSTM Emotion Feature Extraction  
BiLSTM (Bidirectional Long Short-Term Memory) is a bi-directional long short-term memory network model consisting of two LSTMs that captures both past and future contextual semantics by processing forwards and reverse word vector input sequences separately, effectively linking the text before and after without a gradient descent problem. At this layer, the word vectors \((v_{1},v_{2},v_{3},\ldots,v_{t})\) are passed into the forwards and reverse LSTMs for feature extraction.