# Weibo Text Sentiment Analysis Based on BERT and Deep Learning
## 4 Proposed Method  
Aiming at the shortcomings of most existing sentiment classification models, this paper proposes a new model. The model consists of five parts: Pre-train layer, BiLSTM layer, Attention layer, CNN layer and Full Connected layer. The structure is shown in Figure 4:  
Figure 3: BiLSTM structure. Suppose the input text sentence is \(S=\{w_{1},w_{2},\cdots,w_{i},\cdots,w_{n}\}\), where \(w_{i}\) represents the \(i\)-th word in the sentence \(S\). The task of the model is to assume a given text sentence \(S\), input \(S\) into the model, and the model outputs the sentiment polarity \(P\) of the sentence. We first need to build a sentiment dictionary \(SD\), which consists of two parts, the first part is the sentiment words, and the second part is the sentiment weight \(sw_{i}\) corresponding to each sentiment word. The function of the sentiment dictionary is to assign a sentiment weight \(sw_{i}\) to words that belong to \(SD\) in \(S\), and to assign a sentiment weight one to words that do not belong to \(SD\) in \(S\). The sentiment dictionary used in this article is based on the open source sentiment dictionary provided by Dalian University of Technology [30]. We only retain the positive and negative words in the original dictionary, and removes the words that represent neutral or both genders. The sentiment strength of the word in the original sentiment dictionary is used as the sentiment weight \(sw_{i}\) of the constructed sentiment dictionary. It is worth mentioning that, for negative words, the sentiment weight should be the sentiment strength multiplied by \(-1\). The expression of sentiment weight is as follows:  
\[\textbf{senti}(w_{i})=\left\{ \begin{array}{cc}sw_{i},&w_{i}\in SD\\ 1,&w_{i}\
ot\in SD.\end{array} \right. \tag{10}\]  
In Formula (10), \(w_{i}\) represents the \(i\)-th word in \(S\), \(sw_{i}\) represents the sentiment weight of the word \(w_{i}\) in the constructed sentiment dictionary, and \(SD\) is the constructed sentiment dictionary. In the pre-train layer, through the fine-tuned BERT model, the input word \(w_{i}\) is converted into a word vector \(v_{i}\), and then the sentiment weight in \(SD\) is used to weight \(v_{i}\), as shown in Formula (11):  
\[v_{i}^{\prime}=v_{i}*\textbf{senti}(w_{i}). \tag{11}\]  
Figure 4: Proposed model structure. The weighted word vector carries a stronger emotional color, and we use it as the input of the BiLSTM layer. At this time, the text sentence starts to classify. In the text data, the current word will be affected by adjacent words, and the LSTM network can only extract information in one direction, so this paper chooses to use the BiLSTM network. The BiLSTM network can process a long text sequence and apply it after the pre-train layer which function is to extract the dependencies of the text in the forward and backward directions. It can combine the historical information at the previous moment and the input at the current moment to determine the output information at the current moment, so it can extract contextual features in the sequence data. For the input \(v_{t}^{\prime}\) at time \(t\), the hidden state representation \(\overrightarrow{h_{t}}\) and \(\overleftarrow{h_{t}}\) obtained by the forward LSTM and the reverse LSTM, as shown in Formulas (12) and (13):  
\[\overrightarrow{h_{t}}=\overrightarrow{LSTM}(v_{t}^{\prime}),t\in[1,n] \tag{12}\]  
\[\overleftarrow{h_{t}}=\overleftarrow{LSTM}(v_{t}^{\prime}),t\in[n,1]. \tag{13}\]  
For each word, we can connect its forward and backward context information, and then obtain the annotation of the word. Combining \(\overrightarrow{h}_{t}\) and \(\overleftarrow{h}_{t}\) is the output \(\mathbf{h}_{t}\) of the state at time \(t\), as shown in Formula (14):  
\[h_{t}=\Big{[}\overrightarrow{h_{t}},\overleftarrow{h_{t}}\Big{]}. \tag{14}\]  
In the task of text sentiment analysis, some words have a strong emotional color, and some words have no emotional color or only a small amount of emotional color.