# BERT- and BiLSTM-Based Sentiment Analysis of Online Chinese Buzzwords
## 3 Proposed Model
### BiLSTM Network  
A BiLSTM model is a model composed of a forward LSTM and a backward LSTM, and the LSTM in this model is a variant of the recurrent neural network (RNN). In order to solve the problem of gradient disappearance of traditional RNN, a gating unit is introduced into the LSTM, which provides the LSTM with a stronger ability to capture long-term dependencies and enables the RNN to better discover and utilize the dependencies existing in long-distance data. Each cell unit in LSTM adopts a new structure, which mainly consists of four parts: input gate \(i_{t}\), output gate \(o_{t}\), forget gate \(f_{t}\), and storage unit \(c_{t}\). The internal structure of a single cell of the LSTM module is shown in Figure 4.  
The LSTM update formula is as follows:  
1. Forget gate mechanism: \[f_{t}=\sigma(W_{(f)}x_{t}+U_{(f)}h_{t-1}+b_{(f)})\] (1)  
The forget gate \(f_{t}\) is a reset memory cell. \(\sigma\) represents the Sigmoid activation function, \(W_{(f)}\) is a weight matrix, \(x_{t}\) represents the input at time \(t\). \(U\) is the weight matrix of the hidden layer output, \(h_{t-1}\) is the hidden state, represents the short-term memory. \(b_{(f)}\) is the offset vector. 2. Input gate mechanism: \[i_{t}=\sigma(W_{(i)}x_{t}+U_{(i)}h_{t-1}+b_{(i)})\] (2)  
The input gate \(i_{t}\) represents the input gates that control the input of the memory cell. \(\sigma\) represents the Sigmoid activation function, \(W_{(i)}\) is a weight matrix, \(x_{t}\) represents the input at time \(t\). \(U\) is the weight matrix of the hidden layer output, \(h_{t-1}\) is the hidden state, represents the short-term memory. \(b_{(i)}\) is the offset vector. 3. Current unit status: \[\widetilde{c}=f_{t}\odot c_{t-1}\] (3) \[\widetilde{c}\] is candidate memory cell. The forget gate \(f_{t}\) is a reset memory cell. \[c_{t-1}\] is the cell state of \[t-1\], represents the long-term memory. Figure 4: LSTM cell unit. 4. Update unit status: \[c_{t}=\widetilde{c}+i_{t}\odot\tanh(W_{(c)}x_{t}+U_{(c)}h_{t-1}+b_{(c)})\] (4)  
\(c_{t}\) is the cell state, represents the long-term memory, \(\widetilde{c}\) is candidate memory cell. \(i_{t}\) represents the input gates that control the input of the memory cell. \(W_{(c)}\) is a weight matrix, \(x_{t}\) represents the input at time \(t\). \(U\) is the weight matrix of the hidden layer output, \(h_{t-1}\) is the hidden state, represents the short-term memory. \(b_{(c)}\) is the offset vector. 5. Output gate mechanism: \[o_{t}=\sigma(W_{(o)}x_{t}+U_{(o)}h_{t-1}+b_{(o)})\] (5)  
The output gate \(o_{t}\) represents the output gates that control the output of the memory cell. \(\sigma\) represents the Sigmoid activation function, \(W_{(o)}\) is a weight matrix, \(x_{t}\) represents the input at time \(t\). \(U\) is the weight matrix of the hidden layer output, \(h_{t-1}\) is the hidden state, represents the short-term memory. \(b_{(o)}\) is the offset vector. 6. The current state of the hidden layer: \[f_{t}=\sigma(W_{(f)}x_{t}+U_{(f)}h_{t-1}+b_{(f)})\]0 (6)  
\(h_{t}\) is the hidden state. tanh is the Sigmoid activate function. \(c_{t}\) is the cell state, represents the long-term memory. \(o_{t}\) represents the output gates that control the output of the memory cell. To acquire backward and forward features in a given time, [PERSON] [32] proposed the BiLSTM network. Therefore, in order to solve the problem of feature irregularity in the OCBs, this paper uses the BiLSTM model to capture more features hidden in the contextual deep semantic dependencies. The structure of BiLSTM is shown in \[f_{t}=\sigma(W_{(f)}x_{t}+U_{(f)}h_{t-1}+b_{(f)})\]1 Here, BiLSTM is used to learn the output of the BERT layer to enhance the fitting effect of network features and generalization to new datasets. The formulas for the state at each moment of the model are shown in Equations (7) and (8). The final state is jointly determined by the state of the BiLSTM, as shown in Formula (9) later.