# AB-LaBSE: Uyghur Sentiment Analysis via the Pre-Training Model with BiLSTM
## 3 Methodology
### BiLSTM  
The _BiLSTM_ model was proposed by [PERSON] by combining the bidirectional \(RNN\) models \(BRNN\) and \(LSTM\)[5] unit and verified its effectiveness in speech recognition tasks. The core idea of the _BiLSTM_[6] model is that the current input is related to the sequence before and after, and the realization is to input the data sequence into the model from two directions, respectively, and save the data information in two directions, namely the historical information and the future information, through the hidden layer. The recurrent neural network \(RNN\) is good at capturing long dependent sequence relations. Some outputs of neurons can be transmitted to neurons again as inputs, and the previous information can be effectively utilized. However, in the process of training, the constant multiplication of the function derivative will lead to the problem of \"gradient disappearance\" and \"gradient explosion\".  
_LSTM_ cleverly introduces a \"gate\" mechanism to solve this problem. _LSTM_ is composed of a series of repeated timing modules, each of which contains three gates and a memory cell, namely, the forgetting gate, input gate, and output gate. The forgetting gate determines what information the cell will discard, reads \(h_{t-1}\) and \(x_{t}\), and outputs a value between 0 and 1 to each in cell state \(C_{t-1}\).  
\[f_{t}=\sigma\Big{(}W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\Big{)} \tag{3}\]  
The input gate determines what information is stored in the cell state, and there are two parts to this. First, a sigmoid neural network layer determines what values will be updated, called the \"input gate layer.\" Then, a tanh layer creates a new candidate vector \(C_{t}\), which is added to the state.  
\[\begin{split} i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i})\\ \tilde{C}_{t}=\tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c})\end{split} \tag{4}\]  
When the unit information is updated, the old state is multiplied by \(f_{t}\), irrelevant information is discarded, and \(i_{t}\star\tilde{C}_{t}\) is added to form A new candidate value.  
\[C_{t}=f_{t}*C_{t-1}+i_{t}*\tilde{C}_{t} \tag{5}\]  
The output gate determines which part of the cell state will be output by running a sigmoid layer, and then processes the cell state through the tanh function to get a value between \(-1\) and \(1\), which is multiplied by the output of the sigmoid gate, and finally outputs only part of the determined output.  
\[\begin{split} O_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o})\\ h_{t}=o_{t}*\tanh(C_{t})\end{split} \tag{6}\]  
where: \(\tanh()\) represents the activation function. \(\sigma\) represents the neural network layer of sigmoid. \(x_{t}\) is the unit state input at time of \(t\). \(f_{t}\), \(i_{t}\) and \(O_{t}\) are the settlement results of forgetting gate, input gate and output gate, respectively. \(W_{f}\), \(W_{i}\), \(W_{o}\) and \(W_{c}\) respectively represent the weight of forgotten gate, input gate, output gate, and the updated weight. \(b_{f}\), \(b_{i}\), \(b_{o}\) and \(b_{c}\) are the corresponding offset quantities.  
In the process of text classification, _BiLSTM_ is used to make full use of the context information of text, which is to combine the two _LSTM_ models with opposite sequences. That is, add a layer of reverse _LSTM_ layer to the original forward _LSTM_ network layer and combine them, so the output can be expressed as:  
\[H_{t}=h_{l}\oplus h_{r} \tag{7}\]  
\(H_{t}\) for _BiLSTM_ model outputs the text feature vector. Where \(h_{l}\) is the output of the forward _LSTM_ and \(h_{r}\) is the output of the reverse _LSTM_.  
In this experiment, we added a dropout layer to prevent the model from overfitting. At each iteration, neurons are randomly output to zero at the specified ratio. In the experiment, the output of _BiLSTM_ is denoted as \(H_{T}\), and \(W_{hy}\) stands for the ratio of the dropout layer.  
\[y=W_{hy}H_{T} \tag{8}\]  
Finally, the linear layer performs a linear transformation of the output of dropout, where \(A\) is the weights and \(b\) is the bias.  
\[z=yA^{T}+b \tag{9}\]