## 3 Method
### Pre-Trained Model for the Second-Class Entities
#### 3.2.2 Bi-LSTM Model  
Recurrent Neural Network (RNN) is frequently utilized in the named entity recognition (NER) task to solve such sequence annotation issues. It is challenging for the model to acquire the properties of medium long-term dependence when the sequence length is too long, as the gradient would vanish. The Long Short Term Memory (LSTM) is a significant advance over the conventional RNN. It introduces a gating mechanism to regulate information input and output and a memory unit to learn which data should be remembered and lost during training.Therefore, LSTM can better capture long-distance dependencies and further learn the semantic feature information in the text.  
Figure 6: BERT network structure [47].  
At time \(\mathbf{t}\), the long short term memory (LSTM) model is made up of the following components: the input word \(\mathbf{W_{t}}\), the cell state \(\mathbf{C_{t}}\), the temporary cell state \(\widetilde{\mathbf{C_{t}}}\), the hidden layer state \(\mathbf{h_{t}}\), the forget gate \(\mathbf{f_{t}}\), the input gate \(\mathbf{i_{t}}\), and the output gate \(\mathbf{o_{t}}\). Equations (1)-(7) show the calculation formulas for each of them. Figure 7 depicts the LSTM model's frame diagram.  
Forget Gate:  
\[\mathbf{f_{t}}=\sigma(\mathbf{U_{f}}\cdot[\mathbf{h_{t-1}},\mathbf{W_{t}}]+ \mathbf{b_{f}}) \tag{1}\]  
Input Gate:  
\[\mathbf{i_{t}}=\sigma(\mathbf{U_{i}}\cdot[\mathbf{h_{t-1}},\mathbf{W_{t}}]+ \mathbf{b_{i}}) \tag{2}\]  
\[\widetilde{\mathbf{C_{t}}}=\tanh(\mathbf{U_{c}}\cdot[\mathbf{h_{t-1}},\mathbf{ W_{t}}]+\mathbf{b_{c}}) \tag{3}\]  
Cell State:  
\[\mathbf{C_{t}}=\mathbf{f_{t}}\ast\mathbf{C_{t-1}}+\mathbf{i_{t}}\ast\widetilde {\mathbf{C_{t}}} \tag{4}\]  
Output Gate:  
\[\mathbf{o_{t}}=\sigma(\mathbf{U_{o}}\cdot[\mathbf{h_{t-1}},\mathbf{W_{t}}]+ \mathbf{b_{o}}) \tag{5}\]  
\[\mathbf{h_{t}}=\mathbf{o_{t}}\ast\tanh(\mathbf{C_{t}}) \tag{6}\]  
\[Y_{t}=softmax(\mathbf{h_{t}}) \tag{7}\]  
where sigmoid, tanh and Softmax are activation functions of nonlinear neural network, \(\mathbf{W_{t}}\) represents input characteristics of current network, \(\mathbf{U}\) represents weight matrix of three gates, and \(\mathbf{b}\) represents bias term of three gates.  
Forget gate determines how much the unit state \(\mathbf{C_{t-1}}\) of the last moment is retained to the current moment \(\mathbf{C_{t}}\). Input gate determines how much of the current network input \(\mathbf{W_{t}}\) is saved to the unit state \(\mathbf{C_{t}}\). Output gate controls how much the unit state \(\mathbf{C_{t}}\) outputs to the current output value \(\mathbf{h_{t}}\) of the long short term memory (LSTM). The final output consists of the current cell state \(\mathbf{C_{t}}\), the current output value \(\mathbf{h_{t}}\), and the current prediction \(Y_{t}\) obtained by the softmax function.  
A one-way long short term memory (LSTM) model, however, is unable to process context data concurrently. The bidirectional long short term memory (Bi-LSTM) proposed  
Figure 7: LSTM model diagram [48].  
by [PERSON] et al. [26] takes forward and backward LSTM for each word sequence, combining the output simultaneously. As a result, it contains both forward and backward information for each moment. Figure 8 depicts its model structure.