## 3 Neural Network Algorithm Model
### LSTM Model  
#### 3.2.1 Structure of the LSTM and BILSTM Models  
LSTM introduces self-loops to produce paths where gradients flow continuously over long periods. The algorithm can predict dynamically changing sequences of numbers. The LSTM model contains three gating units, the input gate, the forget gate, and the output gate, which control the flow of information [31]. The internal structure of the LSTM model is shown in Figure 3.  
The relationship between the three gating units and the candidate memory states is as follows:  
\[i_{t}=\sigma(W_{ii}x_{t}+b_{ii}+W_{hi}h_{t-1}+b_{hi}) \tag{4}\]  
\[f_{t}=\sigma(W_{if}x_{t}+b_{if}+W_{hf}h_{t-1}+b_{hf}) \tag{5}\]  
\[o_{t}=\sigma(W_{io}x_{t}+b_{io}+W_{ho}h_{t-1}+b_{ho}) \tag{6}\]  
\[g_{t}=tanh(W_{ig}x_{t}+b_{ig}+W_{hg}h_{t-1}+b_{hg}) \tag{7}\]  
The relationship between \(h_{t}\) and \(c_{t}\) is as follows  
\[c_{t}=f_{t}c_{t-1}+i_{t}g_{t} \tag{8}\]  
Figure 3: Structure of the LSTM model.  
\[h_{t}=o_{t}\cdot tanh(c_{t}) \tag{9}\]  
where \(i_{t}\) indicates the output of the input gate; \(f_{t}\) denotes the output of the forgetting gate, controlling the importance of the memory state at the previous moment; \(o_{t}\) denotes the output of the output gate, controlling the output of the memory state; \(g_{t}\) indicates the activation value of the input at the current moment; \(c_{t}\) indicates the memory state of the current moment, used to store historical information; \(h_{t}\) indicates the output value of the current moment for passing to the next moment; \(\sigma\) denotes the sigmoid function; tanh denotes the hyperbolic tangent function; and \(W\), \(b\) denote the weight and offset, respectively.  
BILSTM is an extended LSTM model that considers not only historical input information from the current moment but also information from future moments. It has greater expressiveness and model generalization capabilities.  
The BILSTM model processes the input sequence from front to back and from back to front in two different LSTM layers and concatenates their outputs to obtain the final output. In the forward LSTM layer, the LSTM model receives the original input sequence, i.e., in left-to-right order. In the backward LSTM layer, the LSTM model receives the inverted input sequence, i.e., in right-to-left order. The structure of both the LSTM and the BISTM are shown in Figure 4.  
\(h_{f}\) represents the hidden state output of the forward LSTM layer. \(H_{b}\) represents the hidden state output of the backward LSTM layer; and \(h_{t}\) denotes the output value at the current moment, which is a concatenation of the outputs of the forward and backward LSTM layers. Suppose the hidden state of the forward LSTM output is \(h_{f,t}\), the hidden state of the backward LSTM output is \(h_{b,t}\), and the input at the current moment is \(x_{t}\). Then, outputs can be expressed as follows, respectively, the output of the final forward and backward LSTM are dot productively stitched together at moment in order to obtain the output of the BILSTM model at that moment as  
\[h_{f,t}=LSTM(x_{t},h_{f,t-1}) \tag{10}\]  
\[h_{b,t}=LSTM(x_{t},h_{b,t-1}) \tag{11}\]  
\[h_{t}=h_{f,t}\oplus h_{b,t} \tag{12}\]  
where \(\oplus\) represents the splicing operation of vectors, and the activation function is chosen as \(relu\).  
Figure 4: Structure of the BILSTM model.