## 3 Method
### Basic Theory and Techniques for LSTM
#### 3.1.1 LSTM Algorithm Flow  
LSTM cell structure is shown in Figure 5. LSTM uses two gates to control the content of cell state c, one is the forget gate, which determines how much of the cell state \(c_{t-1}\) at the previous time is retained to the current time \(\text{c}_{\text{t}}\). The other is the input gate, which determines how much of the network's input \(x_{t}\) is saved to the cell state \(\text{c}_{\text{t}}\) at the current moment. LSTM uses an output gate to control how much of the \(\text{c}_{\text{t}}\) state of the unit is output to the LSTM's current output value \(\text{h}_{\text{t}}\).  
\[\text{Forgotten gate}:f_{t}=\sigma\Big{(}\mathbf{W_{f}}\cdot[h_{t-1},x_{t}]+b_{f} \Big{)}, \tag{1}\]  
In the above formula, \(\mathbf{W_{f}}\) is the weight matrix of the forget gate, \([h_{t-1},x_{t}]\) represents joining two vectors into a longer vector, \(b_{f}\) is the bias term of the forget gate, and \(\sigma\) is the sigmoid function. If the dimension of the input is \(\text{d}_{\text{x}}\), the dimension of the hidden layer is \(\text{d}_{\text{h}}\), and the dimension of the cell state is \(\text{d}_{\text{c}}\) (usually \(d_{\text{c}}=\text{d}_{\text{h}}\)), then the dimension of the weight matrix \(\mathbf{W_{f}}\) of the forget gate is \(d_{\text{c}}\times(\text{d}_{\text{h}}+d_{x})\). In fact, the weight matrix \(\mathbf{W_{f}}\) is a concatenation of two matrices: one is \(\mathbf{W_{fh}}\), which corresponds to the input \(h_{t-1}\), whose  
Figure 5: LSTM unit structure.  
dimension is \(d_{c}\times d_{\text{h}}\). The other is \(W_{f\!x}\), which corresponds to the input \(x_{t}\), whose dimension is \(d_{c}\times d_{\text{x}}\). \(W_{f}\) can be written as:  
\[\big{[}W_{f}\big{]}\begin{bmatrix}h_{t-1}\\ x_{t}\end{bmatrix}=\big{[}W_{f\!h}&W_{f\!x}\big{]}\begin{bmatrix}h_{t-1}\\ x_{t}\end{bmatrix}=W_{f\!h}h_{t-1}+W_{f\!x}x_{t}, \tag{2}\]  
\[\text{Input gate}:i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}), \tag{3}\]  
In the above formula, \(W_{i}\) is the weight matrix of the input gate and \(b_{i}\) is the offset term of the input gate.  
\(\widetilde{c_{t}}\) is used to describe the cell state of the current input, which is calculated based on the last output and the current input:  
\[\widetilde{c_{t}}=tanh(W_{c}\cdot[h_{t-1},x_{t}]+b_{c}), \tag{4}\]  
\(c_{t}\) is the state of the cell at the current time. It is produced by multiplying the previous cell state \(c_{t-1}\) by element by the forget gate \(f_{t}\), then multiplying the current input cell state \(\widetilde{c_{t}}\) by element by the input gate \(i_{t}\), and adding the two products:  
\[c_{t}=f_{t}\circ c_{t-1}+i_{t}\circ\widetilde{c_{t}}, \tag{5}\]  
The symbol \(\circ\) means multiplication by element.  
LSTM can combine the current memory \(\widetilde{c_{t}}\) with the long-term memory \(c_{t-1}\) to form the new cell state \(c_{t}\). Thanks to the control of the forget door, it can save information from a long time ago, and because of the control of the input door, it can avoid the current insignificant content in the memory.  
\[\text{Output gate}:o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o}), \tag{6}\]  
The final output of the LSTM is determined by the output gate and the cell state:  
\[\text{Output}:\ h_{t}=o_{t}\circ tanh(c_{t}), \tag{7}\]  
Table 2 shows the formula of the activation function.