## 2 Methodology
### Long Short-Term Memory Neural Networks  
Recurrent neural networks (RNNs) (Figure 1) have been successfully applied to a variety of problems, such as speech recognition, language modeling, translation, and image captioning [28; 29]. RNNs can utilize historical information and apply it to the current output, which allows for feedback in the networks [30]. The recurrent connections in the RNN implement passing information from one step of the network (time step \(t-1\)) to the next one (time step \(t\)). For a given input sequence \(x=(x_{1},x_{2},\cdots,x_{T})\), the hidden state vector sequence is \(h=(h_{1},h_{2},\cdots,h_{T})\), and the output sequence is \(y=(y_{1},y_{2},\cdots,y_{T})\). The output sequence \(y\) can be obtained by iterating the following equations from time \(t=1\) to \(T\):  
\[h_{t}=\Re(W_{th}x_{t}+W_{th}h_{t-1}+b_{h}) \tag{3}\]  
\[y_{t}=W_{hy}h_{t}+b_{y} \tag{4}\]  
where \(x_{t}\) and \(y_{t}\) are the input and output at time step \(t\); \(h_{t}\) and \(h_{t-1}\) are the hidden states at time step \(t\) and \(t-1\), respectively; \(T\) is the prediction period; \(W_{sth}\) is the weight matrix between input and hidden vectors; \(W_{th}\) represents the weight matrix between different time steps of the hidden vectors; \(W_{hy}\) represents the weight matrix that connects the hidden information to the output vectors; \(b_{h}\) and \(b_{y}\) are the corresponding bias vectors of \(W_{th}\) and \(W_{hy}\), respectively; and \(\Re\) is the nonlinear activation function for hidden nodes.  
However, in practice, RNNs are not capable of handling a problem as long-term dependencies because of the vanishing gradient or exploding gradient problems [31]. The LSTM neural networks are a special type of RNN, which are capable of learning long-term dependencies [32]. LSTMs replace every single unit in an RNN with a memory block (Figure 2). There are three types of gate functions in this memory block [33]. The input gates control the flow of input activations into the memory cell, and  
Figure 1: Standard recurrent neural network (RNN) module contains a single layer.  
the output gates control the output flow into other cells or as the final results [34]. The forget gate controls whether the value from the previous moment is forgotten or remembered. The combination of the three types of gates controls the output state. The key difference between RNN and LSTM lies in the formulation of the hidden vector (\(h_{t}\)). The hidden vector (\(h_{t}\)) in LSTM neural network can be obtained as follows:  
\[\left\{\begin{array}{c}i_{t}=\sigma(W_{xi}x_{t}+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b _{i})\\ f_{t}=\sigma(W_{xf}x_{t}+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_{f})\\ c_{t}=f_{t}c_{t-1}+i_{t}\text{tanh}(W_{xc}x_{t}+W_{hc}h_{t-1}+b_{c})\\ o_{t}=\sigma(W_{x0}x_{t}+W_{ho}h_{t-1}+W_{co}c_{t-1}+b_{o})\\ h_{t}=o_{t}\text{tanh}(c_{t})\end{array}\right. \tag{5}\]  
where \(i_{t}\), \(f_{t}\), \(o_{t}\), and \(c_{t}\) are the values of the input gate, the forget gate, the output gate, and the memory cell in the memory block; \(b_{i}\), \(b_{f}\), \(b_{o}\), and \(b_{c}\) are their corresponding bias values; \(\sigma\) is the sigmoid function; \(W_{x}\) represents the weights between input nodes and hidden nodes; \(W_{h}\) represents the weights between hidden nodes and memory cell; \(W_{c}\) represents the weights that connect memory cell to output nodes. The output \(y_{t}\) can then be obtained by Equation (2). The output sequence \(y=(y_{1},y_{2},\cdots,y_{T})\) will be obtained by iterating Equations (2) to (7) from times \(t=1\) to \(T\).