# Long Short-Term Memory Neural Network Applied to Train Dynamic Model and Speed Prediction
## 3 Methodology
### LSTM Network Structure  
The LSTM network is a variant of deep neural networks, which was first introduced by [PERSON] [20]. Distinguished from traditional neural networks, there exists an internal hidden state in the units that constitutes the LSTM network. Generally, an LSTM network is made up of one input layer, one recurrent hidden layer and one output layer. The unit in the recurrent hidden layer is termed as the memory block, which consists of memory cells and adaptive, multiplicative gating units. In the memory cell, the state vector is saved to aggregate the previous input data. By tuning the parameters of gating units, the input data, the output data and the state data are mixed to update the current state. The control mechanism is summarized as follows. Note that all the symbol definitions can be found in Table 1.  
Without ambiguity, at each time iteration \(t\), let \(\mathbf{x}_{t}\) denote the hidden layer input feature which is directly fed to the LSTM cell through the input gate. The input gate can be written as [46]  
\[\mathbf{i}_{t}=\sigma_  
To deal with the gradient diffusion and the gradient explosion problems [47], in LSTM networks, the forget gate is deliberately designed  
\[\mathbf{f}_{t}=\sigma_{f}(\mathbf{W}_{f}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}]+ \mathbf{b}_{f}). \tag{8}\]  
According to Equation (8), once the contents of the LSTM cell are out of date, the forget gate helps to update parameters by resetting the memory block. Similarly, the dimension of \(\mathbf{W}_{f}\) is \(d_{c}\times(d_{h}+d_{x})\). The output gate is defined as  
\[\mathbf{o}_{t}=\sigma_{o}(\mathbf{W}_{o}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t}] +\mathbf{b}_{o}). \tag{9}\]  
The definitions of \(\mathbf{o}_{t}\), \(\sigma_{o}(\cdot)\), \(\mathbf{W}_{0}\) and \(\mathbf{b}_{0}\) are listed in Table 1.  
To update the cell statement, \(\tilde{\mathbf{c}}_{t}\) is defined as the state update vector,which is calculated as  
\[\tilde{\mathbf{c}}_{t}=tanh(\mathbf{W}_{c}\cdot[\mathbf{h}_{t-1},\mathbf{x}_{t }]+\mathbf{b}_{c}). \tag{10}\]  
In Equation (10), \(tanh(\cdot)\) represents the hyperbolic tangent function. The reason for using this particular function is that other activation functions (e.g., the rectified linear unit) may have very large outputs and cause the gradient explosion problem [48]. The hyperbolic tangent function can be written as  
\[tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}. \tag{11}\]  
Based on the results of Equations (7) and (11), at a certain time step \(t\), the current cell state is updated by  
\[\mathbf{c}_{t}=\mathbf{f}_{t}\circ\mathbf{c}_{t-1}+\mathbf{i}_{t}\circ\tilde{ \mathbf{c}}_{t} \tag{12}\]  
where \(\circ\) represents the scalar product of two vectors. By observing Equation (12), thanks to the control of the input gate and the forget gate, one can find that the current input and long-term memories are combined to form a new cell state. At last, the hidden layer output is decided by the output of the output gate and the current cell state  
\[\mathbf{h}_{t}=\mathbf{o}_{t}\circ tanh(\mathbf{c}_{t}). \tag{13}\]  
The internal structure of an LSTM cell is shown in Figure 2.  
Figure 2: The internal structure of an LSTM cell. The training process can be regarded as a supervised learning process and many objective functions are designed to minimize the average differences between the outputs of the LSTM network and the actual values. In the training process, the parameters (such as the weight matrices and the biases vectors) are optimized. Errors are intercepted at the LSTM cells and are vanished thanks to the forget gate. The gradient descent optimization algorithm is adopted to obtain the gradient of the objective function, and then the Back Propagation Through Time (BPTT) [49] algorithm is used to train and update all the parameters to increase the prediction accuracy. The derivations of the training process are not covered in this paper due to space restrictions. One can refer to the work in [46] for detailed execution steps.