# Multivariate Time Series Data Prediction Based on ATT-LSTM Network
## 3 ATT-LSTM Model  
### Long Short-Term Memory Neural Network  
Long short-term memory neural network (LSTM) is a special kind of recurrent neural network (RNN). LSTM was first proposed by [PERSON] and [PERSON] [7], which effectively solves the RNN network time delay and gradient disappearance. LSTM [24] is widely used in text generation, machine translation, speech recognition, generated image description, and video tagging. LSTM is more potent than ordinary RNN because it can selectively record or forget the input information. On the one hand, it has a powerful memory block, which mainly contains three gates (memory gate, forgetting gate, and output gate), on the other hand, it also has a memory unit, which can control the transfer of information to the next moment [25]. The network structure of the LSTM is shown in Figure 1.  
The unit structure of LSTM is mainly composed of input gate, forget gate, output gate, and unit state [26]. The unit structure diagram of the LSTM is shown in Figure 2.  
\(Z\) represents the input, and \(Z_{i}\) represents the control signal of the input gate. \(Z_{f}\) represents the control signal of the forget gate, \(Z_{o}\) represents the control signal of the output gate, and the \(f(x)\) function is usually the Sigmod function:  
\[f(x)=\frac{1}{1+e^{-x}}. \tag{1}\]  
This function is used to indicate the degree of opening of the door, and the value range is within [0, 1]. Both \(g(x)\) and \(h(x)\) are activation functions. First, \(Z\) passes the activation function to get \(g(Z)\), \(Z\) passes the Sigmod function to get \(f(Z_{i})\) and then multiplies it to get \(g(Z)f(Z_{i})\). \(Z_{f}\) gets \(f(Z_{f})\) through the Sigmod function and then multiplies it with the value a of the memory unit at the last moment to get \(\mathcal{G}(Z_{f})\). Finally, update the value of the memory unit to:  
\[c^{\prime}=g(Z)f(Z_{i})+cf\left(Z_{f}\right). \tag{2}\]  
Figure 1: LSTM network structure diagram.  
Figure 2: LSTM unit structure diagram.  
\(c^{\prime}\) passes the activation function to get \(h(c^{\prime})\), \(Z_{o}\) passes the Sigmod function to get \(f(Z_{o})\), and multiplies it to get the output:  
\[\text{a}=h(c^{\prime})f(Z_{o}). \tag{3}\]  
The input \(x_{t}\) at time \(t\) and the output \(h_{t-1}\) of the hidden layer neuron at time \(t-1\) are jointly used as the input part of the hidden layer at time \(t\), multiplied by different weight vectors. The activation function is used to obtain the control signals of the three gates \(Z_{f}\), \(Z_{i}\), \(Z_{o}\), and the input value \(Z\). The formula is as follows:  
\[Z_{f} =\omega_{f}\cdot[h_{t-1},x_{t}]+b_{f} \tag{4}\] \[Z_{i} =\omega_{i}\cdot[h_{t-1},x_{t}]+b_{i}\] (5) \[Z_{o} =\omega_{o}\cdot[h_{t-1},x_{t}]+b_{o}\] (6) \[Z =\omega_{x}\cdot[h_{t-1},x_{t}]+b_{x}. \tag{7}\]  
Among them, \(b_{f}\), \(b_{i}\), \(b_{o}\), and \(b_{x}\) are the biases of different connection weights. After the LSTM unit is operated, the value \(c\) of the memory unit is updated Equation (1) and the Formula (8) is obtained. The output of hidden layer neurons is Formula (9):  
\[c^{\prime} = g(\omega_{x}\times[h_{t-1},x_{t}]+b_{x})f(\omega_{i}\times[h_{t -1},x_{t}]+b_{t})\] \[+cf(\omega_{f}\times[h_{t-1},x_{t}]+b_{f}) \tag{8}\] \[h_{t} = h(c^{\prime})f(Z_{o})\] (9) \[= h(g(\omega_{x}\cdot[h_{t-1},x_{t}]+b_{x})f(\omega_{i}\cdot[h_{t -1},x_{t}]+b_{i})\] \[+cf(\omega_{f}\cdot[h_{t-1},x_{t}]+b_{f}))f(\omega_{o}\cdot[h_{t -1},x_{t}]+b_{o})\]  
It can be seen from the above equations that the input of LSTM not only includes the output \(h_{t-1}\) of the hidden layer neuron at the last moment, but also includes the value of the memory unit in the LSTM unit. The LSTM network can effectively avoid the occurrence of gradient disappearance, can memorize long-term historical information, and fit long-term time series data more effectively [27].