## 3 LSTM Attention Decoder  
RNN is an important branch of DL that can be used not only for the processing of time series data, but also to focus on the timing of the feature model. In addition, it is useful for processing sequence data where the front input affects the behind output [24]. However, there are problems of gradient disappearance and gradient explosion with the expansion of timeline in traditional RNN. The gradient disappearance occurs because of the Sigmoid function. The Sigmoid function is usually employed in the output layer, but the derivative of this function ranges from 0 to 0.25. When the BP algorithm is utilized to calculate the gradient, the gradient of each layer will be reduced to 1/4 of the original. If there are many network layers, the gradient is going to become really small. The value of the initial network weight needs to be set larger than 1 to avoid this phenomenon, but it will lead to gradient explosion [25]. So, it has great limitations in the prediction of long time series data. Figure 2 shows the diagram of the RNN network unfolded along the time line. In order to solve the problems existing in RNN, the LSTM network is proposed to solve the ubiquitous long-term dependence problem in the network, which has been proved to be effective in solving the gradient disappearance and gradient explosion problems caused by RNN [24]. The biggest difference between LSTM and RNN is that RNN has only one state inside a single recurrent structure, while LSTM has four states and each structure  
Figure 2: RNN network structure unfolded along the time line. is composed of an input gate, forget gate, output gate, and cell state. The diagram of the LSTM network is shown in Figure 3. \(\otimes\) denotes the multiplication of vector elements and \(\oplus\) denotes the addition of vectors elements. Both the input and output gates open and propagate signals only when previous information is needed. In this way, the previous information can be saved selectively. The function of the forget gate is to receive the error from the memory unit and \"forget\" the value stored in the memory unit when needed, so as to achieve the control of the network weights. LSTM can prevent the gradient disappearance problem by defining \"gate\" operations (i.e., \(f_{t}\), \(i_{t}\), \(o_{t}\)) as follows:  
\[f_{t}=\sigma(W_{f}[h_{t-1},x_{t}]+b_{f}) \tag{6}\]  
\[i_{t}=\sigma(W_{i}[h_{t-1},x_{t}]+b_{i}) \tag{7}\]  
\[\widetilde{C}=\tanh(W_{c}[h_{t-1},x_{t}]+b_{c}) \tag{8}\]  
\[C_{t}=f_{t}*C_{t-1}+i_{t}*\widetilde{C} \tag{9}\]  
\[o_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o}) \tag{10}\]  
\[h_{t}=o_{t}*\tanh(C_{t}) \tag{11}\]  
where \(\sigma\) denotes the sigmoid() activation function; \(*\) denotes the element-wise multiplication; \(f_{t}\) denotes the probability of forgetting the previous information, and ranges from 0 to 1; \(h_{t-1}\) denotes the output of the previous moment; \(x_{t}\) denotes the input at the current moment; \(W_{f}\) and \(b_{f}\) denote the weight and bias of the forget gate, respectively; \(i_{t}\) denotes the retained probability of information from input gate; \(W_{i}\) and \(b_{i}\) denote the weight and bias of the input gate, respectively; \(\widetilde{C}_{t}\) denotes the information from the input gate, and the tanh activation function normalizes the values to the range \(-1\) to \(1\); \(W_{c}\) and \(b_{c}\) denotes the weight and bias of the cell state, respectively; \(C_{t}\) denotes the cell state; \(o_{t}\) denotes the probability of information being sent from the output gate; and \(W_{o}\) and \(b_{o}\) denote the weight and bias of the output gate, respectively. As shown in Equation (9), the value of \(f_{t}\) multiplied by \(C_{t-1}\) denotes the selectively forgotten information from the previous moment. The second term, \(\widetilde{C}_{t}\) multiplied by \(i_{t}\) denotes the selectively forgetting information in the present moment. At this moment, the cell state \(C_{t}\) is updated.