## 2 LSTM Neural Network  
### LSTM Neural Network  
The LSTM neural network was proposed in 1997, which is a time-domain deep learning neural network. Compared with traditional recurrent neural network, there are two special parts: the forget gate and the memory unit in the hidden layer of the LSTM neural network. A long-term information stream from the input to output can improve the memory capacity of the neural network in the process of training. The structure of the LSTM cell is shown in Figure 1. It consists of four computing units: namely, the output gate, forget gate, memory unit, and input gate, respectively.  
Based on the output \(h_{t-1}\) of the last hidden layer and the current input \(x_{t}\), a new value of \(f_{t}\) is generated based on the activation function \"Sigmoid\", which determines whether to let the information \(C_{t-1}\) learned in the last moment pass through; that is, how much of the last cell state \(C_{t-1}\) is saved to the current time \(C_{t}\). The function between \(h_{t-1}\),\(x_{t}\) and \(f_{t}\) can be written as:  
\[f_{t}=\sigma\Big{(}W_{f}\cdot[h_{t-1},x_{t}]+b_{f}\Big{)} \tag{1}\]  
Figure 1: LSTM cell structure.  
where \(\sigma\) is the \"Sigmoid\" function, and the range of the output value of the \"Sigmoid\" function is [0,1]. \(W_{f}\) is the weight matrix of the forget gate, \(b_{f}\) is the bias of the forget gate, \(f_{t}\) is the value of the forget gate which decides the forgetting factor of long-term memory information. The value of \(f_{t}\) is between [0,1].  
The threshold of LSTM consists of a sigmoid activation function and dot multiplication operation. After the hidden layer of the previous moment enters the forget gate, the function gives the judge information about whether it is updated or not. However, the cell state rolls continuously and runs in the horizontal direction.  
\[I_{t}=\sigma(W_{t}\cdot[h_{t-1},x_{t}]+b_{i}) \tag{2}\]  
\[\widetilde{C}_{t}=\tanh(W_{\rm C}\cdot[h_{t-1},x_{t}]+b_{c}) \tag{3}\]  
\[C_{t}=f_{t}*C_{t-1}+I_{t}*\widetilde{C}_{t} \tag{4}\]  
where \(\tanh\) is the hyperbolic tangent activation function, \(\widetilde{C}_{t}\) is the temporary unit state of \(C_{t}\), \(W_{\rm C}\) is the weight matrix of the memory unit, \(b_{c}\) is the bias of the memory unit, and \(I_{t}\) is the output value of the input gate. The current cell state \(C_{t}\) is the sum of the original state and the updated state.  
\[o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o}) \tag{5}\]  
\[h_{t}=o_{t}*\tanh(C_{t}) \tag{6}\]  
where \(W_{o}\) is the weight matrix of the output gate, \(o_{t}\) is the output value of the output gate, and \(b_{o}\) is the bias of the output gate. The initial output \(h_{t}\) is obtained through the sigmoid layer, \(\tanh(C_{t})\) is between \(-1\) and \(1\).  
The signal passes through the input gate, output gate, and forget gate in turn, and it realizes information storing and maintaining in the current time period. The LSTM structure of the neural network shows that the input variable is transmitted horizontally from input to output directly. Hence, the prediction error will accumulate continuously and suddenly swell in direct proportion with the previous time in the prediction model. Figure 2 shows the LSTM prediction accumulation error in the power load forecasting in one day. The short-term load forecasting method usually takes a day or a week as the training dataset. Three days' data are taken as sample forecast in Figure 2. The error accumulation occurs in the LSTM prediction results with the increase in the step of the prediction data, and the error will become larger and larger as time goes on.  
Figure 2: Accumulate error of the forecasting model.