## 2 LSTM Neural Network
### Bidirectional LSTM Neural Network  
In order to overcome the accumulative error problem, a bidirectional LSTM is proposed here, which is shown in Figure 3. The bidirectional LSTM neural network consists of two layers of LSTM structure; one is used to calculate the hidden vector from the front to the back, and the other is used to calculate the hidden vector from the back to the front. The output of the bidirectional LSTM neural network is determined by these two layers.  
The bidirectional LSTM neural network is different from the traditional feed-forward mechanism neural network. The internal nodes in each layer do not connect with each other in bidirectional LSTM. A directional loop is introduced in the connection of hidden layers, foregoing information; results are memorized and stored in the memory unit, which can improve the association of single pieces of information in different time series. The current output of the neural network is determined bycombining the previous output and the current input. However, with the increasing amount of input data in the time series, there will be gradient disappearance and gradient explosion problems due to the lack of delay window width.  
Based on the traditional LSTM model, the bidirectional LSTM neural network will fully consider the front and back correlation of the load data in time series and improve the model performance for the sequence classification problem especially. During the training process, the input data sequence of the forward layer is the training data, and the backward layer is the reverse copy of the input data sequence. The results of bidirectional structure prediction are determined by the previous input and the latter input, which increases the dependence between the training data to avoid the forgetting of the order information.  
Figure 3 shows that the forward layer calculates the forward direction from \(1\) to \(t\), and it saves the output of the forward hidden layer at each moment. The backward layer calculates the reverse time series and saves the output of the backward hidden layer at each moment. Finally, the output of the bidirectional LSTM neural network is calculated by combining the corresponding output results of the forward layer and backward layer at each time point. The bidirectional LSTM neural network can be written as:  
\[s_{t}=f(Ux_{t}+Ws_{t-1}) \tag{7}\]  
Figure 3: Basic structure of bidirectional LSTM.  
\[s_{t}^{\prime}=f\big{(}U^{\prime}x_{t}+W^{\prime}s_{t+1}^{\prime}\big{)} \tag{8}\]  
\[o_{t}=g\big{(}Vs_{t}+V^{\prime}s_{t}\big{)} \tag{9}\]  
where \(s_{t}\) is the state variable of the hidden layer at time \(t\), \(o_{t}\) is the state variable of the output layer at time \(t\), \(s_{t}^{\prime}\) is the state variable of the reverse hidden layer at time \(t\), \(x_{t}\) is the input vector, \(g\) and \(f\) are activation functions, \(V\), \(W\), and \(U\) are the weight matrix from the hidden layer to the output layer, the hidden layer, and the input layer to the hidden layer, and \(V^{\prime}\), \(W^{\prime}\), and \(U^{\prime}\) are the corresponding reverse weight matrix. The state weight matrix of the forward layer and the backward layer is not shared information between the two. The forward layer and backward layer are calculated in turn and give the result of each time. The final output \(o_{t}\) depends on the sum of the forward calculation result \(s_{t}\) and the reverse calculation result \(s_{t}^{\prime}\).