## 2 Methodology
### Bi-LSTM-Attention Model for Residential Daily Load Forecasting
#### 2.3.2 Self-Attention Mechanism  
The attention mechanism is a probability-weighted mechanism that mimics the attention of the human brain [39]. When the human brain observes things, it will focus on specific places and ignore other places. There is an intrinsic correlation between hidden features. Hidden features of different time steps have different effects on the prediction results, which is unrecognizable by the Bi-LSTM network. Therefore, the self-attention mechanism is well suited to load prediction methods involving LSTM networks. This method highlights the more important factors by assigning different probability weights to the inputs, ultimately further improving the prediction accuracy of the model.  
The attention layer assigns the feature weight of the model learning to the input vector of the next time step to highlight the impact of key features on the sequence. The final data are entered into the fully connected layer. After the virtual function processing of the fully connected layer, the predicted load value is obtained. The implementation process is as follows:  
Step 1 Calculate the correlation between each current input feature and the current load.  
Step 2 Use the Softmax formula to convert each correlation into a probabilistic form.  
Step 3 Multiply each obtained probability by the implicit representation of the corresponding input feature to represent the contribution of the feature to the predicted load. The contributions of all the input features are then added together as the input parts to predict the next load data.  
The process can be expressed by Equations (22)-(24):  
\[e_{t}=Vtanh(Wh_{t}+b) \tag{22}\]  
\[\alpha_{t}=\frac{exp(e_{t})}{\sum_{j=1}^{n}exp(e_{j})} \tag{23}\]  
\[C_{t}=\sum_{t=1}^{n}a_{t}h_{t} \tag{24}\]  
Figure 3: The structure of the Bi-LSTM model.  
where \(e_{t}\) and \(\alpha_{t}\) are, respectively, the weight score and the attention weight corresponding to different features at the current time t; \(V\) and \(W\) is the weight of the multilayer perceptron when calculating the attention weight; \(b\) is the bias parameter of the multilayer perceptron when calculating the attention weight; N is the dimension of the input vector of the prediction model; and \(C_{t}\) is the output of the attention mechanism at time t. This paper introduces attention mechanisms based on past and future input characteristics. Thus, the model can give different weights to the input features to highlight the influence of strong correlates and reduce the shadow of weakly correlated factors.  
As seen in Figure 4, after receiving the input for a time window, the model passes the sequence data to the forward LSTM hidden layer and the reverse LSTM hidden layer, and the two combine to output the processed vector. The attention layer takes the data processed by the LSTM layer as input, calculates the weight vector, and then combines the weight vector with the shallow output to obtain a new vector input into the linear transformation layer. The green boxes in the figure indicate the resulting attention weight parameters. Finally, the linear transformation layer calculates the predicted value.  
This paper combines the Bi-LSTM algorithm with an attention mechanism. This method mines the contribution of the contextual information in the time series data, the historical resident data in different time dimensions, and the external features to predict the results. Thus, it highlights the effective features and further improves the accuracy of residential electricity forecasting.  
Figure 4: The structure of the Bi-LSTM-Attention Model.