# A Novel NODE Approach Combined with LSTM for Short-Term Electricity Load Forecasting
## 4 Data and Methodology
### Model Settings  
The dataset is a univariate time series collected from 2 May to 4 July (2019-2021). One year of data has 3024 time steps, and 1814 time steps are used as training set. The size of the sliding window is set to 20. The hidden dimension of the NODE block is set as 100, which means that it has two fully connected layers with 100 hidden nodes. In order to compare the function of NODE, we use the same NODE block in different model groups. The output size of the final fully connected layer after the LSTM layer is set to 1.  
The overall parameters are listed in Table 1. In OCBL and CBL, the convolutional size of the kernel is set as 3 and the number is set as 64. Then, we conducted a grid search over the size of hidden states for LSTM and BiLSTM among [20; 30; 40; 50; 60; 70; 80; 90; 100], and the number of hidden nodes of the LSTM and BiLSTM layers achieving the best performance the over validation set is used for test (30 and 50, respectively). In the end, a fully connected layer with one node is used as the output.  
In OBLL and BLL, we gradually reduce the size of hidden nodes to shorten the length of generated series to mix the rich information. In detail, we select the number of hidden nodes of the BiLSTM layer by conducting grid searches among [50; 60; 70; 80; 90; 100]. To decide the number of hidden nodes for the two LSTM layers, we conduct two grid searches over [30; 40; 50] and [10; 20; 30], respectively. The number of hidden nodes of the BiLSTM and stacked LSTM layers achieving the best performance over the validation set is used for test (50, 30, and 10, respectively). After the LSTM layer, two fully connected layers with 50 and 20 hidden nodes are used. Similarly, a fully connected layer with single hidden node is used as model output.  
In OBBLL and BBLL, similarly, we gradually reduce the size of hidden nodes. In detail, we select the number of hidden nodes of these two BiLSTM layers by conducting two grid searches among [50; 60; 70; 80; 90; 100] and [30; 40; 50], respectively, and we select the number of hidden nodes of the first LSTM layer by conducting two grid searches among [20; 30; 40] and [10; 20; 30], respectively. The number of hidden nodes of the stacked  
Figure 8: The model structures of OBBLL and BBLL.  
BiLSTM and stacked LSTM layers achieving the best performance over the validation set are used for test (50 and 30 for stacked BiLSTM; 20 and 10 for stacked LSTM, respectively). A fully connected layer with 20 hidden nodes is used and a fully connected layer with single hidden unit is used as output in the end.  
In this paper, all fully connected layers use rectified linear units (RELUs) as activation function. Adam optimizer is adapted for model training. The initial learning rate is 0.001. In addition, mean squared error (MSE) is used as loss function to perform gradient descent. In order to avoid overfitting, an early stopping strategy is used to supervise the loss in the validation dataset, and the patience of early stopping technology is 10. The size of the minibatch is 32, which is selected by conducting a grid search among {16, 32, 64, 128}. Maximum epochs are set to 1000. When the loss in the validation dataset keeps not descending in more than 10 epoches, the early stopping strategy will end training. We train each model five times to obtain an average experimental result.