# A Novel NODE Approach Combined with LSTM for Short-Term Electricity Load Forecasting
## 3 Background Knowledge
### NODE  
Neural ordinary differential equation (NODE) is a new member of the neural networks family and is considered to be a continuous version of the ResNet [25]. Generally, the residual module of ResNet can be expressed by Equation (9):  
\[h_{t+1}=h_{t}+f(h_{t},\theta_{t}) \tag{9}\]  
The residual module indicates that ResNet is discrete and undefined between different levels, and \(h_{t}\) denotes hidden state at time \(t\). Assuming that the network is continuous and defined between layers, naturally we will obtain:  
\[\frac{\mathrm{d}h(t)}{t}=f(h(t),t,\theta) \tag{10}\]  
\[h(t_{1})=h(t_{0})+\int_{t_{0}}^{t_{1}}f(h(t),t,\theta)dt \tag{11}\]  
Equation (11) indicates that by inputting any given initial value in time \(t_{0}\), we can obtain the final value after an arbitrary length of time by using a black box ODE solver such as simple Euler's method or higher-order variants of the Runge-Kutta method [26]. Equation (10) shows that the function \(f\) defines the neural network, which is parameterized by the derivatives of \(h(t)\). In this paper, we use a two-layer dense neural network to represent the derivatives of \(h(t)\). Therefore, essentially, NODE can be considered as using a neural network to describe a continuous and very deep neural network.  
Figure 3 shows the latent trajectory when inputting an initial value at time \(t_{0}\) into NODE to propagate through Equation (11). The numbers of arrows express how many time points we set in order to observe the latent trajectory between \(t_{0}\) to \(t_{1}\). In actuality, the latent trajectory means that NODE can convert any initial value into a continuous time series that exists on latent space. The ODE solver in NODE selects an appropriate step size to approximate the true solution according to the given error tolerance. Forecasting accuracy can be improved in real-time or low-power applications. Hence, NODE has two main characteristics: it produces continuous latent trajectory in time series and dynamically adapts to the real solution, which allows the NODE-based model to learn rich dynamics and continuity of time series.  
Figure 2: The structure of BiLSTM.  
In this paper, the structure of the designed NODE block is shown in Figure 4. Firstly, we select a time series \(H_{t}\) ranging from \(t_{1}\) to \(t_{N}\) in time. Then, each value of the time series is inputted into the same ODE solver to generate a latent trajectory in latent space, which can be considered as a time series. When each value in the original time series is input into the ODE solver, the ODE solver will produce \(N\) (\(N\) is the length of time series) latent trajectory in latent space. After that, we select the last value of each latent trajectory and combine them into a new time series in latent space. Therefore, every value in the original time series is input into the same ODE solver and generates the same number of conversions in the latent space. The information about continuous time will not be discarded. Finally, a fully connected layer is used to shorten the length of the new time series in order to refine the new time series further in latent space. In fact, NODE actually maps the original time series into a new time series in latent space, having higher continuity, dynamic, and feature information in time. It will be easier for the following LSTM or BiLSTM layer to learn the series features processes by NODE.