# Recurrent Neural Network Based Short-Term Load Forecast with Spline Bases and Real-Time Adaptation
## 2 Methodology
### Recurrent Neural Network with Selected Bases  
RNN introduces loops in the network and allow internal connections among hidden units to enable exploration of the temporal relationships among the data [24]. The RNN structure with selected model bases taken from the resulting SPM described above is presented in the following. 1. General Structure of RNN  
Each of the RNN layers uses a loop to iterate over the time steps of the sequence. An RNN with a single hidden layer is illustrated below. The input training data is the adaptive lasso estimator effects, it is given as:  
\[\mathbf{x}=\left\{\mathbf{x}^{(1)},\mathbf{x}^{(2)},\dots,\mathbf{x}^{( \mathbf{t})},\dots,\mathbf{x}^{(\mathbf{T})}\right\},\ \mathbf{x}^{(\mathbf{t})}=\left(x_{1}^{(t)},x_{2}^{(t)},\dots,x_{p}^{(t)} \right)^{\mathsf{T}},\ t=1,\dots,T. \tag{11}\]  
The mapping of the output \(\mathbf{o}^{(\mathbf{t})}\) can be represented as:  
\[\mathbf{o}^{(\mathbf{t})}=\phi\Big{(}\mathbf{u}\mathbf{x}^{(\mathbf{t})}+ \mathbf{w}\mathbf{o}^{(\mathbf{t}-1)}+\mathbf{b}\Big{)} \tag{12}\]  
where \(\mathbf{o}^{(0)}=0\), \(t=1,\dots,\,T\), \(\phi\) are the activation functions, \(\mathbf{u}\) is the input weight for \(\mathbf{x}^{(\mathbf{t})}\), \(\mathbf{w}\) is the input weight for \(\mathbf{o}^{(\mathbf{t}-1)}\), where both are the same for all time points \(t\), and where \(\mathbf{b}\) is  
Figure 2: The scatterplot of temperature and load. a parameter in the model representing the bias of the hidden layer and the output layer. The RNN uses the hidden state \(\mathbf{o}^{(t)}\) at time step \(t\) to memorize the network. When \(t=1\), an RNN is the same as an ANN. Consider a RNN model with \(k\) multilayer perceptrons, where in the \(i\)th hidden layer there are \(m\) neurons and the time step is \(T\). Then the \(i\)th layer vector for these \(m\) neurons in the (\(i-1\))th layer at time \(t\) is expressed as: \(\mathbf{h}_{i}^{(t)}=\left(h_{i,1}^{(t)},\ldots,h_{i,m}^{(t)}\right)^{\top},t= 1,\ldots,T\), which can be obtained through the following equation:  
\[\mathbf{h}_{i}^{(t)}=\phi\left(\mathbf{U}_{i}\mathbf{h}_{i-1}^{(t)}+\mathbf{W }_{i}\mathbf{h}_{i}^{(t-1)}+\mathbf{b}_{i}\right),\;i=1,\ldots,k, \tag{13}\]  
where \(\mathbf{h}_{0}^{(t)}=\mathbf{x}^{(t)},\mathbf{h}_{i}^{(0)}=0,\;\mathbf{W}_{i}\) and \(\mathbf{U}_{i}\) are the transition and input weight matrices. The mapping of the output \(\mathbf{o}^{(t)}\) can be represented as:  
\[\mathbf{o}^{(t)}=\phi\left(\mathbf{V}\mathbf{h}_{k}^{(t)}+\mathbf{b}_{o} \right),\;t=1,\ldots,T, \tag{14}\]  
where \(\mathbf{V}\) is output weight matrices and \(\mathbf{b}_{0}\) is a parameter in the model representing the bias of the hidden layer and the output layer. With suitable choices for the parameters, such as the number of layers \(k\), number of neurons in each layer \(m\), and time steps of a sequence \(T\), the RNN is expected to perform better than a more general model structure considering time effects in the neural network framework for STLF problems. We build an RNN model with \(k\) multilayer perceptrons in Python using the tensorflow library. 2. Configuration Architecture  
The RNN training process is heavily influenced by the choice of hyper parameters: sequence size, number of hidden layers and number of nodes per hidden layer. Efforts were made to search for a hyper parameter space to test different parameter combinations most suitable for TPC system. The experiment was conducted using standard RNN network to provide a best set of hyper-parameters. The results shown in Table 1 indicate that the best number of hyper-parameters units is 14, Layers is 3, and Time steps is 4. Figure 3 shows the structure of the RNN used in this work, where the input \(\left\{\mathbf{X}^{(1)},\ldots,\mathbf{X}^{(t)}\right\}\) are values obtained from the selected model bases in the SPM model.