# Deep Learning for Short-Term Load Forecasting--Industrial Consumer Case Study
## 3 Materials and Methods
### Gated Recurrent Units (GRU)  
Many articles such as [34] present a general literature review of ML and work done by [35; 36; 37] details a comprehensive review for DL algorithms used for forecasting. The GRU combines input gate and forget gate of LSTM into an update gate \(Z_{t}\) and the output gate in LSTM is called a reset gate \(R_{t}\) in GRU [38] as shown in Figure 4.  
The difference between GRUs and simple RNN is the implementation of gating for the hidden state, which uses a few steps to determine when the hidden state needs to be updated and when to reset.  
The input is a sequence of data, for a given time step \(t\), \(X_{t}\in\mathbb{R}^{n\times d}\) (n is the number of sequences, \(i\) is the number of inputs) and the hidden state of the previous time step is \(H_{t}\in\mathbb{R}^{n\times h}\) (\(h\) is the number of hidden units). Then, the reset gate \(R_{t}\in\mathbb{R}^{n\times h}\) and update gate \(Z_{t}\in\mathbb{R}^{n\times h}\) are implemented as follows in Equation (3):  
\[\begin{split} R_{t}&=\sigma(X_{t}W_{xx}+H_{t-1}W_{hr }+b_{r})\\ Z_{t}&=\sigma(X_{t}W_{xz}+H_{t-1}W_{hz}+b_{z}) \end{split} \tag{3}\]  
where \(W_{xx},W_{xz}\in\mathbb{R}^{i\times h}\) and \(W_{hr},W_{hz}\in\mathbb{R}^{h\times h}\) represent weight parameters and \(b_{r},b_{z}\in\mathbb{R}^{1\times h}\) are biases. In Equation (4) the reset gate \(R_{t}\) updates the candidate hidden state \(\widetilde{H}_{t}\in\mathbb{R}^{n\times h}\) at time step \(t\):  
\[\widetilde{H}_{t}=\tanh(X_{t}W_{xh}+(R_{t}\odot H_{t-1})W_{hh}+b_{h}) \tag{4}\]  
Figure 4: Structure of the GRU cell.  
where \(W_{xh}\in\mathbb{R}^{d\times h}\) and \(W_{hh}\in\mathbb{R}^{h\times h}\) are weight parameters, \(b_{h}\in\mathbb{R}^{1\times h}\) is the bias, and the symbol \(\odot\) is the Hadamard (elementwise) product operator. For the nonlinearity of the values in the candidate hidden state, \(\tilde{H}_{t}\) uses the tanh to maintain the values in the interval \((-1,1)\). Hidden state at time step \(t\), \(H_{t}\in\mathbb{R}^{n\times h}\), is a combination of previous hidden state \(H_{t-1}\) as presented in Equation (5) and current time step candidate hidden state:  
\[H_{t}=Z_{t}\odot H_{t-1}+(1-Z_{t})\odot\tilde{H}_{t} \tag{5}\]  
Activation functions used in the GRU cell are sigmoid and hyperbolic tangent Equation (6):  
\[\begin{split}\tanh(x)&=\frac{e^{2x}-1}{e^{2x}+1}\\ \sigma(x)&=\frac{1}{1+e^{-x}}\end{split} \tag{6}\]  
RNN model was implemented by [39] for forecasting non-residential loads (catering, electronic, and hotel industry), concluded that predicting each consumer is not as accurate as of the forecasts for substation loads, and obtained the best results using LSTM with MAPE ranging from 15.45% to 19.57%. Extreme gradient boosting regressor (XGBoost) for STLF was implemented by [40] and achieved MAPE of 3.74% for a horizon of one week for substation loads, with hourly steps, which is a total of 168 h. For one-step forecasting [41] are presenting a model based on FFNN and LSTM for air compressor electricity usage. The authors in [42] have also been used LSTM for nonlinear, non-stationary, and nonseasonal univariate electric load time series over 96 steps ahead with a MAPE of 5.35%, but it does not mention what type of load is forecasting. The RNN model fails on vanishing gradient decent as mentioned above, so LSTM and GRU are designed to compensate for its failure by using gates such as the ones in Figure 4. GRU is designed to provides a longer-term memory [43].  
Multiple hidden RNN layers can be stacked on each another. The main reason for stacking is to allow for greater model complexity. Deep RNN work better than shallower networks, as presented by [44], a multiple-layer deep architecture was better for machine-translation in an encoder-decoder framework and authors in [45] also showed improved results by using an architecture with several stacked recurrent layers for RNN.