# Deep Learning for Short-Term Load Forecasting--Industrial Consumer Case Study
## 3 Materials and Methods
### Deep Learning (DL)  
There is a vast spectrum of terminology that tends to be confusing because of the interchangeability of utilization: artificial intelligence, machine learning, deep learning, artificial neural networks, or reinforcement learning. Machine learning is considered a subdomain of artificial intelligence [26]. Deep learning is a subdomain of machine learning, and neural networks are at the core of deep learning algorithms. The dissimilarity between a simple neural network and a deep learning algorithm is the number of neurons and structure of hidden layers (deep learning must have more than two hidden layers). ML techniques can be broadly grouped in two large sets--supervised and unsupervised. The methods related to the supervised learning paradigm classify objects in a pool using a set of known annotations/attributes/features. The unsupervised learning techniques  
Figure 2: Hourly industrial load curve used for forecasting. form groups among the objects in a batch by identifying similarities and then use them for classifying the unknowns. Reinforcement learning is a behavioral algorithm similar to supervised learning, not using sample data for training but by trial and error. A sequence of successful outcomes will develop the best recommendation or policy for a given problem. DL models were developed to map a complex function between the last \"n\" hours (timesteps--also called lag) and predict how the time series can continue in the future, as presented in Figure 3. Most machine learning algorithms have hyperparameters; by setting the parameters, the ML algorithm can offer the desired results. The values of hyperparameters should not be calculated in the learning stage (because of the overfitting problem). To evaluate the generalization of the DL methods on the training data, we use a testing set of time series that the built network in the training stage did not experience prior. In our work we use deep recurrent neural networks (DRNN) and variations of the algorithm. RNN is a sequential data neural network processor because it has internal memory to update the state of each neuron in the network with the previous input. Because RNN train with backpropagation, this can fail because of vanishing gradient descent. Deep networks combine multiple layers into the architecture and provide more significant benefits. Neural networks build functions by multiplying a weight matrix to the input vector, add bias, and then apply the activation function to obtain non-linearity in the output. To calculte the current state, we can use the following Formula (1). \[h_{(t)}=f(h_{(t-1)},x_{(t)};\theta)=tanh\Big{(}Wh_{(t-1)}+Ux_{(t)}+b\Big{)} \tag{1}\]  
In the equation above, the parameters \(\theta\) include \(W\), \(U\), and \(b\). The \(W\) and \(U\) are parameters representing weight matrices, and \(b\) is the bias vector. Hyperbolic tangent is the activation function \(\tanh\) for the hidden state; other activation functions could be used. The output of the RNN cell is:  
\[o_{(t)}=g(h_{(t)};\theta^{\prime})=Vh_{(t)}+c \tag{2}\]  
where \(V\) and \(c\) denote the weight and bias, the parameters \(\theta^{\prime}\) of the output function \(g\). Matrix \(V\) and vector \(c\) are multidimensional outputs. The same set of parameters is applied at each time step for every RNN-cell [27]. LSTM was developed to improve the vanishing or exploding gradient problem and has become one of the most popular RNN architectures to date and was introduced by [28]. GRUs were later introduced by [29] as a simpler alternative and have also become quite popular. We will use both architectures in the context of the vanishing or exploding gradient problem. Many variants of LSTM and GRUs exist in the literature, and even the default implementations in various deep learning frameworks often differ. Performance is often similar, but this can confuse when reproducing results.