## 3 Problem Formulation
### Day-Ahead Hourly Demand Estimatio  
In this study, the deep neural networks (DNNs), which have widely been established and used in different fields, especially macroscopic short-term load forecasting (STLF) of electrical demand, are utilized and trained to perform the demand prediction [48,49]. DNNs are neural networks with more than three layers [50], used for supervised learning, in which the algorithm trains the computer to learn from given data and make future predictions or classifications. As can be seen in a [28], due to their intricate connections and nonlinear activations, DNNs can model any non-linear function [51]. Neurons are the building blocks of a neural network, as shown in Figure 7b [50], where the output \(a_{j}^{[l]}\) for the \(j\)th neuron in hidden layer \(l\) with an activation \(g^{[l]}\) is depicted as follows:  
\[a_{j}^{[l]}=g^{[l]}\Big{(}w_{j}^{[l]T}*a_{i}^{[l-1]}+b_{j}^{[l]}\Big{)}=g^{[l ]}\Big{(}z_{j}^{[l]}\Big{)} \tag{5}\]  
\(z_{j}^{[l]}\) represents the output of the neuron \(j\) prior to the activation in the hidden layer \(l\), where \(w\) is the weight vector of that neuron, \(b\) is the bias and \(a_{i}^{[l-1]}\) is the input to the neuron from the \(i\)th training example. The choice of \(g^{[l]}\) greatly affects the performance of training the network. A detailed comparison of the popular activation functions currently in use and their comparison can be found in [52]. In this study, the exponential linear unit (Elu), which is an improved variant of the RELU (rectified linear unit), is used as the activation functions due to its higher speed of converging and performance [53], better sparsity [54] and avoiding the dying neuron problem where the saturation plateau in its negative region allows for more enhanced learning of robust representations [52]. The main objective of training a neural network is to update all the weights and biases in the hidden and output layer to decrease the error of the neural network, where the mean squared error (MSE) is used for regression problems. The total error is calculated by forward propagating the input data into the network and is used to measure the performance of the neural network at each training iteration, and the error function is used with the backward propagation algorithm to update the parameters (\(\theta\))s of the model using gradient descent [55]:  
\[\theta_{t+1}=\theta_{t}-\eta\cdot\
abla_{\theta_{t}}J(\theta_{t}), \tag{6}\]  
In the above equation, each weight \(\theta_{t}\) at training step t is updated by calculating the partial derivative \(\
abla_{\theta_{t}}J(\theta_{t})\) of the error function \(J(\theta_{t})\) relative to each parameter \(\theta_{t}\) in the network using the chain rule. \(\eta\) denotes the learning rate and determines the speed of gradient descent, where choosing a very big value might cause the model not  
Figure 7: (a) Neural network representation, and (b) a single neuron. to converge towards the global minimum or too low a value slows down the learning process. Currently, there are advanced versions of gradient descent, with the most popular being momentum [56], RMSprop [57], and adaptive moment estimation (Adam) [58]. Since Adam is a combination of both momentum and RMSprop, it has been used in this research for its well-known high performance, with more details found in [55]. Finally, Figure 8 depicts the training and tunning process for the proposed deep neural network model used in this study to predict the day-ahead 24-h demands. The model predicts the day-ahead demand at a specific hour \(h\) starting from the end of the previous day; the model is run 24 times to predict the next day's hourly demand hour by hour. The selection and detailed analysis of the input features is discussed in detail in the following section.