## 3 Rationale of Deep Learning Implementation
### Feedforward Neural Networks (FNNs)  
The basic architecture of an FNN is depicted in Figure 1. In this architecture, the input vector \(x_{i}(t)\) is associated with a weight \(w_{i}(t)\) and bias \(b_{i}(t)\). The activation function \(f()\) is applied to each neuron, and the network learns by adjusting the weights. The weight updates are accomplished through the backpropagation error \(E\), allowing the network to iteratively refine its predictions. Normally, an FNN is constructed by applying a Back-Propagation (BP) learning algorithm; the BP learns the neurons' connecting weights, which are adjusted over the input-output dataset, as shown by Equation (1). This helps the FNN model to learn the behavior of the data very quickly. Suppose that \(\mathbf{X_{i}^{l}}\) is the input vector, where \(x_{1,\dots,l}^{[1,\dots,L]}\) for layers \(l\in{1,\dots,L}\) for input \(i\) represents the set of half-hourly demand data, including the calendar and weather variables passing into the layers. The output \(Y_{i}^{L}=X_{i}^{l}\) is equivalent to \(h^{L}[W^{[L]}X^{[L-1]}+b^{[L]}]\), where \(h^{[l]}(. )\) are the activation functions, \(b^{[l]}=(b_{1}^{[l]},\dots,b_{j}^{[l]})^{T}\) are the bias, and \(W^{[l]}\) are the weights [48]. These parameters are optimized using the gradient descent algorithm, as follows:  
\[p_{i+1}=p_{[i]}-\lambda\
abla E(p) \tag{1}\]  
where \(p\) represents the parameters, \(\lambda\) represents the learning rate, and \(E(p)\) represents the mean square error (MSE), sometimes called the loss function. In Equation (2), \(x_{i}\) is the predicted value, \(t_{i}\) is the target value, and \(n\) is the amount of half-hourly demand data. The search for the minimum loss function is commonly performed by computing its gradient \(\frac{\partial}{\partial p}E(p)\), which indicates how the function changes with small changes in the parameters. \[E(p)=MSE=\frac{1}{n}\sum_{i=1}^{n}(f(x_{i},p)-t_{i})^{2} \tag{2}\]  
Figure 1: Architecture of a Feedforward Neural Network. From above two equations Equations (1) and (2), the final equations for updating the weights and biases are  
\[ \begin{split}\text{\emph{for weight update:}}\\ \text{\emph{W}}_{[i+1]}^{[L]}&=W_{[i]}^{[L]}- \lambda\frac{\partial E(p)}{\partial W^{[L]}}\\ \text{\emph{for bias update:}}\\ \text{\emph{b}}_{[i+1]}^{[L]}&=b_{[i]}^{[L]}- \lambda\frac{\partial E(p)}{\partial b^{[L]}}\\ \end{split}  \tag{3}\]  
The backpropagated errors \(\frac{\partial E(p)}{\partial W^{[L]}}\) and \(\frac{\partial E(p)}{\partial b^{[L]}}\) for Equation (3) can be obtained using the chain rule:  
\[ \begin{split}\frac{\partial E}{\partial W^{[L]}}&= \frac{\partial E}{\partial X^{[L]}}\circ\frac{\partial X^{[L]}}{\partial A^{[ L]}}\frac{\partial A^{[L]}}{\partial W^{[L]}}^{T}\\ \frac{\partial E}{\partial b^{[L]}}&=\frac{\partial E }{\partial X^{[L]}}\circ\frac{\partial X^{[L]}}{\partial A^{[L]}}\frac{ \partial A^{[L]}}{\partial b^{[L]}}^{T}\end{split}  \tag{4}\]  
where the \((. )\) symbol stands for matrix multiplication and the \((\circ)\) symbol represents the Hadamard or element-wise product. The backpropagated error represented by \(\delta^{[L]}=\frac{\partial E}{\partial X^{[L]}}\circ\frac{\partial X^{[L]}}{ \partial A^{[L]}}\) is passed to the \(L-1\) layer, the updated backpropagated error in layer \(L-1\), that is, \(\delta^{[L-1]}=\frac{\partial A^{[L]}}{\partial X^{[L-1]}}\delta^{[L]}\circ \frac{\partial X^{[L-1]}}{\partial\delta^{[L-1]}}\) (represented by \(\delta^{L-1}\)), is passed to \(\delta^{L-2}\) layer, and so on. Finally, the general form of the updated error is represented as shown below. \[ \begin{split}\frac{\partial E}{\partial W^{[l]}}&= \delta^{[l]}\frac{\partial A^{[l]}}{\partial W^{[l]}}\\ \frac{\partial E}{\partial b^{[l]}}&=\delta^{[l]} \frac{\partial A^{[l]}}{\partial b^{[l]}}^{T}\end{split}