## 3 Neural Network Selection
### Feed-Forward Back Propagation Neural Network  
A more complex type of neural network examined is a feed-forward back-propagation neural network (BPNN) with two fully connected hidden layers [47]. This type of network has already been successfully applied to forecasting problems of complex systems with highly nonlinear behavior affected by several parameters [14; 48; 49]. For a single-layer network, the error is an explicit function of the network weights, and its derivatives with respect to the weights can be easily computed. In multilayer networks with nonlinear transfer functions, the relationship between the network weights and the error is more complex [39]. The goal of BPNN is to update each of the network weights so that the neural \begin{table}
\begin{tabular}{l c} \hline \hline
**ANN Type** & **FF** \\ \hline ANN Dimensions & 7 \\ Inputs & 2 \\ Layers & 1 \\ Outputs & 0 \\ Layer Delays & 0 \\ Weight Elements & 160 \\ ANN connections: & 160 \\ Bias Connections: & [1; 1] \\ Input Connections: & [1; 0] \\ Layers Connections: & [0; 1 0] \\ Output Connections: & [0 1] \\ ANN Training hyper-parameters & \\ Maximum Epochs & 1000 \\ Maximum Training Time & Inf \\ Performance Goal & 0 \\ Minimum Gradient & \(1.00\times 10^{-7}\) \\ Maximum Validation Checks & 6 \\ \(\mu_{k}\) & 0.001 \\ \(\mu_{k}\) decrease ratio & 0.1 \\ \(\mu_{k}\) increase ratio & 10 \\ Maximum \(\mu_{k}\) & \(1.00\times 10^{10}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Design parameters of the specific type of FF ANN applied, along with the hyper-parameter values related to the training procedure. network can approximate its output to the desired target. The error between the neural network output and the desired target can be written in the form of a cost (or loss) function, where \(\text{O}_{\text{k}}\) is the output and \(\text{t}_{\text{k}}\) is the desired target for a specific time step:  
\[\text{E}=\frac{1}{2}\sum_{\text{k}\in\text{K}}\left(\text{O}_{\text{k}}-\text{ t}_{\text{k}}\right)^{2} \tag{3}\]  
The objective is to minimize this cost function by updating the weights and biases during the training process, which, for the specific type of network, is an iterative process. It starts with forward propagation with the training dataset, which computes an initial output and then compares it to the reference values (targets) to calculate the loss function. The loss is back-propagated using a series of partial derivatives with respect to each FNN's internal parameters (weights and biases). Thus, the values of these parameters are updated, and a new iteration starts. The iterative process ends with the fulfillment of convergence criteria or when the maximum number of epochs is attained. At regular intervals during the training iterations, a separate test dataset is employed to validate the accuracy of the BPNN. This procedure is similar to the one employed to train the feed-forward ANN described in Section 3.1. The Adam optimization algorithm is employed for training the specific ANN, as mentioned in Table 2. It is an extension of the classical stochastic gradient descent procedure, which computes individual adaptive learning rates for different parameters from estimates of the first and second moments of the gradient [50]. Training methods with a derivative-based optimization algorithm, such as Levenberg-Marquardt or Adam, may be trapped in local minima; hence, they should be repeated to ensure they lead to an appropriate ANN. A comparative analysis of the results of applying the two types of networks with the actual demand for various periods of 2022 takes place in the next section and is based on the MAPE and the nRMSE error metrics. This type of performance metric is routinely applied in load forecasting problems. They are expressed as follows:  
\[\text{MAPE}=\frac{100\%}{\text{N}}\sum_{i=1}^{\text{N}}\frac{\left|\text{Pf}_ {\text{i}}-\text{Pm}_{\text{i}}\right|}{\text{Pm}_{\text{i}}} \tag{4}\]