## 3 Data and Methodology
### Methodology
#### 3.2.2 Genetic Algorithm Parameters  
As previously stated, in order to find a sub-optimal set of hyper-parameters, described in the previous section, for the deep learning algorithm, we use a GA. In particular we use the implementation provided by the GA R package [50]. So our proposal lies within the field of neuroevolution.  
The GA package contains a collection of general-purpose functions for optimization using genetic algorithms. The package includes a flexible set of tools for implementing genetic algorithms in both the continuous and discrete case, whether constrained or not. However the package does not allow to simultaneously optimize continuous and discrete parameters, so we had to treat all the parameters as continuous, which caused the dimension of the search space to increase drastically.  
The package allows us to define objective functions to be optimized, which, in our case, is the forecasting results obtained by a deep neural network built with a specific set of parameters. In fact, each individual of the population encodes the values of the eight parameters shown in Table 2.  
Each parameter setting yields a specific deep neural network, which is then applied to the data and the forecasting result represent the fitness of the individual.  
In particular, the fitness of an individual is equal to the \(MRE\) obtained by the deep neural network on the validation set, being the \(MRE\) defined as:  
\[MRE=\frac{1}{n}\sum_{i=1}^{n}\frac{|Y_{i}-\hat{Y}_{i}|}{Y_{i}}, \tag{2}\]  
where \(\hat{Y}_{i}\) is the predicted value, \(Y_{i}\) the real value and \(\overline{Y}_{i}\) is the mean of the observed data, and \(n\) is the number of data.  
Several genetic operators are available and can be combined to explore the best settings for the current task. After having performed a set of preliminary experiments aimed at setting the GA's parameters, we used, in our implementation, a tournament selection mechanism (with tournament size of 3), the BLX-a crossover (with \(a=0.5\)), which combines two parents to generate offspring by sampling a new value in a defined range with the maximum and the minimum of the parents [51]. We used the random mutation around the solution, which allows us to change one value of an element by another value.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Values** \\ \hline Layers & From 2 to 100 \\ Neurons & From 10 to 1000 \\ Lambda (\(\lambda\)) & From 0 to 1 \(\times\) 10\({}^{-10}\) \\ Rho (\(\rho\)) & From 0.99 to 1 \\ Epsilon (\(\epsilon\)) & From 0 to 1 \(\times\) 10\({}^{-12}\) \\ Activation function & From 0 to 3 \\ Distribution function & From 0 to 7 \\ End metric & From 0 to 7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Search space of the neural network parameters.  
The setting of the parameters used in the GA are reported in Table 3. The value shown are those that obtained the best performances in the preliminary runs, but the population size. In fact, better results were achieved with higher population size. However, the computational cost increases dramatically the higher the population size is. In fact, the deep learning algorithm takes around 89.42 s for a number of layers between 2 and 100 and for a number of neurons between 10 and 1000.  
The execution of the GA with the deep learning algorithm as a fitness function and with the parameters defined in Table 3 takes around five days. If the population size is doubled, the execution can take more than one week. It is necessary to enhance one of the parameters (population size or number of generations) but not both. Moreover, if the fitness of the best individual does not improve after 50 generations, the GA is stopped.  
At the end of the execution, the best individual is returned and used in order to build a deep learning network.