## 5 Experimental results and discussion
### Convolutional neural network hyperparameter optimization experiments
#### 5.2.2 Convolutional neural network hyperparameter optimization results and analysis
However, the methods presented in [18] were tested under different experimental conditions, but for the same MNIST dataset. For this reason, comparative analysis with these approaches was not entirely realistic. In [18], as one of the main drawbacks of CNN models, the over-fitting problem was addressed. In order to overcome this issue, the authors proposed four swarm intelligence methods (FA, BA, CS, and PSO) for proper selection of the regularization parameter dropout and reported the average accuracy over 20 independent runs. Our first approach, EE-TGA-CNN, established an average accuracy over 10 independent runs of 99.18%, while \(E^{3}FA\)-CNN for the same metric obtained a result of 99.16%. At the other hand, the basic FA and TGA metaheuristics that we implemented obtained an average accuracy of 99.13% and 99.14%, respectively. We note that the results reported for the FA in [18] were different from our paper since different experimental conditions were used. The results of the comparative analysis between EE-TGA-CNN and \(E^{3}FA\)-CNN and the approaches presented [18] are summarized in Table 11, while the visual representation is given in Figure 9. \begin{table}
\begin{tabular}{l c} \hline \hline
**Method** & **Accuracy (\%)** \\ \hline PSO [18] & 99.17 \\ FA [18] & 99.16 \\ CS [18] & 99.14 \\ BA [18] & 99.13 \\ FA-CNN [72] & 99.13 \\ TGA-CNN [71] & 99.14 \\ EE-TGA-CNN & **99.18** \\ \(E^{3}FA\)-CNN & 99.16 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Comparative analysis with other swarm algorithms. CS, cuckoo search. Figure 8: Convergence graph of the accuracy. Although it was not an entirely realistic comparative analysis, we could conclude that the average accuracy of EE-TGA-CNN was the greatest among all compared approaches. Finally, in the experimental section of this paper, we wanted to give some information regarding the computational burden (costs) of the proposed approaches. The computational burden can be calculated in many ways, for example by determining the time and space complexity, computation time, number of fitness function evaluations, etc. The time complexity uses the number of elementary operations executed by the method, while the space complexity takes into account the memory required by an algorithm to run as a function of the input size. Computational time is a simple measurement in time units taken by the algorithm to perform, and the drawback of this approach is that it highly depends on the overall speed and performance of the computer platform. However, in the case of metaheuristics, it is more suitable to use the number of fitness function evaluations, which was especially emphasized in the simulations that were presented in this paper. Every time the quality of the solution from the population is estimated (fitness calculation), the evaluation of a particular CNN is performed, which is a very time and resource consuming process. For this reason, inspired by [18], for calculating the computation burden of the proposed approaches, we used the metric of the number of evaluation function (fitness function) calls. In both proposed approaches (EE-TGA-CNN and \(E^{3}FA\)-CNN), as well as in the TGA-CNN and FA-CNN methods, the total number of function evaluations depended on the population size \(N\) and the maximum number of iterations (\(t_{max}\)) in one run. In the initialization phase, \(N\) fitness function evaluations were performed, and then, in each iteration, \(N\) more evaluations were executed (that is a total in one run \(N\cdot t_{max}\), one for each solution from the population), and finally, at the end of a run, one final evaluation was executed. For calculating the computational burden, we used the following equation, as in [18]:  
\[burden=N+(N\cdot t_{max})+1 \tag{28}\]  
Since in our simulations, we used values \(N=8\) and \(t_{max}=20\) in each run of both proposed methods, as well as in TGA-CNN and FA-CNN, 169 fitness function calls were used (\(8+8\cdot 20+1\)).