## 5 Experimental results and discussion
### Convolutional neural network hyperparameter optimization experiments
#### 5.2.2 Convolutional neural network hyperparameter optimization results and analysis
Within the comparative analysis, we also encompassed manually designed state-of-the-art networks, namely LeNet-5 [6], deeply supervised net [83], shallow CNN [84], recurrent CNN [85], and gated pooling CNN [86]. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Best 5 solutions** \\ \hline Kernel 1 & 2-4 \\ Output 1 & 43-110 \\ Kernel 2 & 3-4 \\ Output 2 & 96-128 \\ Kernel 3 & 2 \\ Output 3 & 100-120 \\ FC Layer 1 & 50-128 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Ranges of the parameters for the best network configurations (solutions). The EE-TGA-CNN approach resulted in the best accuracy and lowest test error rate, that is 0.19%; \(E^{3}\)-FA-CNN resulted in the classification error rate of 0.21%; TGA-CNN 0.23% [71]; and the fourth approach, FA-CNN, was 0.23% [72]. IEA-CNN [17] obtained the best classification error rate of 0.34%, while CEA-CNN [17] generated the best architecture with a classification error rate of 0.24% with penalization \(k\) set to two. With \(k\) set to one and three, CEA-CNN managed to establish classification error rates of 0.26% and 0.28%, respectively [17]. On the other side, some manually designed CNN configurations established a much higher classification error rate. The highest error rate of 0.95% was produced by LeNet-5 [6], while the deeply supervised net [83] and shallow CNN [84] obtained error rates of 0.39% and 0.37%, respectively. The recurrent CNN, which managed to establish an error rate of 0.31%, and the gated pooling CNN with an error rate of 0.34% generated better results than the IEA-CNN approach. In comparison to the peer competitors and other architectures that were manually designed, besides the high classification accuracy, the proposed enhanced metaheuristic techniques (TGA-CNN and \(E^{3}FA\)-CNN) had fewer parameters and were computationally less expensive. The results of the proposed methods with other comparable techniques that utilized metaheuristics algorithms and manually designed state-of-the-art architectures are presented in  
Figure 6: Some of the generated network architectures by TGA-EE-CNN and \(E^{3}\)FA-CNN. Table 10. We note once again that the best network configuration (solution) from all 10 independent runs is reported. A visual representation of the comparative analysis is depicted in Figure 7. For the sake of better visibility, CEA-CNN results with \(k\) set to one and three are excluded from the graph. Since we implemented two methods in our current study and two in previous studies [71; 72] and with the goal of establishing the visual representation of the quality of the proposed approaches, the convergence speed graph of the accuracy for TGA-CNN, FA-CNN, EE-TGA-CNN, and \(E^{3}\)-FA-CNN is given in Figure 8. The results presented in the figure were taken from one of the best runs. \begin{table}
\begin{tabular}{l c} \hline \hline
**Method** & **Test Error Rate (\%)** \\ \hline LeNet-5 [6] & 0.95 \\ Deeply Supervised Net [83] & 0.39 \\ Shallow CNN [84] & 0.37 \\ Recurrent CNN [85] & 0.31 \\ Gated Pooling CNN [86] & 0.29 \\ IEA-CNN [17] & 0.34 \\ CEA-CNN, k=1 [17] & 0.26 \\ CEA-CNN, k=2 [17] & 0.24 \\ CEA-CNN, k=3 [17] & 0.28 \\ FA-CNN [72] & 0.23 \\ TGA-CNN [71] & 0.23 \\ \(E^{3}\)-FA-CNN & 0.21 \\ EE-TGA-CNN & **0.19** \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Comparative analysis with other metaheuristics. IEA-CNN, evolutionary-based hyper-parameter optimization for independent CNNs ; CEA-CNN, joint optimization for a committee of multiple CNNs. Figure 7: Visual representation of the error rates by different methods. From the presented figure, it was clear that EE-TGA-CNN and \(E^{3}\)-FA-CNN established better convergence than the original TGA and FA algorithms. Moreover, the enhanced metaheuristics managed to converge to the best solutions after only 15 iterations. In order to further access the solution quality of the proposed approaches, we also compared the EE-TGA-CNN and \(E^{3}FA-CNN\) methods with another swarm intelligence-based CNN optimizer presented in [18].