## 5 Experimental results and discussion
### Convolutional neural network hyperparameter optimization experiments
#### 5.2.2 Convolutional neural network hyperparameter optimization results and analysis  
After testing the algorithms on the unconstrained benchmark functions, all four metaheuristics were applied to the CNN hyperparameter optimization problem. The performance of the method could not be assessed by the result of a single test, due to the random character of EE-TGA and \(E^{3}\)-FA. More than one run with independent random population initialization should be made in order to estimate the performance of the introduced approaches. Accordingly, in this study, the results were collected in 10 independent experiments, where for each run, different pseudo-random seeds were employed. Within the each run, the best CNN structure was generated in 20 iterations. The best reported result was taken as the network configuration (solution) that obtained the lowest classification test error from all 10 runs. The algorithms' control parameters, as well as the CNN hyperparameter range of values that were taken in the simulation are presented in detail in Subsection 4.2. As already stated, in all conducted experiments, the objective function that was subject to the process of optimization was the classification test error rate. The results of both proposed metaheuristics (EE-TGA and \(E^{3}FA\)) were compared to the peer competitors that were tested under the same simulation conditions [17], the original implementations of the TGA [71] and FA [72] metaheuristics and with the same state-of-the-art network configurations that were manually designed. Figure 5: Samples of the MNIST datasetIn [17], the authors proposed a CNN design framework based on the evolutionary algorithm: evolutionary-based hyper-parameter optimization for independent CNNs (IEA-CNN). Furthermore, the authors developed an extension of their framework towards a joint optimization of a committee of CNNs with the goal of leveraging specialization and cooperation between individual networks (joint optimization for a committee of multiple CNNs (CEA-CNN)). A committee represents a set of \(K\) trained CNNs where the classification is performed by merging the individual CNN scores. In the proposed approach [17], the average of the decision values over \(K\) CNNs was employed. In all experiments, the committee size was set to 34 (\(K=34\)). Moreover, the authors conducted simulations with CEA-CNN by varying penalization \(k\) and found that the best CNN configurations were generated when \(k\) was set to two. The results with \(k\) set to 1, 2, and 3 were presented in [17]. According to the results presented in [17], CEA-CNN produced larger filter sizes than both our proposed approaches, which extended from 3 \(\times\) 3 - 4 \(\times\) 4 to 8 \(\times\) 8 -7 \(\times\) 7. On the other hand, some of the best generated solutions by EE-TGA-CNN and \(E^{3}FA\) consisted of three convolutional layers and one fully-connected layer with the filter sizes 2 \(\times\) 2, 3 \(\times\) 3, and 4 \(\times\) 4, while the number of filters per convolutional layer ranged from 42 to 128, and the number of hidden units in the fully-connected layers were between 50 and 128. The network structures were significantly less complex than the architectures in [17]. We performed careful analysis of the empirical results generated from the conducted tests and concluded that the kernel size of the convolutional layers had a very significant influence on the network performance compared to the number of filters (kernels). The details of the five best performance network architectures (solutions) from the total of 80 generated solutions (eight solutions in the population \(\cdot\) 10 runs) are summarized in Table 9.  
Some of generated network architectures by TGA-EE-CNN and \(E^{3}\)FA-CNN are given in Figure 6.  
Since the study presented in this paper was an extension of our previous study where we tested the basic TGA [71] and FA [72] metaheuristics for the same problem and under the same experimental conditions, we also included in the comparative analysis these two methods.