## 5 Hyperparameter Optimization of the CNN  
### Experimental Settings  
To show the effectiveness of ADGMPSO in optimizing the hyperparameters of the convolutional neural networks, this study applied the improved algorithm to the classic LeNet-5 CNN model and the more popular ResNet-18 CNN network model. The MNIST handwritten digit dataset [18] and the CIFAR-10 dataset [35] were the benchmark datasets. The MNIST dataset contained 10,000 test images and 60,000 training images, all of which were \(28\times 28\) single-channel grayscale images of the numbers zero to nine. This study used MNIST as the benchmark dataset for optimizing the hyperparameters of LeNet-5. Each hyperparameter of LeNet-5 that needed optimization was used as a dimension of the individual ADGMPSO particle, and the hyperparameter information encoded by the different particle dimensions is shown in Table 8.  
The CIFAR-10 dataset is a multi-channel RGB image dataset of ten different types of images. It has 10,000 test images and 50,000 training images, all of which are \(32\times 32\) pixels in size. It was used as the benchmark dataset for the hyperparameter optimization experiments of ResNet-18. This paper used hyperparameter tuning for the first convolutional layer and pooling layer of the ResNet-18 network since they were responsible for the feature extraction and the dimensionality reduction from the original data, respectively [36]. Considering the computational cost, the tuning was limited to these two layers. The information on the hyperparameters that were to be optimized is shown in Table 9.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dimension** & **Hyperparameters** & **Search Range** \\ \hline \(x_{1}\) & Number of first layer convolution kernels & [1–128] \\ \(x_{2}\) & Size of first layer convolution kernels & [\(3\times 3,5\times 5,7\times 7\)] \\ \(x_{3}\) & Type of first layer activation function & [Sigmod, ReLu, Tanh] \\ \(x_{4}\) & Type of second pooling layer & [max. pooling, avg. pooling] \\ \(x_{5}\) & Number of third layer convolution kernels & [1–128] \\ \(x_{6}\) & Size of third layer convolution kernels & [\(3\times 3,5\times 5,7\times 7\)] \\ \(x_{7}\) & Type of third layer activation function & [Sigmod, ReLu, Tanh] \\ \(x_{8}\) & Type of fourth pooling layer & [max. pooling, avg. pooling] \\ \(x_{9}\) & Number of neurons in the fifth layer & [1–128] \\ \(x_{10}\) & Type of fifth layer activation function & [Sigmod, ReLu, Tanh] \\ \(x_{11}\) & Number of neurons in the sixth layer & [1–128] \\ \(x_{12}\) & Type of sixth layer activation function & [Sigmod, ReLu, Tanh] \\ \hline \hline \end{tabular}
\end{table}
Table 8: LeNet-5 to be optimized for the hyperparameters.  
To compare the performance of the base CNN model, PSO-CNN, LDWPSO-CNN, GA-CNN, and the model optimized by the proposed algorithm with the hyperparameters, five experiments were conducted on the same device to reduce the experimental error, and the results were averaged. Given that the hyperparameters to be optimized were entirely integers, all the algorithms in this paper were encoded using integers. To represent the discrete hyperparameters, such as the pooling layer types, integer one corresponded to the max. pooling, and integer two corresponded to the avg. pooling. Additionally, the rounding operation was applied to each dimension of the evolving individual to ensure the correctness of the network structure. Furthermore, every optimization algorithm in this study employed the classification accuracy of the dataset images as the fitness function. The population size for all the selected optimization algorithms was 10, and the maximum number of evolutions was 30. Details of the relevant parameters for all the algorithms are listed in Table 10.  
All the algorithms used random initialization except ADGMPSO-CNN, which used cat-based chaos to initialize the population. All the algorithms were terminated by reaching the maximum number of iterations. The operating environment of the hyperparameter optimization experiment is shown in Table 11.