## 5 Hyperparameter Optimization of the CNN
### LeNet-5 Hyperparameter Optimization Experiments  
The MNIST handwritten digit dataset was used as the benchmark dataset for this experiment. The hyperparameters were optimized every five epochs, and the network was trained based on the optimized hyperparameters. The LeNet-5 hyperparameters obtained after optimization using ADGMPSO-CNN are shown in Figure 6.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm** & **Parameter** \\ \hline PSO-CNN & \(\omega=0.9\), \(c_{1}=c_{2}=2\) \\ LDWPSO-CNN & \(\omega\) decreases linearly from 0.9 to 0.4, \(c_{1}=c_{2}=2\) \\ GA-CNN & Mutation possibility rate = 0.1, crossover probability = 0.9 \\ ADGMPSO-CNN & \(\omega_{max}=0.9\), \(\omega_{min}=0.2\), \(c_{max}=2\), \(c_{min}=1\), \(\gamma_{min}=0.2\), \(\gamma_{max}=1\), \(\gamma\) \\  & Decreases linearly from 1 to 0.2 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Algorithm parameter settings.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Running Environment** & **Details** \\ \hline Operating System & Ubuntu 18.04.5 \\ CPU & Intel(R) Xeon(R) Gold 5218 [EMAIL_ADDRESS] GHz \\ RAM & 32 GB DDR4 RAM \\ Software & Python 3.8.5, Pytorch 1.12 \\ GPU & NVIDIA Tesla T4 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Hyperparameter optimization experiment running environment.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dimension** & **Hyperparameters** & **Search Range** \\ \hline \(x_{1}\) & Number of first layer convolution kernels & [1–28] \\ \(x_{2}\) & Size of first layer convolution kernels & [\(3\times 3\), \(5\times 5\), \(7\times 7\)] \\ \(x_{3}\) & First layer convolution kernel step & [1–2] \\ \(x_{4}\) & First layer convolutional layer filling type & [valid, same] \\ \(x_{5}\) & Type of second pooling layer & [max. pooling, avg. pooling] \\ \(x_{6}\) & Size of third layer convolution kernels & [\(3\times 3\), \(5\times 5\), \(7\times 7\)] \\ \(x_{7}\) & Second pooling layer step & [1–2] \\ \(x_{8}\) & Second layer filling type & [valid, same] \\ \hline \hline \end{tabular}
\end{table}
Table 9: ResNet-18 to be optimized for the hyperparameters.  
Compared to the standard LeNet-5, the optimized hyperparameters had a larger number of convolution kernels for extracting more feature information. The ReLU activation function was more effective in addressing the vanishing gradients problem than the original Sigmoid activation function used in the neural networks. The mixture of the max. pooling and avg. pooling enhanced the model's generalization performance by preserving the image texture and background information while also reducing the dimensionality. The optimized number of neurons in the fully connected layer was selected to prevent overfitting issues.  
As can be seen from Figure 7, The hyperparameter optimization algorithms based on evolutionary algorithms showed promising results compared to the base CNN. Among them, PSO-CNN and GA-CNN achieved similar results in optimizing the hyperparameters. However, the performance of LDWPSO-CNN was slightly better due to its linear decreasing inertia weighting strategy. The algorithm ADGMPSO-CNN proposed in this paper achieved a significantly better performance in the search for optimization than PSO-CNN, LDWPSO-CNN, and GA-CNN through three improvement strategies. The optimized standard LeNet-5 model achieved a high accuracy from the first iteration and finally reached 99.11% accuracy after the five training iterations, showing a significant improvement.  
Figure 8 illustrates the average loss curve of the LeNet-5 model optimized by ADGMPSO-CNN on the MNIST dataset, indicating a well-fitted model.