# Genetic Algorithm Based Deep Learning Neural Network Structure and Hyperparameter Optimization
## 4 Results and Discussion
### Results  
The results of the experiment are shown in Table 3. For each of the generations stated in the table, maximum, minimum and average accuracy of 10 individuals are shown. The best performing combination and network structure and hyperparameter set is also listed as encoded form. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Image Type** & **NII Images** \\ \hline
**Classes** & **Number of Subjects** & **Number of Images** \\ \hline  & **Train Set** & \\ \hline AD & 159 & 10,812 \\ \hline MCI & 130 & 8840 \\ \hline NC & 85 & 5780 \\ \hline  & **Test Set** & \\ \hline AD & 17 & 1156 \\ \hline MCI & 14 & 952 \\ \hline NC & 9 & 612 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Details of the dataset. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Activation Function** & \multicolumn{2}{c}{**Optimization Algorithm**} \\ \hline
**Hyperparameter** & **Bit Representation** & **Hyperparameter** & **Bit Representation** \\ \hline ReLU & 00 & SGDM & 00 \\ \hline Leaky ReLU & 01 & RMSProp & 01 \\ \hline Clipped ReLU & 11 & Adam & 11 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Encoding rule for hyperparameters. To evaluate the performance of our algorithm with other state-of-the-art GAs, we ran the original Genetic-CNN with the same input dataset, and compared the average accuracy of individuals for different numbers of generations. This algorithm uses ReLU and Adam as hyperparameters to calculate their fitness values. The result is shown in Table 4. In early generations, individuals in the Genetic-CNN algorithm seem to learn faster. However, after the final generation, our algorithm found a neural architecture, including set of hyperparameters, that had 11.73% higher classification accuracy for given input data. Our algorithm has four further bits compared with Genetic-CNN, due to the larger search space. However, the increase in processing time is minimal. The processing times required to obtain the result shown in Table 4 for Genetic-CNN and our algorithm were 618,278 s and 619,784 s, respectively. Only an increase of 0.24% was observed, which is negligible. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Number of Generations** & \multicolumn{2}{c}{**Average Accuracy (\%)**} & \multicolumn{2}{c}{**Difference (\%)**} \\  & **Genetic CNN [7]** & **Ours** & \\ \hline
0 & 23.29 & 43.61 & 20.32 \\
1 & 23.29 & 42.78 & 19.49 \\
2 & 28.95 & 46.68 & 17.73 \\
5 & 51.91 & 48.24 & \(-\)3.67 \\
10 & 58.51 & 61.56 & 3.05 \\
20 & 63.61 & 69.67 & 6.06 \\
50 & 70.01 & 81.74 & 11.73 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of our algorithm with Genetic CNN [7]. Figure 6: Graph showing the accuracy of networks against the number of generations. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Number of** & \multicolumn{2}{c}{**Accuracy (\%)**} & \multicolumn{2}{c}{**Best Combination of Network Structure and**} \\
**Generations** & **Maximum** & **Minimum** & **Average** & **Hyperparameters** \\ \hline
0 & 46.45 & 40.89 & 43.61 & 0-11 \(|\) 1-01-111 \(|\) 1-01-110-1101 \(|\) 10 \(|\) 00 \\ \hline
1 & 46.45 & 38.25 & 42.78 & 0-11 \(|\) 1-01-111 \(|\) 1-01-110-1101 \(|\) 10 \(|\) 00 \\ \hline
2 & 47.81 & 44.89 & 46.68 & 0-11 \(|\) 1-10-111 \(|\) 1-01-110-1101 \(|\) 10 \(|\) 00 \\ \hline
5 & 47.81 & 48.63 & 48.24 & 0-11 \(|\) 1-10-111 \(|\) 1-01-110-1101 \(|\) 10 \(|\) 00 \\ \hline
10 & 64.64 & 58.49 & 61.56 & 1-01 \(|\) 1-10-111 \(|\) 1-01-1101 \(|\) 01 \(|\) 01 \(|\) 01 \\ \hline
20 & 71.48 & 67.86 & 69.67 & 1-01 \(|\) 0-11-010 \(|\) 1-01-101-1101 \(|\) 10 \(|\) 00 \\ \hline
50 & 82.64 & 80.84 & 81.74 & 1-11 \(|\) 1-10-111 \(|\) 1-11-011-011 \(|\) 10 \(|\) 10 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of the genetic algorithm, showing accuracy of generated networks. The final network architecture is visualised in Figure 7. The first stage of the network is a residual architecture that is used in ResNet. The next stage has a multiple-path network similar to GoogLeNet, as outputs of three previous layers are used as input of other convolutional layers.