## 6 Results and Discussion
### Refining the Search Space with HyperGE  
As the objective of HyperGE is to reduce the computational complexity of the HPO, we aim at reducing the resources used in the experimental settings. [PERSON] et al. (2016) and [PERSON] et al. (2017) have shown that while fine-tuning the models, a small subset of the dataset is enough to learn the hyperparameters. Hence, we followed the same settings and used 5% of the training set of the dataset during HPO. However, the validation accuracy, viz., the fitness function, was calculated on the entire validation set, making it a robust hyperparameter learner. For each trial, the subset of 5% of data was trained over five epochs with the configuration, and validation accuracy over the entire validation set was measured as metrics.  
With the budget of Stage 1 (50 trials), the insights of the individuals' performance were obtained and plotted in Figure 10. The plot shows the distribution of performance of individuals across the hyperparameters for VGG-19 and ResNet-50 across all three datasets. It is evident that although the search space and dataset are the same, the performance of the hyperparameters is still very different across the models (e.g., batch size for cifar10). Hence, the choice of model, hyperparameters, and datasets is equally significant in HPO.  
Using the insights obtained from these plots, the least-performing hyperparameters (with validation accuracy \(<25\%\)) were eliminated and a refined search space was designed for each model across the datasets as illustrated in Figure 11. In Stage 2 of HyperGE, the models are further fine-tuned using the refined grammar for the budget of 30 trials. The best-performing individual among these 30 trials (the individual with the highest validation accuracy) is then trained for 50 epochs using the hyperparameters.  
Figure 12 illustrates the performance of individuals across Stage 1 and Stage 2 against the number of trials. From the graphs, it can be seen that in Stage 1, the individuals have more variance and have a global explorative nature. This is helpful for them not to get stuck in local minima and to learn all the hyperparameters. In Stage 2, with the refined search space, the variance of the performance is significantly reduced, and it can be seen that the accuracy of the individuals lies within a certain range thus guiding the search. The values  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Evolutionary** & \multicolumn{2}{c}{**Values**} \\ \cline{2-3}
**Hyperparameters** & **HyperGE-Stage 1** & **HyperGE-Stage 2** \\ \hline Population size & 10 & 10 \\ Number of generations & 5 & 3 \\ Search Engine & Genetic Algorithm & Genetic Algorithm \\ Crossover Type & Variable One-Point & Variable One-Point \\ Mutation Type & Integer Flip Per Codon & Integer Flip Per Codon \\ Crossover probability & 0.95 & 0.95 \\ Mutation probability & 0.01 & 0.01 \\ Selection Type & Tournament & Tournament \\ Initialisation method & Position Independent grow & Position Independent grow \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evolutionary hyperparameter settings for Stage 1 and Stage 2 in HyperGE.  
of mean and standard deviation of the fitness values are presented in Table 3. The average reduction in search space with Stage 2 of HyperGE is 90%, as presented in the last column of Table 3, showing the potential of HyperGE in reducing the computational resources.  
Figure 11: Refined search space of VGG-19 and ResNet-50 models across datasets in Stage 2 of HyperGE.  
Figure 10: Data-driven analysis of exploration in Stage 1 of HyperGE across all datasets.