## 5 Experiment and Results
### Results and Analysis
#### 5.4.1 Results Exceeding a Certain Threshold
The algorithm marked with a blue line is the GA that achieved the highest score in terms of reward and optimisation duration. The orange line marks the algorithm that obtained the highest average agent score. The best algorithm relative to time and reward quickly identified superior hyperparameter configurations in the initial generations. In later generations, the high scores were updated less frequently. The first five value updates were significant for the score and the last two led to minor improvements. This indicates the importance of exploration in the initial iterations of the GA, which allows the algorithm to find promising areas of hyperparameter configuration. The algorithm marked with an \begin{table}
\begin{tabular}{c c} \hline
**Crossover Method** & **Number of Occurrences** \\ \hline arithmetic & 3 \\ single-point & 2 \\ uniform & 3 \\ multi-point & 4 \\ \hline \end{tabular}
\end{table} 
Table 10: Crossover methods with a score greater or equal to 400. \begin{table}
\begin{tabular}{c c} \hline
**Selection Method** & **Number of Occurrences** \\ \hline roulette & 5 \\ tournament & 7 \\ \hline \end{tabular}
\end{table} 
Table 9: Selection methods with a score greater or equal to 400.  
orange line optimised hyperparameters slower than the previous GA, but avoided getting stuck in a local optimum and experienced more frequent value updates. The Euclidean distance on the Y-axis quantifies hyperparameter changes across generations, providing insight into the convergence process. Figure 4 depicts the convergence of two GAs optimising DQN hyperparameters. Both employed boundary mutation, leading to periodic jumps in Euclidean distance. * Figure 4a The orange plot represents the GA that achieved the highest agent score. It exhibits rapid initial convergence, followed by sustained exploration, as evidenced by the fluctuations beyond generation 50. This suggests that continued search contributed to improved agent performance. * Figure 4b The blue plot corresponds to the GA with the best computational efficiency. It also shows a sharp initial drop in Euclidean distance but stabilises earlier, particularly around generation 100. This indicates a more exploitative approach, leading to faster convergence at the cost of reduced exploration. These differences highlight the exploration-exploitation trade-off in genetic algorithms. The GA maximising agent performance retained longer exploration, leading to potentially better hyperparameters. In contrast, the time-efficient GA prioritised early convergence, reducing computational costs but possibly missing optimal configurations. Figure 4: Convergence of the two best GAs in hyperparameter optimisation. (**a**) GA achieving the highest agent score. (**b**) GA with the best composite score. Figure 3: Best individual’s performance over generations. Four visualisations representing the relationship between the value of each hyperparameter and the agent's score have been prepared. The data shown were generated in the GA optimisation process with roulette selection, arithmetic crossover, and boundary mutation. These visualisations facilitate the analysis of hyperparameter values' impact on agent scores and the exploration of the hyperparameter configuration space. The visualisation in Figure 5 illustrates the exploration of the gamma (\(\gamma\)) hyperparameter by the GA. It is evident that the GA explored a wide range of gamma values, with a notable concentration of high-performing results around values close to 1. This suggests that the GA effectively identified the importance of high gamma values, which tend to produce significantly better scores for the agent. The clustering near 1 suggests that the GA not only explored the configuration space but also successfully converged on an optimal gamma value, reinforcing its critical role in improving the agent's learning process. The visualisation in Figure 6 shows the exploration of learning rate values by the GA.