## 5 Experiment and Results
### Results and Analysis
#### 5.4.1 Results Exceeding a Certain Threshold
It is evident that the GA effectively explored a range of learning rate values, identifying distinct clusters of results. In particular, it found that certain higher learning rate values, close to 0.08 and 0.1, correspond to significantly improved agent performance, as evidenced by the higher result scores in these areas. This pattern indicates that the GA not only explored diverse values within the configuration space but also identified values that optimised agent performance. The visualisation in Figure 7 shows the relationship between the replay buffer size and the agent's performance. The GA explored a wide range of buffer size values, from small to large capacities. In particular, there is a concentration of high-performance scores around a buffer size of approximately 60,000, indicating that this particular setting yielded significantly better results for the agent. This suggests that the GA effectively explored the configuration space and was able to identify an optimal range for the replay buffer size  
Figure 5: Dependency of result on the \(\gamma\) hyperparameter. Figure 6: Dependency of result on the hyperparameter: learning rate. that enhances the agent's performance. Clustering at this value demonstrates the ability of GAs to converge toward configurations that optimise the agent's learning process. As shown in Figure 8, the relationship between the neural network update frequency and agent performance reveals that the GA explored a wide range of update frequencies. The results indicate a clear concentration of high-performance outcomes at lower frequencies, particularly around a value of 1. This clustering suggests that more frequent updates contribute to significantly better performance. The GAs' exploration of various configurations led to the identification of an optimal update frequency, demonstrating its effectiveness in navigating the configuration space to enhance the agent's learning efficiency. Table 12 shows the hyperparameter configurations obtained as a result of the GA optimisation process. Both configurations share the use of boundary mutation. They were selected based on the highest achieved result and the time-to-result ratio. Both sets have the hyperparameter value \(\gamma\) set to 1, the replay buffer size set to 65,125, with the agent's neural network updating every one episode. The only difference in hyperparameters is the learning rate 0.0217 in the first configuration and 0.1 in the second. There is also a small difference in the average reward obtained from the last 50 episodes, where the first configuration obtained an average reward of 979.16, and the second 936.66. However, a significant difference exists in the GA compilation time. Despite the fact that, in both cases, the GA was to perform the optimisation process in 200 generations, the first configuration took almost twice as long as the second. Therefore, the configuration with the best time-to-score ratio was considered the best. Figure 8: Dependency of result on the hyperparameter: neural network update frequency. Figure 7: Dependency of result on the hyperparameter: replay buffer size. To further substantiate why Table 12 indicates that varying the learning rate is crucial for optimising performance, an extended series of experiments was conducted with multiple random initialisations and repeated runs. Specifically, the seed set {10, 20, 30, 40, 50} and five different learning rate values {0.001, 0.005, 0.01, 0.05, 0.1} were utilised. For each combination of seed and learning rate, five independent runs were performed, yielding a total of 25 data points per learning rate. In each run, the DQN agent was trained for 150 episodes, and mean score was defined as the average reward over the last 50 of those 150 episodes. Table 13 summarises the descriptive statistics for each learning rate, showing how substantially the choice of learning rate influences the agent's final performance.