## 5 Experiment and Results
### Results and Analysis
\begin{table}
\begin{tabular}{c c} \hline \hline
**Descriptive Statistic** & **Value** \\ \hline mean & 3 h and 30 min \\ median & 2 h and 13 min \\ minimum & 33 min \\ maximum & 14 h and 31 min \\ standard deviation & 3 h and 37 min \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Basic descriptive statistics of the GA compilation time. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline \(\gamma\) & 0.85 \\ learning rate & 0.0758 \\ replay buffer size & 65,125 \\ neural network update frequency & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Best randomly selected hyperparameters in the first generation. methods. The red dashed line indicates a certain threshold of acceptable results indicating the agent's ability to understand the environment. These results are sorted by the time it took the GA to obtain a given result. In cases where the optimisation duration exceeded 4 h, the agent's average reward was close to the maximum reward. Such a result suggests that out of 1000 possible actions, the agent made only a few incorrect decisions. According to the specified constraint, exactly half of the GA configurations successfully optimised the hyperparameters. Table 7 shows the results of the hyperparameter optimisation of the DQN algorithm by the implemented GAs. It consists of five columns, where the first three are unique combinations of selection, crossover, and mutation methods. The next column represents the optimisation time for 200 iterations of GA. The last column shows the value of the fitness function for the best chromosome of the last generation. The values in this table are sorted in descending order based on the score column. Analysing the configurations presented, it is evident that the lowest scores were obtained with methods that exhibit minimal exploration of the configuration space. Most configurations employing one-point crossover failed to exceed a reward threshold of 400 points. Combining this crossover method with boundary mutation, which also exhibits low exploration, resulted in some of the worst scores. This observation highlights the importance of exploration over exploitation, particularly within certain hyperparameter ranges. In terms of selection methods, they do not seem to have a significant impact on exploration, as various selection methods appear throughout the rankings. Uniform mutation, however, relies heavily on randomness, resulting in a very high level of exploration in the hyperparameter configuration space. Although this high exploration rate can be beneficial, it also introduces challenges in preserving promising solutions, as excessive exploration can prevent the algorithm from converging effectively on optimal configurations. Considering the relationship between time and score, the best combinations needed a much longer time to optimise. As mentioned earlier, this difference is largely due to the characteristics of RL algorithms and, more precisely, the way the agent communicates with the environment. Knowing that in the CartPole environment, the reward is awarded at each timestep for taking an optimal action, the duration of the episode increases proportionally to the number of actions taken in the episode. Considering the second example in the comparison, it is assumed that the agent took an average of 971.04 optimal actions in the last 50 learning episodes. Figure 2: Optimisation time and average agent performance. Table 8 shows the three best configurations of selection, crossover, and mutation methods in terms of optimisation time and score. The score column represents the difference between the normalised time values and the average reward in the last 50 episodes of the best individual from the final population. The best configuration turned out to be roulette selection, arithmetic crossover, and boundary mutation. Finding this configuration in Table 7, it is noticeable that the optimisation time in this case was surprisingly low, totalling 4 h and 44 min.