## 5 Experiment and Results
### Results and Analysis  
Determining the chromosomes that achieved satisfactory results in the hyperparameter optimisation process required establishing a certain threshold value of the fitness function. It was assumed that the threshold value indicating a sufficient result for the best agent would be an average reward from the last 50 episodes greater than 400. This value was \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter** & **Lower Bound** & **Upper Bound** \\ \hline reward discount factor \(\gamma\) & 0.01 & 1 \\ learning rate \(\alpha\) & 0.0001 & 0.1 \\ size of replay buffer & 10,000 & 100,000 \\ update rate of the neural & 1 & 100 \\ network & 1 & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Selected hyperparameters and their ranges. \begin{table}
\begin{tabular}{c c} \hline \hline
**Attribute** & **Value** \\ \hline number of generations & 200 \\ population size & 16 \\ number of hyperparameters & 4 \\ probability of mutation & 0.05 \\ probability of crossover & 1 \\ initial population gene values & random \\ fitness function & \(\frac{1}{50}\sum_{i=100}^{150}R_{i}\) \\ random seed & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Genetic algorithm configuration. determined based on an analysis in which random hyperparameter values very rarely allowed the agent to exceed this threshold. The comparison carried out included iterations through two implemented types of selection, four types of crossover, and three types of mutation. A total of 24 results of the operation of these methods were collected, and the results of the configurations that stood out in terms of the obtained agent result and the quality of optimisation were described in more detail. Table 5 shows the best configuration of hyperparameter values obtained by creating a random zero population. The DQN agent with such hyperparameter values in the training process lasting 150 episodes obtained an average reward of 68.26 points over 50 episodes. Considering that the agent could have obtained a score of 1000 points if it had always taken optimal actions, the obtained score was too low to cope in this environment. Table 6 provides basic descriptive statistics of the compilation time of the implemented GA. The optimisation time for 24 combinations of selection, crossover, and mutation methods was 3 days, 12 h, and 27 min. On average, the optimisation of the presented GA took 3 h and 30 min. The median compilation time was 2 h and 13 min. The median value was smaller than the average HPO time, which means that in most cases the configuration of methods took less than 3 h and 30 min. The minimum duration of this process was 33 min, while the maximum was as high as 14 h and 31 min. The duration deviates from the average value by 3 h and 37 min, which indicates the instability of the algorithm resulting from the structure of RL algorithms. The optimisation time variability suggests significant differences in the efficiency of genetic algorithm configurations, with certain combinations of selection, crossover, and mutation methods potentially more time-efficient due to better alignment with RL needs. The skewed distribution of optimisation time, where the median is below the average, indicates a subset of configurations with substantially longer processing times, potentially influenced by the complexity of the environment in which the agent operates. The large standard deviation points to instability, suggesting that GA performance is highly sensitive to environmental conditions. This variability highlights potential for refining configurations to improve stability and reduce computational demands in RL hyperparameter optimisation. Figure 2 shows a bar graph of the obtained average reward from the last 50 episodes of optimised hyperparameters for 24 configurations of crossover and mutation selection