## 5 Experiment and Results
### Results and Analysis
#### 5.4.2 Hyperparameters Results  
All results obtained from the optimisation process over generations for each combination of selection, crossover, and mutation methods were recorded including gamma, learning rate, target update frequency, replay buffer size, and the corresponding agent score. This approach allowed for a comprehensive analysis of the impact of different hyperparameter settings on agent performance. Table 15 contains information on the number of records concerning the values of selected hyperparameters and the corresponding agent results. According to the experiment conducted, GA iterated through 200 generations, of which there were 16 individuals in each population. The implemented algorithm was to optimise 24 configurations of selection, crossover, and mutation methods. As a result, 76,800 records were obtained. In order to verify whether the randomness problem had disappeared, duplicate removal was performed on this set. As a result of this operation, 17,866 results were obtained, which also confirmed that for the same values of selected hyperparameters, the DQN agent achieves the same results with the applied solution to the randomness problem. Figure 10 illustrates the distribution of the four optimised hyperparameters, where each subfigure corresponds to a specific parameter: Figure 10a represents the learning rate, Figure 10b the target update, Figure 10c the replay buffer size, and Figure 10d the gamma value. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Column** & **Before Removing Duplicates** & **After Removing Duplicates** \\ \hline Number of records & 76,800 & 17,866 \\ \hline \hline \end{tabular}
\end{table} 
Table 15: Comparison of dataset size before and after removing duplicates. Figure 9: Agent rewards for optimised hyperparameters: 150 episodes. * Figure 10a The distribution of the learning rate indicates that the algorithm mainly focused on the range between 0.01 and 0.02, which suggests that the optimal value for this parameter is in this range. There is also a peak around 0.07, which is caused by the combination of arithmetic crossover and edge mutation. * Figure 10b The distribution of the agent network update frequency indicates that it should be updated at least every 20 steps. Values above 20 occurred insignificantly, indicating that less frequent updates are not beneficial in this context. * Figure 10c The distribution of the replay buffer size values shows a concentration around 60000, which means that other values did not improve the agent's performance. * Figure 10d The distribution of the gamma hyperparameter clearly indicates that GA, while looking for optimal solutions, indicated values close to 1. This indicates the preference of the DQN algorithm in the CartPole environment to take into account long-term rewards. Figure 11 shows the correlation matrix, which shows that the learning rate and target update frequency hyperparameters have a moderate negative correlation with the agent score, which indicates that smaller values of these hyperparameters have an impact on the agent performance. The buffer size and gamma hyperparameters show a weak positive correlation with the score, which indicates that their larger values have a better impact on the algorithm performance. The correlations between the selected hyperparameters indicate that they can be optimised individually. Figure 10: Distribution of hyperparameters. (**a**) Distribution of learning rate. (**b**) Distribution of target update. (**c**) Distribution of replay buffer size. (**d**) Distribution of gamma. Figure 12 shows scatterplots of hyperparameter values and obtained results. These plots are consistent with the distributions shown in Figure 10. The specific trends are as follows:  
* Figure 12a Higher agent scores are concentrated at lower learning rate values. Scores tend to decrease with increasing values of this hyperparameter, which aligns with the distribution shown in Figure 10a. * Figure 12b The highest number of high scores is observed for very frequent updates.