## 4 Methodology
The control dynamics revolve around balancing an inverted pendulum, a concept applied in practical challenges such as rocket trajectory design, robotic arm manipulation, and autonomous systems [40]. Another key reason for this choice is that CartPole is one of the primary benchmarks for comparing reinforcement learning algorithms. Among all hyperparameters, it was decided to optimise those with a clear and interpretable impact on the DQN agent, ensuring a balance between computational feasibility and meaningful optimisation. While the genetic algorithm allows for optimising a larger set of hyperparameters, the selection was limited to four due to the computational cost of exploring a broader space. For instance, incorporating parameters such as the number of neural network layers would significantly extend training time, making the process impractical. The selected hyperparameters were chosen based on their well-defined influence on learning dynamics. The reward discount factor (\(\gamma\)) determines how future rewards influence decision-making, while the learning rate (\(\alpha\)) affects training speed and stability. The replay buffer size controls the number of stored experiences, and the target network update frequency regulates weight updates. Additionally, including both floating-point and integer hyperparameters enhances diversity in genetic search and optimisation. By selecting these four hyperparameters, this study maintains computational efficiency while optimising parameters relevant to both reinforcement learning and neural network training. From the hyperparameters described in Section 2.1.2, the selected ones include the following:  
* Gamma;
* Learning rate;
* Buffer size;
* Target network update frequency. As is known, GA aims to find the chromosome that achieves the best results. In the case of RL algorithms, the agent typically seeks to maximise the reward function. Therefore, in the initial stages of testing the implemented algorithm, a surrogate fitness function, described in Section 2.2.1, was introduced to allow a quick evaluation of individuals within the population. This function utilised a modified vector distance metric in n-dimensional Euclidean space:  
\[U_{\text{test}}(X_{i}^{m})=-\sqrt{(a_{1}-b_{1})^{2}+(a_{2}-b_{2})^{2}+...+(a_{n}- b_{n})^{2}}, \tag{3}\]  
where \((a_{1},a_{2},\dots,a_{n})\) represents the reference hyperparameter values, \((b_{1},b_{2},\dots,b_{n})\) denotes the hyperparameter values of the chromosome \(X_{i}^{m}\) at the \(m\)-th generation, and \(n\) is the total number of hyperparameters. By introducing a negative sign, the Euclidean distance minimisation problem is naturally converted into a maximisation problem, aligning with the typical GA objective to maximise fitness. Achieving \(U_{\text{test}}=0\) indicates that the two vectors are identical, implying an optimal match between the reference values and those of the current chromosome. This surrogate function (3) is not intended to replace the actual reward function in the RL setting. Instead, it serves as a simplified metric to quickly verify how effectively the GA operators can converge towards a target vector of hyperparameters. In particular, using this metric helps determine whether the primary source of computational overhead stems from the GA mechanics themselves or from the RL training process, which can be time-consuming. In doing so, \(U_{\text{test}}\) reveals whether the GA mechanisms themselves drive the computational cost or whether the RL training loop is responsible, providing valuable insights for efficient hyperparameter tuning. The simplest approach to evaluate an RL agent, as outlined in [15], is to measure its performance in the environment. Therefore, it was decided that the fitness of a chromosome would be determined based on the performance of the DQN agent during training. This approach required additional analysis to ensure that the results reflected the agent's efficiency rather than its exploration strategy.