## 4 Hyperparameter Optimization  
As reinforcement learning (RL) algorithms exhibit inherently stochastic behavior, a hyperparameter optimization process was carried out. For this purpose, the Optuna library was employed, offering advanced automation for hyperparameter tuning. Each algorithm features its own specific hyperparameters, yet some remain common across all methods. Table 3 summarizes both the fixed and tunable parameters. To identify the best-performing model in terms of its success rate, 12 Optuna trials were run (4 per available GPU). The training duration was fixed at 600,000 steps in each trial. Within every trial, the Optuna optimizer aimed to maximize the average success rate by iteratively adjusting the hyperparameter values. Table 2 presents the results of hyperparameter optimization for the three well-established reinforcement learning algorithms (SAC, TD3, and TQC) and their variants without Hindsight Experience Replay (HER). This optimization was conducted using the Optuna library, which automatically adjusts the hyperparameters to maximize the success rate. The table includes key hyperparameters, such as the discount factor \(\gamma\), the learning rate, the neural network architecture, and the use of Stochastic Differential Equations (SDEs) and HER. Additionally, for TD3, the exploration noise level \(\sigma\) is provided. The results indicate that the algorithms without HER (e.g., SAC, TD3, and TQC) frequently attained higher peak and final success rates compared to those of their HER-enabled counterparts. For instance, TQC achieved a highest success rate of 1.0 and a final result of 0.9992, whereas TQC with HER reached a maximum success of 0.996 and a final \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline
**Algorithm** & **Gamma** & **Learning Rate** & **Net Arch** & **Use SDE** & **Sigma** & **Use HER** & **Max Success Rate** & **End Success Rate** & **Training Time [h]** \\ \hline SAC-HER & 0.9622 & 0.001586 & [256, 256, 256, 256] & False & - & True & 0.98 & 0.9738 & 9.44 \\ TD3-HER & 0.9844 & 0.001231 & [1024, 1024, 1024, 1024] & - & 0.0370 & True & 0.9517 & 0.9517 & 8.59 \\ TQC-HER & 0.9674 & 0.001268 & [256, 256, 256, 256] & False & - & True & 0.996 & 0.9903 & 10.31 \\ SAC & 0.9515 & 0.001291 & [512, 512, 512, 512] & False & - & False & 0.9782 & 0.9603 & 6.45 \\ TD3 & 0.9687 & 0.001450 & [1024, 1024, 1024, 1024] & - & 0.1126 & False & 0.9791 & 0.9715 & 6.69 \\ TQC & 0.9896 & 0.001581 & [1024, 1024, 1024, 1024] & False & - & False & 1 & 0.9992 & 7.21 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The hyperparameters after the optimization process using Optuna (12 runs) for various RL algorithms with their success rates and training times. value of 0.9903. A similar trend was observed for TD3: the final success rate was 0.9715 without HER, outperforming TD3 with HER, which yielded 0.9517. In the case of SAC, the HER variant reached a slightly higher maximum success rate (0.98) relative to that for SAC (0.9782). However, their final rates were comparable (0.9738 vs. 0.9603), though the HER variant required a significantly longer training time (9.44 h vs. 6.45 h for SAC). Figure 4a shows how the success rate evolves with the number of training steps. Notably, algorithms incorporating HER tend to learn more slowly than those without HER. Despite achieving higher maximum success rates in some instances, they required more training steps, as was especially evident for TQC. Among all of the approaches, TQC (both with and without HER) converged most rapidly, reaching its performance plateau at around 150,000 steps. The weakest performer in terms of its success rate was TD3, for which the non-HER variant surpassed the HER-based variant in both its final success rate and total training time. This analysis suggests that using HER does not consistently improve the performance of reinforcement learning algorithms. In the case of TD3 and TQC, the non-HER variants not only achieved higher success rates but also required shorter training times.