## 4 Hyperparameter Optimization
For SAC, although the HER version attained a slightly higher maximum success rate, it required substantially longer training, and its final results were comparable to those of the non-HER variant. This finding implies that for tasks of a moderate complexity or in environments where the goals are relatively easy to reach, HER may not yield significant benefits and might even add an unnecessary training overhead. Furthermore, these results indicate that deeper network architectures (e.g., [1024, 1024, 1024, 1024]) can more effectively capture the complex dependencies in a simulated environment, leading to higher success rates overall. All of the reported training times (e.g., 6.45 h, 7.21 h) refer to the wall-clock timesobserved in the physical workstation employed for both the environment simulations and deep RL training, as opposed to any simulated in-environment times. In particular, these measurements reflect the total duration from the start of each learning run (initialization of the RL algorithm and environment) to the completion of the final training step. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Hyperparameter** & **SAC** & **TD3** & **TQC** \\ \hline
**Buffer size** & 1,000,000 & 1,000,000 & 1,000,000 \\
**Batch size** & 256, 512 & 256, 512 & 256, 512 \\  & [256, 256, 256, 256], [512, 512, 512], [1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2056, 2056, 2056] & [256, 256, 2056, 2056, 2056, 2056, 2056, 2056, 2056, 2056, 2056, 2056, 2056, \\
**Number of critics** & 1 & N/A \\ Learning rate & 0.001–0.004 (opt.) & 0.001–0.004 (opt.) & 0.001–0.004 (opt.) \\
**Gamma** & 0.90–0.99 (opt.) & 0.90–0.99 (opt.) & 0.90–0.99 (opt.) \\
**Learning starts** & 100 000 (const) & 100 000 (const) & 100 000 (const) \\
**Use side** & True / False (opt.) & N/A & True / False (opt.) \\
**Use side at warmup** & True / False (opt.) & N/A & True / False (opt.) \\
**Sde sample freq** & 1 & N/A & 1 \\
**Action noise** & N/A & Normal (sigma opt.) & N/A \\
**Sigma** & N/A & 0.01–0.2 (opt.) & N/A \\ \hline \hline \end{tabular}
\end{table} 
Table 3: RL algorithms’ hyperparameters. Figure 4: An overview of the training performance metrics for the examined algorithms. Panel (**a**) shows the success rate and reward trends; panel (**b**) gathers five additional learning metrics. While Table 3 outlines the parameter ranges and final selections for each algorithm, additional investigations were conducted regarding the impact of specific hyperparameters on the training performance in the AUV environment:  
* **The Discount Factor (\(\gamma\)):** Experiments were carried out with \(0.90\leq\gamma\leq 0.99\). Empirically, \(\gamma\geq 0.95\) was found to be necessary for the agent to account for longer-term rewards (e.g., avoiding distant obstacles). Lower \(\gamma\) values frequently led to short-sighted maneuvers. Long-term maneuver planning is most suitable for complex harbor environments like that in the test. * **The Learning Rate:** The observations from the Optuna trials indicated that rates in the range of \(1.2\times 10^{-3}\) to \(1.6\times 10^{-3}\) provided a balance between stable convergence and training speed. Higher rates sometimes accelerated the initial improvement but increased the risk of divergence in deeper networks, whereas lower rates produced more stable yet slower learning. As seen in Figure 4a, even with a slow learning rate, the algorithms take enough steps to reach a high success rate, while the learning process remains stable. No catastrophic forgetting phenomena are observed either. * **The Network Architecture:** The 6-DOF dynamics of the environment, combined with partial observability from 24 sonar beams, benefited from larger networks (e.g., [1024, 1024, 1024, 1024]). Smaller architectures (e.g., [256, 256, 256, 256]) occasionally plateaued at moderate success rates, presumably due to their limited representational capacity and the complex environment (a 6-DOF AUV model and a continuous and large map).