## 8 Conclusions  
The presented research clearly demonstrates that reinforcement learning algorithms can effectively address trajectory planning tasks for underwater vehicles (AUVs), while incorporating both a model of the vehicle's motion and real-world constraints derived from an actual ENC port map. Among the tested methods, the Truncated Quantile Critic (TQC) algorithm performed the best, achieving high success rates alongside low average path distances and short mission times. This indicates that accounting for the uncertainty distribution in the action values--by means of quantiles--results in enhanced stability and higher-quality decisions, in line with the AUV's dynamic characteristics.  
Compared to the classical approaches (A*, APF, GA, PSO, and RRT*),t he RL-based methods more frequently produced smoother trajectories, thereby avoiding abrupt heading changes and excessive speed fluctuations. Consequently, the vehicle achieved shorter mission times and avoided obstacles more efficiently. Nonetheless, certain classical algorithms--such as the GA and PSO--attained a one-hundred-percent success rate in generating near-optimal paths in terms of their length. Methods such as the RRT* and APF offer significantly faster computation times. However, while the RRT* typically produces feasible trajectories, these often deviate considerably from optimality. In contrast, algorithms like the APF occasionally result in collisions or entrapment in local minima, highlighting their limitations in complex, confined environments.  
The use of Hindsight Experience Replay (HER) did not always improve the performance--on some occasions, it increased the training time without significantly boosting the success rate. Curriculum learning, however, enhanced the training stability and effectiveness, preventing the system from failing to reach the goal at longer distances. Moreover, this study confirms that a realistic model of vehicle motion is crucial for deploying RL solutions in real-world navigation tasks, as it ensures that the trained policies transfer effectively to genuine operational conditions.  
Using RL for route planning on an actual port map may be yet another factor bringing the application of this group of methods in maritime navigation closer. Research has shown that RL algorithms can effectively mirror the complexity of port environments and enable collision-free navigation, competing with and sometimes surpassing the classical algorithms. Additionally, they plan routes without abrupt maneuvers, which distinguishes them from the classical approaches. Moreover, the results show that RL algorithms have strong abilities to generalize path finding in an unseen environment, achieving 0.79 to 0.84 success rates. In future work, the plan is to validate the RL algorithms under more challenging conditions, including environmental factors such as ocean currents and uncertainties in determining the position.  
Conceptualization: L.M. and R.K. Methodology: L.M., P.S. and R.K. Software: L.M. and R.K. Validation: P.S. Formal analysis: L.M., P.S. and R.K. Investigation: L.M., P.S. and R.K. Resources: L.M., P.S. and R.K. Data curation: L.M., R.K. and P.P. Writing--original draft preparation: L.M. and R.K. Writing--review and editing: L.M., P.S. and R.K. Visualization: L.M. and R.K. Supervision: P.S. Project administration: L.M., P.S., P.P. and R.K. Funding acquisition: L.M., P.S., R.K. and P.P. All of the authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Not applicable.  
The original contributions presented in this study are included in the article; further inquiries can be directed to the corresponding author.  
The authors declare no conflicts of interest.