## 4 Simulation Results and Discussion  
The USV is considered to have successful berthing if it approaches the target berthing point with an error within the allowable range established in Section 3.3. The training process stops when the number of training episodes reaches 50,000. Figures 9 and 10 sequentially present a set of information that includes the trajectory, surge, sway, yaw rate, heading angle, and control input as the propeller revolution and rudder angle. The USV starts from the random state shown in Section 3.3. Under the automatic control of the TD3 model, the ship has successfully berthed at the target location. The ship states gradually change from the initial state to the required range. In particular, the combination with the adjusted reward function made the ship berthing process happen faster and more optimally. The time series of surge velocity was shown in the first phase of the berthing process. The ship's speed was increased to reduce the distance to the berthing point. However, the sway velocity and yaw rate did not change much in this phase. In the last phase of the berthing process, the importance of surge velocity, yaw rate, and heading angle is much more important than the distance in the first phase. So, the change in this value seems more sudden to adapt to the required value of the berthing process. \begin{table}
\begin{tabular}{c c} \hline
**Item** & **Value** \\ \hline \(x\) (\(m\)) & \(150\)\(\pm\) 0.5 \\ \(y\) (\(m\)) & \(30\)\(\pm\) 2 \\ \(\psi\) (\({}^{\circ}\)) & \(90\)\(\pm\) 3 \\ \(u\) (\(m/s\)) & \(0\)\(\pm\) 0.1 \\ \(v\) (\(m/s\)) & \(0\)\(\pm\) 0.05 \\ \(r\) (\({}^{\circ}/s\)) & \(0\)\(\pm\) 1 \\ \hline \end{tabular}
\end{table} 
Table 10: Target point (case 2). \begin{table}
\begin{tabular}{c c} \hline
**Item** & **Value** \\ \hline \(x\) (\(m\)) & \([-20,150.5]\) \\ \(y\) (\(m\)) & \([-10,32]\) \\ \hline \end{tabular}
\end{table} 
Table 11: Boundary values (case 2). Figure 9: Simulation results in a parallel berthing task (case 1). Figure 11a,b shows the learning performance of TD3 case 1 with 50,000 episodes. Although the warmup episode number is 20,000, the average reward shows that the model seems to be successfully berthing and stable before the warmup phase is finished. This demonstrates that TD3 provides good reinforcement learning for the automatic berthing process. To account for the difference in penalty values affecting the training process, too high a penalty will cause the model to misjudge the states, for instance, in cases where the vessel has moved reasonably close to the berthing position. However, it then collides with the wall and receives a heavy point deduction, causing the model to judge that process as wrong and try actions other than that process. This makes the training process longer. These results show that the method proposed in this paper has a high performance and success rate with a low penalty value. The simulation results show that the combination of TD3 and the maneuvering simulation proposes a powerful and accurate system for the automatic berthing process. Comparing the shape of the average reward to the results shown in [15] demonstrates that the stability of the TD3 algorithm is better than the older algorithm in the field of reinforcement learning. In particular, the method proposed in this paper is easier to apply because of the ability to learn, explore, and optimize the policy automatically. It can be used for another ship model if we know the ship's hydrodynamic characteristics. However, the limitations in this paper are evident. Firstly, the simplification of the port condition: due to the simplicity of the model, the effect of the disturbance was ignored in the simulation. This causes non-accuracy if there is wind or waves. The second limitation of this approach is the simplification of the obstacles and port geometry. It has a significant effect on the determination of the initial state and the berthing point. The presence of moving obstacles can make the training time increase significantly.