# Improved DQN for Dynamic Obstacle Avoidance and Ship Path Planning
## 4 Experiment
### Comparison of Test Results  
According to the above experimental simulation environment and parameter settings, the ship path drawn by improved DQN, original DQN, Q-learning algorithm in the case of obstacles is shown in Figure 3-5. It can be seen from the figure that the ship draws the shortest navigation path with the improved DQN. Although all three algorithms finally find the end point, their costs are different. Figures 4 and 5 show the capabilities of the original DQN and Q-learning in ship path planning, and the two algorithms produce repeated exploration in some sections. The original DQN performs better than the Q-learning algorithm. The costs of the three algorithms at given starting point and end point are shown in Table 2.  
Figure 3: The improved DQN for path planning.  
Figure 4: The original DQN for path planning.  
Figure 5: The Q-learning for path planning.  
In the simulation, when the ship arrives at the destination, the reward value is \(1\), and when the ship moves towards the obstacle, the reward value is \(-1\). Moreover, the reward value is \(0\). The maximum episode setting in the paper is \(2000\). With the execution of the episode, the steps and cost of each episode of the ship are shown in Figures 6-11. In Figure 6, the ship based on the improved DQN algorithm is constantly explored, and the steps will change dramatically. When the algorithm is executed to the \(1750\)th episode, it reaches stability. The value of steps is \(21\). The steps of the original DQN algorithm are compared, as shown in Figure 7. In the whole episode process, the original DQN has been in dynamic exploration and can not move towards the established goal due to uniform sampling. Figure 8 shows the steps performance of Q-learning algorithm. Q-learning has explored the longest step, but the algorithm is not stable. Figures 9-11 shows the cost of the whole episode. In this paper, cost is the reward obtained by the ship when it travels to the target point. It can be seen from the figure that the reward of the improved DQN algorithm has always been at the maximum in the \(1750\)th episode. The performance of the other two algorithms is weaker than that of the improved DQN.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm** & **The Length of Path** \\ \hline The improved DQN & \(21\) \\ The original DQN & \(32\) \\ The Q-learning & \(124\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental environment  
Figure 6: Episode via steps of the improved DQN.  
Figure 8: Episode via steps of the Q-learning.  
Figure 7: Episode via steps of the original DQN.  
Figure 10: Episode via cost of the original DQN.  
Figure 9: Episode via cost of the improved DQN.  
In order to further verify the performance of the improved DQN algorithm, this paper dynamically generates obstacles in the simulation environment except that the meteorological environment remains unchanged. The simulation results are shown in Figures 12-17. The improved DQN algorithm also performs well in the ship path planning problem with randomly generated obstacles. Under the complex environment, the improved DQN can also find a safe and reliable path. As shown in Figures 13 and 16, after little exploration, the improved DQN algorithm can find a feasible path and reach a steady state. Since the goal of the algorithm is to obtain the maximum reward, the improved DQN algorithm aims to pursue the maximum reward. The cost of each episode is shown in Figures 13 and 16.  
Figure 14: Episode via cost of the improved DQN for scenario 1.  
Figure 13: Episode via steps of the improved DQN for scenario 1.  
Figure 16: Episode via steps of the improved DQN for scenario 2.  
Figure 15: The improved DQN for the path planning of dynamic obstacle scenario 2.