## 3 DDQL with Retrospective Mechanism for Path Planning
### Other Details
#### 3.5.2 Overall Training Framework
The comparative analysis results are provided in Table 5.

Table 5 provides a comprehensive quantitative comparison of the path-planning performance of the three algorithms. As shown in the table, the path generated by DQL requires 59 steps to reach the target and involves 29 turns, resulting in a cumulative reward of 156.5. In contrast, the path generated by DDQL requires only 56 steps to reach the target and involves 25 turns, accumulating a reward of 162. However, the R-DDQL

Figure 11: Results of two algorithms. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **DQL** & **DDQL** & **R-DDQL** \\ \hline Steps & 59 & 56 & 47 \\ Turns & 29 & 25 & 15 \\ Cumulative reward & 156.5 & 162 & 170.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: DDQL and R-DDQL path comparison. algorithm stands out with a superior performance, reaching the target in only 47 steps and involving only 15 turns, while accumulating a higher reward of 170.5. Notably, DDQL significantly outperforms DQL, while R-DDQL outperforms DDQL in terms of the number of steps taken, the number of turns involved, and the cumulative reward, with improvements of 16%, 40%, and 5.2%, respectively. These results demonstrate that while all three methods can plan paths, R-DDQL is optimal. In order to further assess the efficacy of the R-DDQL algorithm in typical scenarios, we conducted simulation experiments on five additional maps. The corresponding data-comparison plots are presented in Figure 12. Our results clearly demonstrate that the R-DDQL algorithm outperforms both the DDQL and DQL algorithms. Specifically, across all five experiments, R-DDQL exhibited a superior step performance by 32.5%, steering performance by 45%, and cumulative rewards by 69.4% compared to DQL. Similarly, when compared to DDQL, R-DDQL achieved a 21.3% improvement in step performance, a 48% improvement in steering performance, and a 58% improvement in cumulative rewards. Figure 12: Cont. ### Performance of R-DDQN in a Dynamic Environment

In order to demonstrate the effectiveness of our proposed method in dynamic scenarios, we conducted a dynamic path-planning experiment where the mountain collapses while the agent is navigating through the path. As shown in Figure 13, the highest mountain collapses and obstructs the canyon that the agent had previously passed through. We trained the R-DDQL model, which was previously trained under static conditions, for an additional 2000 episodes under this dynamic scenario. The training parameters remained consistent with the previous training. Figure 14 displays the performance of R-DDQL in this dynamic scenario. Remarkably, the agent successfully selects a flat road from the collapsed terrain instead of following the previous path. This demonstrates that our proposed method can still perform well in dynamic environments. Figure 12: (**a**). The number of steps. (**b**). The number of turns. (**c**). The cumulative rewards. Figure 13: The location and direction of the landslide. Figure 14: The performance of R-DDQL in a dynamic environment. ## 5 Discussion

This study aims to propose a novel method for decoupling the path-evaluation and -planning algorithms through deep reinforcement learning algorithms. Our proposed method includes a 3D terrain-processing technique and an enhanced deep Q-learning method. Furthermore, we compared our approach with two other deep reinforcement learning methods. The experimental results indicate that the R-DDQL method exhibits superior performance compared to the other two methods. Specifically, the R-DDQL algorithm converges faster and achieves higher rewards in a shorter period. Moreover, the paths generated by R-DDQL are shorter and flatter than those produced by the other two methods. Finally, the generalization ability and real-time performance of the proposed method were evaluated in both static and dynamic planning tasks.