## 3 DDQL with Retrospective Mechanism for Path Planning
### Other Details
#### 3.5.2 Overall Training Framework
The improved performance of R-DDQL can be attributed to its retrospective mechanism, which enables it to identify and avoid the previously encountered overestimation problems during the learning process. The retrospective mechanism reduces overestimation errors during Q-value estimation, which is a common challenge faced by Q-learning-based algorithms. As a result, R-DDQL can more accurately assess the best action to be taken, resulting in a more efficient and effective path-planning strategy. Specifically, the improvement of R-DDQL can be attributed to two factors. First, R-DDQL optimizes the samples in the memory to increase the validity of samples and reduce the proportion of invalid samples. This ensures that the training process is based on high-quality samples, which improves the overall learning performance of the algorithm. Second, while DDQL employs a depth-first approach with the help of heuristic rules, R-DDQL incorporates a retrospective mechanism based on the combination of depth-first and extended breadth strategies. The retrospective mechanism filters the high-quality solutions that are most useful for expanding the training breadth, which improves the overall efficiency of the algorithm. Figure 15 illustrates the retrospective mechanism, with the red grid indicating the raster filtered by the retrospective mechanism. These high-quality solutions are sent to the memory D, which enables the algorithm to expand the training breadth and improve its performance. Although the retrospective mechanism guides R-DDQL, the path generated by R-DDQL is entirely different from the guidance mechanism. This implies that the retrospective mechanism can effectively guide DQL and improve its convergence speed and stability. The limitation of this method is that an algorithm needs to be developed to help DDQL improve its performance in the training phase, which will take more effort in the training

Figure 15: Path planning based on depth-first and extended breadth. phase. In future research, better retrospective mechanisms (not limited to the methods in this article) can be developed to enhance the performance of DQL. ## 6 Conclusions

In this study, a novel approach was proposed to improving the performance of self-learning algorithms by incorporating timely retrospection. The proposed retrospective mechanism is introduced to the DDQL algorithm to enhance its learning efficiency. This mechanism enables the algorithm to filter high-quality solutions, thereby improving the overall performance of the algorithm, resulting in a more efficient and effective path-planning strategy. In order to be able to integrate the path costs into the map and to be able to input the R-DDQL algorithm as a whole for path planning in 3D surface terrain, we propose a 3D terrain-processing technique that uses a digital elevation model to represent the integrated costs of paths and uses functions to simulate this 3D terrain for planning. This approach is able to separate the path cost from the planning algorithm and facilitate the robot to plan according to different path-evaluation methods. The experimental results show that R-DDQL exhibits a superior performance compared to two other deep reinforcement learning methods. Specifically, R-DDQL achieves higher rewards in a shorter period and generates shorter and flatter paths. The proposed method's generalization ability and real-time performance were evaluated in both static and dynamic planning tasks, demonstrating its effectiveness in a wide range of scenarios. The proposed method has been demonstrated for pathfinding in 3D terrain, but future work should focus on bridging the reality gap. The reality gap poses a challenge in obtaining a reliable training set in real-world scenarios with many uncertain factors. For DQL, a large number of episodes is necessary to obtain stable policies, which increases the risk of collision and equipment damage.