## 3 DDQL with Retrospective Mechanism for Path Planning
### Other Details
#### 3.5.2 Overall Training Framework  
In this article, we adopt a retrospective mechanism to improve the stability and performance of the DDQL model. Algorithm 2 is the overall training framework of the proposed model. ```
Input layer \(47\times 47\times 3\) Convolution layer \(3\times 3\times 40\) Feature map after convolution \(24\times 24\times 40\) Feature map after pooling \(12\times 12\times 40\) Convolution layer \(3



## 4 Experiments and Analysis

In this section, we apply the R-DDQL algorithm proposed in Section 3 to solve the problem of 3D path planning and compare its performance with that of the DDQL and DQL algorithms. For the purpose of comparison, all three algorithms are assigned the same parameters, except that R-DDQL incorporates the retrospective mechanism proposed in Section 3.4. The comparison is divided into two parts: the training phase and the training results. ### Parameter Setting

All models adopt the same parameters, which are presented in Table 4. The maximum number of steps, denoted by M, is set to 200, indicating that if the agent fails to reach the target within 200 steps, the episode will terminate, and the next episode will begin. Moreover, the value of the greedy factor, denoted by \(\varepsilon\), is initially set to \(0.001*IT\) (where \(IT\) is the number of episodes at that moment). This means that after 1000 episodes, the heuristic rules are disabled, and the agent continues to explore according to the greedy algorithm for another 1000 episodes. In the pre-training phase, the model is executed ten times to fill the memory, and thus the initial value of _Pre_episodes_ is set to 10. Therefore, all three algorithms are trained for 2010 episodes. However, since the model is not trained during the pre-training phase, it is not included in the total training count in this study. ### Training-Phase Comparison

The figure depicting the cumulative rewards obtained during the training of the three algorithms is presented in Figure 10. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Network learning rate & 0.0001 \\ Discount factor & 0.9 \\ Memory capacity D & 20,000 \\ Greed factor \(\varepsilon\) & \(0.001*IT\) \\ Max steps M & 200 \\ _Pre_episodes_ & 10 \\ Max episodes & 2000 \\ Mini-batch size & 400 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The parameters of the neural network. Figure 10: Cumulative rewards obtained by the two algorithms. The cumulative reward curves of the three algorithms during training are presented in Figure 10. As can be observed, after 1000 episodes (line 1), the heuristic search rules are disabled, and all three algorithms experience a sharp decrease in the cumulative rewards. However, R-DDQL achieves higher rewards at a faster rate. Specifically, R-DDQL tends to converge after line 2 (at 1406 episodes) and remains stable after line 3 (at 1602 episodes), whereas the remaining two algorithms start to converge after line 4 and do not stabilize before the end of the training. Therefore, R-DDQL exhibits higher stability compared to DDQL and DQL, indicating that the proposed retrospective mechanism can mitigate the overestimation issue and enhance the training stability. ### Training Results Comparison

The paths of the three algorithms are presented in Figure 11, where the details of the image are consistent with those introduced in Section 3.1. Specifically, the red grid in the upper-left corner of the image denotes the starting point of path planning, while the blue grid in the lower-right corner represents the target area. Moreover, the black line indicates the DQL, the blue line indicates the path of DDQL, whereas the red line illustrates the path of R-DDQL. Notably, all three algorithms select the option to traverse over the lowest slope to reach the end, instead of circumventing around the side. However, the DQL and DDQL algorithms manifest hesitation before opting to enter the mountainous area (as indicated by the black arrow). In contrast, the R-DDQN algorithm is notably smoother than DQL and DDQL.