## 3 The Improved Path Planning Method  
RRT* has the characteristics of continuous iteration and path replanning in the environment. However, due to its random nature, each exploration or iteration of RRT* is independent of each other, so the new node positions generated by each iteration may be different, leading to a tortuous and suboptimal path. The reward function in DRL plays a pivotal role in shaping the learning effectiveness of the agent. It is imperative to define an appropriate reward function for each task to guide the agent toward successful task completion. The variation of the task can also easily lead to the failure of the reward function. Currently, there are many improved variations of RRT*, such as informed RRT* ([PERSON] et al., 2018), RRT*-Smart ([PERSON] et al., 2018), and real-time RRT* ([PERSON] et al., 2018), which have advantages in specific scenarios. However, they still face the problem of independent exploration and iteration. Additionally, introducing target biasing methods in RRT* can yield favorable results, and this approach is relatively easier to implement and debug compared to algorithms such as informed RRT*. Based on these issues, a path planning framework that combines the strengths of DRL and RRT* have been proposed, and is built on the basis of RRT* with target bias. Performing a path search between two adjacent waypoints is called a dynamic point-to-point (D-P2P) task. The agent defines a spherical range centered at the intermediate waypoint as an explorable region and performs a D-P2P task between the preceding and following waypoint. The initial base path \(R^{0}=\left\{r_{1}^{0},r_{2}^{0},\ldots,r_{N}^{0}\right\}\) is obtained by RRT*, \(N\) is the total number of waypoints. The length of the original path can be characterized as \(l_{0}^{1}=\sum_{n=1}^{N-1}|r_{n}^{1}r_{n+1}^{1}|\), where \(\left|r_{n}^{1}r_{n+1}^{1}\right|\) represents the Euclidean distance between waypoints \(r_{n}^{1}\) and \(r_{n+1}^{1}\). The steps of the DRL-RRT* algorithm is described as follows. Step 1: Initialization. The number of iterations for RRT* is \(m\), DRL path search task is \(K\) times. D-P2P task for each iteration is \(J=N-2\) groups. Using RRT* for planning iteration. Step 2: Using DRL for path search tasks. When the agent first enters the \(jth\)\((1\leq j\leq J)\) explorable region in a certain steps, the position \(c_{j}^{k}\) is retained as the end position of this D-P2P task, and the waypoint replacement task in Step 3 is performed. If the explorable region is not accessed in the specified number of steps, go to Step 4. Step 3: The procedure of waypoint replacement. The first waypoint replacement is considered, resulting in point \(c_{2}^{1}\) being obtained after completion of the D-P2P task; then, the impacts of \(r_{2}^{1}\) and \(c_{2}^{1}\) on the original path are computed individually. If \(c_{2}^{1}\) makes the path shorter, \(c_{2}^{1}\) replaces \(r_{2}^{1}\), and the length becomes \(l_{0}^{1}=\left|r_{1}^{1}c_{2}^{1}\right|+\left|c_{2}^{1}r_{3}^{1}\right|+ \sum_{n=3}^{N-1}\left|r_{n}^{1}r_{n+1}^{1}\right|\). The waypoints of the path become \(R^{1}=\left\{r_{1}^{1},c_{2}^{1},\ldots,r_{N}^{1}\right\}\); at the same time, \(c_{2}^{1}\) also serves as the starting position for the next D-P2P task; otherwise, the original path is unchanged. Step 4: Termination condition check for a round of DRL search task. If D-P2P task iteration is larger than the iterative number \(J\), break to Step 2. Step 5: Get the new path \(R^{k+1}\). Assuming that only \(c_{2}^{1}\) can shorten the path during the first iteration (\(k=1\)), the path for the second iteration is \(R^{2}=\left\{r_{1}^{2},r_{2}^{2},\ldots,r_{N}^{2}\right\}=\left\{r_{1}^{1},c _{2}^{1},\ldots,r_{N}^{1}\right\}\). Step 6: Terminating condition check. If the iteration is larger than \(K\), the path explored is retained as \(P=R^{K}\), and the algorithm stops. Otherwise, restart from Step 2. RRT* with DRL is utilized by [PERSON] ([PERSON], 2018); however, the waypoints were unchanged during the process.