## 3 Methodology
### Search-Based Optimization  
To take advantage of the search-based method, we use it as a demonstration to put data into an experienced pool in the same format as reinforcement learning approaches. In this way, DDPG has advantageous learning samples in the early stage of learning. The experiences of search-based method planning and self-learning are also in the experience replay buffer of DDPG, and normal random extraction cannot guarantee that the advantages of the search-based method can be learned every time. As experience increases, early search-based method data may even be taken out of the buffer. In order to improve the utilization of effective experience, we add a Prioritized Experience Replay (PER) [41] mechanism. PER enables efficient propagation of reward information [42], which is critical for training in the initial stage under training sparse rewards. PER sets the agent's experience priority so that it takes more important samples from its replay buffer more frequently.  
Priority is set with four parameters. Set \(\varepsilon_{d}\) as a constant that boosts the priority of good experience to increase the probability of being sampled. Although the priority exists, in order to guarantee the generalization performance of the learning, it is necessary to ensure that every experience can be sampled. So, adding small constants \(\varepsilon\) enables all experiences to be sampled with some probability. The last TD error \(\delta\) in the conversion process should also be considered. Finally, join the loss of the network which can be expressed as the gradient descent of the action-value function \(J(\theta)=\
abla_{\alpha}Q\big{(}s_{i},a_{i}\mid\theta^{Q}\big{)}\) and weigh it using \(\lambda\). The priority of the experience is calculated as:  
\[p_{i}=\varepsilon_{d}+\varepsilon+\delta_{i}^{2}+\lambda|J(\theta)|^{2}. \tag{9}\]  
Then, the probability that the experience is sampled can be calculated as:  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum_{k}p_{k}^{\alpha}}, \tag{10}\]  
where \(P(i)\) is the sampling probability, \(p_{i}\) is the priority, and \(\alpha\) is the distribution factor. Changes in sampling probability also affect the network. To account for the change in the distribution, updates to the network are weighted with importance sampling weights:  
\[w_{i}=\bigg{(}\frac{1}{N}\cdot\frac{1}{P(i)}\bigg{)}^{\beta}, \tag{11}\]  
where \(N\) is the batch size and \(\beta\) is a constant value for adjusting the sampling weights.  
Figure 4: Actor–critic network: (**a**) The actor network with three fully connected layers as hidden layers, with the current state as input and action instructions as output. (**b**) The critic network, consisting of three fully connected layers as hidden layers, with the current state and action as inputs and Q value as output.  
The data from the search-based method and the data from autonomous training are prioritized according to the above rules, which can naturally control the ratio of data sampling between the two during the training process, and efficiently utilize the advantages of the search-based method.  
We choose two representative search-based methods, A* algorithm and EGO-Planner [29]. A* algorithm provides a basic solution to path planning and intuitively demonstrates the advantages of SO-DRL over unoptimized methods. The EGO-Planner contains more optimizations for real physical motion, which allows SO-DRL to achieve impressive performance.