## 4 Simulation Experiments
### Evaluation For Search-Based Optimization  
The simulation of SO-DRL included two experiments using the A* algorithm and EGO-Planner for optimization. To verify the effectiveness of the search-based algorithm on the strategy in the framework, our method was compared with the original DDPG method and trained according to the same strategy. Table 1 clearly shows the training setup, where the number of training times for stage 1 and stage 2 are shown in the Episode. For each method, we followed a two-stage model for training. SO-DRL was trained by loading the corresponding experience in the first stage, and we chose the first convergence weights as the initial weights for the second stage of training. For the original DDPG algorithm, we trained the first stage directly and also selected the convergence weights for the second stage of training. The average reward function and the average Q value for each episode of the first stage are shown in Figure 8. It can be seen that SO-DRL had a negative reward at the beginning of training and some downward floating. This is because there were already some positive and negative rewards of experience in the early stage of training, and the SO-DRL network needed to integrate with the experience and did not achieve good rewards at the early stage. However, as the training progressed, SO-DRL converged relatively faster, especially SO-DRL with EGO-Planner optimization, which was ahead of the original DDPG at 100 episodes, and SO-DRL with A* algorithm optimization, which was ahead of the original DDPG by about 220 episodes. From the performance of the final convergence, the rewards of SO-DRL reached more than 60, while the original DDPG could not break 60, and there was an obvious gap between the two in terms of rewards. As can be seen from the Q values, it is the uneven sample distribution at the beginning that caused the quantifiers to not be very good at the beginning, but after about 50 episodes, the Q values of SO-DRL were able to exceed the original DDPG. SO-DRL with EGO-Planner optimization converges in less than 200 episodes. Although SO-DRL with A* algorithm optimization converged slightly later than the original DDPG method, SO-DRL eventually converged to a higher Q value than the original DDPG. It can be seen that the overall performance effect of SO-DRL with EGO-Planner optimization was better than SO-DRL with A* algorithm optimization than the original DDPG. \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{\begin{tabular}{c} **Experience** \\ **Buffer** \\ \end{tabular} } & \multicolumn{2}{c}{\
\begin{tabular}{c} **Additional** \\ **Experience** \\ **Achieved** \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} **Episode** \\ **Consuming** \\ \end{tabular} } \\ \hline Original DDPG & Only DDPG & None & 600 + 600 & 363 min \\ SO-DRL(A*) & A* + DDPG & A* control & 600 + 600 & 240 min \\ SO-DRL(EGO) & EGO + DDPG & EGO control & 600 + 600 & 214 min \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Training settings by evaluation for search-based optimization. The average reward function and the average Q value for each episode of the second stage are shown in Figure 9. As can be seen by the reward, at the beginning of the second stage, all three methods had certain initial values, and the SO-DRL with EGO-Planner optimization was better than the SO-DRL with A* algorithm optimization than the original DDPG in terms of initial value level, convergence speed, and convergence final value, and the trend and gap of the three curves are obvious. As for the Q values, we can see that the Q values produced some unstable jitter at the beginning due to the new environment, but in the subsequent training process, the advantage of SO-DRL was obvious, indicating that SO-DRL can cope better in longer distances and denser environments.