## 4 Experimental Validation
### TD3 Algorithm Optimization  
To validate the potential of deep reinforcement learning algorithms in enhancing path quality, this research further explored the application of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for BIT* path optimization. This experiment aimed to evaluate the effectiveness of TD3 as a post-processing optimizer in improving paths generated by the BIT* algorithm. In the experiment, we designed and implemented a TD3-based agent to iteratively optimize the initial paths generated by the BIT* algorithm. To ensure the validity of the experiment and the reliability of the results, meticulous hyperparameter tuning was conducted, and a set of parameter configurations that balance algorithm performance, stability, and computational efficiency was finally determined, as shown in Table 4. Table 1 details the specific hyperparameter values used for training the TD3 agent. Key hyperparameters include the discount factor (0.95), which determines the importance of future rewards; the soft update coefficient (0.005), which controls the rate at which the target networks are updated; and the learning rates for both the Actor and Critic networks (0.0002). These values were chosen based on a combination of prior work in the field of deep reinforcement learning and empirical tuning to achieve a balance between exploration and exploitation, ensuring stable and efficient learning. The focus of the experiment is to observe and analyze the effect of the TD3 algorithm on improving path quality at different training stages and to evaluate the learning performance and convergence of the TD3 algorithm through the variation curve of the reward function. During the training process, we recorded the average reward for each episode and plotted the reward function convergence curves as shown in Figures 11-14. As can be seen from the figures, the average reward exhibits a clear upward trend with the increase in training episodes and gradually converges to a relatively high level after approximately 600 episodes. This indicates that the TD3 agent effectively learns strategies for optimizing paths through interaction with the environment. Notably, the reward increases relatively rapidly in the early stages of training, suggesting that the agent can quickly grasp basic path optimization skills. In the later stages of training, the fluctuations in reward gradually decrease and eventually stabilize around a relatively optimal value, indicating that the agent has learned a relatively stable strategy capable of generating high-quality paths. It is worth noting that some minor fluctuations exist in the reward curve, which may be attributed to the presence of exploration noise in the TD3 algorithm and the inherent stochasticity of the environment. Overall, the convergence curve of the reward function clearly demonstrates the effectiveness of the TD3 algorithm in the path optimization task, validating its feasibility and superiority when combined with the BIT* algorithm. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter Name** & **Value** & **Description** \\ \hline Maximum Steps per Episode & 1000 & The maximum number of steps allowed within each episode. \\ Episodes & 200 & The total number of episodes for training. \\ Discount Factor & 0.95 & The discount factor used for calculating future rewards. \\ Soft Update Coefficient & 0.005 & The parameter controlling the soft update of the target networks. \\ Policy Noise & 0.1 & The standard deviation of the noise added to the target actions. \\ Noise Clip & 0.3 & The range to which the noise is clipped. \\ Policy Update Delay & 4 & The frequency at which the policy network is updated relative to the value network. \\ Actor Network Learning Rate & 0.0002 & The learning rate for the Actor network. \\ Critic Network Learning Rate & 0.0002 & The learning rate for the Critic networks. \\ Replay Buffer Size & 2E6 & The size of the experience replay buffer. \\ \hline \hline \end{tabular}
\end{table}