# Dynamic Path Planning for Vehicles Based on Causal State-Masking Deep Reinforcement Learning
## 5 Comparison of Simulation Results
### Convergence Analyses  
The simulation parameters for different DRL algorithms are shown in Table 1. The discount factor determines the weight of future rewards relative to immediate rewards. The soft update coefficient controls the rate at which the target network converges toward the current network. The exploration noise reflects the degree of variability in exploration behavior, with its variation determined by bounded noise. The refresh frequency of the policy network dictates the update rate of the associated parameters. In the path-planning environment, the starting point is fixed at the center of the geographical coordinate system, while the endpoint is randomly generated within the path-planning boundary. The speed of obstacles varies within a specified range to ensure that the DRL algorithm can be trained in a more diverse environment; the training environment parameters are shown in Table 2. To ensure comparability among the algorithms, all the hyperparameters and training environment parameters for the comparative algorithms were kept consistent. Figure 4: A diagram of the proposed path-planning control framework for vehicles based on CSM-TD3. In the established training environment, each dynamic path-planning task was performed 10 times to validate the stability and effectiveness of the DRL algorithms. During training, an evaluation report was generated in each assessment. In the comparison graphs for the reward curves, the solid line represents the average results of the reward curves across the 10 trials, while the shaded area indicates the range of variation in the reward curves. The path curves in the dynamic path-planning comparison illustrate the optimal paths generated by the trained agents of the same algorithm across the 10 trials in the same environment. The average return curves obtained from three DRL algorithms in this convergence comparison experiment--DQN, PPO, TD3, and the proposed CSM-TD3--are compared in Figures 5 and 6, with all the other parameters held constant. The average return of the DQN demonstrates a significantly slower growth rate throughout the training process compared to TD3 and CSM-TD3 and exhibits considerable fluctuations, consistently hovering at a lower level. This is attributed to the DQN's original design, which is tailored for discrete-action decision making, making it challenging to effectively model and optimize policies for high-dimensional continuous control tasks. Furthermore, the high volatility of the DQN indicates that it is heavily influenced by randomness during policy updates, lacking an effective mechanism to smooth and stabilize its optimization process, presenting certain limitations for continuous control tasks in dynamic path planning. PPO is a policy gradient-based method that directly optimizes the policy for continuous actions, avoiding the complexity of discretizing actions in continuous spaces as required by DQN. PPO improves training stability by clipping policy updates, thereby restricting the magnitude of each update. This helps to mitigate performance fluctuations caused by excessively large updates, resulting in better convergence performance compared to DQN in continuous action spaces. In this experiment, TD3 also outperformed the DQN, showing a clear upward trend in average returns and ultimately achieving higher return levels than the DQN. However, the return curve for TD3 exhibited significant fluctuations, indicating a degree of instability, particularly during the initial training phase, where the returns varied greatly. This suggests that TD3's handling of noise during the policy exploration phase and its ability to improve