## 5 Conclusions  
In this paper, we explored an off-policy algorithm, namely DDPG, based on an actor-critic architecture to address the path-following problem for ground vehicles. Our approach not only minimizes the cross-track error between the vehicle and the path, but also prevents excessive steering that can cause severe oscillations. To train the agent, We used a challenging and varied environment where each episode generates a random path. In testing, we evaluated the performance of the trained agents in terms of fast path convergence and smooth steering by selecting three representative paths.  
Conventional methods rely on rules and parameter tuning. As a comparison, three baseline methods mentioned in this paper require parameter adjustment for each path to achieve good path-following performance. In contrast, the trained agent has broader applicability and outperforms the baseline methods. Similar to the baseline methods, our agent focuses only on steering control to achieve the goal of path following, reducing the action dimension but also losing some exploration space. An agent that combines both speed and steering control may find better solutions, which is our future research direction. Furthermore, control strategies that take into account both the path-following and tyre  
Figure 12: The trajectories of the positions for calculating the cross-track error, where the initial postures are all \([0.0,0.5,0.0]\). The right subplot show the changes of steering angles over time.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Delay Time (s)** & **Settling Time (s)** & **Overshoot (\%)** \\ \hline Pure Pursuit & 0.60 & 1.00 & 11.11 \\ Feedback & 0.50 & 0.80 & 10.91 \\ Stanley & 0.50 & 1.80 & - \\ Proposed DDPG & 0.45 & 0.80 & 5.51 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Comparison of convergence performance.  
management in contact with various terrains [37; 38], or more importantly include the pollution due to particles of worn rubber [39; 40], is a practical aspect we aim to investigate in our future work.  
In conclusion, our approach effectively achieves smooth path following by only interacting with the environment to reward long-term returns. Our approach has demonstrated satisfactory performance and could contribute to the development of autonomous driving technology.  
Conceptualization, Y.C., T.K. ([PERSON]) and S.H.; methodology, Y.C., K.N., S.H. and W.J.; software, Y.C., X.J. and H.Z.; validation, K.N., X.J. and T.K. ([PERSON]); formal analysis, Y.C., K.N., X.J. and H.Z.; investigation, Y.C., T.K. ([PERSON]) and W.J.; resources, T.K. ([PERSON]) and S.H.; data curation, Y.C. and S.H.; writing--original draft preparation, Y.C.; writing--review and editing, Y.C.; visualization, Y.C.; supervision, T.K. ([PERSON]), S.H. and W.J.; project administration, S.H.; funding acquisition, T.K. ([PERSON]). and S.H. All authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Not applicable.  
The data that support the findings of this study are available from the author, [PERSON], upon reasonable request.  
The authors declare no conflict of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} RL & Reinforcement Learning \\ DDPG & Deep Deterministic Policy Gradient \\ DPG & Deterministic Policy Gradient \\ DRL & Deep Reinforcement Learning \\ PPO & Proximal Policy Optimization \\ DQN & Deep Q-Network \\ MDP & Markov Decision Process \\ TD & Temporal-Difference \\ OU & Ornstein-Uhlenbeck \\ ReLU & Rectified Linear Units \\ \end{tabular}