## 1 Introduction
The Deterministic Policy Gradient (DPG) algorithm [14] replaces the method of representing policy using probability distribution with a deterministic function, making control in continuous action spaces more practical. The DDPG algorithm [15], which combines the two approaches, has thus found widespread applications in fields such as robotics control. Over the last few years, numerous researchers have attempted to apply deep reinforcement learning (DRL) methods to solve the problem of path following. [PERSON] et al. [5] proposed three sequentially improved methods based on DDPG, which realized the tracking of the path and adaptive velocity control of a quadrotor. It is difficult to fine-tune an agent trained in a simulation environment through experimentation; thus, a correction constant is employed for stable performance of the agent's output in their experiment. [PERSON] et al. [16] accomplished path following and collision avoidance for a nonholonomic wheeled mobile robot in simulation, but the trained agent exerted excessive control effort, resulting in high jerks in robot velocities. More recently, [PERSON] et al. [17] propose a 3D path-following control method for powered parafoils, utilizing a combination of linear active disturbance rejection control and DDPG, to effectively control the parafoils' flight trajectory and counter wind disturbances. [PERSON] et al. [18] present a path-following control scheme based on Soft Actor-Critic for an underwater vehicle, demonstrating successful path tracking. Our work focuses on exploring the application of RL to the path-following problem for ground car-like vehicles that move at medium speeds, with the primary objective of minimizing cross-track error. We use the DDPG algorithm, which combines the advantages of Deep Q-Network (DQN) and DPG, to solely address the steering control of the vehicle, separating lateral and longitudinal control to simplify the problem and reduce the action dimension. We also use a simple reward function to achieve smooth steering and avoid jitters during vehicle operation. This approach has been demonstrated to be easy to train and effective, with better performance on the trained path compared to three baseline methods and comparable performance on untrained paths. The trained agent's control strategy has been shown to respond quickly to lateral offset from the target path with an acceptable overshoot. Given the advantages of this method, it has great potential for practical application. The remainder of this paper is structured as follows. In Section 2, the path-following problem is described using a general approach, allowing for broad applicability. The kinematics model of the car-like vehicle, which relates to the reference path, is then introduced in detail. Section 3 provides an overview of the prerequisite knowledge of reinforcement learning, which forms the foundation of the actor-critic algorithm known as DDPG. This is followed by a description of the design and algorithm of the path-following controller based on DDPG, including implementation details. In Section 4, the training process and corresponding agent performances on three different paths are presented. Each case is compared and numerically analyzed against three baseline methods, with additional discussion on corresponding steering actions. The conclusion is presented in Section 5.