## 2 Related Works
Among these, the Deep Deterministic Policy Gradient (DDPG) algorithm [32] gained significant attention, leading to its application in various autonomous driving studies [37; 54; 55; 56; 57; 58; 59]. In [54], the authors introduced Controllable Imitative Reinforcement Learning (CIRL), a vision-based end-to-end control framework for urban driving. CIRL combines model-based supervised learning with DDPG in a dual-stage training procedure to overcome the exploration inefficiency of RL and the generalization limitations of imitation learning. Ref. [57] proposed a hybrid framework that combines affordance learning and DDPG for urban autonomous driving. In this approach, a high-dimensional camera input is processed by a CNN to extract six key environmental factors: (1) steering angle deviation, (2) traffic light state (e.g., red light detection), (3) distance to the leading vehicle, (4) speed limit, (5) lateral lane offset, and (6) presence of hazards (e.g., pedestrians). These affordances, along with vehicle speed and lane departure information, are used to generate continuous control outputs for steering, acceleration, and braking. Ref. [37] proposed a two-stage DDPG framework for the car-following problem in autonomous vehicles, combining both real-world and simulated data. In the first stage, the agent is trained in the CARLA simulator using standard DDPG. In the second stage, additional fine-tuning is performed using real driving data from the Napoli dataset via a practical replay buffer, enhancing the policy's adaptability to real-world dynamics. While DDPG is effective for continuous control, it is based on a deterministic actor-critic structure, which makes it vulnerable to \(Q\)-value overestimation. This leads to performance degradation in situations involving multi-modal decision-making, which are commonly encountered in complex autonomous driving tasks. In response to the limitations of deterministic actor-critic methods, a new line of research has emerged that seeks to mitigate the \(Q\)-value overestimation problem through stochastic policy-based approaches. Among these, PPO [30], which constrains policy updates via a clipping mechanism, has been applied to various autonomous driving decision-making tasks [60; 61; 62; 63]. In [60], the authors proposed an end-to-end autonomous driving policy learning method that integrates PPO with curriculum learning in the CARLA simulator. The study demonstrated that even without prior knowledge, the agent could learn robust driving behaviors through a staged curriculum with gradually increasing task difficulties. Additionally, the authors addressed issues related to unstable value function updates and the normalization of the advantage function. Ref. [61] applied PPO to the tactical decision-making problem for autonomous vehicles navigating intersection scenarios, with the goal of ensuring both safety and efficiency during maneuvering. In [63], the authors conducted a comparative study of DDPG and PPO in the TORCS simulator, evaluating their performance in lane-keeping, obstacle detection, and collision avoidance tasks. While PPO has shown promise in improving stability and reducing \(Q\)-value overestimation to some extent, it is fundamentally an on-policy algorithm. As such, it requires repeated rollouts with fresh data, making it sample-inefficient compared to its off-policy counterparts. Furthermore, its limited capacity to fully resolve overestimation issues has led to performance bottlenecks in complex autonomous driving scenarios. To overcome the performance limitations of DDPG and PPO, TD3 [33] was introduced as an enhancement to DDPG, specifically designed to address the \(Q\)-value overestimation problem. TD3 improves policy learning stability by employing a double critic architecture and selecting the minimum of the two \(Q\)-value estimates. Furthermore, it introduces policy smoothing by adding clipped noise to the target policy values, which mitigates overfitting to narrow \(Q\)-function peaks.