## 2 Related Works  
This section reviews prior research efforts that have used DRL for autonomous driving, focusing on the methodologies employed and their associated limitations. We first examine foundational studies aimed at developing efficient DRL-based autonomous driving systems and follow this with comparative evaluations of various DRL algorithms. Early approaches to decision-making in autonomous driving primarily relied on IL, where an agent learns to mimic the behavior of a human expert [25; 26; 42; 43; 44; 45; 46; 47]. One of the earliest attempts to apply IL to autonomous driving is presented in [42], where a neural network was trained via supervised learning to predict steering angles from a camera input. This study was also among the first to experimentally highlight the problem of covariate shift in IL-based driving. In [43], the authors proposed PilotNet, a pioneering end-to-end driving model that extracted visual features from camera images using a convolutional neural network (CNN) and directly output steering commands. The model was trained using behavior cloning on a large-scale human driving dataset. However, its trajectory completion performance was limited in complex scenarios. To enhance IL, ref. [25] introduced a hierarchical command-conditioned imitation learning approach, where high-level navigational commands (e.g., turn left/right, go straight) guided the agent's behavior. In [26], the authors proposed Generative Adversarial Imitation Learning (GAIL), in which a discriminator network distinguishes between expert and learner trajectories, thereby guiding the agent to produce expert-like behavior through adversarial training. Similarly, ref. [44] presented a generative model that learns trajectory distributions via Conditional Variational Autoencoders (VAEs), enabling multi-modal trajectory generation through flexible planning. In [45], an end-to-end learning environment was established by integrating the CARLA simulator with OpenAI Gym, where deep imitation learning was empirically validated through autonomous driving experiments. However, these IL-based approaches had significant limitations in their generalization and scalability due to common issues such as covariate shift [22], error compounding [23], distributional drift [24], and recovery failure [25; 26]. To mitigate such limitations, hybrid approaches combining IL and reinforcement learning have been explored. For example, ref. [46] proposed a Hierarchical Behavior and Motion Planning (HBMP) framework, integrating PPO with DAGGER-based imitation learning to enable efficient and safe lane-change planning. Likewise, ref. [47] attempted to overcome the limitations of IL by combining it with SAC, leveraging the strengths of both paradigms. Nevertheless, these hybrid approaches represent transitional efforts made during a period when DRL techniques had not yet fully matured. As DRL methodologies have advanced, the focus of research has increasingly shifted toward direct policy learning through DRL itself, aiming to exploit its full potential in autonomous decision-making tasks. Early research on autonomous driving decision-making using DRL predominantly focused on the Deep Q-Network (DQN) algorithm [28] and its variants, such as Double DQN (DDQN), Dueling DQN (DuIDQN), and Prioritized Experience Replay DQN (PRDQN) [39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53]. While these methods laid foundational groundwork for learning decision policies, their applicability in real-world driving was limited by the discrete nature of the action space inherent to a DQN, making it difficult for it to directly learn fine-grained control signals such as steering angles and throttle commands. To address this limitation, several DRL algorithms were subsequently proposed to handle continuous action spaces, including those based on deterministic actor-critic architectures.