## 3 System Model  
### Policy Network Model  
Figure 1 presents a schematic overview of how the system operates, including the CARLA simulator and the interacting DRL module. Among the core tasks of autonomous driving, i.e., perception, decision-making, and control, the CARLA simulator is responsible for perception by collecting the camera-based visual information required for downstream decision-making. This perceived information is then transmitted to the DRL module, which performs decision-making based on a reinforcement learning algorithm. The resulting control actions are subsequently processed through action smoothing (a weighted sum between the previous and current action values) and sent back to the CARLA simulator to control the ego vehicle. Finally, the resulting reward for the executed action is fed back to the DRL module to evaluate the action and update the policy accordingly. In detail, the CARLA simulator receives front-facing RGB images from the ego vehicle and processes them using the Variational Autoencoder (VAE) network illustrated \begin{table}
\begin{tabular}{p{56.9 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**RL Category** & **Representative Works** & **Advantages** & **Limitations** \\ \hline \multirow{3}{*}{Imitation Learning (IL)} & \multirow{3}{*}{[25, 26, 42, 43, 44, 45]} & Simple training via expert data; good initial policy generation & Poor generalization; covariate shift; error compounding \\ \hline \multirow{3}{*}{Hybrid (IL + RL)} & \multirow{3}{*}{[46, 47]} & Combines strengths of IL (stability) and RL (adaptability) & Still sensitive to initial policy; intermediate complexity \\ \hline \multirow{3}{*}{Value-based DRL (DQN and variants)} & \multirow{3}{*}{[28, 48, 49]} & Effective for discrete action learning; well understood & Not suitable for continuous control; limited fine-grained behavior \\ \hline Deterministic Actor-Critic (DDPG, TD3) & \multirow{3}{*}{[32, 37, 64]} & Supports continuous action; efficient policy learning & \(Q\)-value overestimation risk; limited multi-modality \\ \hline \multirow{3}{*}{Stochastic Actor-Critic (PPO, SAC)} & \multirow{3}{*}{[30, 68, 69]} & Better stability and exploration; robust to noise & PPO is on-policy and sample-inefficient; SAC may still overestimate \\ \hline Regularized Actor-Critic (CrossQ, TQC) & \multirow{3}{*}{[35, 36]} & Conservative updates; reduced \(Q\)-value noise; robust convergence & Relatively new; limited use in autonomous driving \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of DRL approaches in autonomous driving. in Figure 2, producing segmentation outputs that encode essential environmental features such as road lines, road surfaces, and sidewalks. In addition, sensor data that describe the current state of the vehicle--namely RGB camera feeds, GPS signals, and IMU measurements--are used to estimate the vehicle's position and heading angle. These perception-related observations, denoted by \(\mathcal{R}\), are concatenated and forwarded to the DRL module. Furthermore, the DRL module receives navigation-related information \(\mathcal{N}\) from the CARLA simulator, which includes the coordinates of the next 15 waypoints \(n_{up}\) and the state of traffic lights \(n_{tl}\) within an approximate 18 m forward range. Here, the 15 forward waypoint coordinates \(n_{up}\) are initially determined by Global Path Planning (GPP) using the \(A^{*}\) algorithm [73; 74]. The 15 waypoints are selected based on their proximity to the vehicle's current position, while also considering road curvature, lane geometry, and junction structures. Lastly, the DRL module is also provided with the current control information of the vehicle \(\mathcal{C}\), including its steering angle \(c_{st}\in[-1.0,1.0]\), throttle \(c_{th}\in[0.0,1.0]\), brake \(c_{br}\in[-1.0,0.0]\), and velocity \(c_{sp}\).