## 6 Simulation Result  
Each algorithm was trained for 1 million steps, and performance varied according to the structural characteristics and learning paradigms of each method. Figure 7 visualizes the progression of six performance metrics--the Episode Reward Mean, Step Reward Mean, Route Completion Rate, Success Rate, Centerline Deviation Mean, and Speed Mean--across training checkpoints from 200 k to 1 M steps. In particular, the Step Reward Mean graph represents the average reward received per step. As observed in Figure 7, there is no strict proportional relationship between the Episode Reward Mean and Step Reward Mean. This discrepancy arises because the Episode Reward denotes the total cumulative reward over an entire driving episode, while the Step Reward Mean reflects the average reward per time step. Consequently, a high Episode Reward Mean does not necessarily imply a high Step Reward Mean, particularly in longer or partially successful episodes. Therefore, the joint interpretation of both metrics enables a more precise evaluation of each algorithm's performance in terms of both its overall driving success and per-step reward efficiency. In Sections 6.1-6.3, the model that achieved the best performance within the 200 K-1 M steps of learning for each algorithm is selected and compared across key evaluation metrics in tabular format. Furthermore, Section 6.4 provides a quantitative analysis of penalty occurrence rates, as defined in Section 3.4, for each algorithm and town. To ensure the statistical reliability of the penalty evaluation, five additional trials were conducted per route scenario. The average penalty ratio was then computed based on the aggregated results. TQC exhibited the most outstanding performance across all evaluation metrics, achieving top values in Episode Reward Mean, Step Reward Mean, Route Completion, Success Rate, and Centerline Deviation Mean after 1 million training steps. SAC initially demonstrated a learning trajectory similar to TD3 but accelerated its performance improvements in the later stages and ultimately converged stably, securing the second-best performance after TQC. Both TQC and SAC are grounded in the Maximum Entropy reinforcement learning framework, which incorporates entropy terms into its policy and critic networks. This formulation promotes extensive exploration during the early stages of training by maintaining high entropy, while gradually reducing entropy to enable the precise estimation of the optimal action distribution as training progresses. TQC further enhances this approach by adopting Quantile-based Distributional Learning, which learns the return distribution for each state rather than a single \(Q\)-value. This enables more refined value estimation in complex environments. Although this additional optimization of return distributions occasionally caused TQC to underperform compared to SAC in certain intermediate metrics, it ultimately outperformed SAC at the final 1 M steps of learning. TD3, developed to mitigate the \(Q\)-value overestimation issue inherent in DDPG using a Twin \(Q\)-Network structure, consistently outperformed DDPG throughout training. However, its performance remained at a mid-level relative to TQC and SAC. This suggests that TD3's deterministic policy is less capable of effectively covering complex state-action spaces compared to the stochastic policies employed by SAC and TQC. \begin{table}
\begin{tabular}{l c c c c c c} \hline  & **DDPG** & **TD3** & **SAC** & **TQC** & **CrossQ** & **PPO** \\ \hline Policy size (MB) & 0.62 & 0.62 & 0.62 & 0.62 & 0.63 & 0.62 \\ Target policy size (MB) & 0.62 & 0.62 & 0.0 & 0.0 & 0.0 & 0.0 \\ Critic size (MB) & 0.62 & 1.24 & 1.24 & 1.30 & 1.25 & 0.62 \\ Target critic size (MB) & 0.62 & 1.24 & 1.24 & 1.30 & 0.0 & 0.0 \\ \hline Total size (MB) & 2.48 & 3.72 & 3.10 & 3.21 & 1.88 & 1.24 \\ Latency (ms) & 1.05 & 1.29 & 1.46 & 1.80 & 1.41 & 1.10 \\ \hline \end{tabular}
\end{table} 
Table 7: Model size and inference latency. PPO also yielded a mid-range performance comparable to that of TD3.