# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 5 Experiments and Results
### Results  
We trained each algorithm for a total of \(1.2\) billion environment steps and report the corresponding average rewards in Figure 8. It can be observed that the CILRLv3 outperforms all the other variants, achieving the highest average return per training step. This result confirms that our dual-buffer, Side-Tuning approach effectively mitigates Catas  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Entropy coefficient (\(\alpha\)) & 0.2 \\ Actor learning rate & \(3\times 10^{-4}\) \\ Critic learning rate & \(1\times 10^{-3}\) \\ Discount factor (\(\gamma\)) & 0.99 \\ Polyak coefficient (\(\tau\)) & 0.005 \\ Mini-batch size & 256 \\ Replay Buffer Capacity (S) & \(10^{6}\) \\ Epochs (e) & 1 \\ Optimizer & Adam \\ Beta (\(\beta\)) & 0.5 \\ \(p\) & 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 2: T-SAC configuration.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Hidden Layers & 2 \\ Units per Hidden Layer & 256 \\ Activation & ReLU \\ \(\beta\) & 0.5 \\ \(p\) & 0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 3: CILRLv3 configuration table. The hyperparameters \(\beta\) and \(p\) are set to \(0.5\) and \(0.3\), respectively, according to the guidelines of the original papers.  
trophic Forgetting, allowing the pretrained policy to continue improving via exploration even after hundreds of millions of steps.  
The CILRLv3-PPG variant, which substitutes PPG for T-SAC, also attains competitive rewards but exhibits greater volatility and slower convergence due to its alternating actor-critic training phases. This suggests that PPG does not utilize both expert demonstrations and online experiences as efficiently. Similarly, the original CILRL++ with KL-Divergence regularization maintains a stable learning curve, steadily optimizing the reward while preserving expert behavior, but requires a higher per-step computational cost and memory space, since the full trainable CIL++ model must be loaded into both actor and critic.  
In contrast, the CILRLv3-NoST and CILRLv3-NoBC approaches, which, respectively, remove Side-Tuning or behavioral cloning, although they do gain some positive returns, fail to match the KLD loss baseline. This underscores that neither technique alone suffices to simultaneously optimize the reward and retain prior knowledge as robustly. However, when combined in CILRLv3, they yield the best average returns. Finally, both the original CILRL and the T-SAC variant quickly start to forget their prior skills, resulting in the lowest performance.  
Continuing our evaluation, we benchmarked the original CIL++, CILRL++, and our best fine-tuning method, CILRLv3, on the CARLA Leaderboard (Table 4). Across all key metrics, CILRLv3 consistently outperforms both the baseline method and its predecessor. In particular, CILRLv3 achieves higher overall driving scores and completes a greater fraction of the prescribed routes than CILRL++, while CIL++ exhibits the weakest performance, especially in Driving Score and Route Completion. Importantly, all three agents demonstrate the ability to avoid collisions with other vehicles across every test scenario, which is an important safety metric for autonomous navigation systems.  
Figure 8: The performance of the evaluated algorithms after 1.2 billion training steps. The vertical axis displays the average returns, while the horizontal axis presents the current evaluation step.