# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 5 Experiments and Results
### Hyperparameter Tuning  
The hyperparameters of PPG have a large combinatorial space, making an exhaustive search infeasible. Although automated methods--such as the hybrid evolutionary-Bayesian optimisation employed by [19] for deep spatiotemporal prediction models, can systematically explore such spaces, they typically require substantial training time and computing resources, which were beyond our scope. Consequently, we manually fine-tuned only the most sensitive PPG hyperparameters through a small number of targeted trials. More specifically, we set the image sizes to 300 \(\times\) 300 pixels in order to achieve a balance between capturing sufficient visual features and accelerating the training process. Moreover, we set the clipping factor (\(\epsilon\)) of PPG to 0.2, chosen from a range of [0.1, 0.3], and the KL-Divergence penalty factor to 0.1. The discount factor (\(\gamma\)) is set to 0.99, which is a typical value for DRL problems. PPG was trained on mini-batches of 128 samples, with a total batch size of 6144, due to low computational resources.  
The actor and critic networks are trained for 1 epoch per batch using the Adam optimizer, as suggested in the original work of PPG. We fine-tuned the learning rate of Adam, which is set to 0.0004, chosen from a range of \([1\times 10^{-4},1\times 10^{-3}]\). Moreover, we disable entropy regularization, as we found out that it has almost no impact on PPG's policy updates and that PPG already provides the agent with enough exploration. Furthermore, we use the pretrained CIL++ model as the actor and the same architecture for the critic network. Finally, we set the reward function weights according to [PERSON] et al. [7], as shown in Table 1.  
\begin{table}
\begin{tabular}{l l} \hline
**Parameter** & **Value** \\ \hline Clipping factor (\(\epsilon\)) & 0.2 \\ KL-Divergence penalty factor (\(p\)) & 0.1 \\ Discount factor (\(\gamma\)) & 0.99 \\ Mini-batch size & 24 \\ Train batch size & 4048 \\ Epochs (e) & 1 \\ Discount Factor (\(\gamma\)) & 0.99 \\ Optimizer & Adam \\ Learning Rate (\(lr\)) & \(4\times 10^{-4}\) \\ Actor network & CIL++ \\ Critic network & CIL++ \\ \(c_{success}\) & 1.0 \\ \(c_{fail}\) & \(-1.0\) \\ \(c_{lat}\) & \(-0.02\) \\ \(c_{sped}\) & 0.05 \\ \(c_{slope}\) & 2.0 \\ \(c_{angle}\) & \(-0.1\) \\ \(c_{max,peed}\) & 30.0 \\ \hline \end{tabular}
\end{table}
Table 1: CILRL++ configuration.  
The hyperparameters of Twin Soft Actor-Critic (T-SAC) also exhibit a large combinatorial space, so we manually tuned the most sensitive parameters. More specifically, we fix the entropy coefficient \(\alpha\) at \(0.2\), chosen from an initial range of \([0.1,0.5]\), to regulate exploration without destabilizing learning. The actor and critic learning rates were set to \(3\times 10^{-4}\) and \(1\times 10^{-3}\), respectively, each drawn for the range \([1\times 10^{-4},1\times 10^{-3}]\). The Polyak coefficient \(\tau\) was set to \(3\times 10^{-3}\) after sampling from \([0.001,0.01]\) to ensure smooth target network updates. Finally, we set the replay buffer capacity to \(S=1\times 10^{6}\) and the batch size to \(256\) samples. Similar to PPG, both actor and critic networks were trained for one epoch per aggregate batch using the Adam optimizer with the tuned learning rates. We summarize the hyperparameter configuration of T-SAC in Table 2.