# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 2 Related Work  
For many years, researchers have struggled to develop vision-based end-to-end navigation systems relying solely on camera inputs. Recently, [PERSON] et al. [8] introduced CIL++, a Transformer-based architecture trained through Imitation Learning on a large dataset, consisting of (frames, expert trajectory) pairs. CIL++ architecture employs a ResNet-34 network as the backbone to extract key features from three images captured from the left, front, and right perspectives of the vehicle. These features are then processed by a Transformer layer serving as a mid-level attention mechanism, before being passed to a fully connected neural network that generates the actions (Figure 2). Although effective, this architecture exhibits significant weaknesses, including limited generalization ability and making the vehicle oscillate around the center of the lane. Recently, [PERSON] et al. [9] introduced an enhanced variant of CIL++, termed CILv3D. In this work, the authors proposed a sequential-based approach in which CIL++ processes three video streams--one from each camera--using an advanced backbone network called Uniformer. The resulting embeddings are then fed into a Transformer encoder. This design significantly improved the vehicle control pipeline by enabling the controller to capture motion and distance cues from video sequences. However, CILv3D was trained entirely from scratch, without leveraging the skills previously acquired by CIL++. Several recent studies have explored Deep Reinforcement Learning (DRL) pipelines for vision-based autonomous driving, but most share a common limitation: the extraordinary amount of data and simulation time required to learn even a modest set of driving skills. One such example is the work by [PERSON] et al. [10], which benchmarked several algorithms, including DDPG, SAC, TD3, PPO, TQC, and CrossQ in CARLA. Working in a closed-loop setting, the authors crafted scenario-specific reward functions and evaluated the performance of each algorithm. Despite the promising results of their benchmarks, each agent was trained for millions of environment steps before it could reliably traverse a handful of intersection, traffic light, and roundabout layouts. [PERSON] et al. [6] introduced a two-stage approach to improve the generalization of IL-based navigation agents. First, they trained a policy model in a supervised fashion using expert demonstrations. Then, they fine-tuned this pretrained model with a DRL algorithm, allowing the agent to retain core driving skills learned from demonstrations while acquiring new behaviors through exploration. Although this method showed promising results, it  
Figure 2: Illustration of CILRL++ architecture (adopted by [7]). CILRL++ fine-tunes the pretrained CIL++ policy using a DRL pipeline, composed of the PPG algorithm, a crafted reward function, and an additional KLD loss term. achieved lower performance in comparison with CIL++, primarily due to CF during the fine-tuning stage. [PERSON] et al. [5] proposed an effective solution for the CF by incorporating a KL-Divergence penalty during DRL fine-tuning Equation (1). This penalty constrains the updated policy's action distribution to remain close to the initial pretrained policy, ensuring steady performance improvements without erasing previously learned skills. The KL-Divergence (KLD) loss is weighted by a regularization coefficient \(\lambda\in[0.0,1.0]\), which initially takes a high value and decays over time. This term forces the policy to maintain updates within a \"trust region\" during the early stages of the fine-tuning, stabilizing the learning process of new skills. \[L_{total}=L_{RL}+\lambda L_{KD}(\pi_{new}||\pi_{old}) \tag{1}\]  
Another approach combines behavioral cloning (BC) with Deep Reinforcement Learning (DRL), enabling both online and offline training. For example, BC-SAC [11] fine-tunes a policy using the Soft Actor-Critic (SAC) algorithm [12], where the update loss, \(L_{SAC}\), is combined with an additional Behavior Cloning loss term, \(L_{BC}\).