# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 1 Introduction  
Inspired by the breakthroughs in Natural Language Processing enabled by large Transformer models [1], Computer Vision has also adopted Transformer-based architectures. These architectures have been extensively applied to a variety of image recognition tasks [2], demonstrating exceptional ability in generalizing patterns in large datasets [3]. One of the more complex challenges in the CV domain is the development of Visual-Based End-To-End (VBETE) navigation systems [4], which employ multiple RGB cameras, interpret visual information from the surroundings, and navigate the host vehicle (Figure 1). These systems have gained popularity due to their simpler implementation and reduced development costs, compared with traditional multi-layer decision-making modules. To develop efficient VBETE systems, researchers have combined advanced Deep Learning models with Imitation Learning (IL), a process that involves training a model to imitate expert driving behaviors via demonstrations. Although effective in certain driving scenarios, the performance of IL agents is limited,due to their reliance on expert demonstrations [5]. Moreover, insufficient experience in certain scenarios can lead to bad decisions and accidents. Deep Reinforcement Learning (DRL) methods have been proposed to tackle the generalization issues [4], where agents are trained through interaction and exploration with their environments. DRL agents are not explicitly told how to act by experts, but their actions are evaluated by a crafted reward function instead. Despite the benefits, DRL is known tobe less data-efficient and more time-consuming than IL, while often struggling to optimize multiple behaviors simultaneously. To address these issues, [PERSON] et al. [6] have proposed CIRL, a policy model which was initially trained in several navigation tasks using IL, and was subsequently fine-tuned using DRL. This strategy did not exhibit promising results, due to the Catastrophic Forgetting (CF) nature of fine-tuning with DRL, which is a significant challenge when fine-tuning a pretrained policy model on new tasks. More specifically, as the policy is updated to optimize performance on new environments or objectives, the network often rapidly overwrites or forgets the representations and skills it previously acquired, leading to a substantial loss of performance on earlier tasks. This phenomenon occurs because gradient-based optimization does not inherently preserve prior knowledge, causing the model to \"forget\" how to solve tasks it was initially trained on via IL. Addressing this issue is crucial for enabling robust transfer learning, continual learning, and generalization in sequential decision-making systems. Later, [PERSON] et al. [7] introduced CIRL++, an improved version of CIRL. In that work, the authors used a pretrained model, named CIL++, as the initial policy of a DRL agent and designed a reward function targeting its weaknesses. To mitigate CF, the authors applied a KL-Divergence penalty during training, which prevented the new policy from deviating significantly from the original policy, and thus ensured that the new policy stayed close to the original IL-learned behavior. However, as the authors highlighted, this approach requires significant training time and computational resources before the agent fully benefits from it. In this paper, we introduce CIRLv3, an improvement of CIRL++ that integrates multiple Deep Reinforcement Learning techniques to mitigate CF efficiently, while reducing the overall training time. To achieve this, we combine (a) a hybrid learning algorithm that leverages both offline and online experiences with (b) a side-learning method that combines a pretrained policy with a lightweight side network. Our contributions can be summarized as follows:  
* Develop robust, VBETE systems that generalize well in navigation tasks. * Design a fine-tuning methodology immune to CF, without significantly increasing the training requirements. The remainder of this paper is organized as follows.