# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 4 Methodology--CILRLv3
### Fused Actor and Critic Networks (Side-Tuning)  
To improve time efficiency, while also preventing CF from occurring, we replace the heavy CIL++ actor and critic clones with lightweight fused networks. Each fused network consists of two main modules, as illustrated in Figure 5: the Pretrained Policy Branch (PPB) and the Encoder Branch (EB).  
The PPB is composed of the CIL++ pretrained model with its Action Prediction Network removed. This branch processes the current state and outputs a fixed 12,800-dimensional Transformer encoder embedding. Importantly, the PPB is completely frozen during fine  
Figure 4: Illustration of CILRLv3 architecture using Twin-Q Soft Actor–Critic (T-SAC) algorithm, a SAC variation that uses the Actor and the two Critic networks. At each timestep \(t\), the fused Actor policy \(\pi_{\phi}\) generates an action \(a_{t}\) and steps into the environment, while the fused critic networks evaluate that action, by calculating the Q values: \(Q(s_{t},a_{t}),Q(s_{t},a_{t})\).  
tuning to preserve all previously acquired expert knowledge. On the other hand, the EB consists of a frozen ResNet-34 (used as a visual feature extractor for both actor and critic) followed by a trainable Multi-Layer Perceptron (MLP) encoder. The ResNet-34 processes each input image, extracts and flattens the feature maps, and concatenates them into a single vector. This vector is then passed through the MLP encoder, which consists of two hidden layers with ReLU activations. The final layer of the EB produces a 12,800-dimensional embedding, matching the output size of the PPB. The purpose of the EB is to acquire and encode new knowledge relevant to the current fine-tuning task.  
The outputs of the PPB and EB are combined using a learnable fusion parameter (\(\alpha_{\phi}\)) for the actor and \(\alpha_{\theta}\) for the critics. For the actor, \(\alpha_{\phi}\) is updated by gradient ascent, which also updates the parameters of the actor network, while for the critic, \(\alpha_{\theta}\) is updated by gradient descent along with the rest of the critic parameters. The fusion is performed as follows:  
\[f_{in}=\alpha PPB(s_{t})+(1-a)EB(s_{t}) \tag{10}\]  
where \(\alpha\in[0,1]\) is a learnable parameter, allowing the network to dynamically balance the retention of prior skills from the PPB with the integration of new knowledge from the EB. The parameter \(\alpha\) is initialized randomly within the range \((0,1)\). As training progresses, if \(\alpha\) is updated toward 1.0, the network increasingly relies on prior expert knowledge from the PPB; conversely, if \(\alpha\) moves toward 0.0, it indicates a greater reliance on new information learned by the EB. If \(\alpha\) converges to a value near 0.5, the network effectively combines both prior and newly acquired knowledge equally [15].  
The fused feature \(f_{in}\) passes through a final linear layer, which depends on the network type. For the actor, we use a Linear layer of 2 units, which is the same as the number of required actions (for steering and acceleration). On the other hand, the critic heads are Linear layers, consisting of a single unit that represents the Q-value.  
Both the actor and critic fused networks share the same PPB and ResNet-34 outputs, reducing memory requirements and computational overhead, thereby making training both faster and more resource-efficient. Moreover, the proposed architecture allows the agent to retain robust prior skills while efficiently adapting to new environments and tasks.