# Optimizing Pretrained Autonomous Driving Models Using Deep Reinforcement Learning
## 3 Background
### Fine-Tuning Module (FTM)  
The fine-tuning process of CIL++ is formulated as a Markov Decision Problem (MDP), with a 5-tuple \(\text{M}=(S,A,p,\gamma,R)\), where \(S\) and \(A\) denote the state space and action space, respectively, \(p\) is the environment dynamics function, \(\gamma\) is the discount factor, and \(R\) is the reward function [17]. At each timestep \(t\), the CIL++, which is the policy model of a DRL agent, generates an action \(a_{t}\) from a state \(s_{t}\), receives a reward \(R_{t+1}\), and transits into the next state \(s_{t+1}\). The goal of the DRL agent is to learn an optimal policy \(\pi_{\theta}\) that maximizes the discounted cumulative rewards \(G_{t}\), as described in Equation (3).  
\[G_{t}=R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\ldots=\sum_{k=0}^{T}\gamma^{l} R_{t+k+1} \tag{3}\]  
**State Space \((S)\)**. The state \(s_{t}\) represents the observed environment through the camera inputs and vehicle controls, defined as a tuple of \((I_{o,l}{}^{WxHx3},v_{t})\). Since the camera inputs are RGB images, each pixel can have values from 0 to 255. We normalize these values to a range of 0 to 1 by dividing by 255.0.  
**Action Space \((A)\)**. The action \(a_{t}\) denotes the vehicle's control in the next timestep \(t+1\), defined as a tuple of \((th_{t+1},st_{t+1})\), which represent the throttle and steering values of the next timestep. Throttle control values range from \(-1.0\) to \(1.0\), with negative values indicating deceleration and positive values indicating acceleration. Steering control values also range from \(-1.0\) to \(1.0\), with negative values indicating a left steer, positive values indicating a right steer, and 0 indicating a straight path.  
**Reward Function**\((R)\). At each timestep \(t\), the agent is rewarded based on the episode state. If the route of the episode is terminated, then the reward can be described as  
\[R_{t+1}=\begin{cases}c_{success}&\text{Route is completed},\\ -c_{fail}\cdot(1+speed_{t+1})&\text{Route not completed}\end{cases} \tag{4}\]  
where \(c_{success}\) and \(c_{fail}\) are small constants in range \([0.0,1.0]\), and \(speed_{t+1}\) is the vehicle's speed right after the episode termination signal.  
On the other hand, if the episode has not terminated, then the reward function can be expressed as the sum of smaller rewards, defined as \(R_{t+1}=r_{lat_{t+1}}+r_{speed_{t+1}}+r_{angle_{t+1}}\) with each reward focusing on a specific weakness of the CIL++ approach. More specifically:  
* \(r_{lat_{t+1}}\): Responsible for the lateral control of the vehicle, which keeps it inside the boundaries of the target lane and can be defined as \[r_{lat_{t+1}}=c_{lat}\cdot distance_{t+1}\] (5) where \(c_{lat}\) is a constant that controls the magnitude of the reward and \(distance_{t+1}\) expresses the distance of the vehicle from the center of the lane.
* \(r_{speed_{t+1}}\): Responsible for the longitudinal control of the vehicle, providing the agent with proper information about the desired speed of the vehicle. More specifically, if there are no obstacles, as well as traffic signs and traffic lights in front of the vehicle, the desired speed is the maximum speed. Otherwise, the desired speed decreases relative to the distance of these obstacles as follows: \[r_{speed+1}=c_{speed}\frac{1-c_{slope}\cdot(speed_{t+1}-DS_{t+1})}{c_{max\_ speed}}\] (6) where \(c_{speed}\) is a constant which weights the speed reward, \(c_{slope}\) controls the range of the reward, \(DS_{t+1}\) is the desired speed of the agent, and \(c_{max\_speed}\) is the maximum allowed speed of the agent.
* \(r_{angle_{t+1}}\): The purpose of this reward is to discourage the agent from oscillating around the route's center line. It is defined as \[r_{angle_{t+1}}=c_{angle}\cdot DA_{t+1}\] (7) where \(c_{angle}\) is a constant which controls the magnitude of the reward and \(DA_{t+1}\) is the absolute distance in radians between the direction of the vehicle and the route.