# Enhancing Autonomous Driving Navigation Using Soft Actor-Critic
## 4 Methodology
### Soft Actor-Critic  
Soft Actor-Critic (SAC) represents a progressed approach within advanced deep reinforcement learning, characterized by its policy-based and model-free nature. Compared to traditional RL methods, SAC uses the concept of soft strategy optimization. Rather than focusing on maximizing rewards, SAC integrates the notion of entropy into its optimization strategy to strike a balance between exploration and exploitation. This approach allows the agent to handle tasks with a degree of randomness by enhancing adaptability and flexibility. Consequently, SAC excels at tackling continuous action spaces, leading to notable enhancements in learning efficiency and stability [34]. Figure 1: Overview of our RL framework: We receive the state \(s_{t}\) from CARLA and feed it to a CNN model to extract the main features. These features are then concatenated with a destination vector. The SAC algorithm utilizes the concatenated vector from the previous step to make driving decisions, taking action \(a_{t}\), and subsequently receiving the reward \(r_{t}\). Integrating maximum policy entropy into the SAC algorithm aims to encourage the agent's exploration of its environment. This allows the agent to achieve optimal actions, improving the model's capacity for generalization. The primary objective of the RL algorithm is to learn a policy that maximizes the expected future accumulated return \(\pi^{*}=\arg\max_{\pi}V^{\pi}(s)\) and \(V^{\pi}\), which is computed by the Equation (1). In addition to the fundamental objectives outlined earlier, the maximum entropy reinforcement learning approach necessitates maximizing the action entropy for every policy output as follows:  
\[\pi^{*}=\arg\max_{\pi}\sum_{t=0}^{T}\left[\gamma^{t}\cdot r_{t}+\alpha\mathcal{ H}(\pi(\cdot|s_{t}))\right] \tag{3}\]  
The definition of every policy entropy \(\mathcal{H}\) is as follows:  
\[\mathcal{H}(\pi(\cdot|s)=-\int_{a\in\mathcal{A}}\pi(a|s)\log(\pi(a|s))da=\mathbb{ E}_{a\sim\pi(\cdot|s)}[-\log(\pi(a|s))] \tag{4}\]  
This goal is to enhance how effectively the policy explores by boosting both the anticipated future rewards and policy unpredictability. The temperature parameter, represented by \(\alpha\), determines the significance of the unpredictability compared to the rewards. Maximum entropy RL methodically converges toward conventional RL as \(\alpha\to 0\). According to the Bellman Equation (2), we can obtain the updated soft Bellman equation:  
\[Q^{\pi}(s_{t},a_{t})=r_{t}+\gamma\cdot\mathbb{E}_{s_{t+1}\sim\mathcal{P},a_{t+ 1}\sim\pi}[Q^{\pi}(s_{t+1},a_{t+1})-\alpha\log(\pi(a_{t+1}|s_{t+1}))] \tag{5}\]  
One common technique that is extensively utilized in the SAC algorithm is double Q-networks to address the overestimation issue. Hence, the parameters of the critic network of SAC are updated by minimizing the mean Bellman-squared error (MBSE) loss function:  
\[\mathcal{L}_{\text{critic}}(\theta_{i})=\mathbb{E}_{s_{t}\sim\mathcal{P},a_{t }\sim\pi}\Big{\|}Q^{\pi}_{\theta_{i}}(s_{t},a_{t})-\left(r_{t}+\gamma\cdot \hat{Q}^{\pi}\right)\Big{\|}_{2} \tag{6}\]  
Where \(\hat{Q}^{\pi}\) is the state-action value of the next step from double target Q-networks and computed as follows:  
\[\hat{Q}^{\pi}=\mathbb{E}_{s_{t+1}\sim\mathcal{P},a_{t+1}\sim\pi}\bigg{(}\min_ {i=1,2}Q_{\text{target},i}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}|s_{t+1}) \bigg{)} \tag{7}\]  
Combining these equations, the total critic loss is the sum of the individual losses for each Q-network:  
\[\mathcal{L}_{\text{critic}}(\theta)=\mathcal{L}_{\text{critic},1}(\theta_{1} )+\mathcal{L}_{\text{critic},2}(\theta_{2}) \tag{8}\]  
The loss function for the actor in the SAC model is designed to maximize the expected return while integrating an entropy term to encourage exploration. The actor aims to learn a policy \(\pi_{\phi}\) that outputs actions that maximize the expected Q-value as estimated by the critics, adjusted by the entropy of the policy.