# Enhancing Autonomous Driving Navigation Using Soft Actor-Critic
## 4 Methodology
### Training Models
This technique involves deriving action samples from the stochastic policy by evaluating a deterministic function as follows:  
\[a_{t}=\text{tanh}(\mu_{\theta}(s_{t})+\sigma_{\theta}(s_{t})\odot\xi),\quad \xi\sim\mathcal{N}(0,I) \tag{13}\]  
Here, \(\xi\) indicates independent Gaussian noise, while the application of hyperbolic tangent ensures that actions are bounded within a finite range. Our framework utilizes two Q-networks, the critic and target critic networks. Those networks are necessary to ensure the stability of the learning process. The network's weights are updated through the Q-network to ensure the agent's adaptation to the environment over time. Similar to the DQN approach, the tuple \((s_{t},d_{t},a_{t},r_{t},s_{i+1},d_{t+1})\) is stored in a replay buffer. During training, the model randomly selects batches of data from this replay buffer. This allows the agent to learn from past experiences, breaking correlations in the data and promoting more robust and efficient learning. To better illustrate the implementation of our approach, we provide Algorithm 1. The other settings of our models are represented in Table 2:  
```
1:Initialize actor network parameters \(\phi\) and critic network parameters \(\theta\). 2:Initialize the entropy coefficient \(\alpha\). 3:Define the batch size \(N\) and create an empty replay buffer \(\mathcal{D}\). 4:Set target network parameters equal to the critic parameters: \(\theta_{\text{target}}\leftarrow\theta\). 5:for each episode \(e=1\) to \(E\)do
6: Initialize images state \(s_{t}\sim CarlaEnv\). 7: Initialize destination vector \(\text{dest}_{t}\sim CarlaEnv\). 8:for each step \(t\) from \(1\) to \(N_{\text{steps}}\)do
9: Select an action \(a_{t}\sim\pi_{\phi}(a_{t}|(s_{t},\text{dest}_{t}))\)
10:Interact with the environment:
11: Observe the next state \(s_{t+1}\), reward \(r_{t}\), and next destination \(\text{dest}_{t+1}\)
12:Store the transition in the replay buffer:
13:\(\mathcal{D}\leftarrow\mathcal{D}\cup(s_{t},\text{dest}_{t},a_{t},r_{t},s_{t+1 },\text{dest}_{t+1})\)
14:if time to update the critic network then
15: Sample a batch of transitions \((s^{i}_{t},\text{dest}^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1},\text{dest}^{i}_ {t+1})^{N}_{i=1}\) from \(\mathcal{D}\)
16: Compute the loss for the critic \(\mathcal{L}_{\text{critic}}(\theta)\) using Equations (6) and (8)
17: Update the critic network parameters \(\theta\)
18:endif
19:if time to update the actor network then
20: Sample a batch of transitions \((s^{i}_{t},\text{dest}^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1},\text{dest}^{i}_ {t+1})^{N}_{i=1}\) from \(\mathcal{D}\)
21: Compute the loss for the actor \(\mathcal{L}_{\text{actor}}(\phi)\) using Equation (9)
22: Update the actor network parameters \(\phi\)
23:if automatic entropy tuning is enabled then
24: Adjust the entropy coefficient \(\alpha\)
25:endif
26:endif
27:if time to update the target network then
28: Update the target network parameters: \(\theta_{\text{target}}\leftarrow\theta\)
29:endif
30:endfor
31:endfor
```  
**Algorithm 1** Soft Actor-Critic with Destination Vector \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Parameter** & **DQN** & **SAC** & **PPO** \\ \hline Learning rate & 0.001 & - & - \\ Learning rate (actor) & - & \(3\times 10^{-4}\) & \(3\times 10^{-4}\) \\ Learning rate (critic) & - & \(3\times 10^{-4}\) & \(10^{-3}\) \\ Number of Episodes & 2000 & 2000 & 2000 \\ Batch size & 128 & 64 & - \\ Clipping \(\epsilon\) & - & - & 0.1 \\ Num. epochs & - & - & 4 \\ Replay Buffer & \(10^{5}\) & \(2\times 10^{5}\) & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The parameter settings of each model.