# Enhancing Autonomous Driving Navigation Using Soft Actor-Critic
## 4 Methodology
### Training Models  
**DQN model**: Our model training methodology aims to effectively train a neural network for decision-making in dynamic environments. Initially, the neural network is initialized with random weights. The agent starts taking random actions to explore the environment, and during episodes, it exploits the knowledge learned during exploration. Actions taken by the agent are stored in a replay buffer with their corresponding states. \(s_{t}\), destination vector \(d_{t}\), rewards \(r_{t}\), next destination vector \(d_{t+1}\), and next states \(s_{t+1}\), forming tuples of the form \((s_{t},d_{t},a_{t},r_{t},s_{t+1},d_{t+1})\). During the training phase, the model selects a random batch, typically consisting of 128 samples, from the replay buffer. This batch is then fed through the policy network, which predicts Q-values associated with each action. Subsequently, to update the Q-value according to Equation (2), a second pass through the network is necessary to compute the maximum Q-value for the next action, \(a_{t+1}\). This additional pass is facilitated by what is known as the target network. Following this, the loss between the predicted Q-value and the output from the target network is computed using the mean square error (MSE) metric. The ADAM optimizer [36] is then utilized to minimize this loss, enabling the updating of weights through backpropagation. This iterative process allows the neural network to progressively refine its Q-value predictions, thereby improving the agent's decision-making capabilities throughout the training process. By iteratively adjusting the network's weights based on experiences gathered from the replay buffer, the model adapts and learns to make more informed decisions in dynamic environments. **PPO model**: The training process for the PPO algorithm follows several key steps. Initially, two neural networks are initialized: an actor and a critic. The actor-network uses a tanh activation function in the output layer to generate the probability distribution over actions, while the critic network, which estimates the state value, employs a tanh activation function in its output layer. The agent interacts with the environment, then the samples are collected according to the current policy. The advantages are computed and returned for each time step in the collected trajectories. The PPO objective is defined as follows:  
\[L(\theta)=\mathbb{E}_{t}\bigg{[}\text{min}\bigg{(}\frac{\pi_{\theta}(a_{t}|s_{ t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{t})}A_{t},\text{clip}\bigg{(}\frac{\pi_{ \theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{t})},1-\epsilon,1+ \epsilon\bigg{)}A_{t}\bigg{)}\bigg{]}\]  
where \(A_{t}\) the advantage function from the finite-horizon estimation of \(T\) steps is given as:  
\[A_{t}=\mu_{t}+(\gamma\lambda)\mu_{t+1}+(\gamma\lambda)^{2}\mu_{t+2}+...+( \gamma\lambda)^{T-t+1}\mu_{T-1}\]  
and \(\mu_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})\)  
The objective function \(L(\theta)\) is then used to update the parameters of both the actor and critic networks. This objective ensures stable policy updates by preventing large changes in the policy. By iterating through these steps, the policy and value function are gradually refined in order to enhance the agent's performance in the environment. **SAC model**: The agent interacts with the environment according to its probabilistic policy, denoted as \(\pi_{\theta}(\cdot|s_{t})\). This policy allows a balance between exploitation and exploration. In our implementation, we use Gaussian policy, wherein the policy network generates two parameters that define the mean and standard deviation of a Gaussian distribution, thus yielding actions represented as \(a_{t}\sim\mathcal{N}(\mu_{\theta}(s_{t}),\sigma_{\theta}(s_{t}))\). To preserve the differentiability of the stochastic policy, we employ the reparameterization trick.