# Enhancing Autonomous Driving Navigation Using Soft Actor-Critic
## 3 Deep Reinforcement Learning: An Overview  
In recent years, DRL has made significant progress, becoming widely embraced across various fields, particularly robotics. Previous studies have highlighted the effectiveness of the model-free DRL approach in enabling autonomous acquisition of motion control strategies. This recognition has led to substantial efforts to advance methodologies based on DRL. At its core, a reinforcement learning algorithm operates based on the principles of a Markov decision process (MDP), where the system's current state depends solely on the preceding state, regardless of the specific path taken. MDPs are typically represented as a tuple \((\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P})\), where \(\mathcal{S}\) denotes the state space of the environment, \(\mathcal{A}\) represents the available actions, \(\mathcal{R}\) indicates the rewards or feedback received from taking actions at each time step, and \(\mathcal{P}\) characterizes the probability distribution governing state transitions [33].  
At each time step, the RL agent observes the state \(s_{t}\) from the set \(\mathcal{S}\) and takes action \(a_{t}\) from the set \(\mathcal{A}\), subsequently receiving an immediate reward \(r_{t}=\mathcal{R}(s_{t},a_{t})\) where \(\mathcal{R}\) is a function mapping states and actions to real numbers: \(\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\). Additionally, the agent transitions to the next state \(s_{t+1}\in\mathcal{S}\) based on the transition probability \(\mathcal{P}(s_{t+1}|s_{t},a_{t})\), which is a function mapping state-action pairs to probabilities in the range \([0,1]\): \(\mathcal{S}\times\mathcal{A}\rightarrow[0,1]\).  
In RL, the agent typically determines its action through a policy denoted as \(a_{t}\sim\pi(\cdot|s_{t})\) wherein \(\pi\) is considered a probability distribution reflecting the agent's beliefs regarding its decisions at each time step within its environment. The objective for the RL agent is to maximize the cumulative reward or total return as it progresses from an initial state \(s\). This cumulative reward, denoted as \(V^{\pi}(s)\), is expressed as the expected value under the distribution \(\mathcal{P}\), calculated across successive time steps from \(t=0\) to \(T\), wherein each reward \(r_{t}\) is discounted by a factor \(\gamma_{t}\). The expression for \(V^{\pi}(s)\) is as follows:  
\[V^{\pi}(s)=\mathbb{E}_{s_{t}\sim\mathcal{P}}\left[\sum_{t=0}^{T}\gamma^{t}\cdot r _{t}\right] \tag{1}\]  
The function denoted as \(V^{\pi}\) is recognized as the value function, while \(\gamma\) is the discounting factor, which must adhere to the constraint of \(0<\gamma\leq 1\). In a similar vein, the state-value function, \(Q_{\pi}\), is contingent upon the state \(s_{t}\) and the action \(a_{t}\) at time step \(t\), as follows:  
\[Q^{\pi}(s_{t},a_{t}) =r_{t}+\gamma\cdot\mathbb{E}_{s_{t+1}\sim\mathcal{P}}[V^{\pi}(s_{t +1})] \tag{2}\] \[=r_{t}+\gamma\cdot\mathbb{E}_{s_{t+1}\sim\mathcal{P},a_{t+1}\sim \pi}[Q^{\pi}(s_{t+1},a_{t+1})]\]  
DRL algorithms are generally categorized into two main types: _discrete algorithms_, such as Deep Q-Network (DQN) and its variant Double Deep Q-Network (DDQN) and _continuous algorithms_, which include diverse methods like stochastic policy techniques (e.g., Asynchronous Advantage Actor-Critic (A3C)) and deterministic techniques (e.g., Deep Deterministic Policy Gradient (DDPG)). In the actor-critic framework, a fundamental RL paradigm, two distinct neural networks are employed: the actor-network takes the state as input and predicts optimal actions, while the critic network evaluates the quality of the actor's actions.