## 4 Value-Based Methods  
In the domain of mobile robot path planning, recent advances in deep reinforcement learning (Deep RL) have introduced both value-based and policy-based methods to solve complex navigation tasks. These methods leverage deep neural networks to approximate functions like the action-value function \(Q^{\pi}(s,a)\) or directly parameterize policies \(\pi(a|s;\theta)\). This section reviews key value-based methods, such as Deep Q-Networks (DQN), Double DQN (DDQN), and Dueling DQN (D3 QN), as well as policy-based approaches, outlining their mathematical foundations and contributions to mobile robot navigation. Figure 2: Conceptual diagram summarizing the categorization of DRL techniques for mobile robot path planning. Value-based methods in deep RL focus on estimating the action-value function \(Q^{\pi}(s,a)\), which represents the expected cumulative reward of taking action \(a\) in state \(s\) and following the policy \(\pi\) thereafter. These methods solve the Bellman equation:  
\[Q^{\pi}(s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}}\Big{[}\max_{a ^{\prime}\in\mathcal{A}}Q^{\pi}(s^{\prime},a^{\prime})\Big{]}, \tag{28}\]  
by using neural networks as function approximators. A well-known value-based deep RL method is the Deep Q-Networks (DQN) [8] which approximates \(Q^{\pi}(s,a)\) using a neural network parameterized by \(\phi\), denoted as \(Q(s,a;\phi)\). The network is trained by minimizing the mean squared error between the current \(Q\)-value and the target value:  
\[L(\phi)=\mathbb{E}_{(s,a,r,s^{\prime})}\Big{[}(Q(s,a;\phi)-y)^{2}\Big{]}, \tag{29}\]  
where the target value \(y\) is as follows:  
\[y=r(s,a)+\gamma\max_{a^{\prime}\in\mathcal{A}}Q(s^{\prime},a^{\prime};\phi^{- }). \tag{30}\]  
Here, \(\phi^{-}\) denotes the parameters of a separate target network, which is updated periodically for stability. The step-by-step pseudocode for DQN is presented in Algorithm 1. ```
Input :Replay memory \(\mathcal{D}\) (capacity \(N\)), discount factor \(\gamma\), exploration probability \(\epsilon\), learning rate \(\alpha\). Output : Trained Q-network with parameters \(\theta\). Initialize : Action-value function \(Q(s,a;\theta)\) with random weights \(\theta\). Initialize : Target action-value function \(Q^{\prime}(s,a;\theta^{-})\) with weights \(\theta^{-}\leftarrow\theta\). for each episode \(=1,\ldots,M\)do  Observe initial state \(s_{1}\); for each step \(t=1,\ldots,T\)do  Select action \(a_{t}\) using \(\epsilon\)-greedy policy: \[a_{t}= \begin{cases}\text{random action}&\text{with probability }\epsilon,\\ \arg\max_{a}Q(s_{t},a;\theta)&\text{otherwise}.\end{cases} \]  Execute action \(a_{t}\), observe reward \(r_{t}\) and next state \(s_{t+1}\);  Store transition \((s_{t},a_{t},r_{t},s_{t+1})\) in \(\mathcal{D}\);  Sample minibatch of transitions \((s,a,r,s^{\prime})\) from \(\mathcal{D}\);  Compute target for each transition: \[y= \begin{cases}r&\text{if terminal state},\\ r+\gamma\max_{a^{\prime}}Q^{\prime}(s^{\prime},a^{\prime};\theta^{-})&\text{ otherwise}.\end{cases} \]  Update \(\theta\) by minimizing the loss: \[L(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\Big{[}(y-Q(s,a; \theta))^{2}\Big{]}.\] ifmodulo step \(t=C\)then  Update target network: \(\theta^{-}\leftarrow\theta\);  End
```  
**Algorithm 1**Deep Q-Network (DQN) Algorithm
While DQN is effective, it suffers from overestimation bias in \(Q\)-value updates. Double DQN (DDQN) [23] addresses this by decoupling the action selection and evaluation processes. The target value is modified as follows:  
\[y=r(s,a)+\gamma Q(s^{\prime},\arg\max_{a^{\prime}\in\mathcal{A}}Q(s^{\prime},a^{ \prime};\phi);\phi^{-}). \tag{31}\]  
This correction reduces overestimation and improves learning stability. An alternative is the Dueling DQN (D3 QN) method [24] which introduces a novel architecture that separates the representation of state-value and advantage functions:  
\[Q(s,a;\phi,\xi)=V(s;\phi)+A(s,a;\xi), \tag{32}\]  
where \(V(s;\phi)\) is the state-value function and \(A(s,a;\xi)\) is the advantage function.