## 3 Deep Reinforcement Learning  
Deep Reinforcement Learning (Deep RL) leverages deep neural networks as function approximators to address high-dimensional state and action spaces. In scenarios, such as mobile robot path planning, where the state space \(\mathcal{S}\) may include sensor data, such as images or LIDAR scans, and the action space \(\mathcal{A}\) involves continuous control, Deep RL provides a scalable and effective solution [3, 8]. The key elements of Deep RL build directly on the Markov Decision Process (MDP) framework and extend it by introducing function approximation methods.  
In Deep RL, the policy \(\pi(a|s;\theta)\) is parameterized by a neural network with parameters \(\theta\). The policy maps states \(s\in\mathcal{S}\) to a probability distribution over actions \(a\in\mathcal{A}\). The objective remains to optimize the parameters \(\theta\) to maximize the expected cumulative reward (2):  
\[J(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}r (s_{t},a_{t})\right], \tag{24}\]  
where \(\gamma\) is the discount factor, as defined previously [1].  
In addition to the policy, in deep RL, the state-value function \(V^{\pi}(s)\) and action-value function \(Q^{\pi}(s,a)\), which are critical for evaluating the quality of a policy or action, can be approximated using neural networks:  
\[Q(s,a;\phi)\approx Q^{\pi}(s,a), \tag{25}\]  
where \(\phi\) represents the parameters of the neural network. Algorithms such as Deep Q-Networks (DQN) [8] use this approximation to solve the Bellman equation:  
\[Q(s,a;\phi)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}}\bigg{[}\max_{ a^{\prime}}Q(s^{\prime},a^{\prime};\phi)\bigg{]}. \tag{26}\]  
When deep RL combines a policy network (actor) \(\pi(a|s;\theta)\) with a value network (critic) \(Q(s,a;\phi)\), the actor selects actions, while the critic evaluates them. This approach is called the _actor-critic paradigm_. The policy gradient is computed as follows:  
\[\
abla_{\theta}J(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}[\
abla_{\theta}\log \pi_{\theta}(a|s)Q^{\pi}(s,a)], \tag{27}\]  
where \(Q^{\pi}(s,a)\) is provided by the critic network [2, 4].  
Deep RL also addresses the balance between exploration (choosing less certain actions to gather information) and exploitation (choosing actions that maximize known rewards). Techniques such as \(\epsilon\)-greedy policies [1, 8] or entropy regularization [2] are commonly used to manage this trade-off. To stabilize learning, deep RL often employs mechanisms such as target networks and experience replay. Target networks provide a stable reference for value updates, and experience replay reuses past experiences for training, improving sample efficiency [8].  
The key components of a Deep Reinforcement Learning (Deep RL) framework and their interactions are illustrated in Figure 1. The interaction flow in the diagram starts with the state \(s\in\mathcal{S}\), which is processed by both the policy network (actor) and value network. The policy network generates an action \(a\in\mathcal{A}\), which interacts with the environment to produce a reward \(r(s,a)\) and a new state \(s^{\prime}\). The value network evaluates \(Q(s,a;\phi)\) to inform the critic. The critic provides feedback for updating both the value network and the policy network, ensuring iterative improvements in both evaluation and decision-making.