## 6 Actor-Critic Methods
### Deep Deterministic Policy Gradient  
The DDPG method is an actor-critic reinforcement learning algorithm designed for continuous action spaces. It extends the Deterministic Policy Gradient (DPG) method [26] by incorporating deep neural networks to parameterize both the policy and value functions. DDPG has been widely used in robotics, including mobile robot path planning, where the control commands often lie in a continuous domain, such as linear and angular velocities. It uses an actor-critic framework where  
* The **actor**\(\pi(s;\theta^{\pi})\) is a deterministic policy, parameterized by \(\theta^{\pi}\), that maps states \(s\in\mathcal{S}\) to continuous actions \(a\in\mathcal{A}\).  
* The **critic**\(Q(s,a;\theta^{Q})\) is a neural network, parameterized by \(\theta^{Q}\), that estimates the action-value function: \[Q(s,a;\theta^{Q})\approx Q^{\pi}(s,a)=\mathbb{E}\bigg{[}\sum_{t=0}^{\infty} \gamma^{t}r(s_{t},a_{t})\,\big{|}\,s_{0}=s,a_{0}=a\bigg{]}.\] (41) The policy is updated by maximizing the estimated Q-value: \[\
abla_{\theta^{\pi}}J(\pi)=\mathbb{E}_{s\sim\mathcal{D}}\Big{[}\
abla_{a}Q( s,a;\theta^{Q})\big{|}_{a=\pi(s;\theta^{\pi})}\
abla_{\theta^{\pi}}\pi(s; \theta^{\pi})\Big{]},\] (42) where \(\mathcal{D}\) is a replay buffer used to store past transitions \((s,a,r,s^{\prime})\) for sampling during training. The critic is updated by minimizing the mean squared error of the Bellman residual: \[L(\theta^{Q})=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\bigg{[}\Big{(}Q( s,a;\theta^{Q})-y\Big{)}^{2}\bigg{]},\] (43) where the target value \(y\) is computed using a target actor \(\pi^{\prime}(s;\theta^{\pi^{\prime}})\) and a target critic \(Q^{\prime}(s,a;\theta^{Q^{\prime}})\): \[y=r(s,a)+\gamma Q^{\prime}(s^{\prime},\pi^{\prime}(s^{\prime};\theta^{\pi^{ \prime}});\theta^{Q^{\prime}}).\] (44) DDPG uses target networks to stabilize training. The target actor \(\pi^{\prime}(s;\theta^{\pi^{\prime}})\) and target critic \(Q^{\prime}(s,a;\theta^{Q^{\prime}})\) are updated slowly using a soft update mechanism: \[\theta^{\pi^{\prime}} \leftarrow\tau\theta^{\pi}+(1-\tau)\theta^{\pi^{\prime}},\] (45) \[\theta^{Q^{\prime}} \leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}},\] (46) where \(\tau\in[0,1)\) controls the update rate. To encourage exploration in continuous action spaces, DDPG adds noise to the deterministic policy during training. A common choice is Ornstein-Uhlenbeck (OU) noise, which models temporally correlated random processes: \[a_{t}=\pi(s_{t};\theta^{\pi})+\mathcal{N}_{t},\quad\mathcal{N}_{t}\sim\text{OU }(\mu,\sigma).\] (47) Algorithm 3 shows the step-by-step pseudocode for the DDPG algorithm. In mobile robot path planning, DDPG has been applied to learn navigation strategies in continuous control domains. The actor network generates control commands, such as linear velocity \(v\) and angular velocity \(\omega\), while the critic evaluates the navigation performance. The state \(s\) may include the following:  
* The robot's position \((x,y)\) and orientation \(\theta\).
* Sensor inputs, such as LIDAR or depth images, providing information about obstacles.
* The relative position of the goal \((x_{g},y_{g})\). The reward function is typically similar to (21), and designed to guide the robot toward the goal while avoiding collisions.