## 6 Actor-Critic Methods
### Deep Deterministic Policy Gradient
#### 6.1.6 Learning World Transition Model for Socially Aware Robot Navigation  
[PERSON] et al. [32] present an extension of the DDPG framework by integrating a model-based reinforcement learning approach with a deep world transition model, which predicts future states and rewards. This improves sample efficiency and allows socially compliant robot navigation. The world transition model predicts the evolution of the robot's environment, including future obstacle maps and rewards. Let the current state \(s_{t}\) be represented as a sequence of obstacle maps, relative goal position, and current velocity. The predicted next state \(s_{t+1}\) is given by the following:  
\[s_{t+1}=f(s_{t},a_{t};\theta_{w}), \tag{85}\]  
where \(f\) is the world transition model parameterized by \(\theta_{w}\). The obstacle map representation disentangles static and dynamic obstacles using ego-motion transformation. Dynamic features are computed by the following:  
\[m_{t}=o_{t}-o_{t-1}, \tag{86}\]  
where \(o_{t}\) is the obstacle map at time \(t\). Static features are extracted by encoding the last obstacle map. ```
Input :Initial parameters \(\theta^{\pi},\phi_{1},\phi_{2}\): Replay buffer \(\mathcal{D}\); Probabilistic Roadmap  PRM; Training environments: 2D Gym, 3D Gazebo. Output :Trained policy \(\pi(s;\theta^{\pi})\). Initialize actor \(\pi(s;\theta^{\pi})\) and critic networks \(Q_{\phi_{1}},Q_{\phi_{2}}\) with random weights;  Initialize target networks \(\pi^{\prime}(s;\theta^{\pi^{\prime}})\leftarrow\pi(s;\theta^{\pi})\) and \(Q^{\prime}_{\phi_{1}},Q^{\prime}_{\phi_{2}}\gets Q_{\phi_{1}},Q_{\phi_{2}}\);  Initialize replay buffer \(\mathcal{D}\); for each training phase (2D \(\rightarrow\) simple 3D \(\rightarrow\) complex 3D)do for each episodedo  Reset the environment and observe initial state \(s_{0}\); for each time step \(t\)do  Select action \(a_{t}=\pi(s_{t};\theta^{\pi})+\mathcal{N}_{t}\), where \(\mathcal{N}_{t}\) is Gaussian noise;  Execute action \(a_{t}\) and observe reward \(r_{t}\), next state \(s_{t+1}\), and conduct  flag \(d_{t}\);  Store transition \((s_{t},a_{t},r_{t},s_{t+1},d_{t})\) in buffer \(\mathcal{D}\); iftime to updatethen  Sample minibatch \(B=\{(s,a,r,s^{\prime},d)\}\) from \(\mathcal{D}\);  Compute target for the critics: \(y=r+\gamma(1-d)\min_{i=1,2}Q^{\prime}_{\phi_{i}}(s^{\prime},\pi^{\prime}(s^{ \prime};\theta^{\pi^{\prime}}))\);  Update critic networks by minimizing: \(L(\phi_{i})=\frac{1}{|B|}\sum_{(s,a,r,s^{\prime},d)\in B}\big{(}Q_{\phi_{i}}(s, a)-y\big{)}^{2},\quad i=1,2\); ifupdate delaythen  Update actor using policy gradient: \(\
abla_{\theta^{\pi}}J=\frac{1}{|B|}\sum_{s\in B}\
abla_{a}Q_{\phi_{1}}(s,a) \big{|}_{a=\pi(s;\theta^{\pi})}\
abla_{\theta^{\pi}}\pi(s;\theta^{\pi})\);  Update target networks: \(\theta^{\pi^{\prime}}\leftarrow\tau\theta^{\pi}+(1-\tau)\theta^{\pi^{\prime}},\quad\phi^{\prime}_{i}\leftarrow\tau\phi_{i}+(1-\tau)\phi_{i},\quad i=1,2\);  Use PRM to generate waypoints \(W=\{w_{1},w_{2},\ldots,w_{n}\}\) for global planning;  Navigate each waypoint \(w_{i}\) using the trained policy \(\pi(s;\theta^{\pi})\); returnTrained policy \(\pi(s;\theta^{\pi})\)
```  
**Algorithm 5**PRM+TD3 with Incremental Training [31]. The reward function incorporates goal-reaching, collision avoidance, and social compliance:  
\[R(s_{t})=R_{g}(s_{t})+R_{c}(s_{t})+R_{s}(s_{t}), \tag{87}\]  
where \(R_{g}(s_{t})\) encourages goal-reaching:  
\[R_{g}(s_{t})= \begin{cases}r_{\text{arrival}},&\text{if goal reached},\\ w_{1}(\|p_{t}-p^{*}\|-\|p_{t-1}-p^{*}\|),&\text{otherwise}.\end{cases}  \tag{88}\]  
Collision avoidance is penalized by \(R_{c}(s_{t})\):  
\[R_{c}(s_{t})= \begin{cases}r_{\text{collision}},&\text{if collision},\\ w_{2}\big{(}1-\frac{d}{r+1.0}\big{)},&\text{if }d\leq r+1.0,\\ 0,&\text{otherwise}.\end{cases}  \tag{89}\]Social compliance is achieved by penalizing proximity to dynamic obstacles:  
\[R_{s}(s_{t})= \begin{cases}w_{3}\Big{(}1-\frac{d_{\text{ped}}}{r+125}\Big{)},& \text{if }d_{\text{ped}}\leq r+1.25,\\ 0,&\text{otherwise}.\end{cases}