## 6 Actor-Critic Methods
### Soft Actor-Critic (SAC)
#### 6.4.4 Robot Navigation in Constrained Pedestrian Environments  
The study by [PERSON] et al. [49] proposed a method that extends SAC by integrating a sampling-based motion planner with a reactive sensorimotor policy to achieve dynamic navigation in constrained pedestrian environments. The problem is formulated as a Partially Observable Markov Decision Process (POMDP) defined as follows:  
\[M=(S,A,O,T,R,\gamma), \tag{242}\]  
where \(S\) is the state space, \(A\) is the action space, \(O\) is the observation space, \(T(s^{\prime}|s,a)\) is the state transition model, \(R(s,a)\) is the reward function, and \(\gamma\) is the discount factor. Observations \(o_{t}\in O\) include \(o_{t}=\{\text{goal,lidar, waypoints}\}\), where  
* goal represents the 2D coordinates of the target in the robot's frame,
* lidar consists of 128 range measurements from a 1D LiDAR sensor, and
* waypoints are derived from a global planner using a 2D map. The policy \(\pi_{\theta}(a_{t}|o_{t})\) outputs actions \(a_{t}=[v_{x},v_{y},\omega]\), where \(v_{x}\) and \(v_{y}\) are the linear velocities, and \(\omega\) is the angular velocity. The reward function \(R(s,a)\) incorporates terms to encourage goal-reaching, minimize time, avoid collisions, and follow global waypoints:  
\[R=R_{\text{goal}}+R_{\text{timestep}}+R_{\text{collision}}+R_{\text{ potential}}+R_{\text{waypoint}}. \tag{243}\]Here,  
\[R_{\text{goal}} = +1\quad\text{if dist}_{\text{goal}}<d_{g}, \tag{244}\] \[R_{\text{timestep}} = -0.001,\] (245) \[R_{\text{collision}} = -1\quad\text{if collision occurs},\] (246) \[R_{\text{potential}} = f(\text{dist}_{\text{waypoint}}), \tag{247}\]  
where \(d_{g}\) is the goal tolerance distance, and \(f(\text{dist}_{\text{waypoint}})\) is a potential-based reward inversely proportional to the distance to the next waypoint. ```
Input: State \(s_{t}=\{s_{t}^{\text{lidar}},s_{t}^{\text{goal}},s_{t}^{\text{speed}}\}\), actions \(a_{t}=[v,\omega]\), reward function \(R(s_{t},a_{t})\), global path planner Output: Optimized policy \(\pi_{\theta}(a_{t}|s_{t})\) Initialize: Actor \(\pi_{\theta}\), critic \(Q(s_{t},a_{t})\), value network \(V(s_{t})\), and replay buffer \(\mathcal{D}\); while not convergeddo // Step 1: Global Path Planning  Generate global trajectory using A* algorithm;  Compute look-ahead point: \(p_{\text{lookahead}}=p_{\text{path}}(d_{\text{lookahead}})\) Update goal  component: \(s_{t}^{\text{goal}}=[p_{\text{lookahead}}-p_{\text{robot}}]\) // Step 2: Interaction with Environment  Observe state \(s_{t}=\{s_{t}^{\text{lidar}},s_{t}^{\text{goal}},s_{t}^{\text{speed}}\}\);  Sample action \(a_{t}\sim\pi_{\theta}(a_{t}|s_{t})\);  Execute \(a_{t}\) and observe reward \(r(s_{t},a_{t})\) and next state \(s_{t+1}\); // Step 3: Store Transition  Store \((s_{t},a_{t},r(s_{t},a_{t}),s_{t+1})\) in replay buffer \(\mathcal{D}\); // Step 4: Reward Calculation  Compute hybrid reward: \(R(s_{t},a_{t})=R_{g}+R_{c}+R_{\omega}\) where \(R_{g}= \begin{cases}10,&\text{if dist}_{\text{curr}}<0.5,\\ \text{dist}_{\text{prev}}-\text{dist}_{\text{curr}},&\text{otherwise},\end{cases} \) \(R_{\omega}= \begin{cases}-0.1|\omega|,&\text{if }|\omega|>0.6,\\ 0,&\text{otherwise}.\end{cases} \) // Step 5: Policy and Value Updates  Update critic by minimizing: \(L_{Q}=\mathbb{E}\big{[}(Q(s_{t},a_{t})-(r(s_{t},a_{t})+\gamma V(s_{t+1})))^{ 2}\big{]}\)  Update value network: \(V(s_{t})=\mathbb{E}_{t_{t}\sim\pi}[Q(s_{t},a_{t})-\alpha\log\pi(a_{t}|s_{t})]\) Update policy using entropy-regularized objective: \(f(\theta)=\mathbb{E}\big{[}\sum_{t=0}^{T}(R(s_{t},a_{t})+\alpha H(\pi(a_{t}|s_ {t})))\big{]}\) // Step 6: Replay Buffer Sampling  Sample a batch of transitions from \(\mathcal{D}\) for training; // Step 7: Termination Check  If robot reaches the goal or exceeds maximum steps, terminate the episode;  end while return Optimized policy \(\pi_{\theta}(a_{t}|s_{t})\);
```  
**Algorithm 12**The SAC-based Dynamic Obstacle Avoidance and Path Planning of [48].