## 6 Actor-Critic Methods
### Proximal Policy Optimization (PPO)
#### 6.3.4 PPO-Based Real-Time Collision Avoidance in Dense Crowds  
The study by [PERSON] et al. [44] proposes a method, CrowdSteer, that builds upon PPO by incorporating multi-sensor fusion, a hybrid reward function, and specialized training scenarios to address challenges in dense and dynamic crowd navigation. The approach uses data from a 2-D LiDAR and a depth camera to implicitly model interactions with obstacles and pedestrians, generating smoother and collision-free trajectories. The navigation task is formulated as a Partially Observable Markov Decision Process (POMDP) defined by a 6-tuple \((S,A,P,R,\Omega,O)\), where \(S\) represents the state space, \(A\) the action space, \(P\) the state transition probabilities, \(R\) the reward function, \(\Omega\) the observation space, and \(O\) the observation probability distribution. The robot's action at time \(t\) is sampled from the policy \(\pi_{\theta}\) as follows:  
\[a_{t}\sim\pi_{\theta}(a_{t}|o_{t}), \tag{191}\]where \(o_{t}=[o_{t}^{\text{Iid}},o_{t}^{\text{cam}},o_{t}^{g},o_{t}^{\tau}]\) comprises LiDAR data \(o_{t}^{\text{Iid}}\), camera data \(o_{t}^{\text{cam}}\), relative goal location \(o_{t}^{g}\), and robot velocity \(o_{t}^{\tau}\). The hybrid reward function encourages goal-reaching and safe navigation while penalizing collisions and oscillatory behavior:  
\[r_{t}=r_{t}^{g}+r_{t}^{e}+r_{t}^{\text{osc}}+r_{t}^{\text{safe}}, \tag{192}\]  
where the components are defined as follows:  
\[r_{t}^{g} = \begin{cases}r_{\text{goal}},&\text{if }||p_{t}-p_{g}||<\delta,\\ 2.5(||p_{t-1}-p_{g}||-||p_{t}-p_{g}||),&\text{otherwise},\end{cases} \tag{193}\] \[r_{t}^{e} = \begin{cases}r_{\text{collision}},&\text{if }||p_{t}-p_{\text{ obs}}||<\epsilon,\\ 0,&\text{otherwise},\end{cases}\] (194) \[r_{t}^{\text{osc}} = -0.1|\omega_{t}|,\text{ if }|\omega_{t}|>0.3,\] (195) \[r_{t}^{\text{safe}} = -0.1||R_{\text{max}}-R_{\text{min}}||. \tag{196}\]  
The policy is trained using PPO with a clipped surrogate objective:  
\[L_{\text{PPO}}(\theta)=\mathbb{E}_{t}[\min(\rho_{t}A_{t},\text{clip}(\rho_{t}, 1-\epsilon,1+\epsilon)A_{t})], \tag{197}\]  
where \(\rho_{t}=\frac{\pi_{\theta}(\pi_{t}|o_{t})}{\pi_{\text{old}}(a_{t}|a_{t})}\) is the probability ratio, and \(A_{t}\) is the advantage function:  
\[A_{t}=r_{t}+\gamma V_{\theta}(s_{t+1})-V_{\theta}(s_{t}). \tag{198}\]  
Figure 12: Block diagram of the method proposed by [43].  
The policy \(\pi_{\theta}\) is represented by a neural network with separate branches for processing LiDAR and camera data, followed by fusion layers that integrate these features with goal and velocity observations. Lidar data \(o_{t}^{\text{lid}}\) and camera data \(o_{t}^{\text{cam}}\) are processed as follows:  
\[o_{t}^{\text{lid}} = \{l\in\mathbb{R}^{512}:0<l_{i}<4\}, \tag{199}\] \[o_{t}^{\text{cam}} = \{C\in\mathbb{R}^{150\times 120}:1.4<C_{ij}<5\}. \tag{200}\]  
During training, the policy is initialized in static environments and progressively trained in dynamic scenarios with occluded obstacles and high pedestrian densities. This curriculum learning approach improves the robot's ability to generalize to real-world dense crowd navigation while ensuring smooth and efficient trajectories.