## 6 Actor-Critic Methods
### Proximal Policy Optimization (PPO)
#### 6.3.3 PPO-Based End-to-End Path Planning for Lunar Rovers with Safety Constraints  
[PERSON] et al. [43] proposed an end-to-end path planning method based on PPO by integrating a safety-aware reward function, sliding behavior analysis, and curriculum learning to address the challenges of autonomous path planning for lunar rovers. These improvements ensure safety, stability, and adaptability across different terrains. The path planning problem is formulated as a Markov Decision Process (MDP), with the policy \(\pi_{\theta}\) represented by a deep neural network parameterized by \(\theta\). The optimal policy \(\pi_{\theta}^{*}\) is defined as follows:  
\[u_{t}=\pi_{\theta}^{*}(s_{t})=\pi_{\theta}^{*}(s_{t1},s_{t2},s_{t3}), \tag{184}\]  
where \(s_{t1}\), \(s_{t2}\), and \(s_{t3}\) represent depth camera data, LiDAR data, and rover/target state information, respectively. The safety-aware reward function incorporates distance and safety considerations:  
\[r_{t}=r_{t}^{\text{dis}}+\alpha_{s}\cdot r_{t}^{\text{safe}}, \tag{185}\]where \(\alpha_{s}\) is the safety factor. The distance reward \(r_{t}^{\text{dis}}\) is defined as follows:  
\[r_{t}^{\text{dis}}=\begin{cases}10,&\text{if }d_{p}^{t}<d_{\text{goal}},\\ -10,&\text{if }d_{o}^{t}<d_{\text{min}},\\ d_{p}^{t-1}-d_{p}^{t},&\text{otherwise},\end{cases} \tag{186}\]  
where \(d_{p}^{t}\) and \(d_{p}^{t-1}\) are the rover's distances to the target at time \(t\) and \(t-1\), and \(d_{o}^{t}\) is the distance to the nearest obstacle.  
The safety reward \(r_{t}^{\text{safe}}\) penalizes sliding and unstable behaviors:  
\[r_{t}^{\text{safe}}=\begin{cases}-10,&\text{if }(\theta_{t}>\theta_{\text{ max}})\vee(\phi_{t}>\phi_{\text{max}}),\\ -\text{slip}_{t}\cdot\beta_{t},&\text{otherwise},\end{cases} \tag{187}\]  
where \(\theta_{t}\) and \(\phi_{t}\) are the pitch and roll angles, \(\text{slip}_{t}\) and \(\beta_{t}\) are the sliding rate and sliding angle, calculated as follows:  
\[\text{slip}_{t}=Ae^{B\theta_{t}},\quad\beta_{t}=Ce^{D\phi_{t}}, \tag{188}\]  
with \(A,B,C,D\) as constants derived from experimental data. The policy is updated using the PPO objective with a clipped surrogate:  
\[L_{\text{PPO}}(\theta)=\mathbb{E}_{t}\bigg{[}\min\bigg{(}\rho_{t}(\theta)A_{t },\text{clip}(\rho_{t}(\theta),1-e,1+e)A_{t}\bigg{)}\bigg{]}, \tag{189}\]  
where \(\rho_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a _{t}|s_{t})}\), and \(A_{t}\) is the advantage function:  
\[A_{t}=r_{t}+\gamma V_{\theta}(s_{t+1})-V_{\theta}(s_{t}). \tag{190}\]  
Curriculum learning is applied to improve adaptability by incrementally increasing terrain complexity. Training begins in flat terrains and progresses to more complex environments with adjusted safety factors \(\alpha_{s}\). This progressive training ensures robust and efficient learning of safe and optimal paths. Figure 9 shows a block diagram of the method for lunar rover path planning. The input includes depth camera data, LiDAR data, and rover/target state information. Feature extraction is performed using a neural network, followed by safety-aware reward calculations and policy updates using PPO. A curriculum learning strategy incrementally increases terrain complexity, resulting in a safe and efficient navigation policy \(\pi_{\theta}(a_{t}|s_{t})\). Figure 12 shows a block diagram of the proposed algorithm.