## 6 Actor-Critic Methods
### Proximal Policy Optimization (PPO)
#### 6.3.1 Navigation Skills Acquisition for Wheel-Legged Robots  
The study by [PERSON] et al. [41] builds upon PPO by introducing hierarchical training strategies with secondary and primary policies, along with domain randomization to improve robustness and generalization. The approach addresses challenges such as reward sparsity, temporal credit assignment, and data inefficiency in high-dimensional navigation tasks.  
The hierarchical training begins with secondary policies, each trained for specific behaviors such as moving straight, avoiding obstacles, climbing over obstacles, and squeezing through narrow passages. The secondary policies are trained using PPO with a compound loss function:  
\[L(\theta)=\mathbb{E}[\lambda_{1}\eta_{\pi_{\theta}}+\lambda_{2}L_{v}], \tag{166}\]  
where \(\eta_{\pi_{\theta}}\) is the PPO surrogate function:  
\[\eta_{\pi_{\theta}}=\mathbb{E}\big{[}\min\big{(}Z(\psi_{\theta})A_{t},\psi_{ \theta}A_{t}\big{)}\big{]}, \tag{167}\]  
and \(L_{v}\) is the value function loss:  
\[L_{v}=\frac{1}{2}\big{(}V_{\theta}(s_{t})-R_{t}\big{)}^{2}. \tag{168}\]The probability ratio \(\psi_{\theta}\) is defined as follows:  
\[\psi_{\theta}=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_ {t})}, \tag{169}\]  
and \(Z(\psi_{\theta})\) clips \(\psi_{\theta}\) to \([1-e,1+e]\), where \(e\) is a clipping parameter.  
The reward function for secondary policies encourages specific behaviors:  
\[r_{t}=r_{\text{step}}+r_{\text{progress}}+r_{\text{invalid}}, \tag{170}\]  
where  
\[r_{\text{step}} = -0.1, \tag{171}\] \[r_{\text{progress}} = d_{t-1}-d_{t},\] (172) \[r_{\text{invalid}} = \begin{cases}-0.1,&\text{if collision or invalid action occurs},\\ 0,&\text{otherwise}.\end{cases} \tag{173}\]  
The primary policy is trained using samples from the domain-randomized batch and real interactions in complex environments. Trajectories from secondary policies are replayed in randomized environments to generate action-observation-return tuples \((s_{t},a_{t},R_{t})\). The advantage function \(A_{t}\) for primary policy training is as follows:  
\[A_{t}=R_{t}+\gamma V_{\theta}(s_{t+1})-V_{\theta}(s_{t}). \tag{174}\]  
The policy parameters are updated using stochastic gradient ascent on the compound loss:  
\[L(\theta)=\mathbb{E}[\lambda_{1}\eta_{\pi_{\theta}}+\lambda_{2}L_{\text{ }}]. \tag{175}\]  
A block diagram showing the flow of computations involved in this method is presented in Figure 11.  
Domain randomization improves generalization by altering non-essential aspects of the training environments, such as obstacle appearance, number, and configuration. The randomized environments ensure task-relevant sensory features are preserved while diversifying the training data. The final policy maps height-map images and robot poses to motor actions using a neural network with three convolutional layers followed by fully connected layers.  
By decomposing navigation into manageable sub-tasks, utilizing domain randomization, and optimizing with PPO, the proposed method achieves robust and efficient navigation in high-dimensional environments.  
Figure 11: Block diagram of the method proposed by [40].