# Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization
## 3 Methods
### Weighted Near-Optimal Experiences Policy Optimization
#### 3.2.3 Policy Network Optimization  
For the imitation phase, since the policy \(\pi_{\theta}(u_{t}|x_{t})\) obeys a Gaussian distribution \(N(u^{\pi}(x_{t}),\Sigma^{\pi}(x_{t}))\) and the linear Gaussian controller \(p(u_{t}|x_{t})\) also obeys the distribution \(N(u^{\pi}(x_{t}),C_{t})\), the optimization problem in Equation (9) can be further written as follows. \[ \begin{split}\min\frac{1}{2M}\sum\limits_{i=1}^{M}\sum\limits_{t= 1}^{T}E_{p(x_{t})\pi_{\theta}(u_{t}|x_{t})}\big{[}2\lambda_{t}^{T}u^{\pi}(x_{t })-\text{log}|\Sigma^{\pi}(x_{t})|\big{]}\\ +E_{p(x_{t})\pi_{\theta}(u_{t}|x_{t})}\big{[}tr\big{[}C_{ti}^{-1} \Sigma^{\pi}(x_{t})\big{]}+\widetilde{u}^{T}C_{ti}^{-1}\widetilde{u}\big{]} \end{split}  \tag{23}\]  
where \(\widetilde{u}=\mu^{\pi}(x_{t})-\mu_{ti}^{p}(x_{t})\), and \(M\) is the number of trajectories sampled by the system. To simplify the solution of the objective function, the variance \(\Sigma^{\pi}(x_{t})\) of the policy is assumed to be independent of the state \(x_{t}\). A partial derivative of the above equation yields:  
\[\Sigma^{\pi}(x_{t})=\left[\frac{1}{NT}\sum\limits_{i=1}^{N}\sum\limits_{t=1}^{ T}C_{ti}^{-1}\right]^{-1} \tag{24}\]  
Simplifying the optimization issue to:  
\[\min\limits_{\theta}\frac{1}{2N}\sum\limits_{i=1}^{N}\sum\limits_{t=1}^{T}E_{p (x_{t})\pi_{\theta}(u_{t}|x_{t})}\big{[}2\lambda_{t}^{T}\mu^{\pi}(x_{t})+ \widetilde{\mu}^{T}C_{ti}^{-1}\widetilde{\mu}\big{]} \tag{25}\]  
It can be seen that when the output \(\mu^{\pi}(o_{t})\) of the policy is consistent with the output \(\mu_{ti}^{p}(x_{t})\) of the controller, the corresponding policy parameters \(\theta\) are the optimal solution to this optimization problem. For the practice phase, we adopt the PPO-clip algorithm [28]. The whole idea of this approach is to find a functional relationship between policy change and value, take the new policy that can improve the old policy the most and prevent the policy from being updated too much by truncating the policy. In the training process of this phase, the experiences in the weighted replay buffer contains the near-optimal trajectories output by the linear Gaussian controller, as well as the data generated when the policy network interacts with the environment. We describe the implementation steps of WNEPO as follows:1. Initialize the linear Gaussian controller and policy network;
2. Control the robot to walk with the linear Gaussian controller and record the experiences in the weighted replay buffer;
3. Use experiences stored in the weighted replay buffer to update the linear Gaussian controller and policy network in a supervised learning manner;
4. Check whether the cumulative reward obtained by the linear Gaussian controller converges; if it converges, skip to step (5), otherwise return to step (2);
5. Control the robot to walk with the policy network and record the experiences in the weighted replay buffer;
6. Use the PPO algorithm to update the policy network until the cumulative reward converges. The output of the WNEPO algorithm is a policy network, which can directly control the robot to complete the desired task. The implementation steps described above are an experience-based circular process. The pseudo code of the WNEPO algorithm is shown in Algorithm 1.