# Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization
GPS enables the policy network to be updated without direct interaction with the environment and makes it easier to deploy a policy built with neural networks to real robots. Compared with model-free RL, this training method has higher sample efficiency. The learning process of the policy network in GPS is essentially supervised learning. The purpose of supervised learning is to train a model to fit the labeled training data so as to achieve a specific task. The degree to which a model performs a task is directly determined by the data fed to it. Consequently, the main drawback of GPS is that the performance of the policy network cannot surpass that of the linear Gaussian controller. Although the near-optimal trajectory samples generated by the linear Gaussian controller will make the policy network explore the high-reward area more fully, which can be very beneficial to improve the sample efficiency, the policy network may never explore the optimal space. Beyond that, the policy network will not be updated online after being deployed to the robot because the performance of the neural network will deteriorate rapidly if low-reward samples are used for fine-tuning. In addition, the policy network will not be updated after it is deployed to the robot because the distribution of the low-reward trajectory samples generated by online interaction is not balanced with the original high-reward near-optimal ones, which will lead to instability of the policy network training. To solve the problems that GPS encounters, weighted near-optimal experiences policy optimization (WNEPO) is proposed in this paper. The algorithm framework of WNEPO is shown in Figure 1. In WNEPO, the update of the policy network can be divided into two stages. In the first stage, the optimization objective adopted by GPS is used to update the policy network, and the near-optimal experiences generated by the linear Gaussian controller are used as the training samples; in the second stage, the policy network interacts with the environment directly and is trained by the policy gradient method. The first stage of learning is a supervised learning manner--more accurately speaking, imitation learning [20]--and the second stage is a model-free RL manner. Compared with GPS, WNEPO has two improvements: (1) it replaces the traditional replay buffer with a weighted replay buffer; (2) the policy network not only learns from the near-optimal experiences but also uses the data directly interacting with the environment to improve the performance after approaching the performance of the linear Gaussian controller. The pros and cons of the RL algorithms mentioned above are summarized in Table 1.  
The goal of this paper is to introduce our WNEPO method through training a biped robot to walk, and the whole work is performed under a MATLAB simulation environment. The paper is organized as follows: Section 1 discusses the shortcomings of the existing algorithms in solving the problem of robot skill learning and describes the overall idea; Section 2 provides a brief introduction to the underlying theory; Section 3 introduces the \begin{table}
\begin{tabular}{c c c} \hline \hline
**RL Methods** & **Advantages** & **Disadvantages** \\ \hline \multirow{3}{*}{Model-free RL} & - No need for prior knowledge & - Slow convergence speed \\  & - Strong asymptotic performance & - High risk of damage to robot \\  & & and environment \\ \hline \multirow{3}{*}{Model-based RL} & - Less online interaction, safer for robot & - Depends on explicit models \\  & - and environment & - Poor asymptotic performance \\  & - Fast convergence speed & - May never explore the optimal space \\ \hline \multirow{3}{*}{GPS} & - All the above advantages & - Cannot be updated after being deployed \\ \cline{1-1}  & & - Not all of the above, but additional \\ \cline{1-1}  & - Same as GPS & - component is required \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Advantages and disadvantages of different reinforcement learning (RL) methods.