# Efficient Robot Skills Learning with Weighted Near-Optimal Experiences Policy Optimization
## 3 Methods
### Experience Scoring Algorithm  
The original GPS and PPO use random experiences in the replay buffer and discard the old memory according to the first-in-first-out order. This strategy of storing and updating historical data has two major disadvantages: (1) it does not distinguish the quality of the memory experience and uses it indiscriminately, which results in the low efficiency of the replay buffer; (2) discarding memory experiences according to the time sequence and losing the early high-value memory experiences may cause instability of the policy network learning process. In order to resolve these problems, the experience scoring algorithm is proposed. The experience scoring algorithm evaluates the quality of the experience so as to make more effective use of the trajectories in the replay buffer. For a trajectory \(\tau\), we evaluate its quality from the following three aspects:  
1. Cumulative discounted reward \(R(\tau)\). The ultimate goal of reinforcement learning is to obtain the maximum cumulative expected reward. It is intuitive to use \(R(\tau)\) as an indicator to measure the quality of experience data. For one trajectory, the greater the final cumulative expected reward, the better the overall performance of this episode, and the more valuable it is to learn from. 2. The variance \(var_{\tau}(\tau)\) of all single step rewards. If the single step reward value is much larger than the average value, it will guide the network update direction from the positive direction more effectively. If the single step reward value is much smaller than the average value, it can guide the network update from the opposite direction more effectively. The reward information close to the average value is less efficient for network updates. The analogy is that people can accumulate more life experience in great success or frustration. However, experiences with too large \(var(\tau)\) may lead to more radical network updates and increase the instability of network updating. 3. Episode length \(T_{\tau}\). There is a correlation between \(T_{\tau}\) and \(R(\tau)\), but not a positive one. For example, an episode in which the single step reward is always low but lasts for a long time has a larger \(T_{\tau}\) and a smaller \(R(\tau)\). In this way, even if there is a large \(T_{\tau}\), it will not be considered as a valuable trajectory. The quality \(W\) of a trajectory \(\tau\) can be calculated by the weighted sum of the three evaluation indicators above:  
\[W(\tau)=\alpha_{1}R(\tau)+\alpha_{2}\mathrm{var}_{\tau}(\tau)+\alpha_{3}T_{\tau} \tag{6}\]  
\(\alpha_{1}\), \(\alpha_{2}\) and \(\alpha_{3}\) are parameters that need to be tuned according to the task. Figure 2 shows how the weighted replay buffer works based on the experience scoring algorithm. The left side of Figure 2 shows the update process of the traditional experience replay buffer, which stores and discards data similarly to the data structure of a queue, meeting the principle of first-in-first-out. The right side of Figure 2 describes the updating process of the weighted replay buffer. In the weighted replay buffer, the trajectory is the smallest unit to be discarded rather than the MDUs in the traditional replay buffer. More importantly, the data with the highest score are discarded in the weighted replay buffer rather than the data stored first. Besides, the length of the traditional replay buffer is fixed, while the length of the weighted replay buffer is variable. When a new experience with length \(l\) is stored in a full traditional replay buffer, the first MDUs will be discarded. However, if this new trajectory is put into the full weighted replay buffer, a complete trajectory with the lowest score will be discarded. Figure 2: Comparison of updating processes between traditional replay buffer (**left**) and weighted replay buffer (**right**). The yellow line represents the new experiences that will be stored.