# Self-Adaptive Priority Correction for Prioritized Experience Replay
## 5 Conclusions and Future Work  
In this work, we analyzed the priority delay problem in PER and explained the negative impact of this problem on agent learning through theoretical and experimental data. The existing methods solve the problem by calculating and correcting the TD-errors of the entire EM, which the computational cost introduced is huge, and it is not applicable. We propose a simple and general framework Imp-PER. It is a cost-free method compared to existing methods. First, we design a linear model to predict the sum of real TD-error and estimate the real probability of the current batch of data. Compared to predicting the real probability of each sample in EM, the cost is much less. Next, by using IS technology to estimate the DQN or DDPG loss function for the real priority distribution, we improve the data utilization and eliminate the cost of updating the entire EM priority. The experimental results show that our Imp-PER could improve data utilization on most games and only 10% slower than Org-PER. Specifically, our algorithm could improve both value-based and policy-based RL methods. And Imp-PER also handles discrete state and continuous state tasks. Finally, our optimal policy is better than the original PER about 15% on the mean with DDQN and about 51% on the mean with DDPG.  
The PER has a pivotal position in DRL and is widely used in DRL training. From a practical point of view, we improve data utilization while ensuring the training speed, which can solve practical problems more quickly. In terms of scientific research, our research analyzes the priority problem of PER in detail, and at the same time corrects the training of the agent from a new perspective. The Imp-PER framework can be used in any DRL algorithm based on PER, such as SAC with PER, ACER, Ape-x.  
To conclude this article, we present here some issues for future study. First, Imp-PER cannot handle the rank-based prioritized experience replay, which is a variant of PER. The rank-based PER has a better performance in some tasks. We have already started research on this aspect. Second, we use clip technology to reduce the variance of importance weights, but introduce bias. How to ensure the bias and reduce the variance is still a challenge. These will be part of our future work.  
Conceptualization, H.Z. and C.Q.; methodology, H.Z., J.Z. and J.L.; software, H.Z. and C.Q.; writing, H.Z., J.Z. and J.L. All authors have read and agreed to the published version of the manuscript. This research received no external funding.  
We would like to thank Network and Information Center, USTC for providing the GPU servers to run our experiments.  
The authors declare no conflicts of interest.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Tasks** & **Random** & **Vanilla-ER** & **Org-PER** & **Imp-PER** & **Relative Improvement** \\ \hline Walker2d & \(-4.34\) & \(500\) & \(800\) & \(\mathbf{1360}\) & \(70\%\) \\ [PERSON] & \(11\) & \(650\) & \(800\) & \(\mathbf{950}\) & \(18.8\%\) \\ InvertedPendulum & \(7.76\) & \(120\) & \(105\) & \(\mathbf{275}\) & \(161.9\%\) \\ InvertedDoublePendulum & \(54.11\) & \(300\) & \(275\) & \(\mathbf{312}\) & \(13.5\%\) \\ [PERSON] & \(-90\) & \(\mathbf{-9}\) & \(-10\) & \(-10\) & \(0\%\) \\ Humanoid & \(102.01\) & \(\mathbf{284}\) & \(195\) & \(271\) & \(38.97\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Final score and normalized score comparison of all methods on MuJoCo. The best scores are shown in bold font. The Random represents a random policy. The relative improvement represents the improved score compare to Org-PER. Bold font shows the best performance.