# Self-Adaptive Priority Correction for Prioritized Experience Replay
## 2 Related Work
### Priority Experience Replay  
Priority experience replay was proposed in 2016 by DeepMind [9]. Due to its efficient data utilization, considerable works have been proposed to extend and optimize it. Initially, the PER is used to improve the DQN and its family (Double DQN [22], Duling DQN [23], Distributed DQN [11], and Rainbow [16]) algorithms. TD-error is an effective metric of priority and it is easy to obtain in DQNs. Not only does the value-based algorithm use PER, but researchers also try PER on policy-based algorithms, such as DDPG + PER [12]. The metrics used in priority are the same as DQN because DDPG also needs to optimize TD loss. By using PER, DRL's data utilization and policy quality have been significantly improved.  
In PER, as TD-error is not the only choice of priority metrics, the selection of priority metrics will influence the training. To address this drawback, [PERSON] et al. developed a framework for a twice active sampling method in the deep Q-learning [13], which considered the cumulative reward and TD-error together. First of all, they sampled a batch of data proportional to their cumulative rewards. Then, a mini-batch of data was sampled according to their TD-errors. It got better performance on many Atari games. A similar idea was also used in HVPER [24]. HVPER considered the TD-error, value of Q, and data usage together. Different training stages would focus on different importance estimators, and dynamically adjust the weight of each estimator to realize an adaptive importance estimation method. To stabilize the PER, [PERSON] et al. added the latest transitions into the training batch [25]. In addition, they also considered the reward into priority, which balances the TD-errors and rewards. In addition, [PERSON] et al. used reward-shaping to give an additional reward in priority,  
Figure 1: The similarity of distributions of real and stored priorities in Atari game Pong and MuJoCo walk2d. The x-axis is the index of the sample, which is sorted according to the stored TD-error.  
which further improved data utilization [26]. [PERSON] et al. integrated Upper Confidence Bound (UCB) with PER [27]. Using such a method, when choosing the next batch of data, the UCB balanced the exploration and exploitation of data. Compared to the original PER, UCB reached a better final policy. Recent research called Experience Replay Optimization (ERO) which dynamically adjusted the sampling policy to adapt variety tasks [14]. The ERO used the transition's reward, the TD-error, and the current timestep to estimate the priority of data. A similar priority was used to lifelong learning on multi-tasks DRL [28]. Some information theory-based methods were used in DRL, such as entropy-based optimal sampling strategy [29]. Different from the above methods, Ref-ER tried to control the deletion of data from EM but not to improve the sampling strategy [15]. Recently, [PERSON] et al. proposed Attentive Experience Replay (AER), which ranked the data according to the agent's state distribution [30]. AER outperformed PER on many MujoCo tasks. In order to further improve data utilization, [PERSON] et al. not only used priority sampling, but also based on priority when deleting old samples [31]. [PERSON] et al. proposed double prioritized state recycled (DPSR) to sample and replace data in EM, which keeping a high-quality replay memory. Most of the latest state of the art research focuses on how to design better priority indicators, which basically used delayed TD-errors. However, our work is different from the above, we mainly solve the delay of TD-errors to improve data utilization. The above researches can use our method to further improve their performances.