# Self-Adaptive Priority Correction for Prioritized Experience Replay
## 3 Proposed Approach
### Importance Weighted PER  
Unlike ATDC-PER [18], the Imp-PER will not use the predicted value to update EM. Based on the predicted and stored probabilities of data, Imp-PER uses the importance weights to correct the bias of priorities. From the definition in preliminary, the real probability distribution is \(q(x)\) and the stored probability distribution is \(p(x)\). A mini-batch of data is sampled from EM based on \(p(x)\). Our target is to estimate the loss function \(f(x)\) under distribution \(q(x)\). Using the importance weight, we can do that by Equation (11). \[E_{x\sim q(x)}[f(x)]=E_{x\sim p(x)}\left[\frac{q(x)}{p(x)}f(x)\right] \tag{11}\]  
From the definition of expectation, It is easy to prove the correctness of Equation (11). According to Equation (6), the importance weighted loss is defined by Equation (12). \[loss(\mathbf{\theta})=E_{x\sim p(x)}\left[\frac{q(x)}{p(x)}\left(\frac{min_{i}q(i) }{q(x)}\right)^{\beta}(Q(x;\mathbf{\theta})-y)^{2}\right] \tag{12}\]Figure 5 illustrates the update process. Training DQN with uniform sampling from the EM usually converges to a locally optimal solution. It mostly samples the data with low magnitudes TD-errors, which gives a slower direction. PER based on real priority always chooses the data with high magnitudes TD-error, which gives the steepest direction to minimize Q loss. In most cases, it will converge to a position closer to the global solution. However, Due to outdated TD errors, PER based on storage priority sometimes gives the wrong direction. It significantly increases the steps to reach an optimal solution. In this work, we use the importance sampling to correct the direction of the update, which makes the wrong direction closer to the steepest direction. However, the importance weights \(w=q(x)/p(x)\) has an unbounded variance, which would affect the training stability. Figure 6 shows the distribution of importance weight on some Atari tasks. It is easy to observe that about 2% of the importance weights are greater than 10. Like the original PER, we can normalize weights by \(1/max_{i}w_{i}\) so that they only scale the update downwards [9]. However, experiments on multiple tasks show that the method will reduce the training efficiency in most cases. Another way to reduce the variance is to directly truncate the importance weight to limit it between \([\tau_{min},\tau_{max}]\). Where the \(\tau_{min}\) and \(\tau_{max}\) are fixed values calculated in advance. Truncating importance weights will introduce bias. [PERSON] proved that truncation at rate \(\sqrt{n}\) is a good general choice [33], where \(n\) represents the size of mini-batch. For example, in our experiment the  
Figure 5: The convergence of uniform sampling, PER based on store priority, PER based on real priority, and PER with importance sampling in the training process, respectively. The contour lines constitute the space curve formed by the Q loss of all experiences. Figure 6: The distribution of importance weight on Atari tasks. The bars represent statistical histograms, and the curves are the corresponding kernel density estimates. batch size is set to 32, then the truncated importance weight belongs to \([0.0,5.657]\). It can also be seen from Figure 6 that such a setting is reasonable. For better understanding, we extract the stored and real priorities during the training process of BeamRider and verified the effect of truncation. We regard the loss function of DQN based on real priority as the target value \(y\), and calculate the error between the predicted value \(\hat{y}\) and \(y\) at different priorities. Here, we use Mean Absolute Percentage Error (MAPE) to measure the accuracy Equation (13). Figure 7 shows the errors based on uniform distribution, storage priority, and importance correction. The left column shows the comparison without truncation. The y-axis represents MAPE, and the x-axis represents the number of tests. Obviously, through the importance correction, the estimated value of the loss function is closer to the true value (based on the true priority).