# Self-Adaptive Priority Correction for Prioritized Experience Replay
## 4 Experiments
### Performance Comparison
#### 4.2.4 Final Policy Quality  
The optimal policy reflects the final training result of the agent. Ensuring the quality of optimal policy is as important as improving the data utilization. We save the trained Q network at regular intervals. Then, we use the policy to interact with the environment with the exploration rate \(\varepsilon=0\). The highest recorded score represents the quality of the optimal policy. At the beginning of the test, the agent randomly interacts with the environment 1 to 31 times without action, providing a random initial state for the test. As the score range of each game is different, we use Equation (14) to calculate the relative score based on the actual score. \[score_{normalized}=\frac{score_{agent}-score_{random}}{|score_{human}- score_{random}|} \tag{14}\]  
where \(score_{agent}\) is the score got by a trained agent. \(score_{human}\) and \(score_{random}\) represent the score achieved by human and random policy, respectively. If the normalized score is greater than 100%, it means that the agent surpasses the human level. First, we compare the best scores got by different methods on Atari with DDQN. Table 6 shows the final score of each method. Imp-PER not only has higher data utilization but also is not less than the Org-PER algorithm in the final score. On the 7/10 of tasks, our final score was higher than Org-PER. On game Boxing, Robotank, and Tutankham, the score of our method is lower than Org-PER. We guess that TD-error is not a good estimation of priority on these tasks. If we use real TD as a priority, it will limit the training efficiency of DQN. However, it is just conjecture, and we will conduct in-depth research on this issue in the future. We also calculated the median and mean normalized scores to prove the performance of different methods on Atari with DDQN. Table 7 shows the result. Our method is better than both DDQN and Org-PER. Comparing to Org-PER, our method improve about 3% on the median and 15% on the mean normalized score. Next, we compare the best scores got by different methods on MuJoCo with DDPG. In these tasks, we don't have the human score. In this work, we only compare the best scores achieved by Random, \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Game**} & \multirow{2}{*}{**Uniform**} & \multirow{2}{*}{**Human**} & \multicolumn{4}{c}{**Raw Score**} & \multicolumn{4}{c}{**Normalized Score \%**} \\ \cline{3-8}  & & & **DDQN** & **Org-PER** & **Imp-PER** & **DDQN** & **Org-PER** & **Imp-PER** \\ \hline BankHeist & 21.7 & 644.5 & 728.3 & 870.6 & **951.6** & 113.4 & 136.3 & **149.3** \\ Bowling & 35.2 & **146.5** & 50.4 & 53.5 & 60.5 & 13.6 & 16.4 & **22.7** \\ Boxing & \(-1.5\) & 9.6 & 81.7 & **90.6** & 90.3 & 749.5 & **829.7** & 827.0 \\ Breakout & 1.7 & 31.8 & 450 & 420.5 & **451.3** & 1489.3 & 1391.3 & **1493.6** \\ FishingDerby & \(-77.1\) & **5.1** & 3.2 & 3.1 & 3.4 & 97.6 & 97.5 & **97.9** \\ Freeway & 0.1 & 25.6 & 28.8 & 32 & **32.2** & 112.5 & 125.0 & **125.8** \\ Pong & \(-20.7\) & 9.3 & **21** & 20.7 & **21** & **139** & 138 & **139** \\ RoadRunner & 11.5 & 7845 & 48,330.5 & 50,113 & **54,370.6** & 616.8 & 639.5 & **693.9** \\ Robotank & 2.2 & 11.9 & 47.8 & **53.7** & 52.1 & 470.1 & **530.9** & 514.4 \\ Tutankham & 12.7 & 138.3 & 125.7 & **207** & 190 & 89.9 & **154.6** & 141.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Final score and normalized score comparison of all methods on Atari. The best scores are shown in bold font. The uniform and human score are from deepmind [9]. Bold font shows the best performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metrics** & **DDQN** & **Org-PER** & **Imp-PER** \\ \hline Median & 113.4 & 138 & **141.1** \\ Mean & 389.2 & 405.9 & **420.5** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Median and mean normalized score of all games (%). Bold font shows the best performance. Vanilla-ER, Org-PER and Imp-PER. Table 8 shows the result. Our method Imp-PER gains about 51% policy improvement compare to Org-PER.