# Self-Adaptive Priority Correction for Prioritized Experience Replay
## 4 Experiments
### Performance Comparison
#### 4.2.3 Data Utilization Comparison  
The data utilization is defined as the number of training data to reach a fixed score. Each game is run five times using different random seeds until DDQN or DDPG convergence. The mean score is used, and the shaded area represents a mean \(+/-\) standard deviation. Figure 10(a) shows the score with training steps on Pong (DDQN). The x-axis is the training step, which also represents data utilization. The y-axis represents the mean score for the recent 100 episodes. Our Imp-PER has the same data utilization as ATDC-PER, which is better than the Org-PER. The data utilization is improved by about 30% on game Pong. Figure 10(b) shows the training curves with the time cost. The x-axis represents the time (seconds). From Section 4.2.2, ATDC-PER is \(10\times\) slower than Org-PER and Imp-PER is 10% slower than Org-PER. Figure 10(b) shows the same results as our analysis. Although Imp-PER is slower than Org-PER in each iteration, we use less training time to convergence because of higher data utilization. The same result on game Breakout is shown in Figure 10(c),d.  
The results of the other eight Atari games are shown in Figure 11. We ignore the ATDC-PER in the other games and only compare Imp-PER to Org-PER. The first column and third column are \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Method** & **Total** & **Sampling** & **Update** & **DDQN** \\ \hline Org-PER & 0.014 & 0.005 & 0.0028 & 0.006 \\ ATDC-PER & 0.150 & 0.075 & 0.0028 & 0.006 \\ Imp-PER & 0.015 & 0.005 & 0.0029 & 0.007 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The time cost in each algorithm (s) on Atari with DDQN. The total time also includes other processes which are not the main factors. data utilization on each game. Correspondingly, the second and fourth column is time spent on each game. In most games, our Imp-PER has higher data utilization than Org-PER. From the results, we can see that Imp-PER has almost no additional time overhead, which reduces the whole training time on most games. In different stages of training, our algorithm has different degrees of data utilization improvement compared to Org-PER. Imp-PER has better improvement during the early stages. Table 5 shows the average scores achieved by Org-PER and Imp-PER in different training stages (\(0.50\times 10^{7}\), \(1.00\times 10^{7}\), and \(1.50\times 10^{7}\), steps respectively). In order to Intuitively show the performance improvement of each stage, we estimate the relative improvement of scores. Figure 13 shows the results. Each stage is represented by a different color. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Game**} & \multicolumn{4}{c}{**Org-PER**} & \multicolumn{4}{c}{**Imp-PER**} \\ \cline{2-7}  & \(\mathbf{0.50\times 10^{7}}\) & \(\mathbf{1.00\times 10^{7}}\) & \(\mathbf{1.50\times 10^{7}}\) & \(\mathbf{0.50\times 10^{7}}\) & \(\mathbf{1.00\times 10^{7}}\) & \(\mathbf{1.50\times 10^{7}}\) \\ \hline Pong & \(-12.1\) & \(13.5\) & \(18.6\) & \(\mathbf{7.3}\) & \(\mathbf{16.6}\) & \(\mathbf{19}\) \\ BankHeist & \(151.8\) & \(279.2\) & \(400\) & \(\mathbf{178.2}\) & \(\mathbf{332}\) & \(\mathbf{466.6}\) \\ Boxing & \(11.9\) & \(34.6\) & \(55.3\) & \(\mathbf{17.3}\) & \(\mathbf{38}\) & \(\mathbf{60}\) \\ Breakout & \(10\) & \(63.27\) & \(174.9\) & \(\mathbf{16.65}\) & \(\mathbf{93.24}\) & \(\mathbf{249.95}\) \\ FishingDerby & \(-87.15\) & \(-64.29\) & \(-20\) & \(-77.85\) & \(-40\) & \(-7.86\) \\ Freeway & \(8.3\) & \(13.57\) & \(19.28\) & \(\mathbf{8.33}\) & \(\mathbf{18.6}\) & \(\mathbf{25}\) \\ RoadRunner & \(2000\) & \(18,400\) & \(26,800\) & \(\mathbf{4000}\) & \(\mathbf{21},\mathbf{200}\) & \(\mathbf{30},\mathbf{000}\) \\ Robotank & \(3.64\) & \(7.3\) & \(13.1\) & \(\mathbf{4.62}\) & \(\mathbf{10.4}\) & \(\mathbf{21.5}\) \\ Tutankham & \(187\) & \(\mathbf{183}\) & \(\mathbf{187}\) & \(\mathbf{229}\) & \(166.1\) & \(156.4\) \\ Bowling & \(\mathbf{25}\) & \(\mathbf{22}\) & \(20.3\) & \(\mathbf{25}\) & \(20.5\) & \(\mathbf{24.5}\) \\ \hline \hline \end{tabular}
\end{table}