SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: Left: The rewards obtained after 50K episodes across 50 runs in the gridworld domain with an unlearnable point. Right: Number of times a transition containing the unlearnable state and its neighbors were sampled.
4.1.2 Mitigating forgetting with ReLo
To study how ReLo can help reduce forgetting in RL, we design a multi task version of the gridworld. We create a  $6 \times 6$  gridworld consisting of two rooms A and B and a goal state in each room. Task A is defined as starting in Room A and reaching the goal state in Room A, and Task B is defined similarly for Room B. There is a single gap in between the rooms, allowing the agent to explore both rooms but a time limit is introduced so that the agent can not reach both goals in one episode. During training, for the first 100k steps, the agent starts in Room A with access to Room B blocked. So during this stage, the agent observes only Task A. After 100K steps, the agent starts only in Room B, thereby no longer collecting data about Task A and must retain its ability on the task by replaying relevant transitions from the buffer. We train three agents, a baseline DQN agent, a PER agent and a ReLo agent and monitor performance on both tasks during training. We provide average success rates over 60 seeds after 1M steps in Table 1 and Appendix J. It shows how ReLo exhibits the least degradation in performance on Task A compared to the other baselines. From the training curves, we can see that there is a clear drop off in performance on Task A after 100K steps when the agent can no longer actively collect data on the task. However the ReLo agent exhibits least degradation in Task A while also outperforming the baseline and PER agent on Task B. This experiment clearly shows how ReLo helps the agent replaying relevant data points that could have been forgotten.
Table 1: Performance on Forgetting Task. Confidence intervals in brackets.
ALGORITHM	TASK A	TASK B
UNIFORM	0.43 (0.32, 0.54)	0.40 (0.29, 0.51)
PER	0.29 (0.19, 0.39)	0.26 (0.16, 0.36)
RELO	0.63 (0.52, 0.74)	0.74 (0.65, 0.84)
4.2 Comparison of PER and ReLo
We study the effectiveness of ReLo on several continuous and discrete control tasks. For continuous control, we evaluate on 9 environments from the DeepMind Control (DMC) benchmark [Tassa et al., 2018] as they present a variety of challenging robotic control tasks with high dimensional state and action spaces. We also include 3 environments from the OpenAI Gym benchmark for continuous control. For discrete control, we use the MinAtar suite [Young and Tian, 2019] which consists of visually simpler versions of games from the Arcade Learning Environment (ALE) [Bellemare et al., 2013]. The goal of MinAtar is to provide a benchmark that does not require the vast amounts of compute needed for the full ALE evaluation protocol, which usually involves training for 200M frames across 10 runs per game. This can be prohibitively expensive for researchers and thereby the MinAtar benchmark reduces the barriers present in studying deep RL research. We include scores on 24 games from the ALE benchmark for a reduced number of steps to observe if there are signs of improvement when using ReLo over PER. We provide full training curves for each environment in the supplementary material. We compare ReLo with LaBER [Lahire et al., 2021] on the DeepMind Control Suite and MinAtar benchmarks using the codebase released by the authors.
In addition to the per environment scores, we report metrics aggregated across environments based on recommendations from Agarwal et al. [2021] in Fig. 1. We can see that across a diverse set of
tasks and domains, ReLo outperforms PER as a prioritization scheme, with higher IQM and lower optimality gap scores. This highlights the generality of ReLo.
4.2.1 DMC
In the continuous control tasks, Soft Actor Critic (SAC) [Haarnoja et al., 2018] is used as the base off-policy algorithm to which we add ReLo. SAC has an online and an exponential moving average target  $Q$  network which we use to generate the ReLo priority term as given in Eq. 10. For comparison, we also include SAC with PER to showcase the differences in performance characteristics of PER and ReLo. The results are given in Figs. 1a. On 6 of the 9 environments, ReLo outperforms the baseline SAC as well as SAC with PER. This trend in performance is visible in the aggregated scores in Fig. 1a where ReLo has a higher IQM score along with a lower optimality gap when compared to SAC, SAC with PER and LaBER.
Additionally, to study the effect of stochastic dynamics in a non-gridworld setting, we conducted an experiment on stochastic versions of DMC environments. Specifically, we added noise sampled from  $\mathcal{N}(0,\sigma^2)$  to the environment rewards during training. During evaluation episodes, no noise is added to the reward. This is similar to the stochastic environments used by Kumar et al. [2020]. For Quadruped Run, Quadruped Walk, Walker Run we used  $\sigma = 0.1$ , and for Walker Walk we used a higher level of noise ( $\sigma = 1$ ) since there wasn't much change in the performance when using  $\sigma = 0.1$  compared to deterministic version. The results are presented in Fig. 3 and Table 2. These experiments highlight how ReLo outperforms uniform sampling and PER in stochastic dynamics while also attaining lower TD error and reinforce our claim that ReLo can effectively handle stochasticity in environments.
Figure 3: Performance on Stochastic DMC. Performance aggregated over 5 seeds.
Table 2: Validation TD Error on Stochastic DMC. Performance aggregated over 5 seeds.
QUADRUPED RUN	QUADRUPED WALK	WALKER RUN	WALKER WALK
UNIFORM	0.5 (0.34, 0.66)	1.27 (1.07, 1.47)	0.04 (0.04, 0.04)	1.98 (1.96, 2.0)
PER	5.22 (3.39, 7.05)	0.53 (0.47, 0.59)	0.06 (0.05, 0.07)	3.62 (3.47, 3.78)
RELo	0.19 (0.17, 0.2)	0.12 (0.11, 0.12)	0.03 (0.02, 0.03)	1.94 (1.92, 1.96)
4.2.2 OpenAI Gym Environments
In addition to the DeepMind Control Suite, we also evaluate ReLo on environments from the OpenAI Gym suite, namely HalfCheetah-v2, Walker2d-v2 and Hopper-v2 and the results are available in Fig. 1b. There is a general trend where PER leads to worse performance when compared to the baseline algorithm, in line with previous work by Wang and Ross [2019] that shows that the addition of PER to SAC hurts performance. However, this is not the case when using ReLo as a prioritization scheme. It is very clear how the PER degrades performance while ReLo does not adversely affect the learning ability of the agent and in fact leads to higher scores in each of the tested environments. We believe that the degraded performance of PER in the Gym environments is due to instability in training caused by rapidly varying value estimates. We provide experiments to back up this claim in Appendix L. We should that early terminations in the OpenAI Gym environments cause the PER agent to have the highest variance in the estimate of the fail state distribution. These results are presented in Table 3 When we disable early terminations, the PER agent is able to learn in these environments, though it is still worse than the ReLo agent.
Table 3: Variance in Value Estimate of Fail State Distribution.
METHOD	MEAN (CIs)	MIN	MAX
BASELINE	29.055 (-62.551, 120.662)	-87.8392	258.484
PER	149.982 (-76.382, 376.346)	-115.291	793.237
RELO	18.225 (-0.307, 36.756)	-14.4796	49.37
(a) DMC
Figure 4: Comparison of training TD error for PER and ReLo based sampling in a) DMC and b) Atari benchmarks. Calculated over 5 seeds.
(b) Atari
4.2.3 MinAtar
In the MinAtar benchmark, we use DQN [Mnih et al., 2015] as a baseline algorithm and compare its performance with PER and ReLo on the 5 environments in the benchmark. DQN does not have a moving average target  $Q$  network and instead performs a hard copy of the online network parameters to the target network at a fixed interval. Similar to the implementation of ReLo in SAC, we use the online and target  $Q$  network in the ReLo equation for calculating priorities. The results on the benchmark are given in Fig. 1c. PER performs poorly on Seaquest and SpaceInvaders, with scores lower than the baseline DQN. These results are consistent with observations by Obando-Ceron and Castro [2021] which analyzed the effect of the components of Rainbow in the MinAtar environment. In contrast, ReLo consistently outperforms PER and is comparable to or better than the baseline. Our previous observation that ReLo tends to help improve performance in situations where PER hurts performance is also true here.
4.2.4 ALE
As an additional test, we modified the Rainbow [Hessel et al., 2017] algorithm, which uses PER by default, to instead use ReLo as the prioritization scheme and compared it against Rainbow with PER on 24 environments from the ALE benchmark. Instead of the usual 200M frames of evaluation, we trained each agent for 2M frames to study if there are gains that can be observed in this compute-constrained setting. As shown in Fig. 1d, we see that Rainbow with ReLo achieves better performance than Rainbow with PER. These experiments show the versatility of ReLo as a prioritization scheme.
4.3 Analysis of TD Loss Minimization
To verify if using ReLo as a prioritization scheme leads to lower loss values during training, we logged the TD error of each agent over the course of training and these loss curves are presented in Fig. 4. As we can see, ReLo does indeed lead to lower TD errors, empirically validating our claims that using ReLo helps the algorithm focus on samples where the loss can be reduced. Another interesting point is that in Fig. 4a, SAC with PER has the highest reported TD errors throughout training. This is due to PER prioritizing data points with high TD error which might not be necessarily learnable. Data points with higher TD error are repeatedly sampled and thus making the overall
losses during training higher. In contrast, ReLo addresses this problem by sampling data points that are learnable and leads to the lowest TD errors during training.
Table 4: Comparison of validation TD Errors on DM Control Suite
BASELINE	PER	RELO
CHEETAHRUN	0.02 ± 0.002	0.03 ± 0.003	0.12 ± 0.033
QUADRUPEDRUN	0.62 ± 0.055	2.24 ± 0.127	0.35 ± 0.067
QUADRUPEDWALK	3.17 ± 0.252	2.11 ± 0.189	1.07 ± 0.146
WALKERRUN	0.08 ± 0.015	0.15 ± 0.018	0.06 ± 0.008
Table 5: Comparison of validation TD Errors on Atari
RAINBOW	RAINBOW+RELO
ALIEN	2.329 ± 0.408	2.331 ± 0.251
AMIDAR	2.157 ± 0.120	2.055 ± 0.143
BANK HEIST	2.044 ± 0.222	2.018 ± 0.261
JAMESBOND	1.653 ± 0.819	1.142 ± 0.174
Validation TD Error: We also compared the validation TD errors of each method after training in Tables 2, 4, and 5. This was done by collecting  $10^{4}$  frames from the environment and computing the mean TD errors of these transitions. These results show that the validation TD errors [Li et al., 2023] obtained at the evaluation phase are actually lower for ReLo. This analysis is very similar to observations in [Li et al., 2023] where the authors find that lower validation TD error is a reliable indicator of good sample efficiency for off-policy RL algorithms. ReLo achieves lower validation TD error compared to uniform sampling and PER on DM Control Suite and Atari. When we study the correlation between validation TD error and performance in Appendix 11 and observe a strong correlation between the two metrics, confirming the findings of Li et al. [2023]. These results highlight another crucial reason for the robustness and good performance of ReLo across multiple domains.
In this paper, we have proposed a new prioritization scheme for experience replay, Reducible Loss (ReLo), which is based on the principle of frequently sampling data points that have the potential for loss reduction. We obtain a measure of the reducible loss through the difference in loss of the online model and a hold-out model on a data point. In practice, we use the target network in  $Q$  value methods as a proxy for a hold-out model.
ReLo avoids the pitfalls that come with naively sampling points based only on the magnitude of the loss since having a high loss does not imply that the data point is actually learnable. While alleviating this issue, ReLo retains the positive aspects of PER, thereby improving the performance of deep RL algorithms. It is very simple to implement, requiring just the addition of a few lines of code to PER, and similar to PER it can be applied to any off-policy algorithm. Since it requires only one additional forward pass through the target network, the computational cost of ReLo is minimal, and there is very little overhead in integrating it into an algorithm.
While the reducible loss can be intuitively reasoned about and tested empirically, future work should theoretically analyze the sampling differences between ReLo and PER about the kind of samples that they tend to prioritize or ignore. This deeper insight would allow us to find flaws in how we approach non-uniform sampling in deep RL algorithms similar to work done in Fujimoto et al. [2020]. It would provide an analysis of the change in learning dynamics induced by PER and ReLo.
The authors would like to thank the Digital Research Alliance of Canada for compute resources, NSERC, Google and CIFAR for research funding. We would like to thank Nathan Rahn and Pierluca D'Oro for their feedback and comments. We also thank Nanda Krishna, Jithendarraa Subramanian, Vincent Michalski, Soma Karthik and Kargil Mishra for reviewing early versions of this paper.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679-684, 1957. URL http://www.jstor.org/stable/24900506.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rywHCPkAW.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html.
Scott Fujimoto, David Meger, and Doina Precup. An equivalence between loss functions and nonuniform sampling in experience replay. Advances in neural information processing systems, 33: 14219-14230, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018. URL http://proceedings.mlr.press/v80/haarnoja18b.html.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298, 2017. URL http:// arxiv.org/abs/1710.02298.
Zhang-Wei Hong, Tao Chen, Yen-Chen Lin, J. Pajarinen, and Pulkit Agrawal. Topological experience replay. *ICLR*, 2022. doi: 10.48550/arXiv.2203.15845.
Yuenan Hou, Lifeng Liu, Qing Wei, Xudong Xu, and Chunlin Chen. A novel ddpg method with prioritized experience replay. In Systems, Man, and Cybernetics (SMC), 2017 IEEE International Conference on, pages 316-321. IEEE, 2017.
Niel Teng Hu, Xinyu Hu, Rosanne Liu, Sara Hooker, and Jason Yosinski. When does loss-based prioritization fail? arXiv preprint arXiv: Arxiv-2107.07741, 2021.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140-1144, 2018. doi: 10.1126/science.aar6404. URL https://www.science.org/doi/abs/10.1126/science.aar6404.
Samarth Sinha, Jiaming Song, Animesh Garg, and S. Ermon. Experience replay with likelihood-free importance weights. L4DC, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint arXiv: Arxiv-1801.00690, 2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Dale Schuurmans and Michael P. Wellman, editors, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100. AAAI Press, 2016. URL http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12389.
Che Wang and Keith Ross. Boosting soft actor-critic: Emphasizing recent experience without forgetting the past. arXiv preprint arXiv: Arxiv-1906.04009, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. *Dueling network architectures for deep reinforcement learning*. In *Maria-Florina Balcan and Kilian Q. Weinberger*, editors, *Proceedings of the 33nd International Conference on Machine Learning*, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1995-2003. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/wangf16.html.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https://github.com/denisjarats/pytorch_sac, 2020.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176, 2019.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaii, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
We build our experiments on top of existing implementations of SAC, DQN and Rainbow. For the DeepMind Control Suite experiments, we modify Yarats and Kostrikov [2020], adding a prioritized replay buffer and the ReLo version. We use an open source implementation of Rainbow² for the Arcade Learning Environment and the DQN implementation from the MinAtar authors Young and Tian [2019]. Aside from the collected frames and number of seeds, we have not modified any of the hyper-parameters from these original implementations. The hyper-parameters as well as hardware and software used are given in Table 6.
Table 6: Hyper-Parameters of all experiments
Environments	Algorithm	Algorithm Parameters	Hardware & Software
ALE	Rainbow	Frames = 2 × 10^6 seeds = 5	Hardware-
CPU: 6 Intel Gold 6148 Skylake
GPU: 1 NVidia V100
RAM: 32 GB
Remaining hyper-parameters same as Hessel et al. [2017]	Software-
Pytorch: 1.10.0
Python: 3.8
DeepMind Control Suite	SAC	Frames = 1 × 10^6 seeds = 5	Hardware-
CPU: 6 Intel Gold 6148 Skylake
GPU: 1 NVidia V100
RAM: 32 GB
Remaining hyper-parameters same as Haarnoja et al. [2018]	Software-
Pytorch: 1.10.0
Python: 3.8
MinAtar	DQN	Frames = 5 × 10^6 seeds = 5	Hardware-
CPU: 6 Intel Gold 6148 Skylake
GPU: 1 NVidia V100
RAM: 32 GB
Remaining hyper-parameters same as Mnih et al. [2015]	Software-
Pytorch: 1.10.0
Python: 3.8
Figure 5: Visualization of a few environments from each benchmark. Left to right: DeepMind Control Suite, MinAtar, Arcade Learning Environment
B Mapping Functions for ReLo
Prioritized experience replay buffers expect the priorities assigned to data points to be non-negative. While the MSE version of the TD error used in PER satisfies this constraint, ReLo does not. Therefore, there must be a non-negative, monotonically increasing mapping from ReLo to  $p_i$ . In our main experiments, we clipped negative ReLo values to zero. Another mapping we tried was to set  $p_i = e^{ReLo}$ , in which case the probability of sampling a data point  $P_i$ , from Eq. 6, corresponds to the softmax over ReLo scores. However, for this choice the priority would explode if the ReLo crossed
$^{2}$ https://github.com/Kaixhin/Rainbow
Figure 6: Comparison of different mapping functions from ReLo to  $p_i$  on a subset of environments from the DMC benchmark. Performance is evaluated over 3 seeds.
(b) OpenAI Gym
Figure 7: Sensitivity of ReLo and PER to  $\alpha$  and  $\beta$ .
values above 40 which happened occasionally during the initial stages of learning in Rainbow. The second mapping function candidate was exponential when ReLo is negative and linear otherwise, that is,
$$
f _ {E x p L i n e a r} = \left\{ \begin{array}{l l} e ^ {\operatorname {R e L o}} & \text {i f R e L o <   0} \\ \operatorname {R e L o} + 1 & \text {o t h e r w i s e} \end{array} \right.
$$
The linear part is shifted so that the mapping is smooth around  $\mathrm{ReLo} = 0$ . As shown in Fig. 6,  $f_{\mathrm{ExpLinear}}$  performs worse compared to just clipping negative ReLo values to zero. When the ReLo values during training are analysed, we observe that the average of ReLo values (before the mapping) tends to be positive, so clipping does not lead to a large loss in information.
We used the default hyperparameters that were given by Schaul et al. [2016], i.e  $\alpha = 0.5$ ,  $\beta = 0.4$  for all benchmarks for PER and ReLo. Rainbow [Hessel et al., 2017] used these values for the ALE benchmark after extensive grid search. Obando-Ceron and Castro [2021], which studied DQN and Rainbow on the MinAtar benchmark, used the same values for  $\alpha$  and  $\beta$  after hyperparameter tuning, hence we reused these values in our experiments. These values transferred well to the DMC domain and so we used them for all continuous control experiments too. We performed a hyperparameter sweep over  $\alpha$  and  $\beta$  for DMC and OpenAI gym and the results can be found in Fig 7. ReLo consistently outperforms PER in these sweeps highlighting the robustness of ReLo across benchmarks. These sweeps also show that the performance of ReLo varied with changing the value of  $\alpha$  which implies that the priorities assigned by ReLo are not close to zero and hence the sampling behavior of ReLo is not uniform. If the priorities were close to zero, then performance would not change when we vary the value of  $\alpha$  since Eq. 6 would then reduce to  $p_{uniform}$  regardless of the value of  $\alpha$ .
D.1 Prioritization Schemes
Prioritization strategies have been leading to important improvements in sample efficiency. Sinha et al. [2020] proposes an approach that re-weights experiences based on their likelihood under the
stationary distribution of the current policy in order to ensure small approximation errors on the value function of recurring seen states. Lahire et al. [2021] introduces the Large Batch Experience Replay (LaBER) to overcome the issue of the outdated priorities of PER and its hyperparameter sensitivity by employing an importance sampling view for estimating gradients. LaBER first samples a large batch from the replay buffer then computes the gradient norms and finally down-samples it to a mini-batch of smaller size according to a priority.