Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
Authors: Pierluca D'Oro
Max Schwarzer
Evgenii Nikishin
Pierre-Luc Bacon
Marc G Bellemare
Aaron Courville
Venue: ICLR 2023 notable top 5% (2023)
Keywords: reinforcement learning
sample efficiency
resets

Abstract: Increasing the replay ratio, the number of updates of an agent's parameters per environment interaction, is an appealing strategy for improving the sample efficiency of deep reinforcement learning algorithms. In this work, we show that fully or partially resetting the parameters of deep reinforcement learning agents causes better replay ratio scaling capabilities to emerge. We push the limits of the sample efficiency of carefully-modified algorithms by training them using an order of magnitude more updates than usual, significantly improving their performance in the Atari 100k and DeepMind Control Suite benchmarks. We then provide an analysis of the design choices required for favorable replay ratio scaling to be possible and discuss inherent limits and tradeoffs.

Published as a conference paper at ICLR 2023
SAMPLE-EFIIIEENT REINFORCEMENT LEARNING BY BREAKING THE REPLAY RATIO BARRIER
Pierluca D'Oro" Mila, Université de Montréal
Max Schwarzer'
Evgenii Nikishin
Google Brain
Mila, Université de Montréal
Mila, Université de Montréal
Pierre-Luc Bacon Mila, Université de Montréal
Marc G. Bellemare Google Brain, Mila
Aaron Courville
Mila, Université de Montréal
ABSTRACT
Increasing the replay ratio, the number of updates of agent's parameters per environment interaction, is an appealing strategy for improving the sample effi- ciency of deep reinforcement learning algorithms. In this work, we show that fully or partially resetting the parameters of deep reinforcement learning agents causes better replay ratio scaling capabilities emerge. We push the limits of the sample efficiency of carefully-modiffed algorithms by training them using an order of magnitude more updates than usual, significantly improving their performance in the Atari 100k and DeepMindd Control Suite benchmarks. We then provide an analysis of the design choices required for favorable replay ratio scaling be possible and discuss inherent limits and tradeofTs.
INTRODUCTION
In many real world scenarios, each interaction with the environment comes at a cost, and it is desirable for deep reinforcement learning (RL) algorithms to learn with minimal amount of samples (François-Lavee et al. 2018). This can be naturally achieved if an algorithm is able to leverage computational resources during training to improve its performance. Given the online nature deep RL, there is a peculiar way to aim at having such behavior: to train the agent for longer, given a dataset of experiences, before interacting with the environment again.
800
0.6
600
JQM
IQM
0.4
SR-SAC
SR-SPR
200
SAC
0.2
SPR
2
4
16
128
4
Replay Ratio
Replay Ratio
(a) DeepMind Control Suite (DMC15-500k)
(b) Atari 100k
Figure 1: Scaling behavior of SAC and SR-SAC in the DeepMind Control Suite (DMC15-500K) benchmark, and of SPR and SR-SPR in the Atari 100k benchmark seeds for point for SAC and SR-SAC, at least 20 seeds for point for SPR and SR-SPR, 95% bootstrapped C.I.).
"Equal contribution. Correspondance {pi frierl uca doro, schwarzm) @mi quebec.

---

Published as a conference paper at ICLR 2023
A method based on this idea can be said to be scaling the replay ratio, the number of updates of an agent's ' parameters for each environment interaction. Despite generally providing limited benefit when applied to standard baselines (Fedus et al. 2020 Kumar et al. 202I). replay ratio scaling has been shown to bring performance improvements to well-tuned a algorithms. Recent approaches were able to achieve better sample efficiency by increasing it to higher values, up to 8 for discrete control (Kielak, 2019))et or 20 for continuous control (Chen et al. 2021 Smith et al. 2022).
In this paper, we show that it is possible, with minimal but careful modifications
model-free algorithms mostly based on parameter resets (Ash & Adams| 2020; Nikishin et al. 2022), to reach
new levels of replay ratio scaling and push the sample efficiency limits of deep RL. Both continuous control, with SAC in DeepMind Control Suite (Haarnoja et al. 2018:) Tassa et al.] (0188) and discrete control, with SPR in Atari 100k (Schwarzer et al. 2021a] Kaiser et al. (020), we break the replay ratio barrier, unlocking a training regime in which orders of magnitude of additional agent updates can be used to increase the performance of an algorithm for a given budget of interactions with the environment. By doing so, we obtain better aggregated scores than strong baselines, with general blueprint to improve sample efficiency of potentially any off-policy deep RL algorithm.
To understand how this can be feasible, it is useful to reflect on one of the most common patterns observed in the development of deep RL algorithms (Mnih et al. 2015b). With a few exceptions, researchers typically ground their methods on the well-established dynamic programming mathemati- cal machinery, combining it with optimization strategies common in deep learning. However, the RL setting is inherently different from the one in which most deep learning architectures and optimization methods were developed. In deep RL, neural networks have to deal with dynamic datasets, whose composition changes over the course of training; their training actively determines the value of future inputs, but also the value of future targets. We argue that the recently identified tendency of neural networks to lose their ability learn and generalize from new information during training (Chaudhry et al. 2018 Ash & Adams 2020; Berariu et al., 2021: Igl et al., [221]] Dohare et al. 2022] Lyle et al. 2022a by Nikishin et al.] (022), against which most RL methods deploy no countermeasures, has
been the main roadblock in achieving better sample efficiency through replay ratio scaling.
After presenting and evaluating our algorithmic solution leading to better replay ratio scaling, we discuss some of the aspects of thinking about deep RL algorithms under the lens of this paradigm. We show some examples of algorithm design decisions important, or not important, for effective replay ratio scaling to be possible, with particular attention to the role of online interaction. Then, we visualize in an explicit way the computaaiins tradeoff implied by this approach and, after having shown the potential of replay ratio scaling, we discuss its inherent limits.
RELATED WORK
Loss of Ability to Learn and Generalize in Neural Networks A growing body of evidence suggests that artificial neural networks lose their ability to learn and generalize during training. The phenomenon is not clearly visible when learning with a static datasct on a fixed task, but it starts appearing when the data distribution changes. In the continual learning setting, alleviation of the problem by partially resetting the network parameters already provides a consistent improvement (Ash) & Adams 2020). Berariu et al. (021] provides in-depth study of how this phenomenon happens, including how many training updates are required for the performance of a network on future tasks to be unrecoverably damaged. The phenomenon becomes even more prominent in deep RL, where it has been identified in multiple settings. In the context of on-policy algorithms, it has been investigated as a consequence of transient non-stationarity and mitigated via self-distillation (Igl al. [02I]; ;; in
off-policy RL, it has been studied under the name of capacity loss (Lyle et al. 2022a), counteracted by the use of auxiliary tasks; in the sparse reward setting, it has been mitigated by post-training policy distillation (Lyle et al. 2022b). To address what they call loss of plasticity, /Dohare et al. (2022) proposes a variation of backpropagation compatible with continual learning, also applying it to the continual RL context. In this paper, we primarily leverage a periodic hard resetting method (Zhou et al. 2022), as investigated in Nikishin et al. 2022) to address the primacy bias phenomenon. Our work demonstrates that addressing this phenomenon allows for increased sample efficiency by scaling the replay ratio to much higher values than other model-free methods. We report in Appendix Ala more precise summary and glossary of the different related definitions from previous work.
2

---

Published as a conference paper at ICLR 2023
Scaling in Deep and Reinforcement Learning The topic of understanding and exploiting the scaling behavior of deep learning algorithm'ss 's performance with respect to the amount of resources used for training has recently gained attention. HHsstness et al. 2017) pioncered the idea of empirically studying and predicting performance when increasing a model 's size, and subsequent work investigated scaling
with respect to both model and dataset size, as well as training time (Kaplan al.] 2020 BBhhii et al.)
2021] Djolonga et al. (022]). Recent work in language modeling has also highlighted the importance of having high-quality data and the right training setup for efficient scaling be possible (Hoffmann) et al. 2022). In RL, scaling with respect to model size has been investigated in the offline setting for decision transformers (Lee et al., 2022) and with respect to planning -ime in model-based RL (Hamrick et al. 202T). For what concerns replay ratio scaling, moderately increasing the replay ratio for standard baselines has been shown to be a competitive data-efficient baseline for both discrete and continuous control when compared to model-based RI. methods (Holland et al. 2018]]
Van Hasselt et al. 2019 Kielak| 2019 D Oro & Jaskowski] 2020), despite clear limitations (Kumar
et al. 2021). Recent approaches in continuous control leveraged high replay ratios
strategy
to improve sample efficiency through the use of ensembles of value functions (Chen et al. 2021
Hiraoka et al., 2022 Wu et al. 2022) normalization strategies (Smith et al.) [022); we argue that explicitly alleviating the progressive loss of ability to learn and generalize pushes the replay ratio
scaling capabilities much further than those techniques can achieve.
EFFECTIVE REPLAY RATIO SCALING WITH RESETS
Most off-policy deep RL algorithms make of replay buffer Lin 1992 for storing transitions encountered over (a window of) an agent's 's lifespan. At fixed frequency, such methods sample a batch of transitions from the buffer, update the parameters of the agent by following the gradient of some loss function, and let the agent interact again with the environment before adding new experience to the buffer. The number of agent updates per environment step is usually called replay ratio' (Wang et al., 2016, Fedus et al. 2020), and most standard algorithms are trained with a value around 1 (Mnih et al.j 201 55, Haarnoja et al., 2018). It is natural to view increasing the replay ratio beyond these values as way improve sample efficiency. For ease of discussion, we now explicitly state and give a name to this idea, which has been an object of interest in previous studies (Van Hasselt
et al. 2019 Kumar et al. 1021
Replay Ratio Scaling
Change in an agent's performance caused by doing more updates for fixed number of environment interactions.
This definition does not have any positive connotation per se; any deep RL algorithm will have certain replay ratio scaling behavior, and a desirable property for an algorithm is to have particularly favorable replay ratio scaling, so that its performance can improve by increasing the replay ratio.
In contrast to other performance scaling properties analyzed for deep learning algorithms (Kaplan al., 2020), replay ratio scaling is intertwined with the online RL paradigm: if the agent has significantly better data-collection policy due to more training, the next collected sample will be potentially different with respect to the one collected if doing less training before the interaction; by this virtue, also future learning will be directly impacted by the presence of different data in the replay buffer. In other words, this type of scaling can only be understood by considering the interaction of an agent with an environment: training more on a small dataset of interactions, without any further collection of data, will eventually lead to challenges associated to off- oeflcolicy learning (Ostrovski et al., 2021); but training more while the data is collected can drastically change the stream of incoming data and the overall learning dynamics.
Given its appeal, what the limiting factors increasing the replay ratio? We argue that the main factor inhibiting effective replay ratio scaling in existing deep RL algorithms has been the progressive loss of the ability learn and generalize in neural networks (Dohare et al. 2022 Lyle et al. 2022b] Nikishin el al. 2022). It has been shown that this property hinders a neural network's performance under task switches (Ash & Adams 2020 Berariu et al. 2021) and, from the perspective
Reeatedd quantitics are also known as update-to-data (UTD) ratio CCen et al.
2021 Smith al.
2022
3

---

Published as a conference paper at ICLR 2023
SAC
DMC15-500k
800
REDQ
SR.SAC DOPG
Method
IQM
Median
Meun
SR-SAC
740 (642, 818)
(573, 742)
658 (589, 722)
500
REDQ
511 (440, 577)
493 (442, 544)
494 (452, 534)
10M
SAC
391 (334, 448)
424 (376, 468)
424 (386. 461)
400
DDPG
392 (334, 445)
410 (364, 454)
408 (371, 442)
DMC15-IM
200
Method
IQM
Median
Mcan
SR-SAC
805 (726, 867)
729 (628, 790)
710 (643, 775)
REDQ
(514, 649)
546 (490, 596)
539 (498, 576)
SAC
535 (467. 597)
525 (471,
519 (480, 557)
0.0
0.2
0.4
0.6
0.8
1.0
DDPG
514 (450, 572)
492 (440, 540)
489 (450, 526)
Environment Stope: (x10+)
Figure & Table 1: Performance of SR-SAC and of standard baselines on the DMC15 benchmark. (5 seeds for SR-SAC, 20 for all other algorithms, 95% bootstrapped C.I.).
of a neural network employed by the agent, what is deep RL if not long sequence of related but distinct tasks (Dabney et al. 2
Recent studies showed that, even under smooth task changes, the more training has been done on previous task, the worse the performance will eventually be in a new task (Ash & Adams 2020 Berariu et al. 202T). Since higher replay ratio correspond an increased amount of training. this gives a natural explanation to the limit in increasing it. The ability to learn and generalize can, however, be restored. For instance. Nikishin et al. (2022) periodically reset the network's parameters, with frequency that is fixed with respect the number of environment steps. In this work, argue that the key to surprisingly effective replay ratio scaling is a periodic restoration of the ability to learn and generalize of the network, via partial (Ash & Adams 2020) or total (Nikishin al.) 2022) resets of its parameters, with a reset frequency that only depends on the number of updates and thus implicitly also on the replay ratio. This means the more an algorithm updates its neural networks, the more frequent the restoration of its ability to learn and generalize will be, leading to better performance, as we now show in practice.
4 REPLAY RATIO SCALING DRASTICALLY IMPROVES SAMPLE EFFICIENCY
We apply two different reset strategies to two standard continuous control and discrete control algo- rithms and study their replay ratio scaling behavior. We consider Soft Actor-Critic (SAC) (Haarnoja jet al. 2018), which optimizes an actor and a critic by maximizing policy entropy alongside the environment's reward, and SPR (Schwarzer et al.] (021a). model-free DQN-based reinforcement learning algorithm that augments sample-efficient variant Rainbow (Van Hasselt et al. 2019) with model-based latent dynamics prediction objective designed improve representation learning in the low-data regime. The two curves in Figure I show that it is possible, with the same algorithm, to almost double the performance for the same number of environment steps, by just varying the replay ratio. We call the modified versions of these two algorithms Scaled-by- Bel-byyRReeettig SAC (SR-SAC) and Craled-by-Reseleee
SPR (SR-SPR). In the rest of this section, we are going describe the precise the details of the reset strategies that we employ for the two algorithms, as well as the benchmarks to which they are applied. by describing our decisions first in continuous control and then in discrete control. For evaluation and comparisons, we follow the protocol suggested by Agarwal et al. (221))
4.1 CONTINUOUS CONTROL
The DMC15 Benchmark To appropriately compare the performance of different algorithms, we consider a benchmark based on 15 environments from DeepMind Control Suite (Tassa et al. 2018). Our selection of tasks, reported in Table is a set for which discussing sample efficiency is sensible (i.e.. neither immediately solvable nor unsolvable by common deep RL algorithms). For ease of comparison, we specialize the benchmark to DMC15-500K, in which x 102 interactions with the environment are allowed, and DMC15-1M, in which 106 interactions are allowed.
4

---

Published as a conference paper at ICLR 2023
Reset Strategy We adapt the approach of Nikishin et al. (2022). and completely reset all the agent parameters every 2.56 x 10 of its updates. This lets us avoid individually specifying the moments at which resets should happen for different replay ratios. In terms of environment steps, resets will just occur more often at higher replay ratios. For instance, for replay ratio 128 (1 (128x higher than what typically used by SAC), reset occurs once every 20000 steps of interaction with the environment.
Results In Figure 2 we compare a version of SR-SAC that uses a replay ratio of 128 to standard deep RL baselines. This also includes the recently proposed REDQ (Chen al. 2021). which obtained state-of-the-arr sample efficiency by using a replay ratio of 20. At any budget of interactions with the environment, SR-SAC compares favorably with REDQ, despite being a simpler algorithm. SR-SAC establishes new state-of- thee--ffthhhaeeeeeee art result for model-free continuous control. Following (Agarwal et al. ) we focus on interquartile mean (IQM) performance, defined as the 25% trimmed mean performance over all runs on all considered tasks, and report 95% bootstrap confidence intervals.
4.2 ATARI 100K
Reset Strategy We follow Nikishin et al. (2022) in performing one reset every 40,000 updates; at replay ratio 16, the highest considered, this corresponds to a reset every 2.500 environment steps. or roughly once every three minutes of interaction. However, Nikishin et al. (2022) only reset subset of the agent's 's parameters when training on the ALE, leaving the agent's 's convolutional encoder untouched by resets. While this leaves the encoder vulnerable to plasticity loss, fully resetting the encoder impractical, as Nikishin et al. (2022) observe. As an intermediate solution, we apply soft resets, using variant of Shrink and Perturb (Ash & Adams 2020) in which encoder parameters are interpolated between their previous value and a random re- re-imitialized parameter vector on each reset: 8 = a0 (1 a) o initialize ier. This formulation is different from that used by (Ash) & Adams 2020) but allows easy interpolation between completely resetting a layer and leaving it
unchanged; we use 0.8 by default. We examine the impact of this decision in Section 5.2
Target Networks By default, SPR does not employ a separate target network, unlike traditional DQNs (Maih al.) 2015a).. However, we find that this leads replay ratio scaling stop improving performance at relatively low replay ratios, which we hypothesize is due to fundamental variance in optimization limiting the accuracy which the value function may be estimated. To alleviate it, we directly adopt the target strategy employed by SR-SAC, with an exponential moving average (EMA) target network with coefficient T 0.005, which we find allows beneficial replay ratio scaling out to at least replay ratio 16. Moreover, following (Ghavamzadeh et al.) 20.1). SR-SPR also uses its target network for action selection. We elaborate on this design decision in Section 5.2.
Results Figure shoows performance profiles of SR-SPR at various replay ratios, demonstrating that replay ratio scaling consistently improves performance up to at least replay ratio 16. We also compare a version of SR-SPR that uses replay ratio 16 to standard baselines (DrQ, DER, Kostrikov
1.00
SR-SPR: 16
SR-SPR: 8
Atari 100k
Frectoo of rans with with score St
SR.SPR: 4
Method
IQM
Median
Mean
0.75
SR.SPR: 2
SR.SPR:
SR-SPR
0.632 (0.60, 0.66)
0.685 (0.60, 0.77)
1.272 (1.18, 1.37)
SPR
IRIS
0.501 (0.44, 0.56)
0.289 (0.25, 0.41)
1.046 (0.96. 1.13)
SPR
0.380 (0.36, 0.39)
0.433 (0.38, 0.48)
0.578 (0.56. 0.60)
0.50
DrQ(c)
0.280 (0.27, 0.29)
0.304 (0.28, 0.33)
0.465 (0.46, 0.48)
DER
0.183 (0.18, 0.19)
0,191 (0.18. 0.21)
0.351 (0.34. 0.36)
0.25
SPR
IRIS
SR-SPR
0.40 0.48 0.56 0.64
0.30 0.45 0.60 0.75
0.75
1.00
1.25
1QM
Median
Mean
0.0
0.2
0.4
0.6
0.0
Normalized
Figure 3 & Table 2: Performance profiles (left, higher is better) of SR-SPR various replay ratios, and 95% C.I.s of SR-SPR: 16 and of standard baselines on Atari 100k (right, 20 seeds for SR-SPR and SPR, 5 seeds for IRIS, 100 seeds for all other algorithms as taken from Agarwal et al. 2021
5

---

Published as a conference paper at ICLR 2023
SAC
SR-SAC
Tandern SR-SAC
lerattd Offline SR-SAC
800
900
8.0 1
10M
6.1 x
400
200
2.2x10
2.5 x10
10P
10e 10F 300
10
109
10'
10'
10P
109
10'
10
10c 102 10'
10a
Number updates
Figure 4: Scaling behavior of SAC, SR-SAC and its tandem and iterated offline variations in the given number of environment
DMC15 benchmark. Each individual line shows performance steps, denoted by color, across different numbers of agent updates. Each point in a line is obtained by measuring performance with a different replay ratio for that number of environment steps. Each line is computed over 5 seeds.
et al. 2022 Van Hasselt et al., 2019) and recent work (IRIS, Micheli et al. 2022) table SR-SPR establishes new state-of-the-ane for model-free control on Atari 100k, and rivals prior work that has aggressively pretrained on additional data (Liu & Abbeel 2021 Schwarzer et al. 2021b). We present full results and per-game scores for SR-SPR in table 4 and show training curves in fig. [5] We report IQM performance, as well as plotting a performance profile (Agarwal et al.) 202T). which visualizes the full distribution of performance across all runs and demonstrates that increasing SR-SPR's replay ratio comprehensively improves performance.
5
ALGORITHM DESIGN IN LIGHT OF REPLAY RATIO SCALING
5.1 ANALYZING THE IMPORTANCE OF ONLINE INTERACTION
When training with high replay ratios and short reset intervals, the training regime an agent is subjected to begins to resemble offline RL; the agent is primarily learning from data collected by policies unrelated to own, with only a small amount of online data available to correct its policy. Given many classical analyses from offline RL (Levine al., 2020). it is perhaps surprising that an agent trained a pseudo-offline setting with no explicit regularization towards conservatism (e.g.. Kumar et al. 2020) can learn successfully. What is then the role of the incoming stream of interactions? To gain some understanding, in this section we attack the problem from different angles and study the scaling behavior of variants of SR-SAC. We consider different data collection patterns and how interleaving them with agent optimization changes the training dynamics. The appendix also presents comparisons with NFQI (Riedmiller 2005) and with the online use of an offline RL algorithm.
5.1.1 ITERATED OFFLINE SETTING
Changing the replay ratio in deep RL algorithm can be seen specific way of increasing the proportion of offline training an agent is subject to. Specifically, the agent's 's parameters are updated a number of times exactly equal the replay ratio before a new sample is collected. This implies uniform distribution of the number of offline updates across time steps. Is this an important variable for determining the replay ratio scaling behavior of an algorithm?
To answer this question, we resort to what we call iterated offline RL (Matsushima et al. 202 Riedmiller et al. 2021), which alternates between purely offline updates and data collection. this setting, a certain value of replay ratio is not distributed uniformly during the course of the interactions with the environment. Instead, the agent is not updated during data collection, and an amount of updates equal to the one that would be due in that data collection time frame in virtue of the replay ratio is applied completely offline, right after each reset.
As visible in Figure 4 the iterated offline paradigm has different replay ratio scaling behavior. Applying a very large number of updates with fixed dataset, with an algorithm such as SAC. incurs serious risk of generating degenerate policy, not able to outperform the previous one. As exemplified in Figure 6 in the absence of any mechanism for stopping this natural degeneration to happen, this circle is broken only when enough data is collected. Collecting enough data in this sense is possible
Broadly speaking, transposed and clipped plot of the cumulative distribution function of performance.

---

Published as a conference paper at ICLR 2023
saand
Tamansid stard
queduuped baak
2300
E Boums
752
800
1.00
292
250
0.05
0.10
0:35
0.20
0:00
0.65
0.15
0.20
0:00
0.05
0.10
6.35
0.20
0.00
0.07
0.10
0.35
020
Ensanomnan Skop (K1)
Davkoranca Sage (x10f)
Entraasen: Suge (X30")
Exvaveaaa Skap (X100
SRASA 122
SAG, RR-1
Tanson SRI-SA SAC, RB-128
bacasd Oftice 58- SAC RR-121
Figure 6: Examples of behaviors of SR-SAC and its tandem and iterated offline variations on four environments from DMC15, (5 runs, - std).
for easy tasks such as hopper-stand and walker-run, with a cost in sample efficiency, or impossible on hard tasks such as humanoid sanooodddaandd and quadruped-wall
This is unfortunate: the iterated offline RL paradigm can be quite useful in practical settings, in which the agent is allowed to only collected batches of data without any update (perhaps for safety reasons); however, current backbone algorithms (such as SAC) are not currently compatible with such a setting, that thus leads to favorable replay scaling only when closer to the online setting. This explains the sudden increase in the curve of Figure 4 when the number of agent updates, and consequently the reset frequency, becomes large enough. An interesting question, left for future work, is whether this behavior could change if combined with conservative algorithms created for the offline RL setting.
5.1.2 TANDEM SETTING
With high replay ratios, an agent's training begins to resemble offline RL: although the agent still has the possibility to interact with the environment, it is very infrequent relative to the amount of training. Thus, an agent after reset has a small stream of interactions collected by the agent itself, while the vast majority of its data was collected by potentially unrelated agents. How important, then, is this small stream of online interaction? To answer this question, we leverage the tandem setting, as presented in Ostrovski et al. (202T). Two copies of the same agent, identical apart from the initialization, are created. With the same algorithm (SR-SAC in this case), they are trained on the replay buffer collected by the active agent. The passive agent thus never directly interacts with environment, and cannot collect data to correct its own misconceptions about the environment.
As shown in Figure + the behavior of Tandem SR-SAC offers an alternative perspective on the importance of online interactions: de- spite the performance of the algorithm being hurt, the overall replay ratio scaling capabilities remain similar. We can look the perfor- mance of the passive agent to understand what the exact effect of on- line interactions is on training. As evident in the environments from Figure especially in hopper-ste and and quadruped- wall k, there is a qualitative difference between the behavior of an active agent (blue curve) and passive agent (green curve): right after a reset, with the initial high replay ratio training, the performance of both agents is greatly improved; after few thousands steps, training remains stable for the active agent but causes performance collapse in the passive agent. This experiment thus demonstrates the power of having online interactions as an implicit regularization mechanism.
0.8
SRSSPR: 16 Otfine
Training 10M
SR-SPR. 16
0.6
SPR:
0.4
0.2
0.0
40000
80000
Stop
SPR: 16
SR-SPR: 16 Offiine SRSPR:
0.30 0:45 0.60
Figure 5: Learning curves (top) and evaluation perfor- mance (bottom) at replay ratio 16 for SPR and SR-SPR with and without offline updates af- ter each reset.
For the design of future replay ratio-scalable algorithms, one should keep in mind that it is indeed possible to scale an algorithm poten- tially affected by extreme off-poolcyness: however, online data col- lection slows down performance collapse when training aggressively, as shown in both the iterated offline and the tandem experiments.
5.1.3 ALTERNATIVE COMBINATIONS OF OFFLINE AND ONLINE UPDATES
The iterated offline setting can be seen as the extreme in which all of the updates are done offline, compared to the even distribution used in the online setting. What if we use an intermediate strategy?
7

---

Published as a conference paper at ICLR 2023
For SR-SPR, find that directly mixing offline and online RL by performing half the updates allotted to each interval immediately after each reset can actually improve performance by some metrics, such training return (see Figure Secuur/Suuperr upper), by mitigating the performance drop otherwise experienced after each reset. Although we find that this has essentially no impact on final evaluation performance (Figure lower), it may allow SR-SPR be used when cumulative regret is important.
5.2 WHAT IS REQUIRED FOR REPLAY RATIO SCALING IN DISCRETE CONTROL?
Although replay ratio scaling is relatively straightforward for SR-SAC, achieving robust replay ratio scaling for SR-SPR requires more complex design decisions due to its shorter train-
0.65
ing period and more complex function approx- imation. As result, unlike SR-SAC, SR-SPR contains additional modifications from the vari-
0.60
of SPR used by Nikishin et (022). We study the impact of these design decisions on scaling behavior and report results in Figure
0.50
Inspired by the findings of Berariu et al. (2021)
10M
that plasticity loss concentrated in the final lay- ers of the network but affects all layers, we apply Shrink and Perturb (SP) to the encoder; this is responsible for roughly constant increase of IQM 0.04 past replay ratio 4. We note however that applying Shrink and Perturb alone all the parameters of the network is not sufficient to enable beneficial scaling; it is important that at least the network's ' final layers be completely reset. We explain this using the observations
0.45
0.40
0.35
Target Exploration
Online Targets
No SP, Online targets
Constant interval
from (Berariu et al. (02T) that the last layers are more responsible for the loss of plasticity.
0.25
SP Only
4 Replay
8
16
That said, the most important factor in allowing SR-SPE to continue scaling well its use of target network. This effect is primarily due to better action selection through the target net- work; we found that the stabilizing effect optimization was less important factor. This is reminiscent of speedy Q-learning (Ghavamzadeh et al.) 201T), where the use an exponential moving average policy was shown to improve convergence speed, and can also be understood in relationship to the policy churn phenomenon (Schaul et al. 2022) (see Figure [0 in the appendix).
Figure 7: The replay ratio scaling behavior of SR- SPR with various components ablated.
Mcanwhile, removing both Shrink and Perturb and the target network is roughly equivalent to taking the method Nikishin et al. 2022) but setting reset intervals as in SR-SPR. As Figure suggests, this alone suffices to yield some replay ratio scaling but not efficient compared to SR-SPR. However, maintaining a fixed reset interval (in terms of environment steps) when varying replay ratio, as done by Nikishin al. 2022). leads poor performance replay ratios above 4.
Intriguingly, we note that these modifications are beneficial specifically for replay ratio scaling; replay ratios significantly harm performance either). We thus hypothesize that there may be other modifications to complex algorithms such as SPR that could be made further improve replay ratio scaling properties, but that are today not in widespread use because they do not improve performance in standard low replay ratio settings.
they do not improve performance (although for the most part they do not
5.3 VISUALIZING THE DATA/COMPUTE TRADEOFF
an order of magnitude more of updates can be used for improving the performance of an algorithm, additional tradeoffs start emerge. The type of computations that replay ratio scaling implies fundamentally different than other concepts of scaling, (e.g.. about larger models): scaling here is inherently sequential. Thus, obtaining more hardware does not help faster execution of the algorithm.
8

---

Published as a conference paper at ICLR 2023
When collecting transitions is not very expensive, the choice between collecting samples in the environment and spending more time updating an agent could become nontrivial.
We visualize this tradeoff in Figure 8 The plot is obtained by combining runs of SR-SAC with doubling replay ratio from 0.25 to 128, and considering, for fixed data budget (in terms of environment steps), the total computational budget (in terms of total number of agent updates at that point), as well as the achieved performance. There exists multiple ways to achieve the same level of performance, as denoted by the color. This plot shows that resets provide a knob on replay ratio scaling and allows to tradeoff data for computation. If, for given problem, sample efficiency more important than computational considerations, one can spend about two orders of magnitude of additional agent updates to obtain the same performance that can be obtained by waiting for 800000 additional samples to be collected from the environment. The peculiar feature of the approach
10"
(Nuunber of agent updates)
700
Competational baigee
600
500
109
400
300
0.2
0.4
1.0
Data budget
1e6
(Enviroument steps)
Figure 8: Performance of SR-SAC in DMC15 as a function of the number of interactions and of the number of agent updates, determined by the replay ratio.
advocate for in this paper is that it allows to act on this tradeoff with an algorithm basically as simple the employed backbone.
THE LIMITS OF REPLAY RATIO SCALING
We have seen what becomes possible when higher level of replay ratio scaling are unlocked by resets. What are the limits of this paradigm? First of all, replay ratio scaling is always possible up to finite value, at which there is simply not enough information left to be extracted from the existing dataset of experience. Current methods, including the one proposed in this paper, are not able to automatically identify when this limit is reached, and they are therefore still subject to performance collapse when increasing the replay ratio too much. Second, replay ratio scaling cannot go beyond the intrinsic limitations of the given deep RL algorithm: for example, if the task is simply impossible exploration, then replay ratio scaling only of limited
to solve because of hard credit assignment help. Third, the strategy we proposed for replay ratio scaling is based on keeping the entire history of interactions with the environment in the replay buffer. While this feasible for the kind of sample- efficiency benchmarks that we have used in this paper, it might also require special consideration to be applied to larger problems; for instance, it is possible to keep large replay buffer on permanent storage, albeit at the cost of slower batch retrieval. Lastly, replay ratio scaling can inherently become time-consuming for training agent, which can limit the applicability of methodologies like ours to settings requiring high-frequency interactions with an environment.
CONCLUSIONS
In this paper, have shown that, by leveraging partial full resets of agent's parameters, is possible to unlock new levels of favorable replay ratio scaling and, consequently, of sample- efficiency for model-free deep RL algorithms. We demonstrated this by careful evaluation on the DeepMind Control Suite and Atari 100k benchmarks, where our approach (SR-SAC and SR- SPR) demonstrated far superior performance compared strong baselines, with minimal amounts of additional algorithmic complexity. Then, we discussed which algorithmic design choices are important for achieving such levels of replay ratio scaling with deep RL algorithm, as well as the tradeoffs implied by this paradigm. Through our empirical analysis, we showed the value of online data collection, offering a perspective on its relationship with offline RL (Levine et al. 2020).
More generally, this paper is about how to leverage discovery for the design of future deep RL. algorithms. We believe this work to be an example of how the development of effective deep RI. ones,
methods should be achieved not only through extending existing algorithms or creating but also through the discovery of new phenomena related to deep RL systems, and of techniques for exploiting them to increase performance. It is natural wonder whether deeper understanding or exploitation of surprising empirical properties (Ostrovski et al., 2021 Schaul et al. 2022) beyond the one behind this work could lead to the emergence of new capabilities in deep RL algorithms.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors thank Zhixuan Lin for adapting the REDQ baseline, Nathan U. Rahn, Rishabh Agarwal, David Yu- Yu-Tung Hui, Jesse Farebrother for insightful discussions and useful suggestions on the early draft, the Mila community for creating a stimulating research environment, Digital Research Alliance of Canada and Nvidia for computational resources. This work was partially supported by CIFAR, Samsung, Hitachi, Facebook Al Chair, Borealis, IVADO, and Gruppo Ermenegildo Zegna.
We acknowledge the Python community (Van Rossum & Drake Jr. 1995; Oliphant 2007) for developing the core set tools that enabled this work, including JAX (Bradbury et al. 2018 Babuschkin et al.) 2020), Jupyter (Kluyver et al. 2016), Matplotlib (Hunter 2007), numpy (Olihhant 2006 Van Der Walt et al. 2011). pandas (McKinney 2012), and Scily (Jones et al. 2014).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro. Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Thirty- Conference on Neural Information Processing Systems, 2021.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information Processing Systems, 33:3884-3894, 2020.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL
com/ htt.ccom////eeeee deepmind
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint 2:X1::2102.06701, 2021.
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2 202.00000022 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2 w.xXiv2208..0272 078.0728, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:://ithme httt:://ggtaubb.coo/ooggg////// com/google/a/a.....oooooo
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A research framework for deep reinforcement learning, CoRR, abs/1812.0611 2018.
URL http::tttt/xii...t/aa http//awwwv.rr..raa/a//// httpg//bb////1//// 06110
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprini arXiv:2101.05982. 2021.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The aalue-improvement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7160- 7168, 2021.
10

---

Published as a conference paper at ICLR 2023
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16458-16468, 2021.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. Continual backprop: Stochastic gra- dient descent with persistent randomness, 2022. URL http::/ss//oppeeeeeeeeeeett/e/es
Pierluca D'Oro and Wojciech Jaškowski. How to learn a useful critic? model-based action-gradient- estimator policy optimization. Advances in Neural Information Processing Systems, 33:3 2020.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. International Conference on Machine Learning, pp. 3061-3071. PMLR, 2020.
Vincent François-L Peter Henderson, Riashat Islam, Mare G Bellemare, Joelle Pineau, et al. An introduction deep reinforcement learning. Foundations and Trends@ in Machine Learning, 11 (3-4):219-354, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Mohammad Ghavamzadeh, Hilbert Kappen, Mohammad Azar, and Rémi Munos. Speedy q-learning. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Asso- ciates, Inc., 2011. URL
abl Ba440dddd44Ba2bba 1tt.:///444.22..7777777777444449
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Jessica B Hamrick, Abram L.. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither- spoon, Thomas Anthony, Lars Holger Buesing, Petar Veličković, and Theophane Weber. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations, 2021. URL httstss:::/ haattps::/openneeiieee net/ forum Saciumiidi IMMG44G26
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint 17Xiv:1712.00449,, 2017.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout f-functions for doubly efficient reinforcement learning. In International Confer- ence on Learning Representations, 2022. URL https::/p///.... h/tp:::ss///peeeiee.... www.eefoofriiiiommooo
XCVOMMPV3RT
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203.11555, 2022.
G Zacharias Holland, Erin Talvitie, and Michael Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv preprint arXiv: 1806.01825, 2018.
John Hunter. Matplotlib: A 2d graphics environment. IEEE Annals of the History of Computing, (03):90-95, 2007.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In ICLR, 2021.
Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python. 2014.
11

---

Published as a conference paper at ICLR 2023
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, K. Czechowski, D. Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, G. Tucker, and Henryk Michalewski. Model-base reinforcement learning for atari. ArXiv, 190000033,,7, 2020.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, Abs/2001.0836 2020.
Kacper Piotr Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? 2019.
Thomas Kluyver, Benjamin Ragan- Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter Notebooks-a publishing formaaa for reproducible computational workflows. volume 2016. 2016.
Hya Kostrikov. JAXRL: Implementattinns of Reinforcement Learning algorithms in JAX, 10 2021.
URL http:httpt//gttt// http:///....ooooootttooooooeccoe
com/
Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. ArXiv, abs/21 10.061669 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems. 33:1179-1191, 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parametterizaion inhibits data-efficient deep reinforcement learning. ArXiv, abs/2010.14498, 2021.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, L. Y. Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian S. Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. ArXiv, 0bbs/2055.12411 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement leammin Tutorial, review, and perspectives on open problems, 2020. URL htRl://pee//aaaa.orr///bb/////// 01643
Jjng-Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-6747 PMLR, 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. ArXiv, abs/2204.09560, 2022a.
Clare Lyle, Mark Rowland, Will Dabney, Marta Z. Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in reinforcement learning. ArXiv, abs/2206.02 126. 2022b.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2021.
Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O' O'Reilly Media, Inc.". 2012.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models, 2022. URL wttptttss///axxv.cooo www.://////.///o//a//..//eaeaeee
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540:529-553 2015a.
12

---

Published as a conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik. Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-5333 2015b.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In ICMI., 2022.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances Neural Information Processing Systems, 2021.
Martin Riedmiller. Neural fitted q iteration- first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Martin A. Riedmiller, Jost Tobias Springenberg, Roland Hafner, and Nicolas Manfred Otto Heess. Collect & infer- fresh look at data-efficient reinforcement learning. In CoRL, 2021.
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. arXiv preprini arXiv:2206.00733 2022.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR, 2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34: 14:12666-12699, 2021b.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. ArXiv, abs/2208.077 077660 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. ArXiv, abs/1801 0006000 2018.
Stefan Van Der Walt, Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in science & engineering, 13(2):22-30, 2011.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Guido Van Rossum and Fred Drake Python tutorial, volume 620. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv: (6:::611.000 1.1120224 2016.
Yanqiu Wu, Xinyue Chen, Che Wang, Yiming Zhang, Zijian Zhou, and Keith W. Ross. Aggressive learning with ensembles: Achieving both high sample efficiency and high asymptotic performance, 2022.
net www.:reeetffrooee forum? Now.mmiii-o-AAAAAAA :
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville, Fortuitous forgetting connectionist networks. In International Conference on Learning Representations, 2022. URL https://
revnreview net www./neettton/neeeeeeoo
zY.E
13

---

Published as a conference paper at ICLR 2023
Expression
Definition
Used In
[Phenomenon for which] "a warm-started network performs worse test samples than network trained the same data but with new random initialization"
from Warm-Starting
Damage
Ash &
Adams
2020
"A memory effect where these transient non -tatttttionaistes can permanently impact the latent representation and adversely affect generalisation performance"
Damage from
ct al.
2021
Non-Stations Stattiitinn
Lyle
2022h
Capacity Loss
"Reduced ability to fit new targets in deep neural networks"
Lyle ct a
2022a
Berari
al.
Loss of Plasticity
"Loss of the ability of the model to keep leamnin"
2021
Dohare
et al. (2022
Primacy Bias
"A tendeney to overfit initial experiences that damages the rest
Nikishin
et
of the learning process"
2022)
Table 3: Definitions of coinciding and related phenomena from previous work justifying the effective- ness of our strategy for replay ratio scaling.
RR:
RR-10
0.65
B2
0.60
620
100
0 0.45
420
0.30
0.25
5
Intevel
166
Reoee (nterna)
150203
(a) DeepMind Control Suite (DMC15-500K)
(b) Atari 100k
Figure 9: Sensitivity of the IQM to varying reset intervals (in terms of gradient updates) of SR-SAC on the DeepMind Control Suite (DMC15-500K) benchmark, and of SR-SPR on the Atari 100k benchmark. (10 seeds, 95% bootstrapped C.I.).
A DEFINITIONS FROM RELATED WORKS
To further clarify description of previous work from the related work section, we report in Table 3 definitions of the different terms used to refer to the loss of the ability to learn and generalize neural networks. Each definition directly taken from one of the papers corresponding despite their overlap, they reflect slightly different perspectives on the nature of this phenomenon, and it can be worth for future investigations to pin down which one of these more relevant for replay ratio scaling or reinforcement learning as a whole.
it. Note that,
B ADDITIONAL EXPERIMENTAL RESULTS
B. ADDITIONAL STUDIES
Reset Interval important hyperparameter for both SR-SAC and SR-SPR is the interval at which resets are performed, as denominated in terms of number of agent updates. In Figure we study how performance is impacted by this choice, at different replay ratios. Overall, both SR-SAC and SR-SPR perform well for a vast range of reset intervals, with favorable replay ratio scaling and generally smooth performance degradation. Note that, for large intervals (e.g., the last point on the right for SR-SAC with R.R 16, and last two points in the bottom right for SR-SPR), this is equivalent to actually performing no resets, just rumning the unmodified baseline algorithms. Thus, performance experiences non-smooth drops only in these easily avoidable cases.
14

---

Published as a conference paper at ICLR 2023
RR 16.0
RR 8.0
RR 4.0
RR 2.0
RR 1.0
0.06
0.04
0.25
Unlice Chum
0.03
0.02
0.10
0.01
0.05
0.00
20000
100000
40000
60000
80000
100000
Stop
Stup
0.65
0.7
ORtPatyyloa Iate
0.90
0.6
Online-Tugget
0.75
0.4
20000
40000
60000
80000
100000
20000
40000
6COCO
80000
100000
Stap
Stup
Figure 10: thlrnerelated diagnostics (based on the policy churn definition from Schaul (2022) for the online and target networks. Different colors and RR denote different values of replay ratio.
Counteracting Policy Churn by Acting with the Target Network Schaul et al. (2022) defined the policy churn the change the agent's policy due to optimization. It was shown that a certain amount of policy churn can be beneficial for exploration in the absence of external noise (e.g.. coming from e-greedy exploration); however, it intuitive that excessive churn can actually hurt performance, for instance by breaking the the consistency of trajectories. We hypothesize that mitigating excessive policy churn is a major reason why SR-SPR performs better when actions are selected with the target network rather than the online network. Figure 10 shows that, if we measure churn before each interaction with the environment, increasing the replay ratio will naturally increase it, while acting with the target network will decrease it. When the replay ratio is too high, it is likely that the benefits coming from additional offline computations might be nullified by the inconsistency in the exploration data; acting with the target network reduces these inconsistencies without giving up on the more efficient optimization, at the cost of introducing minimal delays in the improvement of the data-collecting policy.
Using an offfiie RL algorithm online In Section 5.1 we conducted set of experiments with the goal of highlighting the importance of online interactions, concluding that a consistent stream of online interaction data and neural networks able to learn and generalize from a dataset of experiences are key factors behind effective replay ratio scaling. Online RI. algorithms such as SAC are naturally reliant on the online stream of interactions; but it is natural to ask whether algorithms created for offline RL setting, where no interaction with the environment is assumed to be possible during training, can make the most out of the computational budget granted through high replay ratios in the online setting. To test this hypothesis, 2022) algorithm as an online RL algorithm on DMC15-500 with a replay ratio of 32. Results in Figure T1 show that, despite being by design more robust more aggressive training, the conservative
the Implicit Q-larrning (IQL) (Kostrikov et al.
15

---

Published as a conference paper at ICLR 2023
IQL
JQL + resels
DDPG
SR-SAC
REDQ
SAC
230
500
730
200
400
600
200
400
600
Figure 11: Performance on DMC15-500K of running IQL (RR=32) online, with and without resets.
0.7
0.6
0.5
Pretrwitsed SRsSPR Pretrwited SPR SPR
10M
0.4
8
Reploy Ratio
Figure 12:
The performance of SR-SPR and SPR from scratch and when fine-tuning pre-trained encoder on Atari 100k (10 seeds. 95% bootstrapped C.I.
nature of offline RL algorithms makes them not amenable to effective online learning, regardless of the presence of resets. This shows the effectiveness of online interactions as strong supervision mechanism, able, when supported by resets, make online RI. algorithms robust high replay ratios without the need of overly conservative behaviors.
B.2 FINETUNING PRETRAINED REPRESENTATIONS
Finetuning pretrained representations has become increasing common in reinforcement learning and elsewhere (Schwarzer al. 2021b: Liu & Abbeel 2021 Bommasani al., 2021). One obvious question to ask is whether or not replay scaling as demonstrated in the tabula rasa setting here can also be used make this finetuning more sample efficient. In Figure 12 we answer this question in the affirmative. We initialize SPR and SR-SPR with pretrained encoders (taken for experimental convenience from SPR agents trained at replay ratio for one million steps), and initialize all other parameters randomly. We then train at range of replay ratios for 100k steps. For SR-SPR, we apply shrink and perturb towards the pretrained encoder weights rather than random parameters, but otherwise train as normal.
We find that while both SPR and SR-SPR benefit from the pretrained representations at low replay ratios, only SR-SPR is able to improve fine-tuning performance by replay scaling. Standard SPR with pretrained representations rapidly degrades in performance as the replay ratio is increased, while the performance of SR-SPR steadily increases higher replay ratios. Although the gap between SR-SPR with and without pretraining closes somewhat at higher replay ratios, this to be expected, higher replay-ratio agents have more opportunities improve their representations without pretraining.
B.3 FINETUNING AFTER OFFLINE TRAINING
The efficiency of SR-SAC and SR-SPR makes the general approach behind their design potentially appealing for the setting of offline RI. with additional fine tuning phase. In Figure T3 we provide
16

---

Published as a conference paper at ICLR 2023
antmaze-umaze-v0
artmaz-umaare-diver---00
100
50
80
Nomelized Ream
Nomsire:
-05
6.0
-0.
0.0
0.5
1.0
Step
166
Sep
166
R0L
SR IQL
Figure 13: Performance of SR-IQL and IQL in two tasks from D4RI.. Negative steps denotes the pretraining phase (10 seeds, std).
1e7
IQM 600
(Number of agent updates)
SR-SAC
Tandem SR-SAC
Computational budget
SAC
8
Data budget
1e5
(Environment steps)
Figure 14: Pareto fronts for SR-SAC and Tandem SR-SAC DMC15 seeds).
preliminary evidence that the paradigm we advocated for this paper may indeed be particularly beneficial in this setting. We test IQL with the same pretraining scheme presented in Kostrikov al. (2022), consisting in million offline training steps followed by million interactions with the environment for fine tuning. We implement SR-IQL by using replay ratio of 10 and resetting, every two million updates, all of the parameters its neural networks during the fine tuning phase. We compare SR-IQL to IQL on two tasks from the D4RL benchmark (Fu et al. 2020 As shown Figure 13, SR-IQL is roughly on par with IQL the antmaze- metemaazz-azzeeee vv task, but reaches superior performance during fine tuning in antmaze- -uaaeu -ic ve We believe this sets the stage to experimenting with our replay ratio scaling paradigm in this setting as a promising research direction for future work.
B.4 PARETO FRONTS COMPARISON
With the same approach used for studying the data/computation tradeoffs of SR-SAC, it also becomes possible to directly compare the performance of different replay ratio-scalable algorithms. As simple example, we compare SR-SAC, its tandem version and SAC in Figure [4] The different lines are Pareto curves, obtained by retaining the points that are dominating the other ones in terms of either data computational budget, to reach IQM of at least 600. On this plot, SAC simply appears as a point because, not allowing for effective replay ratio scaling, it can only reach the prescribed performance by using data and relatively small amount of computational resources.
17

---

Published as a conference paper at ICLR 2023
B.5 COMPARISON WITH NEURAL FITTED Q-ITERATION
The approach we demonstrated for replay ratio scaling, for its relationship with offline RL and its use of resets, could resemble the classic NFQI algorithm (Riedmiller 2005), which train from scratch, after each large batch of transitions, Q-function. Our approach propagates information across resets mainly through the use of the replay buffer, having fast target network updated alongside the regular agent training; NFQI instead propagates information primarily through a target network, which is updated once per reset. We implement an-friendly variant of NFQI on SR-SPR at replay ratio 16, performing one target network update upon each reset. However, we find that this leads to very poor performance (IQM 0.350), achieving barely half that of standard SR-SPR. Although we hypothesized that 2,500 environment steps (the standard reset interval for SR-SPR at replay ratio 16) might be too infrequent for target network updates, making this interval shorter did not improve performance. Although we cannot rule out the possibility that NFQI might be competitive dramatically higher replay ratios, its inherent slowness in propagating information is likely to lead it lag in data-efficient settings; any reset interval that is sufficiently long to allow for accurate estimation of the value function may lead to insufficiently rapid value propagation via target updates, and vice versa.
C COMPUTATIONAL CONSIDERATIONS
For DMC, the running time depends the individual environment, due differences in dimen- sionality of the observation as well as physics simulation time. On an NVIDIA V100 GPU, at this highest replay ratio of RR=128, our code takes about 10.5 hours on acrobot-swingee 15 hours on bumanoid-run to complete 500k enviromment steps. For a replay ratio of RR=32, which yields remarkable, even not best, performance, the time goes down to just about hours and about hours respectively. which is well-below the typical demands of modern model-based RL methods. With careful seed parallelization, running SR-SAC with RR=32 for seeds for all tasks in the DMC15-500K benchmark takes less than 4 GPU/days running time depends primarily on the replay ratio chosen. At the highest replay ratio used (16) and with five seeds running in parallel, our code takes roughly 25 hours to complete 100k steps NVIDIA A100, yielding cost roughly 5 GPU/hours per training run.
and about
an NVIDIA V100. For Atari 100k,
D EXPERIMENTAL DETAILS
We report in Table 6 the full list tasks for the DMC15 benchmark. Our implementation of continuous control algorithms is based the jaxrl codebase (Kostrikov) 2021> For REDQ, use the best hyperparameters, as recommended by Chen et al. (021). )), as well as a replay ratio of 20. For discrete control, we use version of SPR implemented in Jax (Bradbury et al. 2018) in Dopamine (Castro et al. 2018). See Table 5
for a full list of the employed hyperparameters.
D.1 FULL EXPERIMENTAL RESULTS
In Figure T6 we show the scaling curve for DMC15-IM.
We report in Table the full per-game results for SR-SPR and in Figure 18 19 20 full experi- mental results for SR-SAC. For completeness, we also report the performance of the modified settings and of REDQ at the same replay ratios.
18

---

Published as a conference paper at ICLR 2023
Game
Random
Human
IRIS
SR-SPR:2
SR-SPR:4
SR-SPR:8
SR-SPR:16
Alien
227.8
7127.7
420.0
877.9
964.4
1015.5
1107.8
Amidar
1719.5
143.0
189.2
211.8
203.1
203.4
Assault
222.4
742.0
1524.4
891.9
987.3
1088.9
Asterix
210.0
8503.3
853.6
836.7
894.2
916.5
903.1
Bank Heist
753.1
53.1
253.6
460.0
472.3
531.7
Battle Zone
2360.0
37187.5
13074.0
14493.5
17800.6
19398.4
17671.0
Boxing
12.1
70.1
36.1
46.7
45.8
Breakout
30.5
83.7
24.5
28.8
25.5
Chopper Command
811.0
7387.8
1565.0
1609.4
1933.7
2201.0
2362.1
Crazy Climber
10780.5
35829.4
59324.2
28004.7
3834177
43122.3
45544.1
Demon Attuck
152.1
1971.0
2034.4
2969.0
3016.2
2898.1
2814.4
Freeway
29.6
31.1
24.1
24.9
25.4
Frostbite
65.2
4334.7
259.1
1450.4
1809.9
1752.8
2584.8
Gopher
257.6
2412.5
2236.1
735.3
717.5
711.2
712,4
Hero
1027.0
30826.4
7037.4
6832.1
7195.7
7679.6
8524.0
Jamesbond
29.0
302.8
462.7
412.9
408.8
392.8
389.1
Kangaroo
52.0
3035.0
1651.2
2024.1
3254.9
3631.7
Krull
1598.0
2665.5
6616.4
5206.4
5364.3
5824.8
5914.4
Kung Fu Master
258.5
22736.3
21759.8
14165.6
17656.5
17095.6
18649.4
Ms Pacman
307.3
6951.6
999.1
1472.6
1544.7
1522.6
1574.1
Pong
-20.7
14.6
14.6
-10.5
-.55
-3.0
2.9
Private Eye
24.9
69571.3
100.0
95.8
97.9
Qhert
163.9
13455.0
745.7
3431.7
3699.8
3850.6
4044.1
Road Runner
11.5
7845.0
9614.6
12199.0
14287.3
13623.5
13463.4
Seaquest
68.4
42054.7
661.3
714.7
766.6
800.5
819.0
Up N Down
533.4
11693.2
3546.2
61851.2
91435.2
95501.1
112450.3
Games Human
0
8
9
IQM (t)
0.000
1.000
0.501
0.444
0.544
0,589
0.632
Optimality Gap (L)
1.000
0.000
0.512
0.516
0.470
0.452
0,433
Median (t)
0,000
1.000
0.289
0.336
0.523
0.560
0,685
Mean (1)
0.000
1.000
1.046
0.910
1.111
1.188
1.272
Table 4: Full results for individual games in Atari 100k for SR-SPR various replay ratios.
SR-SPR:2 SPR.2
SR-SPR44 SPR.4
SR-SPR-8 SPR.8
SR-11 SPR.16
2.12
Figure 15: Learning curves for SR-SPR (solid) and SPR (dashed) at various replay ratios. Note that all SR-SPR runs converge to similar TD errors, gradient norms and parameter norms, while these metrics greatly differ for SPR at different replay ratios. IQM training performance does not match evaluation performance, ongoing training episodes are often disrupted by the reset procedure.
19

---

Published as a conference paper at ICLR 2023
Parameter
Setting
Gray-scaling
True
Observation down- sampling
84x84
Frames stacked
4
Action repetitions
4
Reward clipping
[-1, 1]
Terminal on loss of life
True
Max frames per episode
108K
Update
Distributional Q
Dueling
True
Support Q-distribution
51
Discount factor
0.99
Minibatch size
32
Optimizer
Adam
Parameter
Setting
Optimizer: learning rate
0.0001
Discount factor
0.99
Optimizer: B1
0.9
Minibatch size
256
Optimizer: B2
0.999
Optimizer (all)
Adam
Optimizer:
0.00015
Optimizer (all): learning rate
0.0003
Max gradient norm
10
Optimizer (all): 811
0.9
Priority exponent
0.5
Optimizer (all): A8
0.999
Priority correction Exploration
0.4 >
Optimizer (all):
0.00015
Noisy nets
Networks (all): activation
ReLU
Noisy nets parameter
0.5
Networks (all): n. hidden layers
2
Training steps
100K
Networks (all): hidden units
256
Evaluation trajectories
100
Initial Temperature
Min replay size for sampling
2000
Replay Buffer Size
10
Replay period every
step
Updates per step
Variable (1 to 128)
Updates step
Variable (1, 2, 4. 8, 16)
Target network update period
Multi-step return length
10
(EMA coefficient)
0.995
network: channels
32,
Reset Interval (gradient steps)
2560000
network: filter size
8 4 4, 3 3
Layers getting hard reset
All
Q network: stride
4, 2, 1
network:
hidden units
512
Mon-linearity
ReLU
Target network update period
À (SPR loss coefficient)
2
K (SPR prediction depth)
5
Data Augmentation
Shifts (#4 pixels)
Intensity(scalc=0005)
T (EMA coefficient)
0.995
Reset Interval (gradient steps) Layers getting hard reset Shrink and Perturb a Action selection
40,000
Final 2
0.8
Target network
Table 5: Hyperparameters for SR-SPR and SR-SAC, The ones introduced by this work are at the
bottom of the respective tables.
20

---

Published as a conference paper at ICLR 2023
Environment
Tasks
walker
run run, walk hard
quadruped
reacher
hunanoid
walk, stand swi mmimmer6
swimmer
cheetah
run hop, stand swingup swingup turn_hard swim
hopper
acrobot
pendulum
finger
fish
Table 6: The tasks from the DMCI5 benchmark. We chose commonly-cmployeer DMC tasks for which the optimal policy not immediately found by SAC according to btttps://github.co. com/ Bentlsyaarats/yyooorc._ssseseeee
800
600
IQM
400
SAC
200
SR-SAC
2
8
16
32
64
128
Replay Ratio
Figure 16: Scaling curve for SR-SAC and SAC on DMC15-IM.
213
Figure 17: Evaluation Returns on individual DMC15 environments for replay ratio 8.
21

---

Published as a conference paper at ICLR 2023
325
Figure 18: Evaluation Returns on individual DMC15 environments for replay ratio 16.
Figure 19: Evaluation Returns on individual DMC15 environments for replay ratio 32.
3.5
3.2
(1.77
Figure 20: Evaluation Returns on individual DMC15 environments for replay ratio 64.
22

---

Published as a conference paper at ICLR 2023
535
10 woven
3.5
3.20
Figure 21: Evaluation Returns on individual DMC15 environments for replay ratio 128.
23


Reviews:
[{"review_id": "lww1JxpdGg", "reviewer": "Reviewer_qf9b", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents that we can significantly improve the sample-efficient of prior deep RL approaches by increasing the number of updates per environment steps, and shows that resetting all parameters or part of parameters is critical. The paper shows improved performance on a variety of benchmark tasks, and provides interesting analysis and observations that can be related to the offline RL literature.", "summary_of_the_review": "The paper consists of an intuitive approach, nice explanation, strong performance, well supported claims, and interesting discussions. I recommend the paper to be accepted and has no major concerns.", "strength_and_weaknesses": "Strengths\n- Well written, intuitive approach\n- Strong performance with a very simple idea\n- Exhaustive experiments and interesting analysis and discussions\n\nWeaknesses\n- Given the recent surge of leveraging pre-trained representations for vision-based RL, it would be an interesting investigation to see what would happen when considering such a pre-trained perception module.\n- Investigating the role of replay ratio scaling for model-based approaches, especially the effect on the world models and corresponding policies obtained from the models, would be an interesting addition to the paper, but I don't think this is a necessary and could be a future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IW_1ejvU8M", "reviewer": "Reviewer_MFRL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Overall, the paper provides deeper analyses of the reset method proposed by Nikishin et al. [1].  \nThe paper under review shows that resetting the agent after every certain number of updates prevents performance degradation even at large replay ratio settings.  \nIn addition, several experiments and analyses that had not been done in [1] were made:  \n1. comparison with existing sample-efficient methods such as REDQ, 2. evaluation of the reset method in iterated offline and tandem settings, 3. analysis of implementation-level design decision in discrete control settings, and 4. evaluation of trade-offs between computational resources and performance.  \n\n[1] Nikishin, Evgenii and Schwarzer, Max and D'Oro, Pierluca and Bacon, Pierre-Luc and Courville, Aaron, The Primacy Bias in Deep Reinforcement Learning, Proceedings of the 39th International Conference on Machine Learning, 2022 \n", "summary_of_the_review": "Overall, I lean somewhat toward Accept.\nThe main weakness of the paper is the lack of novelty, i.e., the methods are not very different from the existing one, and the reset interval is still treated as a hyperparameter (and the analysis of how much it affects the results is not provided in the paper under review).\nOn the other hand, the various experiments conducted to gain a deeper understanding of the reset method are useful and this is the main strength of the paper. \nI think that the results/insights from these experiments will accelerate the development of the reset method in the future.\n", "strength_and_weaknesses": "**Strengths:**  \n[Clarity & Quality] The paper is basically well-written and easy to read.  \n[Novelty] Analyses to better understand the reset method [1] are conducted.  \n * In the original reset paper [1], the reset method was not compared to methods with a sample efficient method, such as REDQ. The experiments in the paper under review show that the reset method is generally more efficient than the existing sample-efficient method. \n * Experiments in tandem and iterated offline settings are conducted.  In addition, the effect of combining online updates with offline updates is evaluated. This combination approach addresses the temporal performance degradation of resetting, which was one of the main limitations of the reset method. \n * Implementation decisions for discrete control problems and the trade-off between computational resources and performance are also analysed. \n * The experiment results and findings should be beneficial especially to practitioners. Also, as a reset method for RL is a relatively new one and experiments about it have not been stacked so far, the experiment results provided in the  paper under review will be beneficial for a future advance of the reset method.) \n\n**Weaknesses**  \n[Novelty] Not so much progress has been made from [1].  \n * There is only a few technical difference from the original reset method: for continuous control, the proposed reset method is based on \"updates\" steps rather than \"environment steps\", and for discrete control, the Shrink and perturb style encoder update is used.  \n * As with [1], the reset interval is treated as a hyperparameter, \nand it is still unclear the optimal timing for reset (i.e., we still need hyper-parameter tuning for it).  \n\n[Clarity ] It is unclear how much performance is sensitive to the hyperparameter of the reset interval.  \n * For experiments for both discrete and continuous environments, reset intervals are set to magic numbers. No detailed discussion about how this hyperparameter affects overall performance is provided. This makes understanding how the insights observed in the experiments are generalizable a bit difficult. \n\n\n\n**Minor comments**  \n> Recent approaches in continuous control leveraged high replay ratios as a strategy to improve sample efficiency through the use of ensembles of value functions (Chen et al., 2021) or normalization strategies (Smith et al., 2022). \n\nThe following approaches also use ensemble and regularization in high replay ratio settings. \n\n1. Takuya Hiraoka and Takahisa Imagawa and Taisei Hashimoto and Takashi Onishi and Yoshimasa Tsuruoka, Dropout Q-Functions for Doubly Efficient Reinforcement Learning, International Conference on Learning Representations, 2022\n2. Wu, Yanqiu and Chen, Xinyue and Wang, Che and Zhang, Yiming and Zhou, Zijian and Ross, Keith W, Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance, arXiv preprint arXiv:2111.09159, 2021\n\n> To answer this question, we resort to what we call iterated offline RL, ...\n\nIs this the same setting as the following paper?\n\nMatsushima, Tatsuya and Furuta, Hiroki and Matsuo, Yutaka and Nachum, Ofir and Gu, Shixiang, Deployment-efficient reinforcement learning via model-based offline optimization, arXiv preprint arXiv:2006.03647, 2020\n\n\n> Replay Ratio Scaling: Change in an agent’s performance caused by doing more updates for a fixed number of environment interactions.\n\nRegarding SR-SAC, does replay ratio here mean, critic update ratio, actor update ratio, or both?  In the REDQ paper, replay (UTD) ratio means critic update ratio (actor is updated only once per environment step). \n\n\n> The progressive loss of ability to learn and generalize of neural networks and its interaction with RL was only one of the recent empirical discoveries about deep RL.  \n\nThis sounds like an exaggeration. Why \"only\" one? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Heor5-kdPw", "reviewer": "Reviewer_X4J7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors empirically study how combining replay ratio increases with network parameter resets can greatly improve sample efficiency of off-policy methods. Their experiments demonstrate superior performance on existing benchmarks with simple modifications to existing methods.", "summary_of_the_review": "I think this paper is useful and should be accepted to this conference, however, I have some suggestions for improvements with the experiments. I do think that the authors are missing one key set of analysis, which is why I am recommending a weak accept.", "strength_and_weaknesses": "## Strengths\n\n**Clarity:** The paper is written clearly and generally easy to understand.\n\n********************************************Framing/Contribution:******************************************** The paper’s contribution is clear to understand.\n\n**Experiments/Results:** The authors perform extensive experiments that are convincing of the usefulness of high replay ratios when combined with periodically fully/partially resetting network parameters.\n\n## Weaknesses\n\n**Clarity:** \n\n- In the related works section, first paragraph, many italicized terms are used but not explained. I was familiar with all terms except ************primacy bias************, which I had to look up separately while reading. But in general, these terms should be quickly defined to make the paper easier to read.\n- The first paragraph of 5.1.2 is written poorly and hard to read, please fix it.\n\n**Experiments:** \n\n- IQMs are listed (which is great) but the authors should probably explain IQM and the “Fraction of runs with score > $\\tau$” statistics for readers unfamiliar with the benchmarking paper that introduced these metrics.\n- While the provided analysis on different algorithm design choices in light of high replay ratios is useful, I’m not sure that what the authors analyzed is the most sensible thing to analyze. The paper is about replay ratio scaling through resetting network parameters. Therefore, there should be analysis on two things: 1) different replay ratios, and 2) how/when to reset network parameters. The authors provide analysis of 1) through trying different replay ratios, but insufficient analysis of 2) as in Section 4 they define their reset strategies and just stick with it. I think the paper really should have this analysis to be more useful as an empirical study to inform design choices for other researchers.\n- In combination with the experiments in Section 5, it would be useful to have an experiment with an offline RL algorithm (perhaps IQL or CQL). Offline RL is increasingly popular and also being deployed on real robots, so the authors should study their replay ratio increases with an offline algorithm to make the paper more useful for researchers.\n\n****************************Minor details:****************************\n\n- “The number of agent updates per environment step is usually called ************replay ratio************, and most standard algorithms were designed to have values around or below 1” → not sure if they were ********designed******** to have values around or below 1, I think this statement would be more accurate if it was something like “most standard algorithms are trained with a replay ratio around 1.”\n- “under tasks switches” → “under task switches”\n- “with a same algorithm” → “with the same algorithm”\n- “for evaluation and comparisons, we follow the protocol suggested by  Agarwal et a. (2021)” missing a period", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%