Replay across Experiments: A Natural Extension of Off-Policy RL
Authors: Dhruva Tirumala
Thomas Lampe
Jose Enrique Chen
Tuomas Haarnoja
Sandy Huang
Guy Lever
Ben Moran
Tim Hertweck
Leonard Hasenclever
Martin Riedmiller
Nicolas Heess
Markus Wulfmeier
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
robotics
data transfer
Primary area: reinforcement learning

Abstract: Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL).
We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times.
At its core, Replay across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. 
We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation, including hard exploration tasks from egocentric vision. 
Through comprehensive ablations, we demonstrate  robustness to the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.

Published as a conference paper at ICLR 2024
REPLAY ACROSS EXPERIMENTS: A NATURAL EXTENSION OF OFF-POLICY RL
Dhruva Tirumala 1.2
Thomas Lampe*
Jose Enrique Chen
Tuomas Haarnoja*
Sandy Huang'
Guy Lever
Ben Moran 1
Tim Hertweck 1
Leonard Hasenclever*
Martin Riedmiller*
Nicolas Heess!
Markus Wulfmeier!
'Google DeepMind
Universiiy College London (UCL)
ABSTRACT
Replaying data is a principal mechanism underlying the stability and data effi- ciency of off-policy reinforcement learning (RL). We present an effecctvve yet sim- ple framework extend the use of replays across multiple experiments, minimally controller performance
adapting the RI workflow for sizeable improvements and research iteration times. At its core, Replay across Experiments (RaE) in- volves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. We empirically show benefits across number of RL algorithms and challenging control domains spanning both locomotion and manipulation, in- cluding hard exploration tasks from egocentric vision. Through comprehensive ablations, we demonstrate robustness the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.
INTRODUCTION
In the last few years, reinforcement learning (RL) has transitioned from topic of academic study to practical tool for the generation of controllers across various real-world applications (Degrave) let al. [2022, Bellemare et al. 20200 Lazic et al., 2018; Lichtlé et al. 2022). These advancements have been enabled, in part, by recent advancements in algorithms improving data efficiency, bustness, and controller performance (Haarnoja et al. 2019 Lillicrap et al.) 2016 [Schwarzer al., 2023). Nevertheless, many problems remain hard to solve with RL. For instance, high-dimensional and partial observations (e.g. egocentric camera views), high-dimensional action spaces, reward functions that provide inadequate learning signal can all lead to poor asymptotic performance, high variance, low data efficiency, and long training times.
major challenge in RL remains the interaction between data collection and learning. Unlike in the case of supervised learning where all data is available and static, in online RL both the quality and quantity of data available for policy value function learning changes over the course of an experiment. The difficulty of collecting suitable data, and the complex interactions between data collection and function optimization leads to various problems including failure to learn, instabil- ities, and the premature convergence of function approximators early in learning (Nikishin et al. 2022 Ash & Adams 2020).
Experience replay LLn 1992) has become a popular component of most, modern off-policy RL algorithms. Storing data collected over the course of experiment and continuously training policy and value functions with this growing dataset can greatly increase data efficiency and stability of RL algorithms. It does not, however, address issues around premature convergence of the function approximators (Ash & Adams 2020), does it directly provide solution to reusing data from prior experiments. In this paper, we investigate the possibility of extending the use of replay and
"Correspondence be directed to: [dhruvat, mwulfmeier [anlfmmeier]attgggge..oomm [at]goggle.com

---

Published as a conference paper at ICLR 2024
reusing data in an iterative setting. Our main insight is that minimal change to the RL workflow can greatly improve the asymptotic performance of off off-po reinforcement learning algorithms. By reusing interaction data from prior training runs it can further reduce the overall experiment time and thus speed up research iterations.
number of studies have previously investigated how prior data (including expert data) can be used to kick-start RL training Vecerik et al. 2017 Singh et al. 2020b Nair et al. 22200] Walke et al. 2022 Ball al. 2023). We find that the simplest approach, mixing prior and online data with a fixed ratio, is particularly effective across a wide range of application scenarios and algorithms.
Our method dubbed Replay across Experiments (RaF) performs as well or better than alterna- tive approaches with fewer hyperparameter choices to be made. Furthermore, for domains that normally hard to solve even with state-of tfate-of-the-an art algorithms, we find that can be advantageous to perform multiple training iterations giving each iteration access to all data from prior training runs, effectively realising a minimalist perspective lifelong learning. We hope that RaE's simplicity will enable straightforward integration into existing infrastructure and help improve the efficiency of RL workflows.
The main contributions of our work are as follows:
Introduce and empirically validate simple strategy to solve challenging tasks. Demon- strate state-of-the art performance on number of domains including when reusing data from the publicly available offline RL Unplugged benchmark (Gulcehre et al. 2020).
Demonstrate that our approach works across multiple algorithms including DMPO, D4PG, CRR and SAC-X.
Compare RaE to common baselines and recent state-of-the-un methods. Highlight the combination of factors that leads to the effectiveness of our approach.
Provide additional ablations highlight the robustness of RaE quantity and quality of data collected and prrre--arameters.
2 METHOD
2.1 BACKGROUND
We consider the reinforcement learning (RL) problem in which agent observes the environment, takes an action, the environment changes its state and the agent receives a reward in response. The environment is modeled as a Markov Decision Process (MDP) consisting of the state space S, the action space A, the transition probability p($++1 18,, az) and reward r r(se, at) when taking action a; in state st. The agent's behavior is specified using a deep neural network policy π (a St; 0) with parameters 0; which we will denote (a, (8) for brevity.
We optimize the agent to maximize the sum of discounted future rewards, as denoted by:
( (pp.ee))seeessssee.aaa......
Σ
where > E 1] is the discount factor, re = (St, at) is the reward and po (8o) is the initial state distribution.
Given a policy æ the state-action value function, critic, Q(st, ar) is defined the expected discounted return when taking an action a in state 8t and then following the policy.
Q(sr, ax) = r(,,, + 7 (ePpeeesleeaas)attee (Q (8)+1,4))
Modern off-policy algorithms (Haarnoja et al. 2019; Barth-Maron et al. 2018 Abdolmaleki et al, 2018) usually operate through simulraneous alternating optimization of policy and state-action value function. In this context, experience replay (Lin 1992) is a fundamental mechanism decou- ple data collection from policy and value-function optimization. Data is collected by one or multiple policies (which commonly obtained over the course of single experiment) and is stored in replay buffer from which it can be retrieved compute updates to the policy or value function. This approach has multiple benefits including data efficiency, reduced variance of the updates and smoothing the learning process (Mnih al. 2015).
2

---

Published as a conference paper at ICLR 2024
Replay
across
Exporiments
(RaE)
Data
Deta
Data
x.V
X
X.Ng
Replay
Agent
Replay
Agent
Replay
Agent
Frvironment
Ensironment
Environment
time
In each round, multiple (ag, Nun runs acrosa different random seads, typerparamaters, digorttmms researchers, etc.
Figure 1:
Intuitive schema for the use of regular replay in off-policy RL and the extended appli- cation in RaF. In every experiment, the agent can learn from mixture of data from all previous experiments to accelerate learning and increase asymptotic performance.
The success of off-policy algorithms and the need for data-efficient and safe learning schemes has led to growing interest in approaches that allow to reuse data across experiments. One end of a spectrum aims learn policies entirely offline from a pre-generated. fixed corpus of training data (Siegel al.
2020 Wang et al. 2020 Kumar et al. 2020 [ingh et al., 2020a Ajay et al., 202T). This usually requires specialist algorithmii modifications to avoid instabilities when
online data is available. often desirable to combine prior data with new data gathered during a learning experiment. In the simplest case policy (and value function) pretrained offline can be then finetuned online, on the same or related task. While various offline-RL. algorithms have been optimised for this setting, simpler aff-policy approaches can work well with sufficiently diverse data (Yarats et al. 2022).
The second perspective to re-using data is the straightforward reloading into the replay buffer during online learning. This approach has been successful in bootstrapping learning tasks with expert demonstrations (Vecerik et al. 2017; Nair al. 2018: Walke et al.) 2022) or to transfer to new tasks by incorporating data from scripted controllers (Singh et al. 20206] Smith et al. 2023 Jeong al. 2020). Finetuning offline learned agents and mixing data into a replay is further combined in the AWAC algorithm (Nair et al.) 2020). It operates two phases:
the first phase learning is
performed entirely offline by pre-loading the replay buffer with offline data. Then, after specified number of training steps, online data mixed into the replay and learning proceeds off aff-policy. AWAC proposes a specific algorithm for both phases that is similar to the formulation of the offline- RL algorithm CRR (Wang al. 2020
While these methods have shown promising gains in data efficiency and performance, they each come with specific assumptions that increase complexity, implementation effort, and restrict their application. These include the requirement for multiple training stages (Singh et al., 2020b; Smith al.
2023 ,. the introduction of additional training losses (Vecerik et 20,]2011 Nair et al. 2018
Song et al. 2022 or validation with specific architectures and algorithms Nair et al. 2020 Walke
al. 2022 Ball et al. 2023) which typically require domain-specific hyperparameter tuning. The core idea that we are exploring in this paper is that the reuse prior data can be implemented in the context of most contemporary off-policy RI. workflows in very simple way that is robust and leads to excellent results, without many of the algorithmic complexities hyper-parameter choices of prior work.
2.2 REPLAY ACROSS EXPERIMENTS
The basic insight of this work is that reusing data across multiple experiments in off- learning a very simple but effective way to accelerate training and improve final performance. Our key argument is the extension of this insight across all experimentation during a project. Prior data can be used to bootstrap new training runs, and it can further be beneficial train difficult tasks in multiple iterations, bootstrapping later iterations with the data collected during earlier ones.
This idea is illustrated in Figure I Whereas off-policy RL is generally focused on reusing data dur- ing a single experiment, we consider sequences of experiments where data from earlier training runs
3

---

Published as a conference paper at ICLR 2024
Figure 2: Domains: (from left to right) Locomotion, Manipulation stacking and the RL Unplugged domains Humanoid run, Manipulator insert peg, Manipulator insert ball.
made available during later one At the beginning of each training run, policy and value-function are re-initialized in line with stand-alone experiments. The only required algorithmic change is the availability of a second replay mechanism that allows replaying prior and online data with a partieu- lar fixed ratio throughout the course of training (we use a naive 50/50 mix of offline and online data for our main results, without optimizing this ratio).
This simple approach performs very well without introducing additional algorithmic complexities or the need for parammeeeeeerr tuning. By learning in the mixed online setting, the agent controls the data distribution and in line with recent results in offline RL Yarats al. 2022] Lambert al., 2022), we show that given the right data distribution, existing off-policy algorithms are sufficient for effectively using this offline data. This workflow does not require any changes with respect to the RL agent itself and generally agnostic agent and architecture changes across experiments. In order to emphasize this aspect, for the main results presented in Section 3.3] we integrate RaE into existing algorithmic workflows across all of the domains considered. Consequently, we evaluate
RaE across a set of
agents including DMPO (Abdolmaleki al.
2018). SAC-Q
(Riedmiller et al. 2018), CRR (Wang et al. 2020) and D4PG (Barth-Maron et al.
2018) across
number of complex domains.
3 EXPERIMENTS
In this section, we evaluate the performance of RaE across feature-based and vision-based, simulated robot locomotion and manipulation settings and standardised RL control benchmarks.
3.1 DOMAINS
We consider the following set of challenging domains (visualised Figure
Locomotion Soccer For locomotion, we consider robot soccer task (Haarnoja et al. 2023) where a simulated OP3 humanoid robot (Robotis OP3 2022) is rewarded for scoring goals (with additional shaping rewards) against an opponent. We consider two versions of this task: using proprioception and task-specific features (Locomotion Soccer State) and using proprioception with egocentric vi- sual features (Locomotion Soccer Vision). This second variant is particularly challenging since the agent must learn to score from a sparse reward signal and visual features that make the environment partially observable. Further information on this domain can be found Appendix A. We use MPO (Abdolmaleki et al.] (0188) with a distributional critic (Bellemare et al. 2017 Barth-Maron ett al. 2018) as underlying off-policy RL algorithm. For experiments we gather training data of 4c5 and 2c5 episodes [ for state and vision respectively.
Manipulation RGB Stacking For manipulation, we consider the task of stacking parameterized color-coded objects from Lee et al. (2021). This task builds on visual inputs and only sparse rewards. We use the multi-task SAC-Q off policy RL algorithm (Riedmiller et al. 2018) in this domain. Fur- ther information can be found Appendix A.. The offline dataset here consists of 15e4 episodes of training data from a prior experiment with the same algorithm.
RL Unplugged Finally, we evaluate with the offline RL benchmark and dataset RL Unplugged (Guccehre et al. 2020), which includes offline data on various simulated control domains. Within these, we focus on the three most challenging Control Suite domains (Tassa et al.] 2020): hu- manoid run, manipulator insert peg and manipulator insert ball. The benchmark dataset contains
We store and reuse all training data across experiments, not just trajectories left in the final replay. Throughout the we the scientific notation shorthand where 4e5 refers to 10
4

---

Published as a conference paper at ICLR 2024
3000 episodes for humanoid run and 1500 episodes each for the manipulator domains. We use RaE with the offline RL algorithm CRR (Wang al. 2020), which to our knowledge has sate-off-the-an performance for pure offline learning on this benchmark.
The offline data in this setting is notably different from the other domains. In order to collect the data, the authors follow a sub-sampling criterion (see Gulcehre et al. (2020) for details) to select a subset of episodes which are then randomly stored as single-step trajectories. This is in contrast to our simple protocol of collecting and reusing all data across training with single algorithm. We include this domain to illustrate the robustness and flexibility of RaE as well as to simplify future extensions and comparisons. More details on all domains are presented in Appendix A
3.2 BASELINES
We compare RaE against a number of strong baselines to illustrate its effectiveness.
Fine-tuning We first train a policy and state-action value function offline using CRR (Wang al. 2020). We evaluate a pure behavior cloning agent (BC) and two variants of CRR, CRR binary and CRR exp (see Appendix B). We then choose the best performing seed from across these runs and fine-tune (both policy and value function) online.
AWAC AWAC (Nair et al. 2020) starts by learning entirely offline for fixed number of pre- training steps before switching to online iteration. The main difference to finetuning applies to the online phase, where it uses a offline/online shared replay buffer with a specific algorithmic formulation. The authors of this work prescribe 25c3 pre-training steps offline before incorporating online data. For our sweep we also considered 5e4 steps and le5 steps. We also sweep over values for the Lagrange multiplier À (0.3 and 1) as proposed by the original work.
Random Weight Resetting Reloading data for an experiment restart implicitly involves resetting network weights. Nikishin et al. (202)) observed that network resets on their own can be beneficial for learning. To tease apart the effect of weight resets from the benefit data reloading, we consider a baseline where all weights (policy, critic and optimizer) are reset every K policy updates; where K is set to the number of updates used train the policy that generated the data. We sweep All experiments are run till convergence at least over reset frequencies of K, K/ 10 and K/ (/10 K updates.
For each method, we report results of the best performing hyperparameter variant averaged across 5 seeds. Importantly, note that RaE did not require any tuning and we report results using the same hyperparameter set across all of the main results (50% offline data). While minor performance improvements with different parameters for RaE can be observed in extreme settings (see Table we find RaE has very low sensitivity hyperparameter choices across range of diverse domains.
3.3 MAIN RESULTS
Figure B visualises our results on the Locomotion Soccer and Manipulation domains. It compares asymptotic performance achieved by RaE and other baselines after convergence as a bar plot with the dark solid line showing a 95% confidence interval around the mean averaged across seeds after smoothing over 1000 episodes. As the figure shows, RaF consistently achieves the highest asymptotic performance across these tasks with a notable improvement on the challenging vision- based 'Locomotion Soccer Vision' domain. In the Manipulation settings, both finetuning and RaE perform similarly, although finetuning requires choosing the right algorithm and hyperparameter for offline learning.
Figure 4 compares RaE with all baselines that can use the RL Unplugged offline dataset. The figure shows the accumulated reward over the total number of additional online data consumed for each method. The performance of the best pure offline variant (CRR) is shown as dotted blue line. We observe that RaE performs at-par or better than other comparable methods with the biggest difference being in the manipulator tasks. Methods that use offline learning (CRR and
'When using RaE, the data distribution throughout learning uses fixed ratio hetween offline and online data, in addition to implicitly resetting network weights. *Since the data collection methodology for the RL Unplugged domains is different and the number of updates unknown, we instead chose sweep over values of 1e4, le5 and le6 updates for these domains
5

---

Published as a conference paper at ICLR 2024
RaE
Online
AWAC
Finetune
Random resets
800
Arcumulated Reward
Accumuiated neward
Accumulated Reward
Accumulatad Reward
40
800
800
80
60
20
400
200
20
10
(a) L: Soccer State
(b) L.: Soccer Vision
(c) M: Place
(d) M: Stack Leave
Figure 3: Comparison of RaE against baselines the Locomotion Soccer and Manipulation do- mains (denoted by L: and M: respectively). Y-axis shows accumulated reward (undiscounted episode return) for each method; dotted blue line indicates offline learning performance with CRR. RaE con- sistently performs at-par better than the baselines with notable improvement tasks involving vision.
RaE
AWAC
Finetune
Random resets
200
Iecoutttedd Rewaded
100
520
Acondatedd
100
600
200
80
Totar online steps
leB
Iotal pnlinne stopss
160
Total onlinne steps
Sen
(a) Humanoid Run
(b) Manipulator: Insert peg
(c) Manipulator: Insert ball
Figure 4: Performance of various mechanisms to reuse offline data using the RL Unplugged dataset. Accumulated rewards (undiscounted episode return) on the Y-axis are plotted against the total steps. Each curve plots the mean performance across 5 seeds with standard deviation shown as shaded regions. RaE consistently performs the best across all tasks.
finetuning) work well on the densely rewarded Humanoid run domain but peform poorly in the sparsely rewarded manipulator domains whereas RaE works well across all settings.
3.4 ABLATIONS
In this section, we present set of experiments designed to answer the following questions:
How much data is required for performance improvements?
What kind of data is best for mixing: expert data, early training data or mix?
How sensitive is RaE the ratio of online offline data in different data regimes?
Is there an advantage of applying RaE repeatedly across experiment iterations?
Is RaE agnostic to the underlying choice algorithm?
Unless otherwise specified, we present results on the 'Locomotion Soccer State" task with the DMPO algorithm.
Analysis with varying data We answer the first three of these questions together in this section. For this analysis we use multiple subsets of the data collected for the 'Locomotion Soccer State' task, divided into three regimes:
High Return We consider data generated only from the end of training from a complete training run. This corresponds to highly rewarding trajectories "expert" data.
Mixed Return We consider data sampled uniformly at random throughout learning. This corresponds mixed regime with of high and low return trajectories.
Low Return Finally, we consider data generated only from the start of training. This corresponds early low return data.

---

Published as a conference paper at ICLR 2024
High return
Mixed return
Low return
Episodes
Data mix
10,000
100,000
10,000
100,000
10,000
100,000
50%
Online
51%
90%
80%
119%
97%
112%
70%
Online
80%
110%
101%
124%
108%
121%
80%
Online
98%
118%
106%
126%
110%
123%
90%
Online
108%
120%
113%
120%
108%
121%
Performance comparison when using RaE in different data regimes with various mixes of online and offline data. Each cell represents asymptotic performance improvement as a percentage of the performance reached when learning from scratch to convergencel (100% 100% online). Colors are used to indicate the trend: red for below 90% performance, orange between 90 to 100% and blue when greater than 100%
Table I:
High and Low Return data are sampled according to recency while Mixed return samples data uni- formly. For each regime, consider 2 datasets: one with le5 episodes and another with just le4 episodes. The purpose of this analysis is to understand how best to adapt RaE in a regime with limited data. For each data subset we then consider data mixing at different ratios: in addition to the 50% mix that used in the main results we consider mixes of 70, 80 and 90% online to offline data. For each setting, we consider the asymptotic performance reached when using RaE on the "Locomotion Soccer State' task.
Table shows the final asymptotic reward as percentage of the final reward achieved when learning online from scratch, which corresponds to 4e5 episodes. To improve the ease of interpretation the cells colored based the reward achieved: yellow for below 90% performance, orange for between 90-100% and blue for greater than 100%.
few interesting patterns emerge which we summarize below:
In the lower data regime (1e4 episodes) mixture with more online data is beneficial. However, as more data becomes available, lower ratio works better (of 70-80% le5 episodes). We hypothesize that using more online data for learning prevents over-fitting to small set of offline trajectories.
In the lower data regime, low return data tends to be the most beneficial. As the dataset size increases though, mix of high and low return trajectories provide greater benefit. Surprisingly, expert data is the least beneficial in both regimes. This indicates that the advantage of mixing data may stem from the benefit of having a larger state distribution with mixed rewards.
Gains in performance can be achieved with as little as le4 episodes. This particularly promising since even a small amount of prior data can considerably improve results.
Iterative improvement Figure 5a shows the improvement in performance when iteratively apply- ing RaE on the 'Locomotion Soccer State' domain. For this experiment, we begin by generating a smaller dataset of le4 episodes and then apply RaE to that. We collect the data from this improved run (RaE iteration 1) and apply RaE again. We observe that there is a small gain in asymptotic performance and speed of learning even on the 2nd iteration although the performance plateaus on the 3rd iteration. This result suggests that in some settings it may be preferable to break a single training run into smaller runs for iterative performance improvements.
Indifference to algorithm For the main analysis of Section 3.3 we show that RaE can improve performance across range of domains with different underlying algorithms. Figure (5b reinforces this point on the "Locomotion Soccer State" domain where we show a considerable gain in perfor- mance using RaE with D4PG (Barth-Ne Maron et al. 2018) for both data collection and mixing.
Increased robustness By applying RaE to combine data across seeds and hyperparameters from previous experiments we can add crucial robustness to the underlying agent's 's variance in perfor- mance. Figure [c| demonstrates robust performance when reloading data across random seeds in comparison purely high low performing data sources.
'The original dataset consists of 4e5 episodes.

---

Published as a conference paper at ICLR 2024
Orline
RoE Iteration RoE iteration
Online
RaE jparformant seed)
RaE (mioad)
RaE ieeraiio
RaE
RaE Inon-per formant soed)
1000
1000
500
800
fecumulated Rewerd
100
600
6000
60
400
20
0.0
3.5
8.00
0.30
1.25
0.0
Thtal
ln7
Totnl criline
lnd
online
(a) Multiple iterations of RaE
(b) RaB with D4PG
(c) Reloading across sceds
Figure 5: a) RaE can be applied iteratively on the 'Locomotion Soccer State' domain. In some cases it may be preferable to have multiple smaller iterations although a performance plateau may be reached eventually. b) RaE the 'Locomotion Soccer State' domain continues to improve performance with the D4PG algorithm. c) Combining data from seeds of a high variance experiment. RaE performs well even if just single original seed achieves high reward. Performance of the original data generating experiments plotted dotted lines. All figures plot mean accumulated reward against total online steps with standard deviation shown as a shaded region.
RELATED WORK
The most common paradigms to collect and reuse data in RL. involve either offline agent datasets
(e.g. RL Unplugged (Gulcehre et 2020) and D4RL (Fu et al. (0220) or the use of human or expert
demonstration trajectories (Vecerik et al. 2017: Bohez et al. (2022). While some work has focused on solving benchmark challenges (Wang et al. 2020 Kumar et al. 2020) others have focused on improving online performance by integrating offline data. For example, Singh et al.] (2020a) and Ajay et al. (202T) learn hierarchical architectures offline and reuse them to solve more challenging tasks online. Related to our approach, Smith et al. (2023) mix experience using controllers designed for a different setting with online data to improve learning efficiency for quadruped robots. Our work instead highlights the merits of directly transferring data to improve asymptotic performance on a single domain.
Learning from Data Combinations The idea of mixing expert data sources to bootstrap learning
on the same domain has also been studied in a number of related works (Vecerik et al. 2017:
Nair
let al. et al. domanns the complexity and specificity of various algorithmic assumptions limit their generality.
2018
Singh et al. 2020b; Walke et al. 2022, Davehev et al., 2021 Lee et al. 2022 Ball
2023 Nair et al. 20200. While these methods have shown impressive results on a range of
While Vecerik et al. (017) also mix expert demonstration data into the replay, they include a pri- oritized replay and a mix of 1-step and N-step returns with L2 regularization in their setup. In similar vein, Nair et al. 2018) introduce BC-loss, Q-filter and state-reset mechanism as part of their data mixing approach. Other work relies on multi-stage procedures when mixing data: [Singh et al. (2020b) distill prior data from skill experts to be later mixed with data from a scripted con- troller followed by the application of COL:/Walke Walke al. (2022) train multi-task policy with separate forward and backward policies that are optimized separately. Even work that uses existing offline datasets as opposed to expert demonstrations inevitably include other components that increase their complexity. For instance, Lee et al.] (2022) finetune an ensemble of policies trained offline using a prioritized replay and a density ratio to choose the data mixture to improve performance with the D4RL dataset. On the same domain, Ball et al. (2023) begin with a similar formulation to ours but then argue for random ensemble distillation and per-environment design choices and LayerNorm when training the O-function with which they demonstrate improvements using single off-policy algorithm. Finally, the AWAC algorithm (Nair et al. 2020) that we compare against sits in between finetuning and data mixing but uses a specific algorithmic formulation which introduces a number of domain- -emeddede hyperparameters.
In contrast, our focus is the simplicity of the method: we empirically demonstrate the effective- ness and versatility of mixing previous data with a fixed ratio using many off-policy RL algorithms.
Resetting Network Parameters in Reinforcement Learning A consequence of restarting exper- iments with mixed data is the resetting of neural network parameters. Resetting weights can change the learning dynamics and prevent overfitting which has been shown to be beneficial in both super-
8

---

Published as a conference paper at ICLR 2024
vised learning (Ash & Adams, 2020) and RL (Nikishin et al. 2022: Schwarzer et al. 2023). Under certain settings where parts of network are reset at specific intervals, these methods have been shown to greatly improve learning efficiency on challenging domains like Atari. In this work, we demonstrate that reusing previous experience can improve asymptotie performance tasks. Combined with its simplicity, we hope our approach can be included as part of the natural iteration cycle in RL and robotics in particular.
range of
DISCUSSION
As RL continues to move from the object of study to a practical engineering tool for control (De- grave et al. 2022, Bellemare et al.) 2020 Lazic et al. 2018 Lichtlé et al. 2022, Kaufmann et al. 2023] Osinski 20020p, simple and practical methods that can be generally applied will become increasingly important.
We can apply RaE throughout the lifetime of project, across multiple experiments, algorithms, and hyperparameter settings. In the following paragraphs, we provide some intuition on some practical use-cases for RaE.
Lifelong/ / Project-long learning Research into novel domains nominally involves many iterations of trial-and-errol achieve offe-of-theeart performance. While learnings from early iterations of experimentation inform algorithmic choices, the data collected in these trials is commonly discarded and never reused. Our initial analysis in Section 3.4 indicates that even low-retun data can be useful to boost performance. With costs of data storage typically being far lower than compute, a different workflow where all experimental data (particularly in domains like robotics) is stored and reused to bootstrap learning could improve efficiency across project lifetimes.
Multiple Source Experiments RaE can also be applied in domains with many related source experiments. For example, consider many tasks defined via different reward functions but with the same underlying dynamics (e.g. the family of all manipulation tasks with specific morphology). High-return data on some tasks may result in lower returns in another. However this data still informative and may be useful in improving exploration when transferred.
Multiple Hyperparameters and Seeds Combining data from different variants of the same exper- iment (such as random seeds or hyperparameters) can enable better use large, expensive sweeps. When reloading data with high performance variance over these parameters, the new experiment can benefit from the best option as initially discussed in Section 3.4
Potential Limitations and Strategies for Mitigation Across all experiments, RaE demonstrates better or similar performance to state-of-the-ar baselines for efficient off-policy RL without requir- ing per task hyperparameter tuning. However the applicability of the method is limited by the ability to reuse data. For example, changes dynamics or experimental settings might invalidate previ- ously collected data [6] In such cases, an intermediate step to collect transitional data between old and new settings might be useful. Another challenge that may arise with widespread use of RaE is in the comparison of new algorithms. RL already suffers from issues of reproducibility (Henderson al. 2018); if algorithms incorporate RaE, subtle changes in the data distribution when learning may create large differences in performance. One solution to this may be to specify deterministic orderings for benchmark datasets using fixed random seeds.
CONCLUSIONS
With this work, we introduce an effective and, importantly, simple workflow change for off off-policy RL experimentation. Reloading data requires minimal infrastructure and can greatly improve per- formance as shown in Section This makes it particularly useful from project-long (or life-long) learning perspective in RL experimentation. We believe that as understanding of RL improves and its use as an engineering and control tool becomes more commonplace, simplicity is key to effective integration.
'ooweerr' in practice, this may not be much of issue: see Appendix C.4

---

Published as a conference paper at ICLR 2024
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: 1886066920) 2018.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofrr Nachum. {OPAL): Of- fline primitive discovery for accelerating offline reinforcement learning. International Confer- ence on Learning Representations, 2021. URL
net. wwrmmmorrmmmmmm
V69LGwJ01IN
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information Processing Systems, 33, 2020.
Philip Ball, Laura Smith. Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. arXiv preprint arXiv:2302 02948, 2023.
Gabriel Barth Martn, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tirumala, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gra- dients. In International Conference on Learning Representations, 2018. URL https:// openreview net m
forum? Saterrmmid=yy1yppp 1d=5y71pzb0D
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449-455 PMLR, 2017.
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyun Wang. Autonomous navigation of strato- spheric balloons using reinforcement learning. Nature, 588:77 82, 2020. URL https:
Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasen- clever, Yuval Tassa, Emílio Parisotto, Jan Humplik, Tuomas Haarooja, Roland Hafner, Markus Wulfmeier, Michael Neunert, Ben Moran, Noah Siegel, Andrea Huber, Francesco Romano, Nathan Batchelor, Federico Casarini, Josh Merel, Raia Hadsell, and Nicolas Manfred Otto Heess. Imitate and repurpose: Learning reusable robot movement skills from human and animal behav- iors. ArXiv, abs/2203. 17138, 2022.
Todor Davehev, Oleg O. Sushkov, Jean-Baptiste Regli, Stefan Schaal, Yusuf Aytar, Markus Wulfmeier, and Jonathan Scholz. Wish you were here: Hindsight goal selection for long-horizon dexterous manipulation. ArXiv, abs/21 12.00597, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602((897):414- 419, 2022.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning. pp. 1407-1416. PMLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Caglar Gulcehre. Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi. Matthew Hoffman, Nicolas Heess, and Nando de Freitas. RI unplugged: A suite of benchmarks for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications, 2019.
10

---

Published as a conference paper at ICLR 2024
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H. Huang, Dhruva Tirumala, Markus Wulfmeier, Jan Humplik, Saran Tunyasuvunakool, Noah Siegel, Roland Hafner, Michael Bloesch, Kris- tian Hartikainen, Arunkumar Byravan, Leonard Hasenclever, Yuval Tassa, Fereshteh Sadeghi, Nathan Batchelor, Federico Casarini, Stefano Saliceti, Charles Game, Neil Sreendra, Kushal Pa- tel, Marlon Gwira, Andrea Huber, Nicole Hurley, Francesco Nori, Raia Hadsell, and Nicolas Manfred Otto Heess. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. ArXiv, abs/2304. 13653, 2023.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(), Apr. 2018, doi: 10.16099aaii.v322..111 1694. URL httpp:://ooss //a::/aa...a............. aaai.
orq oxq/index. php/
Rae Jeong, Jost Tobias Springenberg, Jackie Kay, Daniel Zheng, Yuxiang Zhou, Alexandre Galashov, Nicolas Heess, and Francesco Nori. Learning dexterous manipulation from suboptimal experts. CoRR, abs/2010.08587
|Rt
abs 20100.000....000000000 08587
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and
Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 6207797))9882-887, 2023.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Nathan Lambert, Markus Wulfmeier, William F. Whimey, Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi, Tim Hertweck, and Martin A. Riedmiller. The challenges of exploration for offline reinforcement learning, ArXiv, abs/2201.118611 2022.
Nevena Lazic, Tyler Lu, Craig Boutilier, MK Ryu, Echern Jay Wong, Binz Roy, and Greg Imwalle. Data center cooling using model-predictive control. In Proceedings of the Thirty-second Conference on Neural Information Processing Systems (NeurIPS-18), pp. 3818-380 3827, Montreal, QC, 2018. URL bttts:://pppp:eee www..nppsspc....m cttps.cc/ppper 7638 data center cooling wwllinng-uiingg model www.eiiiiieeeeoteeeeeeeeeeeeeeee
Alex X Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. In 5th Annual Confer- ence on Robot Learning, 2021.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistie q-ensemble. In Conference Robot Learning, pp. 1702-1712, PMLR, 2022.
Nathan Lichtlé, Eugene Vinitsky, Matthew Nice, Benjamin Seibold, Dan Work, and Alexandre M. Bayen. Deploying traffic smoothing cruise controllers learned from trajectory data. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2884-2890, 2022. 1109/ICRA46639.2000 292.4466999222281199222
doi: 10.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico May 2-4, 2016, Conference Track Proceedings, 2016. http:tp: Tlaxxiv.org/bbat
02971
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8:293-321, 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292-6299. IEEE, 2018

---

Published as a conference paper at ICLR 2024
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint (0tv:2000.093559 2020.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In ICML, 2022.
Blazej Osinski, Adam Jakubowski, Pawel Ziecina, Piotr Milos, Christopher Galias, Silviu Homo- ceanu, and Henryk Michalewski. Simulation-based reinforcement learning for real-world au- tonomous driving. In 2020 IEEE international conference on robotics and automation (ICRA), pp. 6411-6618. IEEE, 2020.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Robotis OP3, ott.atpps//mmmaaa...roooiissccce Introduction/
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level effi- ciency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), Proceedings of the 40th International Conference on Machine Learn- ing. volume 202 of Proceedings Machine Learning Research, pp. 30365-303333 PMLR, 23-29 Jul 2023. URL httpp:://proceedinggg
mlr press wttpppeeeeeeeeaeeeeeeee 2202//2aarr2222222222222 att.mieer2a...mmllmm
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Confer- ence on Learning Representations, 2020. URL httt:::://peereveeww net wtw.er/wwwweeeeeruummmmmm wtt.foorrmmmii
rke rke
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Par- rot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.1000 2020a.
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 14500, 2020b.
Laura Smith, J. Chase Kew, Tianyu Li, Linda Luu, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Learning and adapting agile locomotion skills by transferring experience. ArXiv, Abs/2304.09834, 2023.
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer. Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Ried- miller, and Matthew M. Botvinick. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2020. URL Satttt::: http:::::////peeeeeeeeeeee net http://rrrmmm--dyyyloo-a
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make efficient. arXiv preprint arXiv:2210.06718, 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. DeepMind control suite. Technical report, DeepMind, January 2018. URL https:
// watxii..rr//ab/11100 00690
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dmcontrol: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12220000..999.. 12983, 2020.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
12

---

Published as a conference paper at ICLR 2024
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 2017.
Homer Walke, Jonathan Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data to automate robotic reinforcement learn- ing. In Conference on Robot Learning, 2022.
Ziyu Wang, Alexander Novikov. Konrad Zolna, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2020.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2001133225 2022.
A ENVIRONMENT DETAILS
A.1 LOCOMOTION SOCCER
The 'Locomotion Soccer' environment introduced by HHarnoja et al. 2023) consists of Robotis OP3 (Robotis OP3 2022) robot which is placed in a 4m x 4m walled arena with a soccer ball. The robot must remain in its own half and scores goals when the ball enters goal area 0.5mtimesIn Im positioned across the center of the back wall in the other half. The robot must also contend with random agent in the other half. The reward for this task is given by:
Rgootscoring
Recore + Rusprigtt + Remazuelocity
where: Recore is 1000 on the single timestep where the ball enters the goal region (and then becomes unavailable until the ball has bounced back to the robot's own half); Rmazveloccity is the norm of the planar velocity of the robot's ' feet in the robot's forward direction; Rupright is 1.0 when the robot's s orientation is close to vertical, decaying to zero outside a margin of 12.5 Additionally, episodes are terminated with zero reward if the robot leaves its own half, or body parts other than the feet come within 4cm of the ground. The agent chooses a desired joint angle every 25 milliseconds (40Hz) based input observations that include the joint angles, angular velocity and gravity direction of the torso. For 'Locomotion Soccer State' the agent also receives the coordinates of the ball, goal and opponent. For the 'Locomotion Soccer Vision domain the robot instead receives a 40 x 30 render of the egocentric camera.
We use the experimental settings and networks described in Haarnoja et al. 2023) for this setting. We use a batch size of 128 with 5 seeds for the main results in Section 3 and a batch size of 256 with 2 seeds for the experiments in Section 3.4
A.2 ROBOT MANIPULATION
The RGB Stacking benchmark introduced by Lee et al. (2021) consists of parametric extruded shapes. which need to be stacked on top of each other by a fixed robotic manipulator, with the role of objects in the configuration being denoted via color coding. The benchmark uses Rethink Sawyer robot arm, controlled in Cartesian velocity space on which a Robotiq 2F-85 parallel gripper is mounted.
We focus on the "skill mastery" challenge of the benchmark as described in Lee al. (0221) where five fixed triplets of objects are used both for training and testing, with the goal being to place the red object on the blue one, while ignoring the green one. Each object triplet highlights different manipulation aspect, such as balancing or reorientation.
For the SAC-X setup, we use a curriculum of sub-tasks that can be chosen by the scheduler. These follow the reward terms of the staged reward terms used in Lee al. (202I), and intuitively contain sub-tasks for reaching, lifting, placing and stacking the red object. For evaluation, we report the final "stack-leave" reward term, which is a sparse reward given when the red object is placed precisely on the blue one, and the arm has been moved away by predefined distance.
13

---

Published as a conference paper at ICLR 2024
The observation provided to the agent consists of the images of three static cameras, well as proprioception data from the robot and gripper, including joint angles. velocities and torques, as well as a simulated force-torque sensor attached to the wrist. It notably does not contain the tracked position of the objects, so the agent must learn to achieve the goal primarily from vision inputs.
In order to solve this task from visual inputs we use a network architecture where both policy and value function use a convolutional neural network with residual blocks (as used in (Espeholt et al. 2018) that is then passed through MI.P for multi-task output for each of the subtasks. Both networks use (2, 2, 2) residual blocks with convolutional layers with 16, 32 and 32 channels respec- tively. The output of the policy network is used to condition multi-headed Gaussian where each Gaussian output represents the policy output for sub-task. Similarly the value function outputs multi-headed categorical for each task. Appendix B describes how this can be used by the SAC-Q algorithm. All experiments are run with 5 seeds for the main results of Section 3.3 and 2 seeds for the ablations in Section 3.4
A.3 RL UNPLUGGED
We consider 3 datasets from the RL Unplugged benchmark (Gulcehre et al. 2020h: Humanoid run, Manipulator insert peg and Manipulator insert ball; which are categorized 'hard' domains by Wang et al. (020). The RL. Unplugged datasets were collected on the DeepMind Control Suite
Tassa al.
2018 implemented in the MuJoCo (Todorov et al. 2012) simulation framework.
Humanoid run consists of 3000 episodes of a 21 dimensional simulated humanoid walker that is rewarded for running forwards while staying upright. The data for this domain is generated using D4PG (Barth-Maron et al. 2018). The Manipulator insert ball and Manipulator insert peg domains consist 1500 episodes each where a 5-dimensional manipulator is sparsely rewarded for inserting a ball and peg respectively into a specified hole. Data for this domain was generated using V-MPO (Song al. 2020) since D4PG is then reduced in size via sub-sampling and the number of successful episodes in each dataset is reduced by 2/3 to ensure the data does not contain too many successful trajectories. For all domains each episode consists of 1000 timesteps.
unable to solve the tasks. The data generated these tasks
We use experimental settings as described Wang et al. (2020) for this domain and use their net- works for the offline and finetuning experiments. For the other baselines and RaE we use an MLP architecture with sizes (256, 256, 128) for the policy and (512, 512, 256) for the value function. The policy network output is used to parameterize a Mixture Gaussian (MoG) with components as in Wang et al. (2020). Since the dataset only involves single-step transitions we use 1-step return for all baselines except the Random resets where we found 5-step returns performs significantly better.
B ALGORITHM DETAILS
In this section we describe the algorithms used in the main text more detail.
Policy Evaluation We use the critic update described Barth-Maron et al. (2018) for N N-step returns:
N TN)) an) r(so, ao) +
Σ "T(sn, ax) + 7 NMss ((s) ((anls))) 00
where the expectation is with respect to N step transition dynamics and the distribution Q and T represents the distributional Bellman operator. For our experiments we parameterize the critic as a categorical distribution with the number of bins set based on the domain. We set N = for our experiments except in the RL unplugged domains where only single-step trajectories exist in the dataset. However, we use N 5 for the random resets baseline in this domain since it can be run online and is not constrained by the limitation in the data.
DMPO For our online experiments in 'Locomotion Soccer' and 'Manipulation Stacking', the Maximum a-posteriori Policy Optimisation algorithm (MPO) (Abdolmaleki al.) (0088) to ad- here to Haarnoja et al. (2023). MPO optimizes the RL objective in an E and M-step. The E-step update optimizes:
14

---

Published as a conference paper at ICLR 2024
max S. uiss) S. a(als))(s, a) da
s.t. S. [(s)) [(ss) ds <c
where g(a an improved non con-paammttri policy that is optimized for states u(s) drawn from the replay buffer. The solution for q is shown to be:
(aass) (als, exp( 2o(s.a)
1)"
where Q is computed as an expectation over the categorical distribution. In the M-step, an improved policy is the obtained via supervised learning:
M
N
π arg max
By log ma(a)))))
8. I. L...L. (al8))][=o(als)
SAC-Q For the online experiments in 'Manipulation Stacking', we use the Scheduled Auxiliary Control (SAC-X) algorithm for multi-task exploration (Riedmiller et al. 2018). SAC-X builds on nulti-headed network architecture to represent both critic and policy, with a single "torso" per network shared across multiple tasks, and separate output "head" for each task. At any given time, the active head use decided by scheduler process. Here, the scheduler itself is also being learned, using the SAC-Q variant of the algorithm. At fixed intervals during each episode, a new sub-task may be chosen, and the scheduler trains Q-function that optimizes the sequence of tasks executed such that it maximizes the total return of one task designated as the main goal.
To update the policy and Q-function, SAC-Q may use any update rule. Here, we use the MPO algorithm with a distributional critic, as described above.
CRR For the offline experiments in Section we use the Critic Regularized Regression (CRR) algorithm (Wang et al.) (020) which achieves state-of-the-art performance on the RL Unplugged benchmark. CRR uses the same policy evaluation step described above learn distributional critic. However the policy in CRR trained by filtering data via the Q function by optimizing:
arg max (s,a)) (Q, ग. s, a) log [[ass)]
where the states and actions are drawn from the data source and f is a non-negative, scalar function whose value increases monotonically with Q. We consider both variants of CRR introduced the paper defined by choices of f::
= 1 A(s a) > 0], == exp (A(s, a)
(4)
(5)
where A(s, a) is the advantage function which can be computed using:
1 ((s ((lls)
A(s. Q(s,
m
Following the authors of CRR, we refer to CRR with f from Equation 4 as CRR binary and Equation 5 al CRR exp.
AWAC For the AWAC bascline in the main text we follow the description from Nair et al. (2020) where learning proceeds in two stages: entirely offline initially and then with online data added to the same replay buffer. The transition between these two stages defined by hyper parameter which is set to 25,000 steps in the original work. For our experiments we sweep over an additional value of 50,000 steps and 100,000 steps for the RL Unplugged domain.
AWAC also introduces an algorithm which is similar the CRR exp formulation described above with a slightly different notation that introduces a temperature À above. As per the original work test the method with À set to 0.3 and 1.
15

---

Published as a conference paper at ICLR 2024
Ra
Online
AWAC
Finetune
Random resets
200
60
Accumulated Roward
aled Reward
150
Accumulate aled Reward
100
Accumuleted Reward
50
150
Accumul Accumulated
100
80
100
60
30
40
20
50
20
10
0
(a) Open
(b) Reach Grasp
(c) Lift
(d) Stack
Figure 6: Comparison of RaE against baselines on Open, Reach Grasp, Lift and Stack sub tasks in
the Manipulation RGB stacking setting. Average performance across 5 seeds is plotted on the Y-axis
with standard deviation as as dark lines.
Finetuning with RaE
RaE
Finetuning
1000
1200
Accumulated Reward
Accumulated Reward
1000
Map
p Like
800
600
600
400
400
200
200
0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Total online steps
1e7
Total online steps
1e8
(a) Locomotion Soccer State
(b) Locomotion Soccer Vision
Figure 7: Comparison of finetuning, RaE and finetuning with RaE the 'Locomotion Soccer state and vision tasks (left and right respectively). Finetuning with RaE combines the best both worlds: improved learning speed with high asymptotic performance. Accumulated reward on the Y-axis is plotted against total online steps on the X. Results averaged across 5 seeds with the shaded region representing the standard deviation.
C ADDITIONAL RESULTS
C.I MANIPULATION RESULTS ON REMAINING TASKS
Figure 6 compares RaE against the other baselines from Section 3.3 on the remaining sub-tasks in the 'Manipulation RGB Stacking' setting. While all methods fare well on the easier tasks like 'Open' and 'Reach Grasp', RaE performs well the harder 'Lift' and 'Stack" tasks similar the trend on 'Place' and 'Stack Leave" that were presented in the main text.
C.2 FINETUNING WITH RAE
The 'finetuning' baseline considered in the main experiments Section orthogonal to the core idea of RaE and as such, we can apply RaE when finetuning. Figure compares RaE, finetuning and finetuning with RaE the 'Locomotion Soccer' tasks from state and vision. We observe that finetuning with RaE performs well across both domains matching the high asymptotic performance of RaF and learning faster by taking advantage of the network weights pre-trained offline.
C.3 EFFECT OF MIXING RATIO ON FULI DATASET
The results presented in Table show the effect of mixing different ratios of data when using RaE with smaller subsets of data (of 10,000 and 100,000) episodes. Figure Sa instead presents the same analysis when using the full dataset (of 4e5 episodes) on the "Locomotion Soccer State' task. In this setting we see a clearer advantage of using offline data where mixture of 50 to 70 percent of offline data works better than using online data (90 %). In fact, the figure shows,
16

---

Published as a conference paper at ICLR 2024
50.00 0% online
80.0% online
100% online
RaE
Online
online
online
1000
400
Accurnuleted Reward
800
300
600
200
200
(a) Mixing ratio with full dataset (b) RaE with changing dynamics
Figure 8: (Left) Comparison with different mixtures of offline and online data when using RaE with 4e5 episodes of data the 'Locomotion (State)' task. Accumulated reward (total undiscounted return) for each ratio of data is plotted. mixture 50 70 percent of online data works best. (Right) Comparison when using RaE to reload data to new task with different dynamics. RaE is suprisingly robust and learns to perform better than learning from scratch indicating it's robustness to changes in dynamics.
mixing ratio of 90 % may slightly degrade performance when compared learning purely online.
C.4 EFFECT OF CHANGING DYNAMICS
In this section we analyze the effect of reusing RaE in a regime where the underlying dynamics change. We reuse the data collected in the 'Locomotion Soccer (State)" task but transfer an environment where a mass is randomly attached to the left and right legs of the walker to perturb it's motion. Figure 8b] shows that RaE continues to show advantage as compared to learning from scratch even when the underlying dynamics of the task have changed. This somewhat surprising finding may be explained by modeling the environment as a partially observed MDP where under some conditions unknown to the agent, the dynamics of walking alter. Reusing data still guide learning in such setting showing the robustness our approach.
17


Reviews:
[{"review_id": "tjFfAv883U", "reviewer": "Reviewer_wVcc", "summary": "The paper discusses a new setting, where data is shared across experimental runs to improve the performance of the policy. A simple strategy, adding data from previous experiments to the initial replay buffer, is explored and shown to be an effective approach. Various factors are investigated including the quality of the data used and the amount of data kept. These experiments are done with a variety of policy optimizatino algorithms and robotics benchmarks.", "questions": "- Aside from RaE, another natural baseline for this across-experiment setting would be to use behaviour cloning (or distillation) on the previously trained agents. Have you considered doing so? \n\n- RaE only incorporates the previous data at the beginning of optimization. What about training on the data throughout the optimization process? For example, keeping the buffer of previous data and sampling from it occasionally or mixing in samples into minibatches. \n\n- As I mentioned in strengths section, I think training across-experiments could be relevant for practical purposes. How do you see it being used in a scientific context? It could be difficult to fairly assess algorithms if the data they have access to depends on the sequence of previous experiments done. i.e. an advantage of discarding previous data is that algorithms start on equal ground in different papers, allowing fair comparisons. \n\n- I'm a bit surprised that doing some offline learning first is so detrimental to the policy (Fig. 4, finetuning and AWAC). Intuitively, I would guess that there should be a jumpstart in the performance due to the additional offline training. Could you clarify this?\nDo you have any hypotheses why these methods don't perform very well here?\n\n- Currently, the experiments that use RaE are quite similar to simply using resets but at the end of each experimental run. It would be interesting to see some experiments where the same algorithm was not used in consecutive experiments or, at least, changing hyperparameters. This could better simulate the development process of an RL algorithm. \nAs a suggestion, I would be curious to see if hyperparameter optimization could be made easier. Since RL agents can be sometimes hard to run initially on a new problem, we might be able to more easily identify which hyperparameter settings are promising by giving the agent additional data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The approach of keeping data from previous experiments seems to be very relevant in practical scenarios where our main goal is to train an agent with strong performance. In that case, given the cheap cost of memory, it would be sensible to keep data from previous runs for the benefit of future experiments. \nThis seems like an understudied topic and it's great that this paper discusses it.\n\n- There's a nice variety of environments that are used, including some more complex ones with a good mix of algorithms too.\n\n- The writing and organization is clear, making the paper easy to read. The paper has a distinct focus which helps gets the message across too.", "weaknesses": "- In the current paper, most of the experiments run the same learning agent on the same environment with the RaE algorithm but the method is pitched as being helpful for boosting learning between different experiments with potential differences in experimental conditions.\nSee Questions.\n\n- There are other simple algorithms for this across-experiment setting that would be interesting to investigate. See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B5kC603qE2", "reviewer": "Reviewer_fgij", "summary": "The authors propose Replay across Experiments (RaE), in which all past experiment data is stored in the replay memory for training. In essence, data is never discarded. Data from all past experiment trials is stored and reused, not just the trajectories left in the buffer at the end of the previous experiment. The authors recommend a default mixture of 50-50 offline/online data (where “online” data is from the current experiment) to obtain good performance without tuning this hyperparameter. The authors compare RaE against other common strategies (fine-tuning, AWAC, and parameter resetting) combined with algorithms such as DMPO and D4PG in control benchmarks including Locomotion Soccer, Manipulation RGB Stacking, and RL Unplugged.\n\n***Rebuttal: score raised from 5 to 6**", "questions": "1. What is meant by “Accumulated Reward” in the y-axes of Figures 3, 4? Is this the undiscounted episode return?\n1. What is meant by “from scratch to convergence” in Table 1’s caption? Does that mean that exactly 100% refers to the final performance obtained by a pure online-data agent?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is simple to implement compared to many existing data-reuse RL methods.\n- The experiments in the paper are comprehensive, testing a multitude of diverse methods in a number of high-dimensional control environments. Several strong baselines from the literature are compared against. In spite of this simplicity, RaE can achieve strong performance in these tasks. The breadth of the results demonstrate the generality of RaE.\n- The paper is well organized and includes nice discussions for related work and practical use cases.", "weaknesses": "- The main insight of the work, while interesting, is a small contribution. As the authors note in the background section, other methods already use data stored from previous experiments, so the only novelty here is that *data is not discarded*. There is no theoretical analysis in the paper. Without significant novelty or theory, the paper depends solely on its empirical results.\n- Storing all previous data is memory intensive, which is why offline RL generally uses more complicated techniques to learn from limited data. The authors do not discuss this drawback of their method, but I could see it being a bottleneck in long experiments with high-dimensional observations.\n- It is unclear that replaying offline data from a long time ago is as beneficial as the authors claim. Table 1 seems to indicate that performance improves almost monotonically as the proportion of online data increases to 90%. The authors claim that “as more data becomes available, a lower ratio [of 70-80%] works better,” but this only happens occasionally, and the performance improvement is small (about 2-5%). Without any measure of significance provided, it does not seem that more than 10% offline data actually helps much.\n- The results are rather noisy and would benefit from increasing the number of trials (which is currently only 5). I would recommend that the authors use a 95% confidence interval instead of standard deviation and apply a 100-episode moving average (if they are not already doing so already) to make the results easier to read. Currently, some of the standard deviations are overlapping, but I think the results would be significant if confidence intervals are used instead.\n\n**Minor edits:**\n- In-text citations are not rendered correctly on the last line of page 1.\n- In the background section, the range of discount values should be mentioned: $\\gamma \\in [0,1]$.\n- I think it would be helpful to the reader to define the scientific notation shorthand the first time it is used, e.g., $\\text{4e5} = 4 \\times 10^{-5}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYQNUkoobt", "reviewer": "Reviewer_u3fY", "summary": "This paper presents a method to reuse experience from prior RL experiments and shows benefits across a variety of RL algorithms and experiments.", "questions": "- I'm a bit confused about the difference between Random Weight Resetting and RaE. The authors write that \"Reloading data for an experiment restart implicitly involves resetting network weights\", which makes me think that RaE and Random Weight Resetting are very similar. However, this clearly isn't the case since RaE performs better. Can the authors clarify the difference?\n- Across all figures does \"Total online steps\" include the steps from prior experiments for RaE? If not, I'm concerned that it may not be a fair comparison.\n- The authors write, \"At the beginning of each training run, policy and value-function are re-initialized in line with stand-alone experiments\". I'm curious if re-initializing vs not re-initializing at the start of each training run makes a difference in performance?\n- Under \"Potential Limitations and Strategies for Mitigation\" the authors write that \"changes in dynamics or experimental settings might invalidate previously collected data.\" Have the authors actually tried experimenting with changing dynamics across training runs. I'd be curious to see how much of a negative effect this would actually have in practice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors present a very simple idea that leads to improved performance across a variety of environments.\n- The proposed method works well even with a small amount of prior data.\n- The proposed method works well even with low return offline data, which makes the method much more useful in practice.", "weaknesses": "- It is not clear to me if \"Total online steps\" in the figures includes the steps from prior experiments or not, so I'm concerned about the fairness of the comparison between RaE and baselines. If the authors can clarify this point then I may be willing to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tHBxwttqn4", "reviewer": "Reviewer_gauC", "summary": "The paper proposes Replay across Experiments (RaE) which is based on the simple concept of reusing experience from previous experiments to improve exploration and bootstrap learning. The proposed approach is employed with a number of existing algorithms in locomotion and manipulation tasks and shown to improve the learning efficiency.", "questions": "1.\tWhat motivated the reuse of previous experimental data? There is no explicit explanation for this.\n\n2.\tDo the age of trajectories matter? Eg: Suppose there is a limit on the data storage, would it be more valuable to add older trajectories to the data buffer or relatively newer ones?\n\n3.\tIn the interest of comprehensiveness, I would have liked to see the benefits of this approach in Atari games as well. Does the approach improve the performance of say, DQN in Atari environments? Performance with methods like prioritized experience replay would also be interesting.\n\n4.\tThe results in Table 1 are interesting. However, the reasons for the observed trends are not clearly explained.\n\n5.\tIn Fig 1, the size of the ‘Data’ blocks is increasing, but is not immediately noticeable. It would be better to exaggerate the increase in block size to bring a reader’s attention to the data accumulation mechanism.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The described approach is simple. It could be impactful as it is mostly algorithm-agnostic.", "weaknesses": "The main drawback is the lack of explicit explanations for the improvements observed. The limitations, perhaps such as increased storage memory could also be emphasized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)