Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
Authors: Rishabh Agarwal
Max Schwarzer
Pablo Samuel Castro
Aaron Courville
Marc G Bellemare
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reusing Computation
Deep RL
Research Workflow

Abstract: Learning tabula rasa, that is without any prior knowledge, is the prevalent workflow in reinforcement learning (RL) research. However, RL systems, when applied to large-scale settings, rarely operate tabula rasa. Such large-scale systems undergo multiple design or algorithmic changes during their development cycle and use ad hoc approaches for incorporating these changes without re-training from scratch, which would have been prohibitively expensive. Additionally, the inefficiency of deep RL typically excludes researchers without access to industrial-scale resources from tackling computationally-demanding problems. To address these issues, we present reincarnating RL as an alternative workflow or class of problem settings, where prior computational work (e.g., learned policies) is reused or transferred between design iterations of an RL agent, or from one RL agent to another. As a step towards enabling reincarnating RL from any agent to any other agent, we focus on the specific setting of efficiently transferring an existing sub-optimal policy to a standalone value-based RL agent. We find that existing approaches fail in this setting and propose a simple algorithm to address their limitations. Equipped with this algorithm, we demonstrate reincarnating RL's gains over tabula rasa RL on Atari 2600 games, a challenging locomotion task, and the real-world problem of navigating stratospheric balloons. Overall, this work argues for an alternative approach to RL research, which we believe could significantly improve real-world RL adoption and help democratize it further. Open-sourced code and trained agents at https://agarwl.github.io/reincarnating_rl.

Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
Rishabh Agarwal 1.2+ Max Schwarzer
Pablo Samuel Castro Aaron Courville"
Mare Bellemare
Google Research, Brain Team MIIL
Abstract
Learning tabula rasa, that is without any previously learned knowledge, is the prevalent workflow in reinforcement learning (RL) research. However, RL systems, when applied to large-scale settings, rarely operate tabula rasa. Such large-scale systems undergo multiple design or algorithmic changes during their development cycle and use ad hoc approaches for incorporating these changes without re- training from scratch, which would have been prohibitively expensive. Additionally, the inefficiency of deep RL typically excludes researchers without access to industrial- scale resources from tackling these issues, we present reincarnating RL alternative workflow class of problem settings, where prior computational work (e.g., learned policies) is reused transferred between design iterations of an RL agent, or from one RL agent to another. As a step towards enabling reincarnating RL from any agent to any other agent, we focus on the specific setting of efficiently transferring an existing sub-optimal policy to a standalone value-based RL agent. We find that existing approaches fail this setting and propose a simple algorithm address their limitations. Equipped with this algorithm, we demonstrate reincarnating RL's gains over tabula rasa RL on Atari 2600 games, challenging locomotion task, and the real-world problem of navigating stratospheric balloons. Overall, this work argues for an alternative approach to RL research, which we believe could significantly improve real-works RL adoption and help democratize it further. Open-sourced code and trained agents at agarwl gi thub. io/reincarnating......
pepuuaatiionllyymmmandeee problems. To address
1 Introduction
Reinforcement learning (RL) is general-purpose paradigm for making data-driven decisions. Due to this generality, the prevailing trend in RL research is to learn systems that can operate efficiently tabula rasa, that is without much learned knowledge including prior computational work such as offline datasets or learned policies. However, tabula rasa RL systems are typically the exception rather than the norm for solving large-scale RL problems [4, 13, 55, 75, 85]. Such large-scale scale RL systems often need to function for long periods of time and continually experience new data; restarting them from scratch may require weeks if not months of computation, and there may be billions of data points to re- process this makes the tabula rasa approach impractical. For example. the system that plays Dota
human-like level [13] underwent several months of RI. training with continual changes (e.g., in model architecture, environment, eic) during its development; this necessitated building upon the previously trained system after such changes to circumvent re-training from scratch, which was done using ad hoc approaches (described in Section 3).
Current RL research also excludes the majority of researchers outside certain resource-rich labs from lackling complex problems, as doing so often incurs substantial computational and financial cost: AlphaStar [85], which achieves grandmaster level in StarCraft, was trained using TPUs for more than a month and replicating it would cost several million dollars (Appendix A. Even the quintessential deep RL benchmark of training an agent on 50+ Atari games [10], with least runs, requires
"Correspondence Rishabh Agarwal wriishabaaggraallggoggeee..
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Nature DQN
DON [Adam) (Fine tune)
Impale CNN Rainbow 100M
1.75
Nature DON (Fine-tume)
DON [Adam) (Tabula Rasa)
10M Normalized Score
1.50
1.25
1.00
DON (Adam) 400M
0.75
0.50
0.25
Impela-CNN Rainbow (Reincaraaionn
DON (Adam, n-3) (Fine-tune) Impala-CNi Rainbow (Tabula Rasa)
0.00
100
200
15 Env. Frames (x 1M)
10
20 1
20 25 30 35 40 45 50 Env. Frames (x 1M)
Env. Frames (x 1M)
Figure I: A reincarnating RL workflow on ALE. The plots show IQM [2] normalized scores over training. computed using 50 seeds, aggregated across 10 Atari games. The vertical separators correspond to loading network weights and replay buffer for fine-tuning while offline pre- pre-training on replay buffer using QDag- ger (Section 4.1) for reincarnation. Shaded regions show 95% confidence intervals. We assign DQN (Adam) trained for 400M frames and to random agent. (Panel 1) Tabula rasa Nature DQN [60] nearly converges performance after training for 200M frames. (Panel 2) Reincarnation via fine-tuning Nature DQN with a reduced learning rate leads to 50% higher IQM with only 1M additional frames (leftmost point). Furthermore, fine-tuning Nature DQN while switching from RMSProp to Adam matches the perfor- mance of DQN (Adam) trained from scratch for 400M frames, using only 20M frames. (Panel 3). modern ResNet (Impala-CNN [26]) with a better algorithm (Rainbow [35]) outperforms further fine-tuning n-step DQN. Reincarnating Impala-CNN Rainbow from DQN, outperforms tabula rasa Impala-CNi Rainbow throughout training and requires only 50M frames to nearly match its performance at 100M frames, See Section 5.
of I
more than 1000 GPU days. As deep RL research move towards more challenging problems, the computational barrier to entry in RL research is likely to further increase.
To address both the computational and sample inefficiencies of tabula rasa RL, we present reincarnat- ing RL (RRL) as an alternative research workflow or a class of problems to focus on. RRL seeks to maximally leverage existing computational work, such as learned network weights and collected data, to accelerate training across design iterations of an RL agent or when moving from one agent to another. In RRL, agents need not be trained tabula rasa, except for initial forays into new problems. For example, imagine a researcher who has trained an agent A, for a long time (e.g., weeks), but now this or another researcher wants to experiment with better architectures or RL algorithms. While the tabula rasa workflow requires re-training another agent from scratch, reincarnating RL provides the more viable option of transferring Ai to another agent and training this agent further, or simply fine-tuning Ai (Figure 1). As such, RRL can be viewed for the rescarch workflow needed for real-world and large-scale RL models.
attempt to provide formal foundation
Reincarnating RL can democratize research by allowing the broader community to tackle larger-scale and complex RL problems without requiring excessive computational resources. As a consequence, RRL can also help avoid the risk of researchers overfitting to conclusions from small-scale RL problems. Furthermore, RRL can enable a benchmarking paradigm where researchers continually improve and update existing trained agents, especially problems where improving performance has real-world impact (e.g., balloon navigation [11], chip design [59], tokamak control [24]). Furthermore, common real-world RI. use case will likely be in scenarios where prior computational work is available (e.g., existing deployed RL policies). making RRL important to study. However, beyond some ad hoc large-scale reincarnation efforts (Section 3), the community has not focused much on studying reincarnating RL as a research problem in its own right. To this end, this work argues for developing veneral-purposs RRL approaches opposed to ad hoc solutions.
Different RRL problems can be instantiated depending on how the prior computational work is provided: logged datasets, learned policies, pretrained models, representations, etc. As a step towards developing broadly applicable reincarnation approaches, we focus on the specific setting of policy-to-value reincarnating RI. (PVRL) for efficiently transferring suboptimal teacher policy to value-based RI. student agent (Section 4). Since it is undesirable to maintain dependency on past teachers for successive reincarnations, require a PVRL algorithm to "wean" off the teacher

---

dependence as training progresses, We find that prior approaches, when evaluated for PVRL on the Arcade Learning Environment (ALE) [10], either result in small improvements over the tabula rasa student or exhibit degradation when weaning off the teacher. To address these limitations, we introduce QDagger, which combines Dagger [71] with n-step Q-learning, and outperforms prior approaches. Equipped with QDagger, we demonstrate the sample and compute-efficiency gains of reincarnating RL over tabula rasa RL, on ALE, a humanoid locomotion task and the simulated real-world problem of navigating stratospheric balloons [11] (Section 5). Finally, we discuss some considerations in RRL as well as address reproducibility and generalizability concerns.
2 Preliminaries
The goal in RL is to maximize the long-term discounted reward in an environment. We model the environment as an MDP, defined as (S, R, P. [69], with a state space S, an action space A.
stochastic reward function R(s, a), transition dynamics P(s'|s [s, a) and a discount factor E [0, 1). A policy r(-ls) maps states to a distribution over actions. The Q-value function Q= a) for policy <--1s) is the expected sum of discounted rewards obtained by executing action a at state s and following r-Iss) thereafter. DQN [60] builds on Q-learning [87] and parameterizes the Q-value function, Qo. with neural net with parameters while following an c-greedy policy with respect to Qo for data collection. DQN minimizes the temporal difference (TD) loss, (DD(Ds), on transition tuples, (s, a, T, s'), sampled from an experience replay buffer Ds collected during training:
CDD(D)= = Es,a, (Qo(s,a) - r max (o(s',a'))
(I)
a'
where Qo is delayed copy of the same Q-network, referred to as the target network. Modern value- based RL agents, such as Rainbow [35], use n-step returns to further stabilize learning. Specifically, rather than training the Q-value estimate Q(se, 0+) on the basis of the single-step temporal difference error r max, ((s++1, a') - Q(s at), an n-step target Ek=0 > + max.' Q( a') Q(s ae) is used in the TD loss, with intermediate future rewards stored in the replay D.
3 Related work
Prior ad hoc reincarnation efforts. While several nation, it has typically been done in an ad-hoc way and has limited applicability. OpenAI Five [13], which can play Dota 2 at a numan-like level, required 10 months of large-scale RL training and went through continual changes in code and environment (e.g., expanding observation spaces) during development. To avoid restarting from scratch after such changes, OpenAl Five used "surgery" akin to Net2Net [17] style transformations to convert trained model to certain bigger architectures with custom weight initializations. AlphaStar [85] employs population-based training (PBT) [42], which periodically copies weights of the best performing value-based agents and mutates hyperparameters during training. Although PBT and surgery methods are efficient, they have they can not be used for reincarnating RL when switching to arbitrary architectures (e.g., feed-forward recurrent networks) or from one model class to another (e.g., policy to a value function). Akkaya et al. [4] trained RL policies for several months manipulate a robot hand for solving Rubik's 's cube. To do so, they "rarely trained experiments from scratch" but instead initialized new policies, with architectural changes, from previous trained policies using behavior cloning via on-policy distillation [20, 67]. AlphaGo [75] also used behavior cloning on human replays for initializing the policy and fine-tuning it further with RL. However, behavior cloning is only applicable for policy to policy transfer and inadequate for the PVRL setting of transferring policy to a value function [e.g., 63, 83). Contrary to such approaches, we apply reincarnation in settings where these approaches are not applicable including transferring DQN agent to Impala-CNN Rainbow in ALE, and distributed agent with MLP architecture to a recurrent agent in BLE. Several prior works also fine-tune existing agents with deep RL for reducing training time, especially real-world tasks such as chip floor-plannin [59], robotic manipulation [43], aligning language models [6], and compiler optimization [82]. In line with these works, we find that fine-tuning a basue- agent can be an effective reincarnation strat- egy (Figure 7). However, fine-tuning is often constrained to use the same architecture as the agent being tuned. Instead, we focus on reincarnating RL methods that do not have this limitation.
RL achievements have used reincar-
Leveraging prior computation. While areas such as offline RL, imitation learning, transfer in RL, continual RL elc focus on developing methods to leverage prior computation, such areas don't 't strive to change how we do RL research by incorporating such methods as a part of our workflow. For completeness, we contrast closely related approaches to PVRL, the RRL setting we study.
3

---

Leveraging existing agents. Existing policies have been previously used for improving data collection [14, 16. 28, 77. 89]; we evaluate one such approach, JSRL [83], which improves exploration in goal-reaching RI. tasks. However, our PVRL. experiments indicate that JSRI. performs poorly on ALE. Schmitt et al. [74] propose kickstarting to speed-up actor-critic agents using an interactive teacher policy by combining on-policy distillation [20, 67] with RL. Empirically, we find that kickstarting is a strong baseline for PVRL, however exhibits unstable behavior without n-step returns and underperforms QDagger. PVRL also falls under the framework of agents teaching agents (ATA) [21] with RL-based students and teachers. While ATA approaches, such as action advice [81], emphasize how and when to query the teacher evaluating the utility of teacher advice, PVRL focuses on sample-efficient
and does not impose constraints on querying the teacher. PVRL is also different from prior work on accelerating RL using a heuristic or oracle value function [9. 19, 78], as PVRL only assumes access to a suboptimal policy. Unlike PVRL. methods that wean off the teacher, imitation-regullriize RL methods [51, 61] stay close to the suboptimal teacher, which can limit the student's performance with continued training (Figure 9).
Leveraging prior data. Learning from demonstrations (LID) [5, 30, 36, 40, 72] approaches focus on accelerating RL training using demonstrations. Such approaches typically assume access to optimal or near-optimal trajectories, often obtained from human demonstrators, and aim to match the demonstrator's ' performance. Instead, PVRL focuses on leveraging a suboptimal teacher policy, which can be obtained from any trained RL agent, that wean off during training. Empirically, that DQID [36], a well-known LfD approach to accelcrate deep Q-learning, when applied to PVRL, exhibits severe performance degradation when weaning off the teacher. Rehearsal approaches [62, 66, 76] focus on improving exploration by replaying demonstrations during learning; we find that such approaches are ineffective for leveraging the teacher in PVRL. Offine RL [1, 49, 52] focuses on learning solely from fixed datasets while reincarnating RL focuses on leveraging prior information, which can also be presented as offline datasets, for speeding up further learning from environment interactions. Recent work [45, 51, 55, 63] use offline RL to pretrain on prior data and then fine-tune online. We also evaluate this pretraining approach for PVRL and find that underperforms QDagger, which utilizes the interactive teacher policy in addition to the prior teacher collected data.
find
Case Study: Policy to Value Reincarnating RL
While prior large-scale efforts have used limited form of reincarnating RL (Section 3), it is unclear how to design more broadly applicable RRL. approaches. To exemplify the challenges of designing such approaches, we focus on the RRL setting for accelerating training of a student agent given access to a suboptimal teacher policy and some data from it. While a policy-based student can be easily reincarnated in this setting via behavior cloning [e.g., 4], we study the more challenging policy-to-value reincarnating RL (PVRL) setting for transferring a policy to a value-based student agent. While we can obtain policy from any RL agent, we chose this setting because value-based RL methods (Q-learning, actor-critic) can leverage off-policy data for better sample efficiency. To be broadly useful for reincarnating agents, PVRL algorithm should satisfy the following desiderata:
Teacher-agnostic. Reincarnating RL has limited utility if the student is constrained by the teacher's s architecture or learning algorithm. Thus, we require the student to be teacher-agnostic.
Weaning. It is undesirable maintain dependency on past teachers when reincarnation may occur several times over the course of a project, or one project to another. Thus, it is necessary that the student's dependence on the teacher policy can be weaned off, as training progresses.
Compute & sample efficient. Naturally, RRL is only useful if it is computationally cheaper than training from scratch. Thus, it is desirable that the student can recover and possibly improve upon the teacher's performance using fewer environment samples than training tabula rasa.
PVRL on Atari 2600 games. Given the above desiderata for PVRL. we now empirically investigate whether existing methods that leverage existing data or agents (see Scction 3) suffice for PVRL. The specific methods that we consider were chosen because they are simple to implement, and also because they have been designed with closely related goals in mind.
Experimental setup. We conduct experiments on ALE with sticky actions [57]. To reduce the computational cost of our experiments, we use a subset of 10 commonly-used Atari 2600 games: Asterix, Breakout, Space Invaders, Seaquest, Q" Bert, Beam Rider, Enduro, Ms Pacman, Bowling and River Raid. We obtain the teacher policy कт by running DQN [60] with Adam optimizer for 400 million environment frames, requiring days of training per run with Dopamine [15] on P100 GPUs.

---

QDagger
Kickstarting
Pretraining
Rehearsal
DQrD
JSRL
Offline
Online
1.00
IQM Noralized Score
Fraction of with score
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
4
10
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Steps (x 100k)
Env. Frames (x 1M)
Teacher Normal zzd Score (T)
Figure 2: Comparing PVRL algorithms for reincarnating student DQN agent given teacher policy (with normalized score of 1), obtained from a DQN agent trained for 400M frames (Section 4), Baselines include kickstarting [74], JSRL [83], rehearsal [66], offline pretraining [46] and DQID [36]. Tabula rasa 3-step DQN student (-- line) obtains an IQM teacher normalized score around 0.39. Shaded regions show 95% Cls. Left. Sample efficiencyy curves hased on IQM normalized scores, aggregated across 10 games and 3 runs, over the course of training. Among all algorithms, only QDagger (Section 4.1) surpasses teacher performance within 10 million frames. Right. Performance profiles [2] showing the distribution scores across all 30 runs at the end of training (higher is better). Area under an algorithm's 's profile is its mean performance while value where it intersects y 0.5 shows its median performance. QDagger outperforms the teacher in 75% of runs.
We also assume access to a dataset Dr that can be generated by the teacher (see Appendix A.5 for results about dependence on D.). For this work, Dr is the final replay buffer (1M transitions) logged by the teacher DQN, which is 100 times smaller than the data the teacher was trained on. For challenging PVRL setting, we use DQN as the student since tabula rasa DQN requires a substantial amount of training to reach the teacher's performance. To emphasize sample-efficient reincarnation, we train this student for only million frames, a times smaller sample budget than the teacher. Furthermore, we wean off the teacher at 6 million frames. See Appendix A.3 for more details.
Following Agarwal et al. [2], we report interquartile mean normalized scores with 95% confidence intervals (CIs), aggregated across 10 games with 3 seeds each. The normalization is done such that the random policy obtains score of 0 and the teacher policy Wr obtains a score of This differs from typically reported Homan-normalized scores, we wanted to highlight the performance differences between the student and the teacher. Next, we describe the approaches we investigate.
Evaluation.
Since the student, in principle, can learn using any off-policy data, we can replay teacher data Dr along with the student's 's own interactions during training. Following Paine et al. [66], the student minimizes the TD loss mini-batches that contain p% of the samples from Dr and the rest from the student's replay Ds (different and n-step values Figure A21.).
Rehearsal:
JSRL (Figure 3, left): JSRL [83] uses an interactive teacher policy as "guide" to improve exploration and rolls in with the guide for random number of environment steps. To evaluate JSRL, we vary the maximum number of roll-in steps, a, that can be taken by the teacher and sample a random number of roll-in steps between [0, a] every episode. As the student improves, we decay the steps taken by the teacher every iteration (IM frames) by a factor of B.
Offline RL Pretraining: Given access to teacher data Dr, we can pre-train the student using offline RL. To do so, we use CQL [46], a widely used offline RL algorithm, which jointly minimizes the TD and behavior cloning on logged transitions in Dr (Equation A.3). Following pretraining, we fine-tune the Iearned Q-network using TD loss on the student's replay Ds.
Kickstarting (Figure 3, right): Akin to kickstarting [74), we jointly optimize the TD loss with an on-policy distillation loss the student's 's self-collected data in Dg. The distillation loss uses the cross-entropy between teacher's ' policy गт and the student policy π (-1ss) = softmax ((s,, ((s..) -//)), where corresponds temperature. To wean off the teacher, we decay the distillation coefficient as training progresses. Note that kickstarting does not pretrain on teacher data.
DQND (Figure 4, left): Following DQfD [35], we initially pretrain the student on teacher data Dr using a combination of TD loss with a large margin classification loss to imitate the teacher actions (Equation A.4). After pretraining, we train the student on its replay data Ds. again using a combination of TD and margin loss. While DQfD minimizes the margin loss throughout training, we decay the margin loss coefficient during the online phase, akin to kickstarting.
5

---

--1
-
n-3 n-5
n- 10
10M Normalized Score
0.4
--10-10
n-3
LQM Normalized Score
0.8
0.3
0.6
0.2
0.1
0.2
0.0
100
1000
5000
2
8 10
Max. Teacher Roll-in Steps
Frames 1M)
Figure 3: Left. JSRL. The plot shows teacher normalized scores with 95% Cls. after training for 10M frames, aggregated using IQM across Atari games with 3 seeds each. Each point corresponds to a different experiment, evaluated using 30 seeds, with specific values of JSRL parameters (a, 3) and rt-tep returns. Right. Kickstarting, with different s-step returns. The plots show IQM scores over the coures of training. Kickstarting exhibits performance degradation, which is severe with 1-step, and is unable to surpass teacher's performance.
n-3, m-10
n-10 m-1.0
n=3, mm3.0
n=10, m=3.0
1QM Norma Score
0.8
Offline
Online
lize lized Score
1.2
Offline
Online
0.6
1.0
0.4
IOM Norma
0.8
0.2
0.6
0.4
1
5
10 2
4
10
10 Steps (X 100k)
8 10
Steps (X 100k)
Env. Frames (x 1M)
Env. Frames (x 1M)
Figure 4: Left. DQID. Here, m is the margin loss parameter, which the loss penalty when the student's action is different from the teacher. Right. QDagger, with different n-step returns. In both, the 1s vertical line separates pretraining phase from online phase while the nnd one indicates completely weaning off the teacher.
Results. Rehearsal, with best-performing teacher data ratio (p = 1/16), is marginally better than tabula rasa DQN but significantly underperforms the teacher (Figure 2, teal), which seems related to the difficulty of standard value-based methods to learn from off-policy teacher data [65]. JSRL does not improve performance compared to tabula rasa DQN and even hurts performance with large number teacher roll-in steps (Figure 3, left). The ineffectiveness of JSRL on ALE is likely due to the ditribiitribtiion mismatch between the student and the teacher, as the student may never visit the states visited by the teacher and as a result, doesn't 't learn to correct for its previous mistakes [16].
Pretraining with offline RL on logged teacher data recovers around 50% of the teacher's performance and fine-tuning this pretrained Q-function online marginally improves performance (Figure 2, pink). However, fine-tuning degrades performance with 1-step returns, which is more pronounced with higher values of CQL loss coefficient (Figure A. 13). We also find that kickstarting exhibits perfor- mance degradation (Figure 3, right), which is severe with 1-step returns, once wean off the teacher policy. Akin to kickstarting, we again observe a severe performance collapse when weaning off the the teacher dependence in DQID (Figure 4, left), even when using n-step returns. We hypothesize that this performance degradation is caused by the inconsistency between Q-values trained using a combination of imitation learning and TD losses, as opposed only minimizing the TD loss. We also find that using intermediate values of n-step returns, such as n (also used by Rainbow [35]), quickly recovers after the performance drop from weaning while larger n-step values impede learning, possibly due to stale target Q-values. These results reveal the sensitivity of prior methods in the PVRL setting to specific hyperparameter choices (n-step), indicating the need for developing stable PVRL methods that do not fail when weaning off the teacher. For practitioners, the takeaway is to consider this hyperparameter sensitivity when weaning off the teacher for reincarnation.
4.1 QDagger: A simple PVRL baseline
To address the limitations of prior approaches, we propose QDagger, a simple method for PVRL that combines Dagger [71], an interactive imitation learning algorithm, with n-step Q-learning (Figure 4, right). Specifically, we first pre-train the student on teacher data Dy by minimizing LoDagger (Dr)) which combines distillation loss with the TD loss, weighted by constant X. This pretraining phase

---

helps the student to mimic the teacher's state distribution, akin to the behavior cloning phase in Dagger. After pretraining, we minimize Degeer (Ds)
on the student's 's replay Ds akin kickstarting. where the teacher "corrects" the mistakes on the states visited by the student. As opposed to minimizing the Dagger loss indefinitely, QDagger decays the distillation loss coefficient X (Ag = A) as training progresses, to satisfy the weaning desiderata for PVRL.. Weaning allows QDagger to deviate from the suboptimal teacher policy тT, as opposed to being perpetually constrained to stay close to тI (Figure 9). We find that both decaying de linearly over training steps or using an affine function of the ratio of student and teacher performance worked well (A (Appendix A.3), Assuming the student policy x(- ()) softmax (Q(s /TT), the QDagger loss is given by:
LQDogger (D) = (DD + A.EE Σ Tr(a)s) [[(())))ooT(())))) lals)
(2)
Figure shows that QDagger outperforms prior methods and surpasses the teacher. We remark that DQID can be viewed as a QDagger ablation that uses a margin loss instead of distillation loss, while kickstarting as another ablation that does not pretrain on teacher data. Equipped with QDagger, we show how incorporate PVRL into our workflow and demonstrate its benefits over tabula rasa RL.
5 Reincarnating RL as a research workflow
Revisiting ALE. As Mnih et al. [60]'s 's development of Nature DQN established the tabula rasa work- flow on ALE, we demonstrate how iterating on ALE agents' design can be significantly accelerated using a reincarnating RL workflow, starting from Nature DQN, in Figure 1. Although Nature DQN used RMSProp, Adam yields better performance than RMSProp [1. 64]. While we can train another DQN agent from scratch with Adam, fine-tuning Nature DQN with Adam and 3-step returns, with reduced learning rate Figure 7), matches the performance of this tabula rasa DQN trained for 400M frames, using a 20 times smaller sample budget (Panel in Figure 1). As such, on a P100 GPU, fine-tuning only requires training for a few hours rather than a week needed for tabula rasa RL. Given this fine-tuned DQN, fine-tuning it further results in diminishing returns with additional frames due to being constrained to use the 3-layer convolutional neural network (CNN) with the DQN algorithm.
Let now consider how one might use a more general reincarnation approach improve on fine-tuning, by leveraging architectural and algorithmic advances since DQN, without the sample complexity of training from seratch (Panel 3 in Figure 1). Specifically, using QDagger to transfer the fine-tuned DQN, we reincarnate Impala-CNe CNN Rainbow that combines Dopamine Rainbow [35], which incorporates distributional RL [12], prioritized replay [73] and n-step returns, with an Impala-CNN architecture [26], deep ResNet with 15 convolutional layers. Tabula rasa Impala-CNi Rainbow outperforms fine-tuning DQN further within 25M frames. Reincarnated Impala- CNNN Rainbow quickly outperforms its teacher policy within 5M frames and maintains superior performance over its tabula rasa counterpart throughout training for 50M frames. To catch up with the performance of this reincarnated agent's 'sts performance, the tabula rasa Impala-CNN Rainbow requires additional training for 50M frames (48 hours on a P100 GPU). See Appendix A.4 for more training details. Overall, these results indicate how past research ALE could have been accelerated by incorporating reincarnating RL approach to designing agents, instead of always re- ve-training agents from scratch.
Tackling a challenging control task. To show how reincarnating RL can enable faster experimen- tation, we apply PVRL on the humanoid:run locomotion task, one of the hardest control problems in DMC [80] due to its large action space (21 degrees of freedom). For this experiment, shown in Figure 5, we use actor-critic agents in Acme [37]. For the teacher policy, we use TD3 [29] trained for 10M environment steps and pick the best run. We find that fine-tuning this TD3 agent degrades performance after 15M environment steps (other learning rates in Appendix A.4), which may be related to capacity loss in basue RI. with prolonged training [47, 56]. For reincarnation, we use single-actor D4PG [8]. a distributional RL variant of DDPG [54], with a larger policy and critic architecture than TD3. Reincarnated D4PG performs better than its tabula rasa counterpart for the first 10M environment interactions. Both these agents converge to similar performance, which is likely a limitation of QDagger. This result also raises the question of whether better PVRL methods can lead to reincarnated agents that outperform their tabula rasa counterpart throughout learning. Nevertheless, tabula rasa D4PG requires additional training for 10- 12 hours on a V V100 GPU to match reincarnated D4PG's performance, which might quickly add up to a substantial savings in compute when running large set of experiments (e.g., architectural or hyperparameter sweeps).
Balloon Learning Environment (BLE) [33]. One of the motivations of our work is to be able to use deep RL in real-world tasks in a data and computationally efficient manner. To this end, the BLE
7

---

110
600
DERG Relicamatioon DIRG
Toasher Normalized Scarets
105
500
TE3 IFins-tune!
Evaluation Return
100
400
95
300
90
200
100
80
OR-DON
I0N
R206
Reincamated
Reincamated
ION
RzDs
0.0 2.5 5.0 7.5 10.00 Environment Steps (x 1M)
15 Environment Steps (x 1M)
10
Fine
Figure 5: Reincarnating RL humanoid:run. (Panel 1). We observe that TD3 nearly saturates in performance after training for 10M environment steps. The dashed traces show individual runs while the solid line shows the mean return. (Panel 2). Rein- carnated D4PG performs better than its tabula rasa counterpart until the first 10M environment steps and then converges to sim- ilar performance (with lower variance). Furthermore, training TD3 for a large number of steps eventually results performance collapse. We use identically parameterized MLP critic and policy networks with 2 hidden layers of size (256, 256) for TD3 but larger networks with 3 hidden layers for D4PG. Shaded regions show 95% CIs based on seeds.
Figure 6: Comparing BLE agents.
See main text. We compare QR-DQN [23] with the same ML. architecture as Perci- atelli, IQN [22] with DenseNet [39], and R2D6. Reincarnated R2D6 outperforms Perciatelli well as the tabula rasa agents, but lags behind ane-ttned Perciatelli. We report the mean score (TWR50) across 10,000 evaluation seeds with varying wind difficulty, averaged over 2 independent runs. Error bars show minimum and maxi- mum scores on those runs.
provides a high-fidelity simulator for navigating stratospheric balloons using RL [II]. An agent in BLE can choose from three actions to control the balloon: move up, down, stay in place. The balloon can only move laterally by "surfing" the winds at its altitude; the winds change over time and vary the balloon changes position and altitude. Thus, the agent is interacting with a partially observable and non-stationary system, rendering this environment quite challenging. For the teacher, use the QR-DQN agent provided by BLE, called Perciatelli, trained using large-scale distributed RL for 40 days on the production-level Loon simulator by Bellemare et al. [11] and further fine-tuned in BLE. For our experiments, train distributed RL agents using Acme with 64 actors for budget of 50,000 episodes on a single cloud TPU-v2, taking approximately 10-12 hours per run.
In Figure 6, we compare the final performance of distributed agents trained tabula rasa (in pink), with reincarnation (in blue), and fine-tuned (in yellow). We consider three agents, QR-DQN [23] with MLP architecture (same as Perciatelli), IQN [22] with Densenet architecture [39], and recurrent agent R2D62 for addressing the partial observability in BLE. When trained tabula rasa, none of these agents are able to match the teacher performance, with the teacher-lookalike QR-DQN agent performing particularly poorly. As R2D6 and IQN have substantial architectural differences from the teacher, we utilize PVRL. for transferring the teacher. Reincarnation allows IQN to match and R2D6 to surpass teacher, although both lag behind fine-tuning the teacher. More details in Appendix A.4.2.
When fine-tuning, we are reloading the weights from Perciatelli, which was notably trained on broader geographical region than BLE and whose training distribution can be considered a superset of what is used by the other agents; this is likely the reason that fine-tuning does remarkably well relative to other agents BLE. Efficiently transferring information in Perciatelli's weights to another agent without the replay data from the Loon simulator presents interesting challenge for future work. Overall, the improved efficiency of reincarnating RL (fine-tuning and PVRL) over tabula rasa RL, as evident on the BLE, could make deep RL more accessible to researchers without access to industrial- scale resources as they can build upon prior computational work, such as model checkpoints, enabling the possible reuse of months of prior computation (e.g., Perciatelli).
6 Considerations in Reincarnating RL
Reincarnation via fine-tuning. Given access to model weights and replay of a value-based agent, a simple reincarnation strategy is to fine-tune this agent. While naive fine-tuning with the same learning rate (Ir) as the nearly saturated original agent does not exhibit improvement, fine-tuning
*R2D6 builds on recurrent replay distributed DQN (R2D2) [44], which uses a LSTM- based policy, and incorporates dueling networks [86], distributional RL [12], DenseNet [39], and double Q-leaming [84].
8

---

Nature DOK (r-5e-05)
tiin--uuue (r=6.25e-5) Ir=leS:
Raiboww iTabula Rasal Dro (Reincamation)
Rainbow (Reinammatoon
Nature
DrD iTabula Rasal
1.5
IQM Normalized Score
1.4
1QM Normalized Score
1.0
1.0
0.5
0.6
0.0
0
4
6
8
10
10
18
22
30
Env. Frames (x 1M)
Env. Frames (X 1M)
Figure 7: Reincarnation via fine-tuning with same
Figure Contrasting benchmarking results under tabula rasa and PVRL. settings.
and reduced Ir, relative the original agent.
with a reduced Ir, for only million additional frames, results in 25% IQM improvement for DQN (Adam) and 50% IQM improvement for Nature DQN trained with RMSProp (Figure 7). As reincarnating RL leverages existing computational work (e.g., model checkpoints). it allows us to easily experiment with such hyperparameter schedules, which can be expensive in the tabula rasa setting. Note that when fine-tuning, one is forced to keep the same network architecture; in contrast, reincarnating RL grants flexibility in architecture and algorithmic choices, which can surpass fine-tuning performance (Figures 1 and 5).
Difference with tabula rasa benchmarking. Are student agents that are more data-efficient when trained from scratch also better for reincarnating RL? In Figure 8, we answer this question in the negative, indicating the possibility of developing better students for utilizing existing knowledge. Specifically, we compare Dopamine Rainbow [35] and DrQ [90], under tabula rasa and PVRL settings. DrQ outperforms Rainbow in low-data regime when trained from scratch but underperforms Rainbow in the PVRL setting well when training longer from scratch. Based on this, we speculate that reincarnating RL comparisons might be more consistent with asymptotic tabula rasa comparisons.
Reincarnation vs. Distillation. PVRL is different from imitation learning or initation-regulariize RL as it focuses on using an existing policy only as launchpad for further learning. as opposed to imitating or staying close to it. To contrast these settings, we run two ablations of QDagger for reincarnating Impala-CNN Rainbow given a DQN teacher policy: (1) Dagger [71], which only mini- mizes the on-policy distillation loss in QDagger, and (2) Dagger QL, which uses fixed distillation loss coefficient throughout training (as opposed to QDagger, which decays it; see Equation 2). As shown in Figure 9, Dagger performs similarly to the teacher while Dagger QL improves over the teacher but quickly saturates in performance. On the contrary, QDagger substantially outperforms these ablations and shows continual improvement with additional environment interactions.
Dependency on prior work. While performance reincarnating RL depend on prior computational work (e.g., teacher policy in PVRL), this is analogous to how fine-tuning results in NLP / computer vision depend on the pretrained models (e.g., using BERT vs GPT-3). To investigate teacher de- pendence in PVRL, we reincarnate a fixed student from three different DQN teachers (Figure 10). As expected, we observe that a higher performing teacher results better performing student. However, reincarnation from two policies with similar performance but obtained from different agents, DQN (Adam) vs. a fine-tuned Nature DQN, results different performance. This suggests that a reincarnated student's performance depends not only the teacher's performance but also on its behavior. Nevertheless, the ranking of PVRL algorithms remains consistent across these two teacher policies (Figure A.1 See Section 7 for broader discussion about generalizability.
Reproducibility, Comparisons and Generalizability in Reincarnating RL
Scientific Comparisons. Fairly comparing reincarnation approaches entails using the exactly same computational work and workflow. For example, in the PVRL setting, the same teacher and data should be used when comparing different algorithms, as we do in Section 4. To enable this, it would be beneficial if the researchers can release model checkpoints and the data generated (at least the final replay buffers), in addition to open-source code for their trained RI. agents. Indeed, to allow others to use the same reincarnation setup as our work, we have already open-sourced DQN (Adam) agent checkpoints and the final replay buffer at gs: ht:://r__checppp

---

DQN - IImpala-CNN Rainbow (Reincamation)
QDagger
Dagger QL
Dagger
Offline
Online
IQM Normalized Score
Offline
Online
formalized Score
2.0
2.0
IQM Normalized
1.5
1.5
1.0
1.0
0.5
DON @ 20M
DON 400M
(Fine-Tunal 20M
1 5 10
10 15
20
25
30
5 10
10 15
20
25
30
Steps (x 100k)
Env. Frames (x 1M)
Steps (X 100k)
Env. Frames (x 1M)
Figure 9: Reincarnation vs. Distillation. Reincarnat- ing Impala- CNN Rainbow from a DQN (Adam) trained for 400M frames, using QDagger, and comparing it to Dagger (imitation) and Dagger + Q-learning (imitation- regularized RL...
Figure 10: Reincarnation from different teachers, namely, a DQN (Adam) policy trained for 20M and 400M frames and fine-tuned Nature DQN in Figure that achieves similar performance to DQN (Adam) trained for 400M frames.
Generalizability. The generalizable findings in reincarnating RL would be about comparing algo- rithmic efficacy given access to existing computational work on a task. As such, the performance ranking of reincarnation algorithms is likely to remain consistent across different teachers. fact, we empirically verified this for the PVRL setting, where we find that while using two different teacher policies, namely DQN(Adam)) vs. a fine-tuned Nature DQN, leads to different performance trends but the ranking of PVRL algorithms remain consistent: QDagger Kickstarting > Pretraining (see Figure 2 and Figure AI11). Practitioners can use the findings from reincarnating RL to try to improve on an existing deployed RL policy (as opposed to being restricted to running tabula rasa RL). For example, this work developed QDagger using ALE and applied it to PVRL on other tasks with existing policies (Humanoid-rus run and BLE).
Reproducibility. Reproducibility from scratch is challenging in RRL as would require details of the generation of the prior computational work (e.g., teacher policies), which may itself has been obtained via reincarnating RI. As reproducibility from scratch involves reproducing existing computational work, it could be more expensive than training tabula rasa, which beats the purpose of doing reincarnation. Furthermore, reproducibility from scratch is also difficult in NLP and computed vision, where existing pretrained models (e.g... GPT-3) are rarely, if ever, reproduced / re-trained from scratch but almost always used as-is. Despite this difficulty, pretraining-on and-finettuning is a dominant paradigm in NLP and vision [e.g.. 18, 25, 34, 38], and we believe that similar difficulty in RRL should not prevent researchers from investigating and studying this important class of problems. Instead, we expect that RRL research would build open-sourced prior computational work. Akin to NLP and vision, where typically a small set of pretrained models are used in research, we believe that research on developing better reincarnating RL methods also possibly converge of open-sourced models/ data on a given benchmark, e.g., the agents and data we released on Atari or the 25, 000 trained Atari agents released by Gogianu al. [31], concurrent to this work.
small set
Conclusion
Our work shows that reincarnating RL is a much computationally efficient research workflow than tabula rasa RL and can help further democratize research. Nevertheless, our results also open several avenues for future work. Particularly, more research is needed for developing better PVRL methods, and extending PVRL to learn from multiple suboptimal teachers [48, 53], and enabling workflows that can incorporate knowledge provided
form other than a policy, such as pretrained models [41, 79], representations [88], skills [50. 58, 68), or LLMs [3]. Furthermore, we believe that reincarnating RL would be crucial for building embodied agents open-ended domains [7, 27, 32). Aligned with this work, there have been calls for collaboratively building and continually improving large pre-trained models in NLP and vision [70]. We hope that this work motivates RL researchers to release computational work (e.g., model checkpoints), which would allow others directly build on their work. In this regards, we have open-sourced our code and trained agents with their final replay. Furthermore, re- re-purposing existing benchmarks, akin to how use ALE in this work, can serve as testbeds for reincarnating RL. As Newton put it "If I have seen further it is by standing on the shoulders of giants", we argue that reincarnating RL substantially accelerate progress by building on prior computational work, as opposed to always redoing this work from scratch.
10

---

Societal Impacts
Reincarnating RL could positively impact society by reducing the computational burden on re- searchers and more environment friendly than tabula rasa RL. For example, reincarnating RL allow researchers to train super- human Atari agents on a single GPU within a span of few hours as opposed training for a few days. Additionally, reincarnatigg RL is more accessible to the wider research community, as researchers without sufficient compute resources can build on prior computational work from resource-rich groups, and even improve upon them using limited resources. Furthermore, this democratization could directly improve RI. applicability for practical applications, as most businesses that could benefit from RI. often cannot afford the expertise to design in-house solutions. However, this democratization could also make it casier to apply RL for potentially harmful applications. Furthermore, reincarnating RL could carry forward the bias or undesirable traits from the previously learned systems. As such, we urge practitioners to be mindful of how RL fits into the wider socio-technical context of its deployment.
Acknowledgments
We would like thank David Ha, Evgenii Nikishin, Karol Hausman, Bobak Shahriari, Richard Song, Alex Irpan, Andrey Kurenkov for their valuable feedback on this work. We thank Joshua Greaves for helping us set up RL agents for BLE We also acknowledge Ted Xiao, Dale Schuurmans, Aleksandra Faust, George Tucker, Rebecca Roclofs, Eugene Brevdo, Pierluca D'Oro, Nathan Rahn, Adrien Ali Taiga, Bogdan Mazoure, Jacob Buckman, Georg Ostrovski and Aviral Kumar for useful discussions.
Author Contributions
Rishabh Agarwal led the project from start-to-finish, defined the scope of the work to focus on policy to value reincarnation, came up with successful algorithm for PVRL, and performed the literature survey, He designed, implemented and ran most of the experiments on ALE, Humanoid-run and BLE, and wrote the paper.
Max Schwarzer helped run DQND experiments on ALE and as well as setting up some agents for the BLE codebase with Aeme, was involved in project discussions and edited the paper. Work done as a student researcher at Google.
Pablo Samuel Castro was involved in project discussions, helped in setting up the BLE environment and implemented the initial Acme agents, and helped with paper editing.
Aaron Courville advised the project, helped with project direction and provided feedback on writing,
Marc Bellemare advised the project, challenged Rishabh come up with an experimental paradigm in which one continuously improves on an existing agent, and provided feedback on writing.
References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do can, not as say: Grounding language in robotic affordances. arXiv preprint arXtv:2204.0166 2022.
[4] Iige Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz. Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113. 2019.
[5] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-485 2009.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05882. 2022.
11

---

[7] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton. Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning act by watching unlabeled online videos. arXiv preprint urXiv:2206.1 11795, 2022.
[8] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint
2018.
[9] Wissam Bejjani, Rafael Papallas, Matteo Leonetti, and Mehmet R Dogar. Planning with receding horizon for manipulation in clutter using learned value fumction. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pages 1-9. IEEE, 2018.
[10] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[11] Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeap Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836)777-22 2020.
[12] Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press. 2022. http://www. distributiinns org.
[13] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer. Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 066600 2019.
[14] Reinaldo AC Bianchi, Carlos HC Ribeiro, and Anna HR Costa. Heuristically accelerated q-learning: new approach to speed up reinforcement learning. In Brazilian Symposium on Artificial Intelligence, pages 244-254. Springer. 2004.
[15] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.061 10. 2018.
[16] Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning, pages 2058-2066.
PMLR, 2015.
[17] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.0564/ 2015.
[18] Ting Chen, Simon Korbliihh, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255, 2020.
[19] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan, Hcuristic-guided guided reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[20] Wojeiech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference Artificial Intelligence and Statistics, pages
PMLR, 2019.
[21] Felipe Leno Da Silva, Garrett Warnell, Anna Helena Reali Costa, and Peter Stone. Agents teaching agents: a survey on inter-agent transfer learning. Autonomous Agents and Multi-Agent Systems, 34(1):1-17, 2020.
[22] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quamtile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018.
[23] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[24] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):4144112 2022.
[25] Jacob Devlin, Ming-We Beig-Weiio Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-traininn training of deep bidirec- tional transformers for language understanding, arXiv preprint arx0v=1810004805, 2018.
12

---

[26] Lasse Espeholt. Hubert Soyer. Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR. 2018.
[27] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang. De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internt-scale knowledge. arXiv preprint arXiv:2206.0883 2022.
[28] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in reinforcement lcarning agent. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pages 720-727, 2006.
[29] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[30] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstratious. arXiv preprint arXiv:1802.0555 2018.
[31] Florin Gogianu, Tudor Berariu, Lucian Buşoniu, and Elena Burccanu, Atari agents, 2022. URL https: //github. wom/foorrngooiaaiieeeooooooo.ooo www.oooiinnngggonooooooaoooaoaaa aataarariieee agentss.
[32] Djordje Grbic, Rasmus Berg Palm, Elias Najarro, Claire Glanois, and Sebastian Risi. Evocraft: new challenge for open-endedness. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar), pages 325-340. Springer, 2021.
[33] Joshua Greaves, Salvatore Candido, Vincent Dumoulin, Ross Goroshin, Sameera S. Ponda, Mare G. Bellemare, and Pablo Samuel Castro. Balloon Learning Environment, 12 2021. URL https://githuu. com/ google/ waw.eeeaaalooooooooo learning- environment.
[34] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[35] Mattco Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
[36] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[37] Matt Hoffman, Bohak Shahriari, John Aslamides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Aeme: A research framework for distributed reinforcement learning. arXiv preprint wxXiv22006.0097) 2020.
[38] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 801.06144 2018.
[39] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4700-4708, 2017.
[40] Peter Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, et al. A data-driven approach for learning to control computers. arXiv preprint 20X.:22002.08137, 2022.
[41] Andrew Hundt, Aditya Murali, Priyanka Hubli, Ran Liu, Nakul Gopalan, Matthew Gombolay, and Gregory D. Hager. "good robot! now watch this!": Repurposing reinforcement learning for task-to-task transfer. In 5th Annual Conference on Robot Learning, 2021. URL http:: htttppa://pennrvieee.. wwwpaaewwwww...eeeeeeeeeeeeeeeee htt.://nntt/toreeee/ee id=Pxe5XwId51n.
[42] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech Czameeki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Jain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 17 2017.
[43] Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning tuning in robotic reinforcement learning. arXiv preprint MoXiv:2004.101900 2020.
13

---

[44] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[45] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 1 2021.
[46] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[47] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- paddteraramttrizzationn inhibits data-efficient deep reinforeement learning. In International Conference on Learning Representations, 2021.
[48] Andrey Kurenkov, Ajay Mandlekar. Roberto Martin,Martia,, Silvio Savarese, and Animesh Garg. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. arXiv preprint arXiv: 19tp,04199 2019.
[49] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45 73. Springer, 2012.
[50] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Picter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.0016/ 2022.
[51] Seunghyun Lee, Younggyo Sco, Kimin Lec, Pieter Abbecl, and Jinwoo Shin. Offline-to-onlin reinforce- ment learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning pages 1702-1712. PMLR, 2022.
[52] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0666. 2020.
[53] Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-aware policy reuse. arXiv preprint arXiv: 1806.03793 2018.
[54] Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint SKKiv11509009711 2015.
[55] Yao Lu, Karol Hausman, Yevgen Chebotar. Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari. Dmitry Kalashnikov, et Aw-opt: Learning robotic skills with imitation andreinforcement at scale. In Conference on Robot Learning, pages 1078-1088 PMLR, 2022.
[56] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint 2244:2200 2022.
[57] Marlos C Machado, Marc G Bellemare, Erik Talvitic, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018
[58] Michael Matthews, Mikayel Samvelyan, Jack Parker-Holder, Edward Grefenstette, and Tim Rocktäschel. Hierarchical kickstarting for skill transfer in reinforcement learning. arXiv preprint arXiv:2207. 15844 2022.
[59] Azalia Mirhoscini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lec, Erie Johnson, Omkar Pathak, Azade Nazi, al. A graph placement methodology for fast chip design. Nature, 594(7862):207-212, 2021.
[60] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 511(7540):59-5533 2015.
[61] Ted Moskovitz, Michael Arbel, Jack Parker-Holder, and Aldo Pacchiano. Towards understanding of default policies in multitask policy optimization. International Conference on Artifiiial Intelligence and Statistics, pages 10661-10686. PMLR, 2022.
[62] Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292 6299. JEEE, 2018.
14

---

[63] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 09359, 2020.
[64] Johan S Obando-Cenne and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning (ICML), 2021.
[65] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficully of passive learning deep reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[66] Tom Le Paine, Caglar Gulcehre. Bobak Shahriari, Misha Denil, Matt Hoffiman, Hubert Soyer, Richard Tan- burn. Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making efficient use of demonstrations to solve hard exploration problems. arXiv preprint 110X:i::99000000 0187 2019.
[67] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimie: Deep multitask and transfer reinforcement learning. arXiv preprint 151 1.13444, 2015.
[68] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 2020.
[69] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley
& Sons, Inc., 1994.
[70] Colin Raffel. A call to build models like we build open-source software. https://colinratttt com/ aalog/-ccall-a to- moiilddmodels---e lieellsslkkeeeeee we- build-open- source- html, 2021.
[71] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artiffcial intelligence and statistics, pages 627 635. JMLR Workshop and Conference Proceedings, 2011.
[72] Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
[73] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 15X.525 2015.
[74] Simon Schmitt, Jonathan Johaitt,Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czamecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.08 1800.03835 2018.
[75] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[76] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and Alcksandr I Panov. Forgetful experience replay in hierarchical reinforeement learning from expert demonstrations. Knowledge- Based Systems, 218: 106844, 2021.
[77] William D Smart and Pack Kaelbling. Effective reinforcement learning for mobile rohots. Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), volume 4, pages IEEE, 2002.
[78] Wen Sun, J Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning, arXiv preprint arXiv:1805.1 12240 2018
[79] Yanchao Sun, Ruijic Zheng, Xiyao Wang, Andrew E Cohen, and Furong Huang. Transfer RI. across observation feature spaces via model-based regularization. In International Conference on Learning Representations, 2022. URL https: http:p::////ppppveeeeeeee
[80] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1800 1801000690 2018
[81] Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 1053-1060, 2013.
[82] Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choromanski, and David Li. Mlgo: a machine learning guided compiler optimizations framework. arXiv preprint arXiv:2101.04580 2021.
15

---

[83] Ikechukwu Uchendu, Ted Xiao, Yao Lu. Banghua Zhu, Mengyuan Yan, Joséphine Simon, Malthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, al. Jump-start reinforcement learning. arXiv preprint arXiv:2204. 02372, 2022.
[84] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, 2016.
[85] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi- agent reinforcement learning. Nature, 575(7782)3350-35, 2019.
[86] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement leaming. International conference on machine learning, pages 2003. PMLR, 2016.
[87] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-292, 1992.
[88] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06/73, 2022.
[89] Linhai Xie, Sen Wang, Stefano Rosa, Andrew Markham, and Niki Trigoni. Learning with training wheels: speeding up training with simple controller for deep reinforcement learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pages 6276-6283 6283. IEEE, 2018
[90] Denis Yarats, lyy Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representations, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 5 (BLE and continuous
control results) and reproducibility and evaluation concerns in Sec 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] agarwl
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.3 and A.4
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] 95% Cls
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Apache License, Version 2.0
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
16

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "xPw3ENBwL7f", "reviewer": "Reviewer_mLNV", "summary": "This paper presents an alternative class of problem statements to classical tabula rasa reinforcement learning, that is called reincarnating reinforcement learning (RRL). In RRL, instead of developing algorithms that learn from scratch (tabula rasa) in the RL environment, algorithms can make use of previous policies, and data collected from those policies, to speed up learning. This goes beyond standard offline reinforcement learning as (1) online learning is allowed, and (2) there's a possibility of access to the previous policy, and not just previous data generated from that policy. They instantiate a specific version of RLL in policy-to-value RL (PVRL), where the aim is to increase the speed of value-based online RL methods (such as DQN) with access to a previous (possibly suboptimal) policy and data from that policy. They present Q-dagger, an adjustment to standard deep Q-learning methods which uses n-step returns combined with a Dagger-style loss for both offline training and online from the pretrained policy to improve learning speed, and show it performs better than a variety of strong baselines in this setting in ALE, BLE and humanoid:run.", "questions": "Most of my suggestions are discussed in the weaknesses section above. \n\n* Do you see RRL and PVRL as research workflows that will produce (improvements to) algorithms that are useful for tabula rasa RL, or only for RRL/PVRL? \n* Could you run an ablation using the Qdagger loss in only the offline pretraining, and then doing standard  and online training stages? From figure 2a it seems that a lot of the benefit of Qdagger comes from the offline pretraining stage - it would be interesting to see how much additional benefit (on top of standard Q-learning) is gained by using the Qdagger loss during online training.", "limitations": "I believe the limitations are adequately addressed, given the fixes recommended in the Weaknesses section (which are more focused on unclear description of the framework that I think leads implicitly to unstated limitations in generality).", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe paper is well-written and clear to read. While (as mentioned in the paper) this kind of problem setting has been tackled before, it has always been in an ad-hoc way, and a more thorough motivation and investigation of it is beneficial and original. Further, it's likely that the vast majority of the usage RL algorithms in real-world settings will be in scenarios where previous data and/or policies are available, which makes this setting an important one to study.\n\nThe PVRL problem/solution setting is interesting, and their algorithm is robustly demonstrated to perform better than existing solutions on a range of benchmarks.\n\nFor all these reasons I'm recommending acceptance, but I think there are several issues, in terms of framing of the problem statement and contribution, that would make the paper clearer and aid future work more easily (see Weaknesses below).\n\n# Weaknesses\n\n## Framing\n\nWhile the RRL class of problem settings in an important and understudied one, I think the framing of the paper slightly confused what exactly RRL __is__. RRL seems like a class of problem settings, of which offline RL is one (in the limit of no online interaction or policy access), and PVRL is another (in which access to a previous policy and data from that policy is assumed). It would be beneficial to describe other possible instantiations of this problem setting; some obvious ones are having access to a previous policy but no data (which is often the setting in kickstarting-style methods), or conversely data but no policy (which is offline RL, possibly plus online fine-tuning); other factors of variation are the quality of the policy and/or data, and the amount of online training allowed.\n\nI think these settings are all valuable, and it's not necessarily the case that one is more general or flexible than another (as claimed for example on line 35; PVRL requires both a fixed dataset and previous policy, making it less general than methods that only require one or the other). It would also make certain distinctions clear that I think aren't in the paper; for example in line 165 it's claimed that assuming access to a previous policy allows one to assume access to a dataset from that teacher \"without loss of generality\", which doesn't seem to be the case, as there are examples where we have a previous policy but not necessarily data from that policy (we could only have data it was trained with, or no data at all).\n\nTo be specific, in the PVRL case, I think it makes more sense to just consider the constraints on the problem setting (previous policy and data, online learning allowed), and not couple that with the type of algorithm being trained (in this case a value-based method). There are many possible solutions to this problem, some of which are based on value-based RL methods and some are not. Hence I'd recommend defining (something like) PDRRL as policy-and-data-RRL as a problem instance within RRL, with PVRL as a class of solutions to this problem.\n\n## Reproducibility\n\nI think it's also important to address the reproducibility and comparability concerns of this research workflow or class of problems. For example consider the passage in lines 47-52:\n\n> For example, imagine a researcher who has trained a deep RL agent A1 for a long time (e.g., weeks), but now this or another researcher wants to experiment with better algorithms or architectures. While the tabula rasa workflow requires re-training another agent from scratch, reincarnating RL provides the more viable option of transferring A1 to another agent and training this agent further, or simply fine-tuning A1 (Figure 1).\n\nWhile this saves computation for the researcher, to make sure their algorithm is fairly compared with others in the same setting, other algorithms need to be trained with the same flow, which might require even more compute (if we need to reproduce the week-long training as well as the faster fine-tuning). As such, it would be beneficial (alongside releasing the code and model checkpoints) to release the data generated from those models - this means future researcher can work in exactly the same setting (i.e. the same inputs to their PVRL algorithm) so that the comparison is fair. Problems of this nature are even raised in the **Dependency on prior work** paragraph in section 6.\n\n## Concrete recommendations\n\n* Adjust the abstract and intro to frame RRL as a class of problems, of which PDRRL is one instance. Describe some other relevant/significant problem instances\n* Adjust the framing of PVRL to be a class of solutions to the PDRRL problem\n* Adjust the claims in L35 and L165, in light of the new framing above.\n* Address the reproducibility/comparability concerns; specifically, if another researcher was to attempt to produce a new method for PVRL, how would they go about doing that, to make sure their algorithm's results are comparable with those reported here.\n\nIf these recommendations and weakensses are addressed (which I think is only a matter of some light rewriting of the paper) then I'm happy to recommend a strong accept.\n\nEDIT: I thank the authors for their detailed response and the adjustments they've made. Given that these addressed my concerns, I'm happy to raise my score to a strong accept (8).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j3YlPJawdD", "reviewer": "Reviewer_jTCn", "summary": "The authors present the concept of \"reincarnating RL\" as an alternative to \"learning tabular rasa\". Reincarnating RL is very generally about using prior data/computation to more efficiently train new agents. They then introduce an algorithm called \"QDagger\" which is meant to solve a specific instance of \"reincarnating RL\" called \"policy-to-value reincarnating RL\" (PVRL). It performs well relative to baselines. \n\nThe main contributions of the paper seem to be:\n\n1. The promotion of \"reincarnating RL\" as an alternative approach to \"learning tabular rasa\"; along with examples of it as a research workflow. \n\n2. QDagger as a solution to a specific instance of \"reincarnating RL\" (PVRL).\n\nI will be writing the review considering both of these contributions separately.", "questions": "1. What example settings (beyond BLE) are you thinking of when it comes to \"real-world RL adoption\"? In what real-world scenarios would a broader community improving existing trained agents be helpful? How do you imagine this looks in practice?\n\n2. How is a real-world practitioner supposed to used the learnings from this line of work on a new task? Suppose that a series of papers for reincarnating RL that build on each other demonstrate SOTA performance on some set of tasks. Is the real-world practitioner supposed to then go through each of the different training / reincarnation phases? Right now, it is easy to run tabula rasa RL on a new environment.", "limitations": "This paradigm introduces its own host of problems around reproducibility and generalization. As mentioned in the paper, \"reincarnation from two distinct policies that perform similarly results in different performance trends\" (Line 336). Thus, it seems as though it may be very easy to overfit to specific implementations and seeds of the pre-trained agents. \n\nFurthermore, the precise details of the generation of the teacher policy / data (which themselves may have been reincarnated) become important for reproducibility; adding on to the challenge of reproducing work from scratch. I do not believe that this is addressed in the paper. \n\n", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality:\n\n## On \"Reincarnating RL\": \n\nStrength: This is an interesting notion that has not been explicitly discussed much in the literature before. \n\nWeakness: There is plenty of work that would fall into this vague category, as seen in the \"Related Work\" section. While this umbrella term / notion has not been explicitly discussed much, I'm not sure if generically promoting (and also coining a term for) a category of works could be considered an original contribution. \n\n## On \"QDagger\": \n\nStrength: The specific setting has not been previously studied much, and the precise details of this algorithm for this specific setting have not been proposed before.\n\nWeakness: This does not seem to be particularly original. The authors write that \"kickstarting [can be viewed] as another ablation that does not pretrain on teacher data\" (Line 236). Indeed when looking at the algorithms, they are very similar. If QDagger is just \"kickstarting with pretraining\" then it is not very original.\n\n# Quality:\n\n## On \"Reincarnating RL\": \n\nI'm really not sure how to evaluate this contribution in terms of quality. \n\nStrength: \n\nThe authors make a coherent argument that more attention in this area would better democratize and improve real-world RL adoption. They also show through case studies that experimenting under this paradigm is computationally cheaper and more accessible. Promoting the release of model checkpoints is helpful to the research community. \n\nWeakness:  \n\n1. I'm not sure if NeurIPS is the correct venue for what seems to largely be posturing / call-to-action. This interpretation may seem uncharitable, but significant portions of the paper are dedicated to this broad discussion instead of experiments, analysis, or theory.\n\n2. I'm not sure if I agree with \"reincarnating RL\" as being a superior approach for democratizing RL research. On the whole, it asks the question of what researchers should care about in reinforcement learning. A large part of the field is about studying and understanding how intelligent agents might/should learn tabula rasa (which is valuable in and of itself), not as much about real-world RL adoption. For example, the breakthrough DQN results in Atari are not interesting because they achieve high scores, but rather because a single algorithm could solve a large portion of the games tabula rasa. Only in the cases in which we \"care\" about ultimate performance (e.g. DOTA, Starcraft, Rubik's cube -- note that these are also not particularly useful in the real-world) do we consider reincarnating RL. Such scenarios currently seem rare in the real-world -- and it's unclear if this is because of a lack of work in reincarnating RL. On the other hand, environments such as [Minatar](https://github.com/kenjyoung/MinAtar) democratize research and allow for cheaper testing of tabula rasa RL.\n\n3. See the limitations section on reproducibility. This is problematic since science is about repeatably testable hypotheses. While it would be neat for a broad community to continually improve existing trained agents, it would not be a scientific endeavor as much as an engineering one.\n\n## On \"QDagger\": \n\nStrengths: The experiments and setup were well-done. \n\nWeaknesses: N/A.\n\n# Clarity:\n\nStrength: The paper as a whole is very well-written and clear.\n\nWeakness: N/A.\n\n# Significance:\n\n## On \"Reincarnating RL\": \n\nSee the discussion above. \n\n## On \"QDagger\": \n\nStrengths: It achieves impressive results in the tested settings. \n\nWeaknesses: Given the seeming lack of originality of the algorithm, it is difficult to say that the contribution itself is highly significant.\n\n# TL;DR\n\nI don't think the large amounts of posturing in the paper is particularly helpful to the broader community; however, the core algorithm and its results are very well-done and clean enough to warrant an accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6BNRJF8QZ-", "reviewer": "Reviewer_kCKQ", "summary": "The authors investigate reincarnating RL, which aims to utilize prior policies to train new agents rather than training RL agents from scratch. This paper formalizes the reincarnation setting. They demonstrate that existing approaches fail in this setting, and their method, n-step QDagger, they are able to outperform prior approaches. ", "questions": "1. Can the authors justify reincarnation in the most common RL settings? Also how should evaluations be reported in this case? Since reincarnating can lead to improved performance, but also has a dependence on an already trained teacher. ", "limitations": "The work sufficiently addresses limitations and societal impacts. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The results demonstrate that n-step QDagger is able to achieve strong performance (match, beat a fully trained DQN with far fewer steps in Atari, train faster than tabular rasa in humanoid:run, and outperform tabular rasa agents in BLE).\n2. They paper includes numerous baselines and shows hyperparameter sweeps over prior agents and demonstrate pitfalls of these agents. \n3. The paper felt quite lucid to read and typically straightforwards to follow. The bulleted list under evaluation was a little dense to read, not sure if it's easy or possible, but a better visualization of each method might make it easier to parse. \n\n## Weaknesses\n1. The main purpose of the paper is to formalize 'reincarnation.' Although I appreciate the extensive baselines and environments tested in the paper, it feels as if this work is formalizing a process that is in some ways well known. This process is valid; however, as noted by the authors already well documented in the RL community e.g. starcraft, dota, robotics for Rubik's cube solving. \n2.  For a lot of RL tasks: single agent RL (atari, mujoco, car racing), robotics (Habitat, Sawyer arms, etc.), multi-agent RL (Hanabi, SMAC, MPE, etc), the training time is typically much shorter e.g. on the order of a dozen hours to a few days per policy. With good engineering, we can also reduce training times. For example, with a well implemented Ape-X or R2D2, we can significantly reduce RL training times -- these also can still be single machine 1 GPU training runs. As a result, in the context of focuses in the broader RL community, I'm not entirely sure how impactful/how big of a role reincarnation will play. \n3. An issue of reincarnating could be related to diversity. A goal in reinforcement learning is to find diverse policies. Although this is not directly related to this work, reincarnating might not be able to fall into the workflow of training diverse policies. \n4. I'm not entirely sure what teacher the paper uses to train the reincarnated agents for BLE? My impression is they're using the weights from Perciatelli et al. In this case it feels like it'd make sense that reincarnated outperforms other variants, as reincarnated bootstraps off of a strong policy that has seen a much wider space then it is basically fine tuned to the BLE environment via reincarnation. This also explains why the fine-tuned Perciatelli model performs well in the BLE environment. \n\n## Minor\n- I think the claim \"JSRL does not improve performance compared to tabula rasa DQN and even hurts performance with a large number of teacher roll-in steps\" feels a little strong, given that it only hurts performance when $n=1$. However, I don't think using multi-step = 1 is very common particularly in atari. \n- I'm not entirely sure if the reference to Panel 3 in Figure 1 on line 251 is correct? Similarly, it would be helpful to reference the appropriate figure for the discussion in the second paragraph of sec. 5 (Fig A. 13, A. 14 I believe). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5V3OFWHXELD", "reviewer": "Reviewer_VM1W", "summary": "This paper proposes a new approach for reincarnating RL agents from pre-trained policies with different architectures and argues that RL papers should also adopt 'pre-training and fine-tuning' approach instead of training everything from scratch. In order to transfer pre-trained policy with different architectures, the method proposes Q-Dagger that learns value functions from a sub-optimal teacher policy and that is capable of continually decreasing the dependency on the sub-optimal teacher. Proposed method is compared to various baseline schemes on Atari and Balloon learning environments.\n", "questions": "- It would be nice to add a discussion on the dependency of the proposed method on the availability of offline datasets. Or I'm open to any follow-up discussion whether it's a limitation or not.\n- More experimental details on BLE would be nice for self-containedness of the paper\n- More clear explanation and take-away message of n-step experiemnts would be nice.\n- Clear explanation about TD3 performance degradation on humanoid run, or additional experiments with SAC without such phenomenon could make experimental results more convincing.", "limitations": "Limitation is not clearly discussed in the paper but potential negative societal impact is well discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Idea of reincarnating RL policies is definitely should be of interest to various researchers.\n- Idea is simple and working well\n- Exhaustive experiments with many baselines\n\nWeaknesses\n- Transferring pre-trained policy is definitely important and the paper proposes an intuitive and effective method for doing so. However, the proposed method is still limited in that it quite heavily depends on the availability of offline datasets, which have size of ~1M samples. This assumption gets often more difficult to hold because usually a replay buffer is implemented as a queue of fixed size in memory instead of storing all transitions in storage. The main benefit of pre-training and fine-tuning scheme in CV and NLP domains is that we can fine-tune a pre-trained model on datasets from target domains, which has a much smaller size than large, often huge, pre-training datasets. In this point of view, it is a bit difficult to see that the proposed method can be a generic solution for reincarnating RL and it's not clear whether only releasing pre-trained checkpoints could enable us to obtain the benefit as stated in the paper. It would be much more nice if the paper can show that the method works without pre-training student on offline dataset. I would not say that this experimental results are necessary because the method and results are still interesting, but more discussion on this front would make paper more interesting and insightful for future researches.\n- The paper is a bit not self-contained because it's missing some experimental details in Balloon Learning Environment, which is quite new benchmark so that it makes a bit difficult to understand the details. For instance, it is difficult to grasp what is Perciatelli and why is it called Perciatelli, and it's even not in the referred paper.\n- Experimental results on n-step returns are quite confusing; what's the main point of n-step return experiments? I understand that some methods are unstable without n-step returns, but why it's so important and what's the conclusion and practical take-away?\n- TD3 performance on humanoid run is also quite confusing; what is the reason for performance decrease of TD3? Does this also happen when we train TD3 from scratch without pausing the training after 10M steps? If so, why does this happen? and what is the reason TD3 is chosen for solving the task? It is known that SAC can solve humanoid run task without such degradation (see https://github.com/denisyarats/pytorch_sac for performance of SAC on humanoid run). This makes results a bit less convincing on humanoid experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept