Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate reincarnating RL, which aims to utilize prior policies to train new agents rather than training RL agents from scratch. This paper formalizes the reincarnation setting. They demonstrate that existing approaches fail in this setting, and their method, n-step QDagger, they are able to outperform prior approaches. 
strengths_and_weaknesses: ## Strengths
1. The results demonstrate that n-step QDagger is able to achieve strong performance (match, beat a fully trained DQN with far fewer steps in Atari, train faster than tabular rasa in humanoid:run, and outperform tabular rasa agents in BLE).
2. They paper includes numerous baselines and shows hyperparameter sweeps over prior agents and demonstrate pitfalls of these agents. 
3. The paper felt quite lucid to read and typically straightforwards to follow. The bulleted list under evaluation was a little dense to read, not sure if it's easy or possible, but a better visualization of each method might make it easier to parse. 

## Weaknesses
1. The main purpose of the paper is to formalize 'reincarnation.' Although I appreciate the extensive baselines and environments tested in the paper, it feels as if this work is formalizing a process that is in some ways well known. This process is valid; however, as noted by the authors already well documented in the RL community e.g. starcraft, dota, robotics for Rubik's cube solving. 
2.  For a lot of RL tasks: single agent RL (atari, mujoco, car racing), robotics (Habitat, Sawyer arms, etc.), multi-agent RL (Hanabi, SMAC, MPE, etc), the training time is typically much shorter e.g. on the order of a dozen hours to a few days per policy. With good engineering, we can also reduce training times. For example, with a well implemented Ape-X or R2D2, we can significantly reduce RL training times -- these also can still be single machine 1 GPU training runs. As a result, in the context of focuses in the broader RL community, I'm not entirely sure how impactful/how big of a role reincarnation will play. 
3. An issue of reincarnating could be related to diversity. A goal in reinforcement learning is to find diverse policies. Although this is not directly related to this work, reincarnating might not be able to fall into the workflow of training diverse policies. 
4. I'm not entirely sure what teacher the paper uses to train the reincarnated agents for BLE? My impression is they're using the weights from Perciatelli et al. In this case it feels like it'd make sense that reincarnated outperforms other variants, as reincarnated bootstraps off of a strong policy that has seen a much wider space then it is basically fine tuned to the BLE environment via reincarnation. This also explains why the fine-tuned Perciatelli model performs well in the BLE environment. 

## Minor
- I think the claim "JSRL does not improve performance compared to tabula rasa DQN and even hurts performance with a large number of teacher roll-in steps" feels a little strong, given that it only hurts performance when $n=1$. However, I don't think using multi-step = 1 is very common particularly in atari. 
- I'm not entirely sure if the reference to Panel 3 in Figure 1 on line 251 is correct? Similarly, it would be helpful to reference the appropriate figure for the discussion in the second paragraph of sec. 5 (Fig A. 13, A. 14 I believe). 
questions: 1. Can the authors justify reincarnation in the most common RL settings? Also how should evaluations be reported in this case? Since reincarnating can lead to improved performance, but also has a dependence on an already trained teacher. 
limitations: The work sufficiently addresses limitations and societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present the concept of "reincarnating RL" as an alternative to "learning tabular rasa". Reincarnating RL is very generally about using prior data/computation to more efficiently train new agents. They then introduce an algorithm called "QDagger" which is meant to solve a specific instance of "reincarnating RL" called "policy-to-value reincarnating RL" (PVRL). It performs well relative to baselines. 

The main contributions of the paper seem to be:

1. The promotion of "reincarnating RL" as an alternative approach to "learning tabular rasa"; along with examples of it as a research workflow. 

2. QDagger as a solution to a specific instance of "reincarnating RL" (PVRL).

I will be writing the review considering both of these contributions separately.
strengths_and_weaknesses: # Originality:

## On "Reincarnating RL": 

Strength: This is an interesting notion that has not been explicitly discussed much in the literature before. 

Weakness: There is plenty of work that would fall into this vague category, as seen in the "Related Work" section. While this umbrella term / notion has not been explicitly discussed much, I'm not sure if generically promoting (and also coining a term for) a category of works could be considered an original contribution. 

## On "QDagger": 

Strength: The specific setting has not been previously studied much, and the precise details of this algorithm for this specific setting have not been proposed before.

Weakness: This does not seem to be particularly original. The authors write that "kickstarting [can be viewed] as another ablation that does not pretrain on teacher data" (Line 236). Indeed when looking at the algorithms, they are very similar. If QDagger is just "kickstarting with pretraining" then it is not very original.

# Quality:

## On "Reincarnating RL": 

I'm really not sure how to evaluate this contribution in terms of quality. 

Strength: 

The authors make a coherent argument that more attention in this area would better democratize and improve real-world RL adoption. They also show through case studies that experimenting under this paradigm is computationally cheaper and more accessible. Promoting the release of model checkpoints is helpful to the research community. 

Weakness:  

1. I'm not sure if NeurIPS is the correct venue for what seems to largely be posturing / call-to-action. This interpretation may seem uncharitable, but significant portions of the paper are dedicated to this broad discussion instead of experiments, analysis, or theory.

2. I'm not sure if I agree with "reincarnating RL" as being a superior approach for democratizing RL research. On the whole, it asks the question of what researchers should care about in reinforcement learning. A large part of the field is about studying and understanding how intelligent agents might/should learn tabula rasa (which is valuable in and of itself), not as much about real-world RL adoption. For example, the breakthrough DQN results in Atari are not interesting because they achieve high scores, but rather because a single algorithm could solve a large portion of the games tabula rasa. Only in the cases in which we "care" about ultimate performance (e.g. DOTA, Starcraft, Rubik's cube -- note that these are also not particularly useful in the real-world) do we consider reincarnating RL. Such scenarios currently seem rare in the real-world -- and it's unclear if this is because of a lack of work in reincarnating RL. On the other hand, environments such as [Minatar](https://github.com/kenjyoung/MinAtar) democratize research and allow for cheaper testing of tabula rasa RL.

3. See the limitations section on reproducibility. This is problematic since science is about repeatably testable hypotheses. While it would be neat for a broad community to continually improve existing trained agents, it would not be a scientific endeavor as much as an engineering one.

## On "QDagger": 

Strengths: The experiments and setup were well-done. 

Weaknesses: N/A.

# Clarity:

Strength: The paper as a whole is very well-written and clear.

Weakness: N/A.

# Significance:

## On "Reincarnating RL": 

See the discussion above. 

## On "QDagger": 

Strengths: It achieves impressive results in the tested settings. 

Weaknesses: Given the seeming lack of originality of the algorithm, it is difficult to say that the contribution itself is highly significant.

# TL;DR

I don't think the large amounts of posturing in the paper is particularly helpful to the broader community; however, the core algorithm and its results are very well-done and clean enough to warrant an accept.
questions: 1. What example settings (beyond BLE) are you thinking of when it comes to "real-world RL adoption"? In what real-world scenarios would a broader community improving existing trained agents be helpful? How do you imagine this looks in practice?

2. How is a real-world practitioner supposed to used the learnings from this line of work on a new task? Suppose that a series of papers for reincarnating RL that build on each other demonstrate SOTA performance on some set of tasks. Is the real-world practitioner supposed to then go through each of the different training / reincarnation phases? Right now, it is easy to run tabula rasa RL on a new environment.
limitations: This paradigm introduces its own host of problems around reproducibility and generalization. As mentioned in the paper, "reincarnation from two distinct policies that perform similarly results in different performance trends" (Line 336). Thus, it seems as though it may be very easy to overfit to specific implementations and seeds of the pre-trained agents. 

Furthermore, the precise details of the generation of the teacher policy / data (which themselves may have been reincarnated) become important for reproducibility; adding on to the challenge of reproducing work from scratch. I do not believe that this is addressed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new approach for reincarnating RL agents from pre-trained policies with different architectures and argues that RL papers should also adopt 'pre-training and fine-tuning' approach instead of training everything from scratch. In order to transfer pre-trained policy with different architectures, the method proposes Q-Dagger that learns value functions from a sub-optimal teacher policy and that is capable of continually decreasing the dependency on the sub-optimal teacher. Proposed method is compared to various baseline schemes on Atari and Balloon learning environments.

strengths_and_weaknesses: Strengths
- Idea of reincarnating RL policies is definitely should be of interest to various researchers.
- Idea is simple and working well
- Exhaustive experiments with many baselines

Weaknesses
- Transferring pre-trained policy is definitely important and the paper proposes an intuitive and effective method for doing so. However, the proposed method is still limited in that it quite heavily depends on the availability of offline datasets, which have size of ~1M samples. This assumption gets often more difficult to hold because usually a replay buffer is implemented as a queue of fixed size in memory instead of storing all transitions in storage. The main benefit of pre-training and fine-tuning scheme in CV and NLP domains is that we can fine-tune a pre-trained model on datasets from target domains, which has a much smaller size than large, often huge, pre-training datasets. In this point of view, it is a bit difficult to see that the proposed method can be a generic solution for reincarnating RL and it's not clear whether only releasing pre-trained checkpoints could enable us to obtain the benefit as stated in the paper. It would be much more nice if the paper can show that the method works without pre-training student on offline dataset. I would not say that this experimental results are necessary because the method and results are still interesting, but more discussion on this front would make paper more interesting and insightful for future researches.
- The paper is a bit not self-contained because it's missing some experimental details in Balloon Learning Environment, which is quite new benchmark so that it makes a bit difficult to understand the details. For instance, it is difficult to grasp what is Perciatelli and why is it called Perciatelli, and it's even not in the referred paper.
- Experimental results on n-step returns are quite confusing; what's the main point of n-step return experiments? I understand that some methods are unstable without n-step returns, but why it's so important and what's the conclusion and practical take-away?
- TD3 performance on humanoid run is also quite confusing; what is the reason for performance decrease of TD3? Does this also happen when we train TD3 from scratch without pausing the training after 10M steps? If so, why does this happen? and what is the reason TD3 is chosen for solving the task? It is known that SAC can solve humanoid run task without such degradation (see https://github.com/denisyarats/pytorch_sac for performance of SAC on humanoid run). This makes results a bit less convincing on humanoid experiments.
questions: - It would be nice to add a discussion on the dependency of the proposed method on the availability of offline datasets. Or I'm open to any follow-up discussion whether it's a limitation or not.
- More experimental details on BLE would be nice for self-containedness of the paper
- More clear explanation and take-away message of n-step experiemnts would be nice.
- Clear explanation about TD3 performance degradation on humanoid run, or additional experiments with SAC without such phenomenon could make experimental results more convincing.
limitations: Limitation is not clearly discussed in the paper but potential negative societal impact is well discussed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an alternative class of problem statements to classical tabula rasa reinforcement learning, that is called reincarnating reinforcement learning (RRL). In RRL, instead of developing algorithms that learn from scratch (tabula rasa) in the RL environment, algorithms can make use of previous policies, and data collected from those policies, to speed up learning. This goes beyond standard offline reinforcement learning as (1) online learning is allowed, and (2) there's a possibility of access to the previous policy, and not just previous data generated from that policy. They instantiate a specific version of RLL in policy-to-value RL (PVRL), where the aim is to increase the speed of value-based online RL methods (such as DQN) with access to a previous (possibly suboptimal) policy and data from that policy. They present Q-dagger, an adjustment to standard deep Q-learning methods which uses n-step returns combined with a Dagger-style loss for both offline training and online from the pretrained policy to improve learning speed, and show it performs better than a variety of strong baselines in this setting in ALE, BLE and humanoid:run.
strengths_and_weaknesses: # Strengths

The paper is well-written and clear to read. While (as mentioned in the paper) this kind of problem setting has been tackled before, it has always been in an ad-hoc way, and a more thorough motivation and investigation of it is beneficial and original. Further, it's likely that the vast majority of the usage RL algorithms in real-world settings will be in scenarios where previous data and/or policies are available, which makes this setting an important one to study.

The PVRL problem/solution setting is interesting, and their algorithm is robustly demonstrated to perform better than existing solutions on a range of benchmarks.

For all these reasons I'm recommending acceptance, but I think there are several issues, in terms of framing of the problem statement and contribution, that would make the paper clearer and aid future work more easily (see Weaknesses below).

# Weaknesses

## Framing

While the RRL class of problem settings in an important and understudied one, I think the framing of the paper slightly confused what exactly RRL __is__. RRL seems like a class of problem settings, of which offline RL is one (in the limit of no online interaction or policy access), and PVRL is another (in which access to a previous policy and data from that policy is assumed). It would be beneficial to describe other possible instantiations of this problem setting; some obvious ones are having access to a previous policy but no data (which is often the setting in kickstarting-style methods), or conversely data but no policy (which is offline RL, possibly plus online fine-tuning); other factors of variation are the quality of the policy and/or data, and the amount of online training allowed.

I think these settings are all valuable, and it's not necessarily the case that one is more general or flexible than another (as claimed for example on line 35; PVRL requires both a fixed dataset and previous policy, making it less general than methods that only require one or the other). It would also make certain distinctions clear that I think aren't in the paper; for example in line 165 it's claimed that assuming access to a previous policy allows one to assume access to a dataset from that teacher "without loss of generality", which doesn't seem to be the case, as there are examples where we have a previous policy but not necessarily data from that policy (we could only have data it was trained with, or no data at all).

To be specific, in the PVRL case, I think it makes more sense to just consider the constraints on the problem setting (previous policy and data, online learning allowed), and not couple that with the type of algorithm being trained (in this case a value-based method). There are many possible solutions to this problem, some of which are based on value-based RL methods and some are not. Hence I'd recommend defining (something like) PDRRL as policy-and-data-RRL as a problem instance within RRL, with PVRL as a class of solutions to this problem.

## Reproducibility

I think it's also important to address the reproducibility and comparability concerns of this research workflow or class of problems. For example consider the passage in lines 47-52:

> For example, imagine a researcher who has trained a deep RL agent A1 for a long time (e.g., weeks), but now this or another researcher wants to experiment with better algorithms or architectures. While the tabula rasa workflow requires re-training another agent from scratch, reincarnating RL provides the more viable option of transferring A1 to another agent and training this agent further, or simply fine-tuning A1 (Figure 1).

While this saves computation for the researcher, to make sure their algorithm is fairly compared with others in the same setting, other algorithms need to be trained with the same flow, which might require even more compute (if we need to reproduce the week-long training as well as the faster fine-tuning). As such, it would be beneficial (alongside releasing the code and model checkpoints) to release the data generated from those models - this means future researcher can work in exactly the same setting (i.e. the same inputs to their PVRL algorithm) so that the comparison is fair. Problems of this nature are even raised in the **Dependency on prior work** paragraph in section 6.

## Concrete recommendations

* Adjust the abstract and intro to frame RRL as a class of problems, of which PDRRL is one instance. Describe some other relevant/significant problem instances
* Adjust the framing of PVRL to be a class of solutions to the PDRRL problem
* Adjust the claims in L35 and L165, in light of the new framing above.
* Address the reproducibility/comparability concerns; specifically, if another researcher was to attempt to produce a new method for PVRL, how would they go about doing that, to make sure their algorithm's results are comparable with those reported here.

If these recommendations and weakensses are addressed (which I think is only a matter of some light rewriting of the paper) then I'm happy to recommend a strong accept.

EDIT: I thank the authors for their detailed response and the adjustments they've made. Given that these addressed my concerns, I'm happy to raise my score to a strong accept (8).
questions: Most of my suggestions are discussed in the weaknesses section above. 

* Do you see RRL and PVRL as research workflows that will produce (improvements to) algorithms that are useful for tabula rasa RL, or only for RRL/PVRL? 
* Could you run an ablation using the Qdagger loss in only the offline pretraining, and then doing standard  and online training stages? From figure 2a it seems that a lot of the benefit of Qdagger comes from the offline pretraining stage - it would be interesting to see how much additional benefit (on top of standard Q-learning) is gained by using the Qdagger loss during online training.
limitations: I believe the limitations are adequately addressed, given the fixes recommended in the Weaknesses section (which are more focused on unclear description of the framework that I think leads implicitly to unstated limitations in generality).
review_title: null

Meta-review: metareview: This paper proposes a novel method for transferring prior policies across design and system changes to improve the sample efficiency of RL algorithms, which could ultimately help unlock RL for real-world use cases. 
There was an active discussion across the reviewing process in which the authors managed to address the concern of the reviewers, leading to updated scores. 

Based on the contributions of the paper and highly positive final reviews I recommend the paper for acceptance. 

confidence: Certain
Decision: Accept