Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the transfer of unsupervised RL agents in the Atari-57 benchmark suite. 

The authors propose behavior transfer (BT), a simple method for transfer that focuses on using the behavior of the pre-trained policy, separately from using the pre-trained policy as an initialization (a.k.a. fine-tuning). BT augments a downstream off-policy learner in two ways: the downstream learner can use the pre-trained policy to explore (for a temporally extended duration), and the pre-trained policy is also added as a pseudo-action to the action space as an option during exploitation (for a single time step).

In the experiments, the authors primarily focus on using the never give up (NGU) objective to obtain the pre-trained policy and assess the impact of BT with this pre-trained policy on downstream learning via recurrent replay distributed DQN (R2D2). The protocol differs from prior works in that each learning phase is much longer: 16B frames per game for pre-training (a 64x increase over [28]) and up to 5B frames per game for transfer (a 12500x increase over [35]). This is shown to be important overall. For standard Atari-57, transfer via BT compares favorably against R2D2 baselines that lack unsupervised pre-training, with more pronounced gains for hard exploration games. Ablations show that both BT methods of using the pre-trained policy are necessary for best performance. The authors also assess on custom tasks in the Ms Pacman and Hero games, with BT causing significant improvement over R2D2 baselines despite the zero-shot pre-trained policy used for BT not achieving any success. Finally, the authors also assess BT with fine-tuning from (some subset of) the pre-trained policy's weights. For standard Atari-57, fine-tuning improves over training from scratch for both R2D2 as well as R2D2+BT. However, finetuning without BT from a partial initialization outperforms any variant involving BT, an interesting negative result. 
main_review: ## Originality
The proposed method, behavior transfer (BT), is, to my knowledge, new. However, it may be helpful to reference and discuss prior works such as [A], which also broadly consider the use of pre-trained behavior in adaptation to new tasks, albeit in a different domain (simulated robotics) and with different problem assumptions (e.g. acquisition of pre-trained behavior via demonstrations). While Atari-57 is a well-established RL benchmark, the authors do bring a new perspective to unsupervised pre-training for Atari-57 by demonstrating the effect of scale in both the unsupervised pre-training and supervised adaptation phases. The authors also generate new reward functions to assess transfer to tasks other than the original games, though this is only done for two games.

## Quality
Given that the novelty of BT is the use of pre-trained policies in specific ways during subsequent off-policy RL and not the intrinsic motivation objectives used to train the policies, I was surprised that the R2D2 baseline agents were run without RND/NGU. To me, this seems vital in order to observe the effect of BT in a controlled manner so that we know what to attribute any improvements to. Otherwise, the experiments appear to be thoroughly executed and documented. There are no theoretical claims to assess.

## Clarity
The writing and presentation of the method and results are clear. However, I am not sure that the contributed method is appropriately named. Certainly, fine-tuning a pre-trained policy involves "behavior transfer" since the behavior of the pre-trained policy collects the data used for transfer learning, but fine-tuning, as the authors point out, is orthogonal to the proposed method. BT also seems too broad for any specific method. I would further recommend updating the title and/or abstract to reflect the scope of the method and experiments, specifically the focus on off-policy RL and Atari-57.

## Significance
Given the simplicity of BT, I imagine that future works will use or build on it. Getting transfer to work well in the context of RL is an open question, and the idea that transferring a pre-trained policy can involve mechanisms distinct from fine-tuning is an interesting idea that I expect the field to consider going forward. However, the fact that BT with finetuning is not the best variant for the overall Atari-57 suite should give us pause. The direct impact of this work is also severely limited by the execution and scope of its experiments: aside from not controlling for the use of intrinsic motivation/exploration objectives, it's not clear whether the effective use of pre-trained policies for exploration is specific to pre-training objectives that focus on encouraging exploration. 

## References
[A] Singh et al., Parrot: Data-Driven Behavioral Priors for Reinforcement Learning, ICLR 2021.

limitations_and_societal_impact: Limitations are discussed above. The potential societal impact of the work was not discussed by the authors.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the problem of transferring pre-trained behavior for exploration in reinforcement learning. They propose an approached called (BT) which relies on the pre-trained policy for collecting experience through temporally-extended exploration, which can be triggered with some probability at any step, or  one-step calls to the pre-trained policy based on value estimates.
The experiments presented in this work show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors.
main_review: *Originality:* The authors have proposed a novel approach for transferring pre-trained behaviour for exploration. While they use existing unsupervised pretraining using an existing method (NGU), how pre-trained policy is applied to downstream RL tasks with reward is novel. The authors made it very clear how their work differs from previous contributions and have adequately cited relevant papers.

*Quality*: The submission is technically sound. The claims are supported using the empirical results. 

*Clarity*: The manuscript is well written and well organised. My only questions related to the Ablation studies Section. Specifically, in the first few sentences of this section, the authors talk about some results without proper reference to any figures. Please clarify this and refer to any figure which illustrates this. 

*Significance*: I think that the results are important and useful for the RL community. The problem setting itself is very well motivated. However, I have the following concerns that I'd like the authors to address:

- The methods are only tested on the Atari benchmark. While I understand that this is what other work in transfer learning might also use for evaluation, I believe the paper can be strengthened if the methods are tested in more interesting environments. Atari tasks are singletons (every episode look the same) and deterministic, whereas for RL methods to generalise to real-world settings, approaches need to handle generalisation and stochasticity. Therefore, I'd suggest the authors try using some procedurally generated environments, such as [OpenAI ProcGen](https://github.com/openai/procgen), [NetHack](https://github.com/facebookresearch/nle), [MiniGrid](https://github.com/maximecb/gym-minigrid) or [MiniHack](https://openreview.net/forum?id=skFwlyefkWJ). In case these approaches don't work well on procedurally-generated tasks, I'd like the authors to explain why is this the case and how this can be improved in future work.

- I'd also be interested in seeing baselines transfer learning results using [Kickstarting](https://arxiv.org/abs/1803.03835) from the pre-trained policy. Since kickstarting is also agnostic to how pre-trained policy is trained, it can be used for transfer learning.
limitations_and_societal_impact: Only one limitation of BT was mentioned in the discussion section (that BT assumes that flights can be started from any state and still produce meaningful behavior.). There are no negative societal impact as far as I'm concerned.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes an approach to transferring pretrained models in a deep RL context. Instead of pretraining a neural network and then simply copying the weights and retraining on a new task (as would be the naive approach taken in supervised learning), the paper instead proposes transferring the pretrained policy's behaviour. This is done by modifying the epsilon-greedy exploration strategy to include the ability to use the action suggested by the pretrained policy, which can also be used as a kind of multi-step option for improved exploration.  Pretraining is conducted using the "Never Give Up" (NGU) intrinsic reward in a reward-free setting, and experiments are conducted in the full set of Atari games. Results and ablation studies indicate that such a transfer approach can drastically improve the performance of an agent on a new task. 

The main contributions are a) the use of NGU as an unsupervised pretraining objective in RL; b) a method for incorporating a pretrained policy when learning a new policy; c) experimental results and ablation studies across the full suite of Atari tasks that demonstrate the improvement resulting from leveraging the pretrained model when exploring in a new task.
main_review: 
There is a lot to like about this paper. I found it extremely polished, well-written and easy to read. The proposed approach is simple, but in a good way, since it can be incorporated into existing work and codebases with very little effort. The experiments are also very extensive, with results reported over all of the Atari domains. However, I have some concerns regarding the novelty, whether the benchmark environments are appropriate, and some missing related work, which I'll discuss below. 

Main comments:

1. As mentioned, one of the main positives of the paper is the ease with which the method can be integrated into existing work. Despite lacking theoretical justification, the empirical results across all the Atari games clearly show that, in practice, we can achieve far better performance with this kind of pretraining and transfer approach. I found the gap between the performance of the pretrained policy and the final policy interesting, since it clearly shows that we don't need a particularly strong pretrained policy (or that it doesn't have to be well aligned with the downstream task) for everything to be effective. 

2. My main concern is that I'm not sure whether the Atari suite is the correct testbed for this method. One reason why pretraining on say ImageNet in the supervised learning setting is so useful is that the models can be reused on other domains that are not necessarily ImageNet, but which operate in the image domain. In this case, though, a policy is pretrained only on the task that we're trying to solve.  So we need to pretrain a network on Montezuma's Revenge in order to play Montezuma's Revenge, but that's all we can do with it. I think the key here is the importance of amortisation - the cost of pretraining must be paid off by being able to solve downstream tasks quickly. But in the Atari suite, we can't just pretrain one policy and then use it for any Atari game - we need to pretrain for each and every Atari game, which defeats the purpose of pretraining in the first place, since no amortising is taking place. Given this, it is perhaps unsurprising that a neural network pretrained for 16bn timesteps on game A then has subsequently improved performance on game A!  Granted, there are two modified domains that attempt to address this issue, but considering just these two limits the experimental results and there's no display of amortisation.    
If the aim is to stick with Atari, then a more convincing approach would be to have a single pretrained model that we can use to solve any subsequent Atari game - the cost of pretraining could then be amortised over 57 subsequent tasks! However, I feel a better domain would have been something like DeepMind lab, where the "base" MDP captures the dynamics of the open world, and then subsequent downstream tasks would all be conducted in the same environment (e.g. collecting different objects). This to me would be more analogous to the ImageNet case, since we could then reuse the policy for each task within the DeepMind lab domain, much as we can reuse the ImageNet model in any natural image task. I realise that suggesting a completely different experimental setup is not immediately feasible, so I would also be open to discussion on why the current Atari experiments are appropriate. 


3. Another issue I'm unsure of is the novelty and contribution of the work, which seems to be combining existing techniques. I thought the discussion of an appropriate pretraining strategy was interesting (since other approaches have attempted to do things like autoencoders etc), and I think this could open many avenues for future work regarding how best to do so in the deep RL setting. However, the actual method of behavioural transfer is a combination and modification of existing techniques, which limits the novelty. For example, using a different policy to solve a new task by modifying the exploration behaviour has been done in prior work (e.g. [1] who call it \pi-reuse). The difference here is simply that a) the policy can be used as an n-step option, and b) by adding in a virtual action, the agent can learn whether or not to explicitly use the pretrained policy. The contribution then feels like it's mainly about putting together these existing ideas and showing that they work well empirically.

4. While there's a good discussion on prior work, I was very surprised to see no mention of reward-free RL [2], which is exactly the setting considered here (agent learns in a domain with no rewards, and then must quickly learn a downstream task after this pretraining phase). Although much of the work in this space is restricted to tabular [2, 3, 4] or linear function approximation [5], the advantage with these approaches is that they have guarantees. While not directly comparable to this work, I think it is definitely worth mentioning them.   

Minor comments:

1. Future work could potentially focus on other ways of incorporating the pretrained policy, such as using entropy regularised RL or similar.

2. On line 164, should it not read that you update a^+ as well as the action chosen by the pretrained policy? At the moment, you have \pi(s) and a', which are the same thing. 

3. A couple of the references have arxiv versions, whereas published ones exist (e.g. Brown et al. [9] was at NeurIPS 2020)

[1] Fernández, Fernando, and Manuela Veloso. "Probabilistic policy reuse in a reinforcement learning agent." Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems. 2006.  
[2] Jin, Chi, et al. "Reward-free exploration for reinforcement learning." International Conference on Machine Learning. PMLR, 2020.  
[3] Zhang, Zihan, Simon Du, and Xiangyang Ji. "Near Optimal Reward-Free Reinforcement Learning." International Conference on Machine Learning. PMLR, 2021.  
[4] Zanette, Andrea, et al. "Provably efficient reward-agnostic navigation with linear value iteration." NeurIPS, 2020.  
[5] Wang, Ruosong, et al. "On reward-free reinforcement learning with linear function approximation." NeurIPS, 2020.


***POST-REBUTTAL***

Thank you for the response, the comparison to prior work and additional experiments that showcase the importance of the *multistep* flight. Despite this, I still have concerns about the use of Atari which result in the following shortcomings (ignoring the Ms PacMan results) :

1. Due to the use of Atari, transfer here is restricted to single target task instances (source task is the game without reward and the target task is the game with reward). However, multitask transfer (for target tasks with different reward functions/dynamics) could have been shown by using a different testbed like Marion, ProcGen, Retro, etc).
2.  Related to this, the motivation in the introduction is that computer vision benefits from this pretraining. However, that's because that these models can transfer to multiple target tasks, not just one. The fact that the multitask transfer does not occur here is a limitation of  Atari, not the approach, which seems like a waste. 
3. I'm not sure I can agree with the assertion that pretraining is "free" because there's no reward function. Why should that be?  Better motivation here would be helpful. But again, if we just had the ability to transfer to multiple tasks, then the pretraining costs could be recouped in the long run. 

If Atari is a must, one final suggestion would be to use the different game "modes" offered by the ALE (Machado et al, 2018). Then, at the very least, multitask transfer could be shown on the different version of the same game.   
  
limitations_and_societal_impact: There are perhaps two limitations to this work that could also be discussed. 

1. Adding a virtual action allows an agent to learn whether or not to reuse the pretrained policy, but it is unclear how we would immediately incorporate this into a continuous-action setting, since we would then have a mixture of continuous and discrete actions (there are, of course, approaches to overcoming this issue, but it wouldn't be straight-forward)
3. The results here show the importance of an extended pretraining period. However, this may simply not be possible given the domain - how would one collect 10 years worth of data on a single robot?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a simple, intuitive method for fine-tuning “behaviors” by pretraining an exploration policy before using it as a temporally extended exploration procedure and a one-step additional action to the downstream policy on a downstream task. They extensively evaluate this on the suite of Atari games and demonstrate strong performance gains. Despite some qualms I have with assumptions and ablations, I think this paper should be accepted to NeurIPS. However, I do hope that the authors address some of my concerns in the rebuttal period. For now, this is a "weak accept" but if my concerns are addressed properly the score will be raised appropriately.

main_review: ## Paper Strengths
**Method and Framing**
The method is simple and intuitive, and demonstrates strong performance improvements. Furthermore, the paper questions the traditional train-then-finetune paradigm that is typical in transfer learning for reinforcement learning. This is an interesting point to focus on and is useful to think about more for transfer learning in reinforcement learning in general, irrespective of the exact method.

**Clarity**
This paper is well written, with a good flow that integrates background knowledge, some related works, and the method itself within the first four sections.

**Experiments**
Demonstrating that hard exploration games like Montezuma’s Revenge benefit from more unsupervised environment frames after transfer despite it not having an effect on 0-shot transfer performance is sensible. Furthermore, Figure 4, which highlights the extra action selection distribution over training during behavior transfer, is intriguing and demonstrates that the policy really learns how to take advantage of the pre-trained exploration policies.

Overall, the number and scope of the experiments is impressive, highlighting many strengths of the method under different conditions.


## Paper Weaknesses
**Pretraining Assumption**
“Interactions between the agent and the environment are often assumed to incur a cost, but we will consider this cost to be relevant only for transitions with reward” - I’m not sure this is a valid assumption. I think the authors should include a comparison/figure which includes the total number of samples including pre-training samples for BT, as sample efficiency is important partially for reasons related to total algorithm running time. Here, having these large numbers of pre-training steps still greatly increases the algorithm running time. 

**Technical Contributions**
Nothing here is particularly surprising. However, that is fine given that this method works well and is relatively easy to implement.

**Experiments**
The “Transfer to Multiple Tasks” experiment would’ve been more enlightening if there were more tasks and/or more reward functions tested. The authors clearly have the computational resources to run comprehensive experiments on the Atari suite, so it seems strange to only perform this experiment on two tasks with just two reward functions each. Can the authors experiment on more environments?

**Ablations**
Why one-step action execution of the pretrained policy? A temporally extended action should still be useful here. Can the authors should include an ablation with different length of action executions of this pretrained policy?


-- UPDATE: score revised from 6 to 7 --
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper describes a simple but empirically effective strategy for unsupervised pre-training policies and utilizing them for exploration, with a temporally extended approach particularly effective for hard exploration challenges. The reviewers agree that the paper's concepts, while simple, are relevant and have potential to impact the community. Several reviewers raised concerns regarding the placement of the paper with respect to existing work and choices of experimental comparison. During responses, the authors presented additional detail (empirical and motivational) that has satisfied some, but not all reviewers. 

In my opinion, this is a paper is just below the NeurIPS borderline, as the papers brings great performance on important problems, but does borrow elements heavily from existing approaches and gives less theory/analytical justification for its advances. Reviewers and authors have had some consensus about the value in further exploration of the multi-task properties, with potentially non-Atari setups or by finding more ways to run multi-task trials in modified Atari levels. I also encourage such exploration to more firmly establish the impact of the ideas.

A side note that has not continbuted to the decision: the term "transfer" led to some misunderstanding during our discussions. For some reviewers (and myself), transfer can refer to many sub-areas of RL where testing is not done in Atari: changes in robotic dynamics, sim2real, multi-task and more. Here "learning from policies learned in an unsupervised pre-trained fashion" would provide the needed context with certainty, despite being a mouthful.

Decision: Reject