Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
#### Summary:

In this work the author focus on transfer in RL via proposing to transfer knowledge through behavior instead of representations. They propose using coverage as an objective for the pre-training procedure. They then employ the NGU (Badia 2020) They also propose a method based on coverage pre-training for transfer and provide empirical evidence in support of their method for transfer in RL on the Atari suite.  


#### Strengths:

- The paper has a good experimentation section, including empirical analysis such as ablation studies and effect or pre-training that is insightful.

#### Weakness:

- **Novelty**: The notion of coverage has been explored in the past, for instance in [1, 2, 3]. The authors fail to differentiate what do they do in their method that fixes the limitations of the previous work. 

- **Incremental Nature of work**: The proposed method relies heavily on the Never Give Up (Badia et al) for finding the policies that maximize the coverage. It is not clear what this work is proposing other than using NGU in the transfer setting directly.
 
- **Cost of pretraining**: The pre-training comes at a cost. In their setup, the agents are allowed as many interactions with the environment as needed. This can be extremely futile for high-dimensional state and action spaces. If the dynamics model doesn't change for the downstream tasks, then this choice of pre-training is a valid choice (that has already been explored in the previous works as the authors mention). However, if there is a change in dynamics then there can be scenarios where the pre-training procedure doesn't result in any benefits. The procedure mentioned in Section 3 is already being employed in many different previous works, and it is not clear what is the setting being considered in this work. Also in Section 3, $\pi_i$ is undefined. 

- **Missing references**: The authors make statements and then fail to give the appropriate references for the same. For instance, the authors quote that "RL techniques have not yet seen the advent of a two-stage setting where task-agnostic pre-training is followed by efficient transfer to downstream tasks", however, there are multiple works that are based on essentially this approach ([4, 5]).
 
- **Reproducibility**: There is no mention of code release and makes me skeptical of their results, especially when this an empirical results-driven work.


#### References:

- [1] Schmidhuber, Jürgen. "Formal theory of creativity, fun, and intrinsic motivation (1990–2010)." IEEE Transactions on Autonomous Mental Development 2.3 (2010): 230-247.
- [2]  Hazan, Elad, et al. "Provably efficient maximum entropy exploration." International Conference on Machine Learning. 2019.
- [3] Conti, Edoardo, et al. "Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents." Advances in neural information processing systems. 2018.
- [4] Lesort, Timothée, et al. "State representation learning for control: An overview." Neural Networks 108 (2018): 379-392.
- [5] Ha, D. and Schmidhuber, J. (2018). World Models. ArXiv e-prints.

review_title: Recommendation to Reject

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper studies a pre-training approach to reinforcement learning. The objective is, first to pre-train a model considering that, without reward, interaction with an environment is cheap, and second, to fine-tune a policy given a particular reward function. 

As a first contribution, the paper proposes two strategies to use a pre-trained policy for discovering an efficient task-dependent policy: i) the action set is expanded such that the policy can choose to follow the pre-trained policy and ii) exploration may be done by following the pre-trained policy on t timesteps, t being randomly chosen. As a second contribution, the paper proposes a criterion to pre-train a policy based on a coverage criterion.  The principle is to encourage a policy to generate trajectories that are going through as many states as possible. 

Experiments are made on different environments. First, the pre-trained policies are evaluated based on how much reward they are able to collect, and compared to other unsupervised approaches. Second, the final policy is compared to epsilon-greedy and epsilon-z-greedy approaches. In the two cases, the proposed approach outperform the baselines. 

== Comments: 
First of all, the paper clearly lacks of details, and it is difficult to be sure about what is really done, and what is the final algorithms. As far as I understand, instead of proposing a really new approach, the paper is more stacking two approaches (i.e NGU and R2D2, just changing a little bit the action space) and it does not really provide any justification about what is done. From my point of view, it is more a paper investigating if coverage may be a good unsupervised training criterion than a paper presenting a new model. 

Concerning the exploration/exploitation of pre-train policies, the paper does not really describe how the 'flights' are generated (section 3). I would advise the authors to provide more details on this aspect of their algorithm. 

Concerning the coverage approach, I am not convinced that the paper allows us to draw any conclusion on the interest of using such a criterion during the pre-training phase. Indeed, the authors are mainly evaluating their approach on Atari games, on which there is a clear relationship between the length of the episodes and the final score achieved by an agent. This is what is shown in Section 5.1: coverage is a good surrogate objective for solving atari games.  The article is thus lacking evaluation on other types of environments, and the performance obtained by the model on Atari games is mainly due to the use of the NGU model which has been developed more specifically for this type of environment.

To conclude, I think that the paper is failing to provide evidence that the coverage approach is a good approach to unsupervised pre-training of policies. Moreover, I have the feeling that the coverage criterion may be good for particular types of environments (like Atari games), but not for some others, making the proposed approach very specific to particular problems. Combined with the lack of novelty, and the lack of details, I recommend to reject the paper. 

Considering the answers from the authors, I decided to not change my score. 
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a methodology to use a pre-training phase (latent learning) where the agent tries to maximize the coverage of the state space to then bootstrap task solving where the agent focuses on the task-defined reward.

The paper doesn’t match conference standards in terms of description of the method. There is no clear explanation (algorithm ? figure?) of the process, how the pre-trained phase and the next phase interact, nor which information is transferred (behaviour = policy ?). No proper definition of coverage is provided. Is the pre-training phase a time when your agent learns based on a different (intrinsic) reward function ? If yes, what is this function? Is it only NGU ?
If I understood correctly, the contribution of the work is limited to training NGU without external reward, then using this pretraining to initialize the agent in an exploitation phase. 

One critical question that is not answered: if you gather data in a pre-training phase, why don’t you estimate a (transition) model ? The properties of model-based RL are quite interesting in terms of finding a more optimal policy compared to learning a reactive policy. They might be arguments in favour of keeping a model-free approach but they are not presented.

The results are in accordance with the claim of the authors that their approach, favouring exploration, improves the final performance. However, you do not evaluate the actual coverage of the agent. Reward itself is a coarse metric, is suitable for the agent performance, but it is a weak claim to say that the differences in performance are due to better coverage. Please provide a measure of the improved coverage to support your claim.

In conclusion, the paper lacks a proper structure to explain the method, make it reproducible by readers; it is unclear what the contribution is, and it seems quite limited. This flaw makes it difficult to understand the results (that are nevertheless supporting the claims). 

review_title: The paper does not provide clear description of the method, despite encouraging results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposed a transfer approach for reinforcement learning. The proposed approach leverages a policy pre-trained via Never Give Up (NGU) approach, and can facilitate learning challenging RL tasks including the ones with sparse reward. This paper presents many strong pieces of evidence that this approach can be used to tackle challenging RL problems including hard exploration and multi-task learning. 

Strengths
* Necessity of transferring behavior is well motivated and experiment showed its strength in challenging RL benchmarks.
* Figure 2 is useful to have intuition on how the proposed transfer method can be useful
* Table 1 set a strong unsupervised RL performance for Atari Suite
* Ablation study in Figure 3 is thought-provoking. It is interesting that the significant gain of pretraining come only when both exploitation and exploration method are used jointly.
* Figure 4 provides a useful intuition that more pretraining is beneficial for transfer to hard exploration task.

Weaknesses
* The "flights" technique is not described in detail in the main text. I managed to find the detail in Appendix A, but the pointer does not exist in the main text.
* The paper claim "coverage" as the desired objective for RL pretraining and tried to support this claim by showing the transfer performance after pretraining via Never Give Up (NGU). I am convinced that NGU is a good pre-training objective but, it is not clear whether a more general claim for "coverage" is supported as well. It is not clear whether NGU is optimizing "coverage" well, and the relation between "coverage" and transfer performance is not studied. 

Comment / Questions to author
* "but little research has been conducted towards leveraging the acquired knowledge once the agent is exposed to extrinsic reward": I'm not sure whether I agree with this description. My understanding is that (Burda et al., 2018) studied a setting where an intrinsic reward is jointly used with extrinsic rewards. The only difference with this work is that the previous work did not study a setting with a clear separation between "pre-training" and "transfer".
* Is there a difference between "ez-greedy with expended action set A+ (using pre-trained policy)" vs "the proposed transfer method (exploitation + exploration)"?
* I'm curious about the comparison between CPT vs joint training with extrinsic reward. How authors would compare CPT vs joint training?

Recommendation
I recommend accepting this paper because this paper presented strong evidence that unsupervised pre-training and transfer may be a powerful approach to solve many challenging RL problems. I believe this observation is likely to catalyze future research of the related approaches, and the proposed method itself may be used for different domains to improve the capability of RL in general.
review_title: Well motivated work with strong experimental results

Meta-review: The paper studies the unsupervised RL problem, where the agent is allowed to interact with the environment for a certain amount of time without any extrinsic reward. The main idea is that the initial unsupervised training phase can be used to learn a set of "skills" that could help both in exploration and zero-shot transfer for any downstream task.

There is general consensus among the reviewers that the paper is studying an important problem and that the empirical validation is solid. Nonetheless, the technical contribution and the positioning wrt to the relevant literature are relatively weak for proposing acceptance for the paper. Before entering into details, I would like to acknowledge the fact that the rebuttal and the revised version did improve the original submission and clarified some aspects (eg, the structure of the algorithm), yet the contribution does not seems strong enough.

The idea of state space coverage is indeed not novel, either for exploration or for transfer (as properly reviewed in the paper). The authors identified some weaknesses of existing methods (eg, estimating state distributions), but it remains unclear whether the algorithm they propose has any significant technical contribution. In fact, as confirmed by the authors, CPT is rather applying any algorithm that *could* perform a good state space coverage and learn policies at the same time and then use the learned policy during the downstream task combined with a relatively basic "option-level" eps-greedy strategy. In this sense, it seems like CPT overcomes the limitations of previous algorithms, just by applying another existing algorithm (NGU) that is more scalable. While the evidence that this is "enough" to obtain good results is indeed interesting, it doesn't seems like it is pushing the algorithmic state of the art forward.

A more substantial contribution would be to dive deeper into the state coverage problem and provide an algorithm that is more specifically designed for the transfer setting considered in the paper. In fact, there is no clear evidence that NGU is the *right* approach to perform good coverage and return "useful" skills. Since this is the core concern of the paper, the technical contribution should be more significant on this part. The "meta" algorithm in itself seems rather standard otherwise.


Decision: Reject