Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a pre-training scheme (APT) for RL with two components: contrastive representation learning and particle based entropy maximization. Experiments are done in DeepMind Control Suite (DMControl) and Atari suite to show improved performance and data efficiency. 

**Strength**: Pre-training good representations and policy initialization without reward is obviously an important direction in RL. The proposed method is conceptually simple and intuitive, yet achieves some promising results. Particle based entropy estimation (eq. (5)) can be a simple and interesting solution to estimating observation entropy in high dimension space.

**Weakness**: Some concerns about experiment results:
- For DMControl results, it seems APT only improves significantly over "From scratch" and "Count-based pre-training" schemes when reward is sparse? I also think the result can be made much stronger if more powerful baselines can be adopted, e.g. DIAYN, VISR, etc.
- For Atari results, APT versus VISR is interesting. APT achieves better median normalized scores but VISR achieves better average normalized scores. Is it possible to show a game breakdown to understand where these methods work better?
- In section 4.3, it is awkward that I find APT and APT (representation) similar across different curves in Figure 4, so I'm not convinced the policy initialization is useful. Maybe some more results (e.g. on Atari) can help with this. 
Also see the **Novelty** part below.

**Novelty**: The two cores of the paper, contrastive representation learning and entropy maximization, are quite established in RL. Particle based entropy estimation is from prior work, but I think it is fairly novel and interesting in the RL domain (despite its usefulness being doubted). 

**Clarity**: I think the paper is written clearly in general.

**Question**: Conceptually eq. (5) looks quite noisy (dependent on batch samples a lot), and like some form of contrastive learning objective still (maybe entropy maximization is equivalent to making representation contrastive?). So is it really a stable reward signal? Is it possible to just use eq. (1) as reward also?
review_title: important direction, simple scheme, but some concerns about experiments and novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission presents a technique for unsupervised pre-training of representations and policies for RL. Unsupervised representation learning has obtained impressive results in supervised scenarios, and adapting these methods to RL is an important research direction. One of the main challenges in the RL setting is that of defining the distribution of data to learn from, as well as sampling from it. The learned representations are unlikely to be useful for observations that are out of the pre-training distribution, so it is desirable to perform representation learning on data that is representative of the full state space. Previous works (Hazan et al., Lee et al.) proposed strategies to train agents that induce maximally entropic state visitation distributions, but they involve density estimation whose underlying assumptions are not well suited for pixel observations. The authors propose to overcome these limitations by using a particle based entropy estimate in the learned representation space. The pre-trained representations and policies can be used for RL from pixels, obtaining faster convergence and higher end scores than the considered baselines in both DMControl and Atari.

The method is novel and the experimental results are strong. The paper includes a proper literature review and it is generally easy to follow. However, it would benefit from a more detailed analysis in order to understand where the reported gains come from. The ablation studies suggest that most of the gains are due to the learned representations: fine-tuning the policy as well only provides slightly faster convergence in two Hopper tasks, but the end performance is the same as when transferring representations only. I would like to suggest the following:
- Reporting the performance of zero-shot transfer for all experiments, i.e. the average return of the pre-trained policy in each task (APT@0). I suggest doing this for both DMControl and Atari experiments.
- It would be very helpful to see per-game scores in Atari in order to understand what type of environments benefit the most from pre-training. Gains could come from faster convergence on dense reward games thanks to the pre-trained representations, or from higher end performance on hard exploration tasks due to the exploratory behavior of the pre-trained policy. This is important given the fact that APT obtains much higher median human normalized scores than VISR while reaching much lower mean scores.
- Related to the previous point, comparing full APT and APT (representation) in a selection of Atari games requiring different degrees of exploration would provide insight on the type of challenges APT is addressing.
- APT (representation) can be understood as pre-training a state representation. How does this version of APT compare to RL from true state on DMControl? A similar analysis was reported by Srinivas et al. (Figure 7, see full reference below).

Please note that most of the suggestions above do not require running new experiments and can be addressed by providing additional information about the ones that are already reported in the paper.


Other comments:
- To the best of my knowledge, there doesn’t exist a standard procedure for fine-tuning policies, especially for actor-critic architectures such as the one in SAC. It would be helpful to include a description on how this is performed. Is the critic fine-tuned as well, or is it initialized from scratch?
- Section 3 mentions a robustness analysis showing the impact of varying k in kNN, but I didn’t see it in the paper. Is it referring to initial experiments, or to some ablation study that was not included for some reason?
- SimCLR fits representations using an objective that is based on cosine similarity, but the reward derived from the particle-based entropy estimate employs L2 distance. Is there a reason for this discrepancy?
- The subtitle “Supervised training @100k” in Table 1 is not very accurate, as two out of five rows use more than 100k interactions. I suggest using “Fully-supervised training” as the title and appending “@100k” to the name of the first three methods.
- A single unsupervised pre-training can be leveraged to solve multiple tasks as long as the environment does not change, as shown in the DMControl experiments. I believe this is one of the most appealing properties of the method and might not be highlighted enough in the paper.


The following works are relevant and might be worth citing:
- Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. "Curl: Contrastive unsupervised representations for reinforcement learning". ICML 2020.
- Lee, Lisa, et al. "Efficient exploration via state marginal matching". arXiv preprint arXiv:1906.05274 (2019).
review_title: Official Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: *SUMMARY*

The paper proposes a method to simultaneously learn effective representations and efficient exploration in a reward-free context. The algorithm iterates between minimizing a contrastive loss and maximizing an intrinsic reward derived from a k-NN entropy estimation of the state distribution. Then, authors empirically evaluate the method over a set of visual Mujoco tasks and Atari games.

*STRENGTHS*
- the paper addresses a very relevant reward-free exploration objective as a preprocessing to RL
- the paper combines representation learning and state entropy maximization into a promising practical method

*WEAKNESSES*
- the paper might be too incremental with respect to previous (albeit unpublished) work
- the paper somewhat fails to empirically illustrate and validate the reward-free phase

*EVALUATION*

Unfortunately, over some concerns regarding the novelty of the presented method and its experimental validation, which I find somewhat weak for an essentially empirical work, I would lean towards rejecting the paper.

*DETAILED COMMENTS*

C1) The exploration component of APT has striking similarities with the method in [1], which also seeks the optimization of a k-NN estimate of the state distribution entropy in a reward-free context. While it would be in general acceptable to overlook unpublished work, I think that the connections with [1] are too many to avoid a deeper discussion over distinctive contributions. The method in [1] does not seem to learn representations, but I am wondering if this contribution alone would be substantial enough.

C2) The paper does not present an explicit empirical evaluation of the reward-free phase, thus I am not sure on how APT is performing in entropy maximization. Especially, what is the impact of the three sources of bias that are introduced in the entropy estimation (avoiding bias correction and constants, avoiding importance weighting, scaling distances with the standard deviation)?
Can authors present state entropy plots, and possibly compare the performance of APT with other methods seeking a similar objective, such as MaxEnt (with representation learning), SMM [2] or MEPOL [1].

C3) I have some concerns on the theoretical and practical implications of considering a reward function that is actually depending on the current policy. It is sound to fit a value function for a reward of this kind? Maximizing an ever-changing reward might cause instability and prevent convergence?

C4) The benefit of employing a non-parametric method to estimate the entropy of high-dimensional inputs is clear, since density modeling would be quite hard. Could density modeling over a reduced latent space be a viable option instead?

C5) I would argue that the idea of simultaneously learning representations and exploration is the main selling point of the presented method, since maximizing the state entropy might help learning superior representations and viceversa. But from the ablation study in Section 4.3 this conclusion does not arise naturally, as learning representations alone seems almost as good as learning both. May I ask the authors to clarify this point?

C6) The scores on Atari games are reported without confidence intervals, leaving some doubts over the statistical significance of the results. Moreover, it is not completely clear from the aggregate performance where and how APT is helping in these experiments. Can authors provide a deeper explanation on why the original performance of SimPLe and VISR is not reproducible?

[1] Mutti and Restelli. A policy gradient method for task-agnostic exploration. Arxiv, 2020.
[2] Lee et al. Efficient exploration via state marginal matching. Arxiv, 2019

*QUESTIONS*

May the authors address the comments listed above in their response?

*ADDITIONAL FEEDBACK*
- For the Atari experiments I would suggest to focus more on hard-exploration games, such as Montezuma or Pitfalls, instead of providing just an aggregate performance over full sets of games. It would be nice to include some visualizations and interpretations on the behavior that APT learns in the reward-free phase.
- I believe that the multi-environment pre-training setting is quite interesting and, to the best of my knowledge, completely novel. The results are promising, and I would suggest to include this setting, together with a deeper analysis, in the main paper.
- Dashing the lines in the reported plots would help visibility, especially without colors.
- typos and rephrasing:
	- when referring to the environment I would use reward-free instead of task agnostic (e.g., page 2, paragraph 3)
	- the 26 games subset instead of the 100k subset (page 7, last lines)
	- I would rephrase "The notable difference is that APT (representation) decouples the action space dimension from pre-trained models" which is not crystal clear (Section 4.3). 


####################

AFTER RESPONSE

I would like to thank authors for their detailed response and for their effort in improving the paper according to reviewers' suggestions. Unfortunately, after authors clarifications, I still have some doubts on the concerns raised with C1 and C2 (see below). Thus, I am keeping a slightly negative evaluation for this paper. 

Authors claim that the main benefit of APT over a prior work method (MEPOL, Mutti er al., 2020) is a lower variance of the gradient estimation, thanks to the choice of avoiding importance weights corrections. However, in Figure 6 MEPOL does not seem to suffer a particularly high-variance. To me, the most likely reason for the improved performance is that APT guarantees an action-level feedback as opposed to a trajectory feedback (see [1]).
However, I am still skeptical about this action feedback: the reward-to-go becomes non-Markovian and Bellman equations does not hold anymore (see [2]). This casts some doubts on the actor-critic procedure APT employs to optimize the rewards. Authors may have a good point on the notion that the encoder is breaking the dependence between policy and rewards, but I think the topic warrant some additional discussion.

I would suggest the authors to rephrase this work to give a more central role to the scalability to high-dimensional observations, which I believe is the main contribution of the paper, and to include a more thorough discussion of (Mutti et al., 2020) in the main text (beyond the related work section).

[1] Efroni et al. Reinforcement learning with trajectory feedback. Arxiv, 2020.

[2] Zhang et al. Variational policy gradient method for reinforcement learning with general utilities. NeurIPS 2020.

review_title: Learning reward-free exploration via k-NN entropy estimation

Meta-review: The authors propose a particle-based entropy estimate for intrinsic motivation for pre-training an RL agent to then perform in an environment with rewards. As the reviewers discussed, and also mentioned in their reviews, this paper bears stark similarity to work of 5 months ago, presented at the ICML 2020 Lifelong ML workshop, namely, "A Policy Gradient Method for Task-Agnostic Exploration", Mutti et al, 2020--MEPOL. What is novel here is the adaptation of this entropy estimate to form an intrinsic reward via a contrastive representation and the subsequent demonstration on standardized RL environments.  The authors have added a comparison to MEPOL, and in these experiments, APT outperforms this method, sometimes by some margin. Unfortunately this work does not meet the bar for acceptance relative to other submissions.

Decision: Reject