Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce Active Pretraining (APT), wherein an agent is encouraged to maximization the entropy of the state distribution represented in its replay buffer. The is accomplished by having rewards that are proportional to distance to nearest neighbours in some embedding space. This space is constructed via a separate contrastive learning objective.

This method is tested in a 2 stage process where the agent initially maximizes this state entropy entropy for a long time, and then finetunes on a down-stream time for a much shorter time, with the final performance on this task being the key metric. By this methodology, APT surpasses both data-efficient methods that train from scratch, as well as other methods that use the same 2 stage training scheme.
main_review: The novelty of the method is a bit slight, as using state-entropy as an intrinsic reward has been attempted elsewhere. That said, the authors acknowledge this prior work, and emphasize the scalability and performance of their system. This puts much more pressure on the empirical results to do the heavy lifting, and luckily they largely deliver on this promise.

The initial negative result showing that naive approaches to pretraining (e.g. ImageNet) are insufficient to accelerate downstream tasks provides some nice motivation. The main results on DMControlSuite and Atari have a large number of baselines whose inclusion is well motivated in the text. The performance of APT is quite convincing, but I would suggest including some notion of spread in all of the table and figures so that the significance is easier to assess. Both ablating contrastive learning from APT and including it in some baseline methods was a nice touch, and the other ablations are also appreciated. That said, Appendix A did give me pause, as the number of architectural flourishes used here feels higher than many other RL papers (e.g. layer norm, spectral norm), but considering you compare favourably to a Rainbow variant (which is probably the fanciest data efficient agent to use as a base) I won't hold that against you.

One significant concern involve proposition 1. It's correct, if a bit trivial, but it's unclear that it matters in practice, as the number of distinct states is far higher than the number of training steps for the pixel-based domains that are core to this method's evaluation. The applicability is further undercut by the reward normalization scheme -- as the reward goes to zero, the normalized reward goes to infinity! And I'm not even convinced one would even want the intrinsic reward to vanish asymptotically, as that suggests that pretraining for too long would result in a random policy. Luckily I think these two problems cancel out; if you don't want your reward to vanish, then just removing the proposition and the surrounding text would alleviate my concern.

Minor points:
1) The "Exploration" tick marks in Table 1 feel very subjective. Maximizing state or action entropy is exploration, but related mutual information terms isn't? Even if there is an argument for this distinction, since just switching to a base RL method that includes an entropy bonus (e.g. SAC) would resolve it, categorizing methods along this axis feels unhelpful.
2) The fine-tuning strategy employed here (when combined with a value function) relies on the intrinsic and extrinsic rewards being of a similar scale. This is in contrast to the regression-style approach of VISR, and should thus probably be mentioned as a weakness of APT unless its already being addressed by other means (e.g. van Hasselt et al).

van Hasselt, Hado P., et al. "Learning values across many orders of magnitude." Advances in Neural Information Processing Systems 29 (2016): 4287-4295.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: * This paper introduces APT (Active Pre-Training), a method for unsupervised pre-training in RL environments. 
* During the pertaining phase, APT explores an environment using an intrinsic reward that incentivizes maximizing entropy in a contrastive representation space. 
* APT is evaluated on DM-Control and Atari to test whether pre-training improves data-efficiency. APT outperforms training RL methods from scratch in DMControl and Atari, and the gains are more noticeable in sparse reward environments.  

main_review: This works focuses on an important research problem: How can we equip RL algorithms with strong priors about the environment so that we don’t start tabula-rasa each time we solve a task? APT approaches this by trying to learn a task-agnostic representation of an environment as well as a policy/Q-function initialization during an unsupervised pre-training phase. The representations are learned via a contrastive learning objective that leverages data augmentation, very similar to CURL, DRIML, SPR, ProtoRL and other works in this space: so this isn’t the novel aspect of this paper. To collect the data for training the encoder, APT uses an intrinsic reward that incentivizes maximizing entropy in this contrastive representation space. 

The results on DMControl (particularly the sparse environments) are quite impressive, but the results on Atari are a bit underwhelming: it seems leveraging a massive amount of data (250M steps) buys little in terms of data efficiency (APT is comparable with SPR which uses no pre-training). 

I have some questions about the paper, and the impact of the novel exploration method in particular which might be useful for the readers to know: 

1. Since the main difference between APT and prior unsupervised RL methods is the introduction of a different intrinsic reward scheme to use for the exploration policy, it would be nice to see the comparison of how well this exploration policy performs against prior methods. In prior works like VISR, this has been well-benchmarked: see Table 1 (first section) of [VISR](https://arxiv.org/abs/1906.05030) [1] which reports Curiosity@0 and VISR@0. Comparing the APT@0 scores would show whether APT performs better exploration than prior methods during the unsupervised phase. 

2. I am concerned that contrastive learning is going to lead to maximally spread-out representations that naturally maximize APT’s exploration reward without any actual exploration at all.  A core part of contrastive learning is the "uniformity on the hypersphere" [2] property, which means that points in the replay buffer should be encoded as maximally distant from other points in the replay buffer. Hence, it’s not clear that APT’s exploration is much more sophisticated than giving the agent a reward for staying alive.  Note that the VISR paper does report this baseline: 
>To rule this out, an explicit “do not die” baseline was run (Pos Reward NSQ), wherein the terminal signal remains and a small constant reward is given at every time-step.

So, if we compare with APT@0 with “Pos Reward NSQ @0” (Table 1 in VISR), we should be able to disentangle if this is actually the case. 

3. The ablations section is limited and I want to better understand which components of APT provide a strong learning prior, and which don’t. Table 3 and Table 4 already shows that CL is indeed quite helpful, but I am not sure how much do the learned policy/q-networks contribute. Do you have an experiment that ablates  the performance of a randomly initialized policy/q-network vs their pre-trained counterparts? 

4. The most interesting experiments in the paper are on the sparse reward environments in DMControl, showing a remarkable improvement in both data efficiency and final performance on sparse environments. I was a bit confused to find out though that these sparse rewards were custom designed, and not evaluated on any of the standard sparse environments already available in DMControl (Cartpole Swingup Sparse, Cartpole Balance Sparse, etc.). Introducing both new envs and new methods in the same paper can often lead to selection bias, and it would be nice to see if the results on already existing DMControl envs so that future works also can benchmark against these results easily. See for example ProtoRL [3], another unsupervised RL method that uses these sparse envs. 

5. Can you clarify what "Value of k" = "search in {3, 5, 10}" in the supplementary refers to? Does this mean the value of k was tuned for each env separately. If so, this is problematic, and deviation from the standard evaluation protocol in Atari where the same algo with the same hyperparams are used for all games. This also makes comparisons to baseline unfair since none of the baselines used per-env hyper-param tuning.

Missing prior work: Though the paper in general does a good job of covering prior work, it misses ProtoRL which is a relatively/concurrent work, and so its okay to not directly compare to it but it should be cited nonetheless.  There should also be a mention of Plan2Exlpore in the related work section. 

Minor comments on the writing: 

1. In Algo 1: “Train neural encoder f on mini batch” is vague: you can reference the contrastive learning equation here. There should also be one more step that clarifies that data augmentation is performed on the mini-batch. Instead of standard Q-learning, you should say standard actor-critic (since both policy and Q-networks are mentioned).
2. "With the models and datasets growing, there is still no sign of saturating performance during pre-training." The scaling law curves do start to saturate after reaching irreducible entropy, perhaps a better way to phrase this is “With increasing dataset and mode sizes, performance continues to improve predictably according to scaling laws.”  
3. The imagenet pertaining experiment in Figure 1 has no details, and in-fact a recent paper ([2107.03380 RRL: Resnet as representation for Reinforcement Learning](https://arxiv.org/abs/2107.03380)) contradicts these results (the paper was released after the NeurIPS deadline). It would be nice to have details of the ImageNet experiment in the supplementary material so that we could compare the differences in experimental setup between these two.  

[1] Hansen, Steven, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. "Fast task inference with variational intrinsic successor features." ICLR 2020

[2] Wang, Tongzhou, and Phillip Isola. "Understanding contrastive representation learning through alignment and uniformity on the hypersphere." ICML 2020

[3] Yarats, Denis, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. "Reinforcement learning with prototypical representations." ICML 2021
limitations_and_societal_impact: The limitations section should include the fact that APT demands a huge data budget (250M steps) for pretraining in Atari: which would be not be practical for real-world RL. It should also be pointed out APT depends on environment specific data augmentation.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explores the idea of doing unsupervised pre-training in a reinforcement learning context. Before giving task rewards to the agent, it is given many orders of magnitude more experience for unsupervised exploration. 

The key idea is to use a non parametric entropy based reward function to efficiently explore the space and amortize it in a Q function. This Q function is then fine tuned on the task reward as a follow up stage. 

Compared to previous unsupervised RL papers, this paper explores a new and interesting particle based formulation instead of count/density based model for intrinsic motivation. The performance metrics are shown on the atari and DMControl suite environments. 
main_review: The particle based entropy reward is novel as far as I understand. Typically previous work has explored the use of count based, density based and structured representations for directed exploration in a similar learning setting. However the particle based method explored here is simple, cheap to compute and asymptotically accurate. I also think that the experiment validate or answer the hypotheses considered in this paper. I also like that this paper ran strong baselines and also highlighted regimes where this approach is not as good as the baselines (table 3). So overall I think this is an interesting paper with thorough experiments and sound hypotheses. 

In Table 3, VISR is much better in terms of the mean score than the proposed method. The authors explain this as -- "APT performs significantly better than VISR in hard exploration games, while VISR achieves higher scores in dense reward games". But if I look at Table 2 and the scores for pong, I would have expected VISR to perform better with this logic as it is fairly a densely rewarded environment. But this is not the case. It would be helpful to analyze and understand where, how and why VISR performs better in more detail. 

 Also I am concerned about the median vs mean scores reported in the results section. How do all the results change if the number of seeds are increased or decreased?


limitations_and_societal_impact: seems adequate
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes an unsupervised (or rather, self-supervised) training method for RL, named active pre-training. This is based on using contrastive learning from pixels, then maximizing state entropy in the abstract space using a k-nearest neighbors density estimate. The authors combine their method with DrQ, demonstrating sample efficiency and asymptotic improvements on DeepMind control suite and Atari, along with several useful ablations.
main_review: **Originality**: The largest limitation of this paper appears to be a relative lack of novelty. The proposed method combines two popular ideas: self-supervised contrastive learning and state entropy maximization. As the authors discuss in section 4, these ideas have been extensively explored in previous works. Indeed, even within state entropy maximization, multiple previous works have used non-parametric density estimators. Note: I would also cite Pitis et al., 2020 as another examples of state entropy maximization via non-parametric estimators. However, in that paper, they perform density estimation in the state space rather than in a latent space. Indeed, the main distinction/contribution of this paper appears to be entropy maximization within a contrastively learned latent space. This differs from previous works that have used contrastive learning purely for pre-trained representations, as well as other works that have used non-contrastively learned latent spaces for exploration. However, given that this is the primary novelty of the paper, it is somewhat unclear and poorly motivated/justified within the paper. For example, as far as I can tell, the main justification for this choice is in sentences like 

*“We hypothesize that the ineffectiveness stems from density modeling at the pixel level being difficult.”*

*“We consider an abstract representation space in order to make the distance meaningful.”*

These statements are not only imprecise, they are also not rigorously analyzed empirically. The main evidence for these claims comes from performance differences between the proposed method and other (state/observation-based) exploration methods. To summarize, this paper combines two useful techniques, contrastive representation learning and state entropy maximization, showing that their combination is even more useful. However, given that this combination is only weakly motivated in the paper, one is left wondering whether this paper provides much in the way of new insights.

Pitis et al., 2020, Maximum entropy gain exploration for long horizon multi-goal reinforcement learning.


**Quality**: Overall, the paper is fairly high quality, particularly the empirical performance evaluation. The authors compare their proposed method with various visual and exploration RL algorithms on both DeepMind control suite and Atari. The number of reported environments is extensive in both domains, and the results are competitive with SOTA algorithms. I see this evaluation as the strongest point of the paper, especially in a benchmark-focused area like RL. I also found the ablations in Table 4 to be helpful for disentangling the benefits of the various components of the algorithm.


**Clarity**: With the exception of the weak motivation issues mentioned above, the paper is clear. For the most part, the background and method are explained well. I found the math fairly straightforward to follow, and I appreciated the diagrams (Figure 2), algorithm box (Algorithm 1), and comparison overview (Table 1). The authors also provide an adequate overview of previous lines of work in the related works section (Section 4).

Some suggestions for improving the clarity of the paper:

It may help to discuss the benefits and drawbacks of parametric vs. non-parametric density estimators in more detail. Currently, the non-parametric estimator is introduced as the obvious choice, but it’s unclear why this should be any better than, for example, using a VAE or other deep generative model, as in state marginal matching [Lee et al., 2019]. When would one estimator be preferred over the other?

The intrinsic exploration reward is non-stationary, as it depends on the on-policy state distribution. It would be helpful to discuss any additional challenges associated with optimizing this reward function in comparison with task-specific reward functions.

The paper discusses entropy maximization in terms of states. Unfamiliar readers are likely more familiar with action/policy entropy maximization. It would be helpful to include a sentence or two describing the distinction between these approaches.


**Significance**: I see this paper as fairly significant for the following reasons: 1) the empirical evaluation provides results across an extensive set of environments with reasonable improvements, and 2) the method combines two fairly popular techniques in the literature, and accordingly, is likely easier to reproduce and build upon. For these reasons, I could see this paper as a useful contribution to the RL community. While this may not be the most novel or well-motivated paper, it provides a fairly clear evaluation of a useful technique.


**Minor comments**:
- Line 80: Why is T a distribution and not a density?
- Line 96: Would be helpful to explain the variable w a bit more.
- Line 131: Would be helpful to describe some of the design choices around the kernel more explicitly.
- Line 155: It’s not clear that entropy maximization is entirely modular from representation learning. For entropy maximization to work, the representation needs to contain a compressed representation of the state.
- Table 1: It’s not clear whether APT necessarily needs to be off-policy. Couldn’t one construct an on-policy version?
- Line 267: incrase —> increase
- Line 278: I’m not sure this is necessarily true for SMM.
- Table 3: Slightly unclear whether APT w/o CL refers to exploration in the pixel space or in a random encoder space.
- Line 332: phrase —> phase

---
**Update**: I have read the other reviews and the authors' responses. I had already rated this paper highly, and the authors provided a satisfactory response. The other reviewers also have increased their scores. I will maintain my score at an 8.
limitations_and_societal_impact: Yes, the authors addressed limitations of their work in the conclusion. Specific negative societal impacts from this work are unclear (other than those associated with machine learning generally).
review_title: null

Meta-review: This paper proposes a novel unsupervised pre-training method for reinforcement learning, using a particle-based entropy estimator on a contrastive-loss trained feature representation. The method is novel and scalable and the empirical results and rebuttal additional experiments are strong. I trust the authors will incorporate the reviewers' comments and discussion points into the final version of the manuscript.
Decision: Accept (Spotlight)