Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an auxiliary task for learning better representations for reinforcement learning. The idea is interesting and is a very active area of research at the moment.

My main concern is that the paper is mostly an empirical evaluation, as the novel algorithm is mostly an extra contrastive loss. Further, it seems their method is almost exclusively meant for pixel-based environments (see point 1 below), so the authors should make this point more explicit. Given this empirical emphasis, I feel the authors could have performed a deeper exploration into understanding _why_ their proposed algorithm performs the way it does in the different environments considered. In particular, some very specific design decisions were made in evaluation (see, for instance, points 2, 5, 9, 11, 12, 13, 14, 15, 16 below). 
The clarity of exposition could also use some improvement, as I detail below.

Main questions/concerns:
1. In Section 3, the authors write "This task encourages the learned encoder to extract meaningful elements of the structure of the MDP from observations.". This assumes some type of continuity in pixel space, relative to MDP dynamics, which is in general not true.
2. In Figure 1, why doesn't the momentum encoder go through a residual predictor?
3. It seems $\theta$, $\phi$, $\bar{\theta}$, and $\bar{\phi}$ are all updated independently, is this the case? What is the actual training regimen? Are the ATC and regular RL networks trained concurrently?
4. It would help if you include a proper algorithm in the paper.
5. Above equation (2), the authors say "In our implementation, the positives from all other elements...". It's not clear what "positives from  all other elements" means.
6. In section 4.1 the authors say "multiple seeds were run", please specify how many.
7. In section 4.2 the authors say they are capable of "training the encoder online, fully detached from the RL agent", but how is it fully detached if they share the conv layers?
8. It's not clear what the difference between ATC and UL training is. In some experiments the authors use ATC, in others UL. Are they the same thing? For example, in Figure 3, which ones are ATC? Also in Figure 14 vs Figures 15 and 16?
9. In Figure 3, why does one environment compare with pri and the other with 2x, but not both in both environments?
10. In the **Atari** subsection on the comment of detached training, the authors point out subpar performance on Breakout and SpaceInvaders. On SpaceInvaders it's possible the screen changes could cause issues, but what do the authors think cause the subpar performance in Breakout?
11. In point (iii) of section 4.3, what's the network architecture used for training the RL part?
12. In section 4.3 the authrs say they "drew expert demonstrations from partially-trained RL agents". Were these all drawn from the same checkpoint?
13. In the **DMControl** section, the VAE is trying to reproduce a frame $T$ steps in the future? What is the value of $T$ used? Did you try different values?
14. Similar questin for the **Atari** subsection. Also for this section, does your VAE try to predict individual frames or stacked frames (as frame stacking is common in Atari experiments)?
15. In Figure 8 top, are these after pre-training the encoder? If that is the case, regular RL would have used fewer frames in comparison, no? Where would RL be if left to train for longer?
16. In section 4.5 please clarify what "random shift augmentations" are.
17. In the **Encoder analysis** subsection, what do you mean by "attention"?
18. In Figures 14, 15, and 16 it's not at all clear what we're supposed to be looking for, nor how they show that ATC/UL is focusing on the score/enemy and the others are not.

Minor comments:
1. At the bottom of page 2, the term "POMDP" has not been introduced yet.
review_title: Official Blind Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of the paper:
The paper aims to define an unsupervised pretraining architecture that can be used to pretrain representations for a reinforcement learning agent. The proposed solution is called "Augmented Temporal Contrast" (ATC). It consists of an encoder-compressor-predictor architecture for an (augmented) observation that is trained in a small latent space by minimizing the InfoNCE loss between that prediction, and the encoding of some future state.
The representation used (the pretrained encoder) is then frozen, and a policy network is trained consuming these representations. The authors then evaluate their approach on several different tasks: a control domain (DMcontrol), Mazenagivation (DMLab) and Atari. They show that their approach - without finetuning the representations - achieves comparable results to end-to-end reinforcement learning on most of the different domains. They also show that adding their defined loss as a regularizer always helps learning. They finally show that their representations can generalize out-of-domain by running several multi-task experiments, while only having pretrained on one domain.

Commentary on the goal of the paper:

The goal of the paper is extremely important: separating learning representations from learning policies would enable better transfer, possible more sample efficiency and lower variance in outcome. 

Strengths:
- The authors propose a well-designed solution that combines existing approaches in a well thought-out way. 
- The paper is extremely well written
- The paper has an extensive empirical section. 
- Results are generally very good.

Weaknesses:
- The goal of the paper is a bit vague. As I said above, I agree and understand the desire for the separation of representation learning and reinforcement learning. However, the paper would have been stronger if it had concentrated on a single benefit of this separation, and evaluated their approach on that. While they say that their goal is to investigate "how to learn representations which are agnostic to rewards", this is too general as well. (One example would be to say that the decoupling makes for better generalizes to new MDPs - but this is not the focus of the analysis, just an aspect. The paper is, unfortunately, less convincing for it.)
- Along a similar line of thought, the results, while strong, are not as convincing as they could be, because the paper does not focus on the benefits of reward-agnostic representations learned by ATC. The results that consider the generalization advantage (the Multi-Task learning experiments) are weaker, without the paper offering an analysis as to why.

In total, I would argue that this is a well-written paper with interesting analysis that could be a lot stronger by narrowing the scope of the contained argument. I argue for rejection, because I can see an updated version of this paper to be a great paper.
review_title: A well-written paper with sound reasoning and mostly good results that should sharpen and narrow its focus

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary:**
This paper presents a new unsupervised learning method for learning latent representations for visual RL control domains. The method, Augmented Temporal Contrast (ATC), can be used alone to learn a representation to be combined with an RL algorithm, or as an auxiliary task in an end-to-end system. ATC matches or outperforms comparable end-to-end systems in several environments. The paper provides an extensive experimental study to support its claims.

**Strengths:**
The paper is clearly written, and all of the main points are well articulated. ATC appears sufficiently novel, and is applicable to a wide variety of domains, and can be deployed in various configurations (e.g. auxiliary task, unsupervised pre-training, etc…). Included is a thorough experimental study that effectively demonstrates the performance of the method.

**Weaknesses:**
Although the experimental study seem thorough. I could not find the actual number of independent runs (seeds) for each domain listed anywhere. This information should be included so that the reader can better evaluate the variance of each method, and make more confident conclusions. 

**Recommendation:**
Overall I vote to accept. The method presented in the paper is not revolutionary, but it appears to be novel and significant enough to be of interest to deep RL practitioners.

**Questions:**
How many independent runs (seeds) were used in each of the domains? Can this information be included in the main text?

**After Author Response and Discussion:**
Thanks to the authors for their responses. After reading the other reviews and the author responses, I am lowering my score to 5. I think that the number of independent runs used (especially on the smaller domains), and the way the results are presented with the min-max extent makes me less convinced of the results than I was in the initial review. Adding many more independent runs (seeds), especially on the smaller domains, would improve my confidence a lot. Overall I think the paper is of interest to the community, but the experiments and their analysis could be improved.
review_title: A new unsupervised learning method for learning latent representations for visual control domains.

final_score: rating: 7: Good paper, accept
review_content: The paper introduces an unsupervised task called Augmented Temporal Contrast which associates pairs of observations separated in time using a contrastive loss. The paper uses the task in several training regimes (in online RL, pretraining and multi-task RL).

Pros:
- Well written and structured paper.
- Interesting, general and simple to implement task.
- Evaluated in several training regimes and on several environments.
- Improves sample efficiency on most of the environments and setups, and improves over prior methods.
- The attention maps in the paper and appendix are great and although they may be hand picked(?) examples, it highlights the issue with many approaches that can't model a goal rarely seen.

Cons:
- You write you use multiple seeds but I don't see anywhere details on this? Consider adding it to the tables in appendix.
- In Figure 3 one agent step is 4x environment frames? I suggest to make it clear in plot or in caption. In Figure 2, environment frames is used.

Comments/questions:
- Wrt. 4.1, to what extent is the "small" replay for DMLab necessary over just using observations in batches/unrolls? Looking at Table 3 I can't see how large the replay is? 10k as in SAC or smaller?

Update: Not all the experiments are particular thorough and the novelty less than expected.
review_title: A good improvement over prior unsupervised RL tasks in many setups

Meta-review: This paper introduces ATC, which is a contrastive learning on observations separated in time, to learn representations that do not need to take rewards into consideration. These learned representations allow, for the first time, a real disentanglement between representation learning and control, as the agent can simply load such a representation, “freeze it”, and still recover performance of end-to-end deep reinforcement learning agents.

Overall, all reviewers agree this is a promising direction. Nevertheless, there has been extensive discussion (with the authors and privately) about the significance of the reported results due to the small number of seeds. On one hand, there’s the argument that there is a wide range of experiments and that should compensate for a small number of seeds in individual experiments. On the other hand, there are experiments with as little as two seeds (e.g., DMControl multi-env) and this can be seen at most as anecdotal evidence. There’s also the argument that we, as a community, should be striving for more reliable and meaningful experiments in reinforcement learning. Moreover, there have been concerns about how “variance” is being reported (max and min performance) and, although the authors replied to that, an alternative plotting was never shown.

Importantly, at this point it is not clear how many seeds were used in each experiment (Figures 6, 7, 9, 11, 12, 13 do not report the number of seeds used). It is said that each curve represents a minimum of 3 random seeds, but that is very informal and not that useful. Exactly stating the number of seeds would be the right thing to do, not to mention that in the rebuttal it is said that 8-game pretraining for Atari multi-env uses 2 seeds, contradicting the original claim. Also, sometimes, different methods, in the same experiment, are  “averaged” across different numbers of seeds (“DMLab offline -- ATC is 4 seeds, PC and CPC are 2 seeds each”). This is particularly problematic because of the small number of seeds and potentially high variance. Reporting the max over 4 numbers drawn from a Gaussian distribution is very likely to lead to a larger number than when reporting the max over 2 numbers drawn from the same Gaussian distribution. 

I do acknowledge the effort to increase the number of seeds during the rebuttal phase, but it is hard to accept a paper with unknown results. We have very little evidence to believe that going from 2 seeds to 5 seeds is not going to change the results. The reviewers couldn’t agree on the variance of this process as well. Some say the variance of PPO is low between runs when using the same hyper parameters while others mention papers (e.g., Deep RL that matters) that show how much variance one can have across these methods. Thus, I cannot accept this paper conditioned on more seeds being added to the final version because we don’t know what the results will look like. Since this paper is mostly an empirical study, it should have thorough experiments and a careful analysis of the results, but the small number of seeds prevents that in my opinion. Thus, as difficult as it is given the promising direction of the paper, I’m recommending its rejection. I strongly encourage the authors to increase the number of runs in their experiments and to use a more standard measure of variability (e.g., standard error, standard deviation) when reporting their results. This will then be a very strong submission for a future conference.

Decision: Reject