Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors claim that the usefulness of the learning components (e.g., contrastive objective) proposed in previous representation learning for RL, is highly dependent on the specifics of the task category, and thus show that they don't provide robustness across different and diverse task categories. They also claim that the reward and transition prediction are the most vital (and minimal) components providing robustness. Based on this observation, they propose to evaluate the representation learning methods by considering the specific properties of task categories instead of evaluating on a whole benchmark level such as DMC and Atari. 
main_review: Strengths 
- The paper performed various experiments analyzing the contribution of each component used in representation learning for RL. 
- The results suggest, which I agree, that we need to analyze based on the categories of the tasks rather than relying on the average or overall performance of the benchmark.
- Some experiments (e.g., crop and part reconstruction) are quite interesting as it shows the effect of the object centering. 

Weaknesses
- One of the main claim of the paper is that reward and temporal prediction are vital components providing the most robust performance. I’m not convinced if this can be taken as a general finding about representation learning in RL. It seems that the author’s argument is highly biased mostly to the dense reward setting. For other settings, the result can be different. Specifically, the authors use 8 categories. The proposed baseline seems to work for these categories. However, the result is highly dependent on how the 8 categories are picked because there exist a much larger number of categories in the space of tasks and we don’t know whether the baseline would be robust generally and globally in this space. For example, the reward prediction may not work well for sparse reward tasks (which is currently considered only in one category out of 8). Also, the temporal prediction may not be useful much for fully observable tasks. Also, for unsupervised RL, the result would be pretty different. 

- While I found some experiment results are interesting (e.g., cropping ), many of the findings from the paper were actually not surprising rather somewhat already known or expected. For example, in dense reward settings, the fact that it does not perform well after removing reward prediction is pretty obvious. Also, the fact that pixel reconstruction does not work well on image observations with distractors has been shown in many previous works (e.g., TPC [1] and TIA and the related works of these papers). 

- For Value-Aware Learning, the paper does not discuss much about the result.

- It would be more interesting if the work also embraces broader classes (e.g., other categories, unsupervised RL, etc.) of tasks to discuss what matters and why about representation learning in RL. 

[1] Temporal Predictive Coding For Model-Based Planning In Latent Space
summary_of_the_review: The argument about the vitality and importance of the reward and temporal prediction seems to be an over-claim, or the experiment results do not support this claim enough. Also, many findings of the paper are quite what can be expected and not much surprising.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for learning representations from pixel data that are amenable for control tasks. The proposed approach is a simple baseline that does not require data augmentation, world models, contrastive losses etc. but only contains two simple sub-tasks that are supposed to contribute heavily towards an effective representation: reward prediction and state transition prediction. Along with evaluating this proposed baseline, the paper also compares it to several prior works on representation learning: i.e., several approaches such as data augmentation, distance metric losses, contrastive losses, relevant reconstruction etc. It is shown that the proposed simple baseline either outperforms several of these methods or at least is very close in performance. Finally, the paper presents an interesting discussion about how evaluating an algorithm is not just about the dataset and the chosen benchmark task, but requires a more nuanced point of view of several factors such as reward sparsity, action continuity/discreteness, relevance and irrelevance of features to the task, and so on. The findings of the paper are not just about the effectiveness of the proposed method, but a more overarching view of which types of representation learning methods work in what conditions. 
main_review: Strengths:

1.	The paper presents an interesting discussion which is useful to the field of representation learning, visuomotor control etc. that is currently of interest to the community. It is interesting to see the success of a what can be considered fairly standard baseline with reward and transition predictions achieve competitive performance.  
2.	It contains a fairly extensive empirical study of several state of the art and quite recent representation learning techniques such as DREAMER, Task informed abstractions, Deep Bisimulation for Control, CURL, SPR etc.  
3.	The discussion on which losses, decodings and augmentations benefit the representation is interesting and useful. Figure 1 is also a convincing way to tell the story that naïve benchmarking of algorithms does not contain the full picture, and to properly evaluate algorithms, one needs to take a data centric view. This can help kick off a wider discussion on how best to evaluate and benchmark representation learning models.   

Weaknesses:

1. The concept of using reward prediction as a way to help reinforcement learning/representation learning is not very new. For example, [1] contains a discussion on how offline pretraining with reward prediction can help downstream RL/imitation learning. Also relevant are the concept of 'successor features' from [2] helping generalization over tasks where dynamics and transitions are similar, the analysis of reward prediction on representation learning from [3]. This class of techniques is probably closer to the proposed baseline, and comparing performances to some of these would have allowed a more pointed comparison about the reward prediction feature and the novelty of the method. 

2. While the main focus of the paper seems to be the discussion itself, and not claiming superiority with the proposed baseline, the main tasks for evaluation are all somewhat visually similar (i.e. DMC agents with distractors). While the distractor backgrounds are likely of different varieties, the underlying task arrangement is still similar: a moving agent on top of a background. It would have been interesting to evaluate how the baseline and other methods perform on more complex tasks such as egocentric visual control.

3. There is not a lot of discussion with regards to generalization of the learned representations. Compared to unsupervised representation learning, one can intuitively understand the advantages of task related supervision such as having access to rewards etc. which in turn benefit the representation. But it is also often hard to handcraft specific tasks, rewards etc. in a way that is beneficial for a model. How well do these representations and associated policies generalize? If several downstream tasks in a given environment are similar to each other (i.e., the task relevant features are similar), can a single trained ‘baseline’ policy generalize? Have the authors considered the case of representation learning where a generalizable representation is learnt in a pre-trained, task-free and reward-free way (one that focuses on agent dynamics, world modeling etc.) and then is finetuned for individual tasks?  

Other comments:

1. The gist of figure 3(a) seems to be the fact that reward prediction plays a major role in creating effective, task-relevant representations which, conceptually is easy enough to understand. I am curious whether such a representation, while beneficial for the task at hand, is detrimental for generalization. For instance, if another way of representation learning (one perhaps not so closely tied to task-specific reward) results in higher sample complexity for individual tasks but better generalization, there might be cases where that is also valuable. Is state transition a more ‘generalizable’ construct than rewards?

2. It is not clear why DREAMER w/o dist in Figure 6, and DREAMER/TIA in Figure 7 have a constant profile of scores. Are they not learning anything?

3. The last part of Section 5 states that performance of a non-contrastive loss often depends on the existence of data augmentations, whereas SPR still maintains strong performance without augmentation. Shouldn’t contrastive loss be the one that is more dependent on data augmentation for sufficient positive-negative pairs? 

4. Section 5 states that reconstruction-based techniques like DREAMER suffer when distractors are present. What would be the case if reconstruction is combined with sufficient data augmentations – i.e., different kinds of distractors? Also, what is the intuition for why TIA performs worse than the baseline even with explicit decoupling between task-relevant and task-irrelevant features? 

[1] Yang, M., & Nachum, O. (2021). Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815.  
[2] Lehnert, L., Littman, M. L., & Frank, M. J. (2020). Reward-predictive representations generalize across tasks in reinforcement learning. PLoS computational biology, 16(10), e1008317.  
[3] Hlynsson, H. D., & Wiskott, L. (2021). Reward prediction for representation learning and reward shaping. arXiv preprint arXiv:2105.03172.
summary_of_the_review: The paper contains a discussion on how to best learn representations from pixel data for control tasks. The authors discuss the insights of  various techniques in the literature and show empirically that simpler constructs that focus on task relevance, system dynamics etc. have the potential to outperform a lot of methods in the literature. While the paper's evaluation focuses on fairly simple tasks, I feel that the empirical results, the extensive analysis and the discussion of strengths and weaknesses of other classes of methods is useful for the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a baseline RL approach for pixel-based control with distraction in background, which adds reward and latent state transition predictions to actor-critic agent (or DQN agent for Atari experiments). The authors further consider many components proposed in previous contributions on pixel-based control (such as contrastive learning, data augmentation, image reconstruction, etc) and conduct ablation studies on top of the proposed baseline approach. They also directly analyze existing methods for pixel-based control with distraction in background. 
main_review: Strengths
* The authors presented large-scale empirical evaluations and ablation studies to analyze various components for pixel-based control with distractors.

Weaknesses:
* Many comparisons presented in this paper are in fact system-wise (such as most comparisons shown in Sec. 5), but the authors use these system-wise comparisons to derive conclusions on the family of methods. I find this unconvincing. For a concrete example, CURL is only one implementation of contrastive learning, and it's known to be an ineffective one such that the original CURL implementation actually works better without the contrastive objective. I don't believe conclusions drawn from CURL experiments can general to contrastive learning. There are other variants such as [a] that consider InfoNCE between the past and future and use momentum encoders similar to SPR, achieving significantly better results on DMC. [44] came after [a] and similarly considered predictive information using InfoNCE. [b] presents an even more complicated implementation of contrastive learning. Just to name a few examples. The same concern applies to other components (such as data augmentation, image reconstruction, etc) and the ablation studies, meaning that your conclusions only apply to specific implementations and that it's not obvious that they can generalize to the whole family of methods.
* Lack of details for experiment set-up: Experiment setup of many figures are unclear to me. For example, are all experiments in Appendix 5 with distractors or not?
* Lack of details for the proposed dataset categorization (table 3): I found it very difficult to follow. For example, since not every DMC task has sparse reward, it's not clear to me what kind of tasks C7 contains and how they are different to C4.
* For one, there isn't a clear story line showing in the contributions paragraph in the introduction section and the conclusion section. The authors propose a new baseline method while analyzing existing methods and studying many different components. There seem to be multiple threads in this paper without having a main story.

Questions:
* Have you noticed your reacher experiments can only work with data augmentation? It's not clear to me whether it's with distractors or not. Do you want to discuss this?
* For the statement in conclusion "Contrastive losses in and of itself does not provide gains when there is a supervised loss available in place of it.", which supervised loss are you referring to?
* What does "non-contrastive loss" even mean? Any loss that is not contrastive can be referred as "non-contrastive loss".
* I was wondering if the authors are aware of performance issues with CURL, no matter it is an implementation flaw or algorithm design choice problem. Why do you choose to compare to CURL as a contrastive baseline?
* Contrastive InfoNCE approximate reconstruction distribution [38, a]. Can you provide more insights for why they could perform differently despite the learning objective is similar?

[a] "Predictive Information Accelerates Learning in RL." NeurIPS 2020.

[b] "DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck." ICLR 2022 submission. (Of course, I don't expect the authors to cite this paper. This is just an example.)
summary_of_the_review: Overall, there are good values in the large-scale studies presented in this paper. However, I'm not convinced by conclusions as the authors try to generalize behavior of specific implementation to a family of methods. If I were to implement a new agent, I don't feel like I can believe these conclusions so that makes me question what knowledge this paper can add to the community. Furthermore, many details are either missing or not made clear, and the main story isn't very strong. Therefore, I don't think this paper is ready for publication in the current status.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work investigates representations for pixel-based reinforcement learning. The authors empirically show how different approaches perform in a variety of domains categorized by their reward structure, presence of distractors and use of data-augmentation techniques. The work further argues in favor of an "exceedingly simple" baseline method consisting of SAC-AE modified with transition and reward prediction modules. 
main_review: [Strengths]

* Representation learning is an important aspect of effective reinforcement learning from observations (here "pixels"). Categorization of environments based on characteristics in which different algorithms perform best has certainly helped in the quick development of RL practice. The C1-C8 chategorization and results of Figure 1 are no different and could be of general interest. It is nice to see that for the most part the results confirm existing "lore" e.g. world-model reconstruction methods work well without distractors but perform significantly worse with distractors, getting closer to the proposed baseline (difference between C4 and C8)

* Data-augmentation is used to learn invariances with respect to non-informative (for the task) transformations of the input. This is well-known in the representation learning literature and I appreciate the investigation of these properties in the context of RL (discussion in Section 5).

* I agree in principle about the potential utility of predicting rewards to improve the quality of transition model representations, and am happy to see SAC-AE with reward transitions performing on par with more involved baselines.

[Weaknesses]

* It unclear if the main contribution of this work is to introduce an improved variant of the SAC-AE baseline (with reward and transition prediction) to improve robustness against distractors or instead investigate the performance of different approaches to RL in tasks with different characteristics. The paper could do well to focus on a single aspect and develop it more thoroughly. Section 4.2 and the introduction of transition prediction is a prime example of an element that feels forced and not supported by the experiments (Fig. 15 in particular does not inspire particular confidence).
  
* To elaborate on above: the paper suffers from tension between attempting to highlight how claiming superiority of a method across all environments is misleading, while at the same time spending much of its time arguing in favor of a specific method that is supposed to "learn meaningful representations" (which is never shown). Much like the other previously proposed methods, `baseline` performance itself wildly varies across domains. The work does not investigate the architectural modifications proposed particularly deeply: there are no comparisons to previous approaches that incorporate reward prediction. 

* The authors bring up contextual decision processes (CDP) without ever utilizing the framework to illustrate a concept or derive a result. In principle, I understand the choice of CDPs as the focus is on distractors. Are there other motivations behind this choice? Could you not extend the standard POMDP formulation to a case where observations are noise corrupted by noise that is not informative for the task (the so-called distractors?)

* The prescriptive claims in Section 7 should be better contextualized given the results provided in the paper (e.g. "Pixel reconstruction suffers massively when distractors are added, as shown in ... where ..."). The degree to which each of these claims is experimentally supported varies greatly. The Section could benefit from providing a summary of experiments done in support of each claim.

[Questions]

* In general, there are only a few domains in which `baseline` appears to beat `baseline-v0` (C4 and C6 and C7), all of which with distractors. Is it correct to state that everything being equal, introducing the proposed modifications to the `baseline` decreases its performance in a setting without distractors, but helps otherwise? The results certainly seem to suggest this is the case.

* The paper is missing a description of what types of distractors are considered and how they have been implemented. Are distractors kept the same (once chosen) across a given training dataset? If not, i.e you sample distractors, why are you treating them any different than data-augmentation (that is guaranteed to retain task information)? This last case would also explain the apparent advantage of introducing the additional reward predictors: you are in effect allowing the representation to learn to ignore backgrounds for example, and building a reward prediction model on this better, "distilled" representation. Methods trained in environments without distractors can choose to also rely on non-relevant information for their predictions, which is inherently a worse representation to learn a reward from, hence the difference. Are the distractors instead only introduced at test time?
summary_of_the_review: In summary, the paper executes a fairly extensive experimental exploration of different RL algorithms in different domains. As is often the case for these types of works, some prescriptive claims made are not fully supported by experimental evidence (e.g Section 4.2). This happens more often when the paper itself argues in favor of a specific algorithmic detail (`baseline` here). Some of the results show evidence for aspects of representation learning in RL that could be of general interest (though they are not all novel in themselves, only summarized together). I am also of the opinion that not including other baselines using reward prediction does not inspire confidence that this approach to its incorporation is any better than the others that exist. I am willing to reconsider my current evaluation.
review_title: null

Meta-review: Meta Review for Learning Representations for Pixel-based Control: What Matters and Why?

In this work, the authors presented large-scale empirical evaluations and ablation studies to analyze various components (e.g. contrastive objectives, model-based approaches, data augmentation) for pixel-based control with distractors. Reviewer 7euW wrote a great summary for this paper:

This paper presents an approach for learning representations from pixel data that are amenable for control tasks. The proposed approach is a simple baseline that does not require data augmentation, world models, contrastive losses etc. but only contains two simple sub-tasks that are supposed to contribute heavily towards an effective representation: reward prediction and state transition prediction. Along with evaluating this proposed baseline, the paper also compares it to several prior works on representation learning: i.e., several approaches such as data augmentation, distance metric losses, contrastive losses, relevant reconstruction etc. It is shown that the proposed simple baseline either outperforms several of these methods or at least is very close in performance. Finally, the paper presents an interesting discussion about how evaluating an algorithm is not just about the dataset and the chosen benchmark task, but requires a more nuanced point of view of several factors such as reward sparsity, action continuity/discreteness, relevance and irrelevance of features to the task, and so on. The findings of the paper are not just about the effectiveness of the proposed method, but a more overarching view of which types of representation learning methods work in what conditions. 

Along with myself, most reviewers (including the critical 61FY) agree that there is great value in the large-scale studies presented in this paper. Furthermore, I personally like how it links a large body of recent work in this topic together in one study. The reviews were mixed (6, 6, 3, 3), and the negative reviews (the 3's) generally had issues with not the study or experiments, but the conclusions the authors drew from them. In the words of 61FY (who managed to have a good discussion with the authors):

*I'm not convinced by conclusions as the authors try to generalize behavior of specific implementation to a family of methods. If I were to implement a new agent, I don't feel like I can believe these conclusions so that makes me question what knowledge this paper can add to the community. Furthermore, many details are either missing or not made clear, and the main story isn't very strong. Therefore, I don't think this paper is ready for publication in the current status.*

Although I really appreciate the effort and detail that went into this nice work, based on the current assessments from the 4 reviewers, I can't recommend it for acceptance in its current state. I feel though, that with a change of narrative, or even with a re-examination of the experimental results, the authors can turn the paper around into a highly impactful paper. The description of all of the methods explored, and experiments performed alone makes a wonderful survey of the field with sufficient impact, so I think the authors are *almost* there in publishing a highly impactful work that can make the community look deeper into pixel-based control methods (with distractors). I hope to read an updated version of this work in the future published at a journal or presented at a future conference. Good luck!
Decision: Reject