Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper performs an empirical study of self-supervised learning for RL from pixels. Specifically, the study considers different SSL objectives (e.g., MoCo, BYOL, etc.), evaluates them using the RL + auxiliary SSL objective framework (like in SAC+AE and CURL), and compares them to RL with data augmentations alone (e.g., RAD and DrQ). The results suggest that auxiliary SSL objectives do not lead to clear gains in tested settings. The paper also performs experiments on using evolutionary search for combining different SSL losses and using SSL pre-training rather than auxiliary SSL objectives.
strengths_and_weaknesses: Strengths:
- The paper considers and important question and would be of interest to the community
- The study perform extensive experiments across different settings
- The paper report robust performance estimates 

Weaknesses:
- Overall, it feels that the paper is trying to do too much by studying three related but different enough aspects and not developing any of them in sufficient depth: (1) RL with auxiliary SSL losses, (2) evolutionary search for combining SSL losses, and (3) SSL pre-training for RL. Most of the paper focuses on (1) and that component is the most thorough of the three (though still leaves a number of questions unanswered; please see the questions section). Sections (2) and (3), and (3) in particular, feel like they have not been explored enough in the current paper and should perhaps be studies of their own (e.g., most of the questions I asked in the context of (1) would equally apply to (3) + additional questions specific to (3), e.g., the choice of pre-training data).
- I will focus on the weaknesses of (1). If my understanding is correct, the main findings of the experiments is that the RL + SSL techniques do not seem to outperform RL + data augmentations in tested settings. This general finding in itself is not new and was reported by DrQ and RAD. Thus, I think that it would be nice to provide a bit more insight on why that might be and/or what specific aspects may be the reason for that (see the questions section below).
- Experiments on the impact of different data augmentations and their hyper parameters in the appendix are nice. It would be good to focus more on that and bring / discuss some of these aspects in the main text. And to try to develop that aspect a bit more (e.g., instead of just saying that data augmentations have a big impact on performance try to provide a bit more specific findings)

Overall, the paper asks an important question and I like it. However, I feel that in its current form it is trying to address too many different aspects (1-3 above) and does not develop any of them in sufficient detail. I think that the paper would benefit from focusing more on RL + auxiliary objectives (e.g. including additional experiments from the questions section) and leaving the remaining ones for future studies.
questions: Questions/suggestions:
- It would be to expand the discussion of the impact of data augmentations (please see my comment in the previous text box)
- Similarly, it would be good to include more experiments / discussion on the impact of different hyper parameter choices (we know from prior work, e.g. DrQ-v2, that this can have a very large effect)
- It would be good to consider the impact of the neural network architecture on performance (e.g., different vision encoders, encoders of different size, etc.)
- It would also be good to study what is the best way to pass / process vision embeddings to policy / critic heads (e.g., like the impact of the tanh discussed on L238)
- It would be good to consider additional RL algorithms (e.g., DDPG, PPO, etc.)
- The paper considers DrQ but not DrQ-v2. Is there are reason for not including DrQ-v2 in the study? (SAC vs DDPG?)
- It seems that action information is not considered by the pairwise approaches but is used in the reconstruction and shuffle approaches. It would be good to try to study the impact of action information.

Minor:
- The term "SAC backbone" may be a bit confusing. People typically use backbone to refer to a neural network architecture used for feature extraction (e.g., common in vision). Maybe something like (learning) "architecture", "framework", or "meta architecture" would be clearer in this context.
limitations: Some discussion of limitations in the appendix (A.9). I think it would be nice to see a more detailed limitation sections perhaps focusing on potential confounding factors and/or methodology aspects that might influence the conclusions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims at answering the question of to what extent can self-supervised learning (SSL) objectives help with online Reinforcement Learning. The authors build a consistent experimentation framework around two popular RL algorithms designed for continuous and discrete action spaces respectively. Experiments in this work are performed over a significant collection of approaches, including state-of-the-art SSL methods, data augmentation methods, and a custom combination of multiple SSL objectives found through evolutionary search. The paper presents results from multiple synthetic domains and a real robot experiment. The authors come to the conclusion that there is not a single golden SSL approach that works the best in all cases, and that the performance of SSL approaches varies across different environments.
strengths_and_weaknesses: Strengths:
- The paper is well organized and well written.
- The authors include an extensive collection of SSL and data augmentation methods for comparison in the experiments.
- The real robot experiment is a good addition to showcase how these methods work with the presence of real-life camera/lighting noises.
- The work provides valuable insight into how the instance-based SSL methods from state-of-the-art vision models don't seem to be super useful in the RL domain.

Weaknesses:
- This work lacks novelty in terms of methods. The authors utilize evolutionary search, but the resulting model is still a weighted mixture of existing methods.
- The discussion on pretraining does not seem necessary because it relies on additional data at the very beginning, as the authors have pointed out, making its setting different from the rest of the methods in comparison. It's still good as an appendix section.
questions: - The paper only shows each model's final performance. Can the authors also include some reward vs. training-step plots? Are there other observations to make from those curves, e.g. ramp-up speed?
limitations: The authors did a great job making a fair comparison among different methods and not overclaiming their contribution. Some potential limitations are discussed above in the weaknesses section.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents an empirical study on the effect of self-supervised learning (SSL) in online and pretraining RL from pixels. Through the experiments on DMControl, Atari and a real-world robot environment, the authors make the following conclusions (i) the existing SSL framework for RL fails to bring meaningful improvement compared with the baselines using data augmentation techniques, while using the same amount of data and augmentation; (ii) the combination of SSL losses for RL also do not bring much gain; and (iii) no single self-supervised learning or image augmentation method dominates all environments. 
strengths_and_weaknesses: Strengths:
1. The authors made a lot of experiments. Some results are interesting.
2. The paper is easy to follow.

Weakness:
1. The conclusions are not very rigorous. For example, the authors test only six environments on DMControl and make the conclusion that augmentation is much more effective than SSL, e.g., CURL is much worse than RAD in Figure 4. However, I tested CURL and RAD on several hard DMControl tasks such as Hopper-hop and Reacher-hard and found CURL is more effective than RAD. Augmentation baseline like RAD may be better than SSL baseline such as CURL on only the six DMControl benchmark environments (in the paper), but that does not mean augmentation is always better than SSL on DMControl, especially on hard DMControl tasks. Therefore, I think the conclusion is biased.
2. Most tested SSL methods in this paper are naively applied to RL. As far as I know, SSL methods applied to RL framework often need specific design. It would be much better if the authors use off-the-shelf SSL for RL methods. For example, when the authors study BYOL or predicting future, they should consider using SPR [56].
3. Readers can hardly learn useful information from this paper. The conclusions that the authors make vary a lot on the test environments, e.g., on DMControl and Atari.
4. The organization of this paper needs to be improved. Too many unimportant contents are put in the main text while some important figures/tables and analysis are in the appendix.

questions: Questions:
1. Line 235: why study 86x86 and 87x87?

Suggestions:
1. Some experiments or methods are not insightful. For example, RotationCLS and ShuffleCLS can cause ambiguities. An input image and the rotated image (with 180 degree) both make sense in some environments, then how do the networks make the discrimination? Besides, "tanh" and many augmentation tricks are presented. They are a little bit off the point of the paper "SSL impact on RL".
2. PSO seems to be an offline search method while RL is an online and dynamic learning process. The search results are likely to be sub-optimal.
3. Studying which SSL benefits which type of tasks is more interesting and insightful.
4. Line 230: "Random" ->"random"
5. SPR [56] has published and the authors should cite the proceeding version.
limitations: The authors have addressed some limitations.
review_title: null

Meta-review: metareview: 
The paper studies an important question, and extends the contrastive reinforcement learning framework to jointly optimize SSL and RL losses. The paper also experiments with various self-supervised losses to empirically validdate the main claim -- "the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentation when the same amount of data and augmentation is used"
The paper presents a surprising result and hopefully provides an interesting platform for others to build on. 

The main novelty of this work in not necessarily in algorithms or systems, but rather providing a thorough experimental evaluation of some of the insights that are known either as 'dark knowledge' or implicit insights from an aggregation of number of previous papers. 
In that it does a good job. The reviewer opinion is split, and rightly so, given the flaws in the presentation and experimentation. 
- conclusions are not very rigorous
- Most tested SSL methods in this paper are naively applied to RL

The rebuttal however has yielded a stronger manuscript, which is likely to be useful to the community. 
The AC strongly advises the authors to make the claims more objective, and less definitive, opinionated, or catchy/click-bait. 
Further the manuscripts should be revised to include the gist of the discussion in the main paper, and addditional clarifications in the appendix. 
confidence: Certain
Decision: Accept