Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses pretrained visual encoders to improve the sample efficiency and generalization performance of visual RL. The main methodological contributions are in the details -- which pretrained layer to transfer, updating of batchnorm parameters, freezing of the encoder, etc. Experiments demonstrate superior performance compared to RL methods that do not use out-of-domain pretraining.
strengths_and_weaknesses: The main strength of this paper are the results -- it works well! The main weakness, in my opinion, is the originality: many other papers have proposed and demonstrated essentially the same high level point. I will elaborte on these points below:

**Originality**
Making use of powerful pretrained representations is ubiquitous in our field right now (c.f. foundation model hype). This paper characterizes the results as "suprising" but I think the opposite is true: these results will be seen as thoroughly expected by a large portion of the field. This is in part due to the many other papers that have shown similar results in the specific domain of visual pretraining for embodied control (e.g., [76, 72, 48] and many more cited by the paper), and partly because the same trend has been so dominant in other areas of ML. I think [48] does a better job of characterizing this finding, in its title, as "unsurprising". So, the main message of this paper -- that visual pretraining can help, and a lot -- is one that has already, I think, been absorbed by the field (some of the cited related work may be concurrent, but there are also many papers that were published well before the neurips deadline on this topic, including [6], [76], [62], etc). That said, I do think the details in the present paper have some originality, especially the use of batchnorm for adaptation. I personally haven't seen that used before in this specific context (although batchnorm for adaptation in general is a well known method, e.g., https://openreview.net/pdf?id=BJuysoFeg).

**Quality**
The quality of results is impressive: the paper delivers on showing that pretraining helps, both in terms of robustness/generalization and in terms of sample efficiency.

**Clarity**
The writing and figures were all clear to me, except one aspect. I would have liked more detail on how the ResNet was pretrained. This is barely mentioned but seems essential to the success of the method. Indeed I would be interested to see how different pretraining methods and datasets affect the results.

**Significance**
Despite that I don't think readers will be surprised by the results, getting the details right, and achieving high performance with a simple system is a significant accomplishment, and could be impactful assuming the code is released. At the same time, I don't think the specific pretrained model in this paper will be used for long, and subsequent work will have to redo the analysis to pick which layers of future models to transfer, and how to tune the other methodological details.




questions: I think it would be very interesting to delve more into the question of "what kind of pretraining helps for what kind of tasks"? The MoCo comparison at the end of the paper was a step in that direction. More pretrained models could be tried. Are there any lessons to glean? Is it always best just to use the SOTA general-purpose vision system? How then does CLIP do, or its various follow-ups? To me these kinds of experiments would strengthen the originality of the paper, and take it beyond what are, to me, expected results.

Along similar lines, the paper could be strengthened by comparing against a variety of methods that _all use the same pretraining dataset_. It's not terribly surprising that PIE-G can outperform methods that have access to less data than it. A more apples-to-apples comparison would be to compare between methods that all are given access to ImageNet, and then see, among those, which is best, and why. The ablations in Section 5.4 are a step in this direction.
limitations: I think the discussion of limitations was adequate. There was not much discussion, but I don't think much is needed for this paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Summary: This paper suggests the use of an off the shelf, frozen, pre-trained ResNet model as an encoder for Reinforcement Learning, as opposed to learning an encoder from scratch. The paper suggests that this enables better generalization as the pre-trained ResNet model serves as a universal representation, which is robust and generalizes well to unseen scenarios. They also show that BatchNorm is critical for better generalizing RL agents.
strengths_and_weaknesses: Strengths:

1. Generalizing well to unseen environments is a fundamentally important problem, and advances in this direction are both extremely important, and useful for the community.

2. The writing is clear and easy to follow. Figures do a good job of explaining the main idea visually.

Weaknesses:

1. Unfair baseline: The main difference in PIE-G is that the encoder is trained on different data, while the baselines are testing different training methodologies. When evaluating generalization to unseen data, control over the training distribution is absolutely essential. For instance, was the generalization better because the testing environment was maybe overlapping with ImageNet? For the unseen backgrounds case, it appears so as the backgrounds are real-world natural images. The typical game in generalization is to improve without access to new data. Here, new and uncontrolled data is being added to PIE-G, but not to the other baselines. This makes the comparison unfair, and also hard to qualitatively gauge.

2. Unsupported claims: The paper suggests that embeddings from a pre-trained ResNet are a universal representations which are both robust and generalize well. However, it is well documented in the computer vision literature that this is not true [1,2,3]. For instance, works have shown that ImageNet trained models do not generalize to new ImageNet images either [1]. Learning generalizable features is an active area of research, and there is significant evidence suggesting that pretrained ResNet features cannot be called universal, robust or generalizable.

In summary, it appears that the performance gains may be due to the additional dataset which baselines do not have access to, as opposed to learning a universal representation.


References:

1. Recht, B., Roelofs, R., Schmidt, L. and Shankar, V., 2019, May. Do imagenet classifiers generalize to imagenet?. In International Conference on Machine Learning (pp. 5389-5400). PMLR.

2. Hendrycks, D. and Dietterich, T., 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.

3. He, K., Girshick, R. and Dollár, P., 2019. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4918-4927).
questions: - Is the fundamental goal to prove that better data distributions are better than better architectures for encoding images in visual RL? That would be a promising direction, but would require significant more experiments than the current manuscript.

- The current experiments show that: (1) Pre-training is better than training from scratch for the same architecture, (2) Simpler architecture with better data performs better than complex architecture with simpler data. Unfortunately, these two are both well known ideas, and building on top of them is necessary to arrive at the point mentioned above.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate using a pre-trained visual backbone in a RL setting. Specifically, they use off-the-shelf ResNet encoders trained in a supervised or self-supervised manner, and use the activations in intermediate layers as input to the policy network, as opposed to the raw pixel values. They find the resulting policy to be significantly more robust to changes in appearance than policies trained on raw pixel values, with fairly dramatic improvements under strongly distracting conditions. 
strengths_and_weaknesses: The paper presents a simple and effective idea. Although most of the components are standard (pre-trained visual encoders, the policy stacked on top), the authors carefully choose and calibrate these components, and the results are very compelling. 

In particular, it is interesting that intermediate feature activations perform better than high-level ones, and that allowing the batch-norm statistics to adapt to the new environment is helpful. These insights make the difference between the method working or not, and will likely apply in a variety of other settings. 
questions: One question we are left with however, is whether higher-level features (e.g. from levels 3 and 4 of the ResNet) simply underperform because of their lower resolution, or whether the more "semantic" information they encode is actually detrimental for the task. Would it be possible to use these features with higher-resolution images, to control for this difference? 
limitations: No ethical or societal risks here.
review_title: null

Meta-review: metareview: This paper contains interesting findings in a research topic currently drawing a lot of interest from the community, i.e., RL with pretraining from large-scale general out-of-domain data. I think the use of low-level features and the batch-norm can be interesting to the community. As pointed by reviewer UwhJ, however, I agree that the authors should moderate and clarify some claims in such a way to acknowledge the fact that this line of research has already been studied recently in many works and thus it is not the first finding. I suggest updating the writing in the camera ready version to focus on the specific contributions such as the low-level features and batch-norm. In particular, the contribution (1) in the last paragraph of Introduction is wrong and thus should be changed because it is already well known but not discovered first in this paper.
confidence: Certain
Decision: Accept