Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper outlines a 3-stage process to train policies. Stage 1 involves training an encoder using large image datasets to obtain task-agnostic visual representation. Stage 2 involves training a policy using offline RL and the pre-trained encoder. This serves a as a good initialization for the final stage 3 that involves further training the agent with online RL.
strengths_and_weaknesses: Strengths: The overall pipeline appears obvious and straightforward. In my view, the main contribution of the paper is putting together the components and demonstrating empirical success. 

Weaknesses: While the overall approach and results look promising, the current submission unfortunately lacks technical depth and rigor. In the introduction section, the authors outline their main contributions. Unfortunately, I do not believe there is sufficient evidence to rigorously substantiate these claims.

1. "Novel framework..." -- I believe there is limited novelty in this submission, especially in light of several recent papers (most of which are published by now). For example, the current submission does not cite or discuss MVP[1], PVR[2], R3M[3] which all explore training encoders from out of domain data.

2. "Novel technical contributions..." the main technical contribution claims are: (a) CCE or convolutional channel expansion; (b) safe Q technique. As for CCE, there are no experiments to benchmark it against simple alternatives like contatenation of framewise embeddings [1,2,3] or their latent differences [4], which have been successfully used in several prior works. The safeQ update does seem to help compared to a naive update rule. However, the challenges in transitioning from offline RL to online RL has been well-documented in prior work, along with different solutions (e.g. AWAC [5]). The authors should consider comparisons with such SOTA baselines as opposed to naive straw-man baselines. Overall, due to the lack of rigor in comparisons and baselines, it is hard to judge the importance or utility of the claimed technical contributions.

"VRL3 achieves an entirely new level of SOTA sample efficiency, parameter efficiency, and computation efficiency on the highly challenging Adroit benchmark while also being very competitive on DMC." -- I would encourage the authors to rephrase such statements to more technically rigorous and falsifiable claims. 

[1] Xiao et al. Masked Visual Pre-training for Motor Control. arXiv 2022.

[2] Parisi et al. The Unsurprising Effectiveness of Pre-Trained Vision Models for Control. ICML 2022.

[3] Nair et al. R3M: A Universal Visual Representation for Robot Manipulation. arXiv 2022.

[4] Shang et al. Reinforcement Learning with Latent Flow. NeurIPS 2021.

[5] Nair et al. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. 2020.
questions: Please see strengths and weaknesses section for details and context. My main questions/concerns are related to:

1. Novelty of VRL3 and its relationship to prior work (that appeared sufficiently before NeurIPS deadline) -- MVP, PVR, R3M etc.

2. Lack of rigorous comparisons to substantiate utility of CCE and SafeQ over current SOTA baselines (as opposed to naive straw-man baselines).
limitations: The limitation section in the paper is too generic (e.g. evaluation in sim vs real robots).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors tackle the problem of improving the efficiency of online learning for visual RL tasks. The main idea of the paper is to pre-train an encoder on non-RL data (Imagenet), then train a value function and policy network on top of the shared encoder with offline RL, and finally fine-tune with online data. With this simple framework and a small number of technical tweaks (e.g. how to pre-train the encoder with single-image input but transfer to tasks with multi-image input, how to stabilize the value function), the authors show that their method achieves SOTA sample efficiency on the Adroit benchmark.
strengths_and_weaknesses: I found this paper a breeze to read. The method is very practical and leverages popular tools in the contemporary ML toolbox (pre-training and fine-tuning, offline RL) in order to achieve impressive sample efficiency on synthetic visual-input robotic tasks. I especially appreciate the careful ablations the authors performed in Section 5 which I believe gives insight into the contribution of each of the 3 stages that the authors propose for their method. I think this paper provides valuable insight for the ML community on the relative importance of leveraging non-RL, offline, and online data for RL tasks.

The main weakness I see for this paper is I think the authors make some claims about the importance of offline RL vs. BC in section 5.2 that I think need an additional experiment to verify. In lines 263-265, the authors claim that BC is the same as entirely ignoring stage 2 altogether "since BC only trains the actor and not the critic". I think BC should provide a stronger baseline than this - maybe the authors could try a stage 2 in which the policy is trained with BC and the value function is trained in a naive way to predict cumulative reward of the offline trajectories? My takeaway from Figure 4 is that as long as we have a decent policy-value match going into stage 3, the tasks will get solved even if the encoder hasn't even been trained yet (with smoother more efficient learning when the encoder is pre-trained on Imagenet). It would be great to disentangle the intricacies of DrQv2 for offline data from the goal of pre-training both the policy and value function in stage 2.
questions: - Lines 66-68 - I missed if the authors ablate CCE? How does performance compare if the first layer is simply randomly initialized for stage 2?
- Lines 249-251 - a 5-layer CNN seems quite limiting in terms of the range of tasks that such a network can solve. What's the Imagenet accuracy of this network after Stage 1 pre-training? Is the framework the authors propose stable for deeper networks?
- Figure 2 - these curves would be easier to read if the legend was moved outside the plot. I also recommend reordering the labels in the legend to match the ordering of the curves in the graph.
- Algorithm 1 - For me at least I found the algorithm box too confusing to be helpful for understanding the method. I personally feel it distracts from the clarity of description in Section 3 and Figure 1.
- Lines 173 - 177 - "To allow more flexibility in the Q network, we can also set the threshold to be higher than this value". Can the authors elaborate more on why it makes sense to allow the Q network to output larger values than what is already a very high upper bound on the range of Q? I'm also wondering why the authors don't parameterize Q with tanh or sigmoid in order to enforce the correct range and instead use the technique they describe in line 176.
- Line 307 - "to" I believe should be "than"
limitations: Yes - the authors discuss that experiments on simulated benchmarks do not imply success in the real world.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose VRL3, a 3 stage data driven technique for pretraining convolutional encoders, performing offline RL and fine tuning with online RL. The authors demonstrate their approach in a number of vision based control benchmarks, include Adroit and DMC. 
The approach greatly improved sample efficiency and outperforms end-to-end methods. 

The usage of offline datasets and pretrained encoders if an interesting field of work, but I am concerned that the problems this framework is evaluated on are somewhat trivial and I would be interested to see the results after more iterations of fine tuning.
strengths_and_weaknesses: Strengths:
* The paper and methodolgy is clear and easy to read.
* Improved sample efficiency.
* The authors share their codebase and plan to release it as open source, with the data, checkpoints etc.
* This method, could potentially be applied to a large variety of benchmarks and achieve SOTA in terms of sample efficiency.

Weaknesses:
* The method requires training an encoder at the observation size of the environment.
* I am not convinced that this approach outperforms pure online methods if they are left to convergence, figure 2 (a) show the performances of RRL still increasing, has this baseline actually converged
* I am not convinced the approach would transfer well to POMDP settings and more challenging benchmarks (DeepMind Lab, Habitat-lab ...)
* It would have been interesting to see comparisons of other pretraining regimes for the encoder, (self-supervised, BOYL, contrastive.. etc)
questions: * The authors retrain their autoencoder on 84x84 images, given they are using the convoluational part, why could they not take the weights from an already existing pretrained resnet 18, for example?
* What is the benefit of a more convoluted method if I can train my baseline for more iterations and achieve comparable performance?
* How applicable is this process to other RL algorithms, could you have made comparisons to on-policy training and memory-based agents?
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a 3-stage visual RL framework, consisting of encoder pretraining with ImageNet, offline finetuning, and online finetuning. The encoder pretraining stage leverages an off-the-shelf large-scale dataset to learn a powerful representation. Then, the offline pretraining uses a conservative method to transfer the learned representation into task-specific embeddings. Finally, the online rl achieves high sample efficiency in accomplishing adroit  tasks. The core contribution of this paper is the usage of imagenet dataset and offline  method to perform pretraining.  The authors also conduct detailed ablative experiments to understand the challenges and important design choices. 
strengths_and_weaknesses: Strength: 
Although all the components are not new, the combined framework is powerful. Within this framework, the authors conduct extensive experiments to understand possible alternatives and analyze the challenges. This shed light on future researchers who is willing to jump into this field. The sample efficiency on adroit benchmark is much higher than previous baselines and their forged stronger baselines with a large margin.

Weakness:
The overall writing quality is slightly concerning. This is an issue throughout the whole paper. For example, when the authors are describing their experimental results, they use very subjective words such as “good “, “very” without supporting evidence. In the introduction, the authors also describe the necessity of their method using the word “we want to …”. These subjective tones heavily reduce the scientific value of this paper. Moreover, there are many grammatical issues.

Slight overclaim, over-selling, and abuse of terms. The author emphasizes their method is a “minimalist” approach; a 3-stage approach with tons of tunings can hardly be regarded as minimalist. When the authors mention offline RL, they should be more careful when they are only using expert demonstrations. This needs clarification somewhere in the paper. The overall paper quality, especially in terms of the experimental aspects, is ok. That being said, I think over-selling the paper is not a good idea. The authors mentioned that it is a ResNet architecture. However, if it does not have skip connect, the naming is inaccurate.

According to the supp material, the framework is brittle to many hyperparameters. For example, the Q threshold is extremely important to tune. What would happen if other people want to use this paper? Does the tuning take a lot of effort?

The data augmentation technique seems important. More reasoning and explanation are needed.
questions: Per the previous section, could you re-write some key sections in your paper?

Can you explain the sensitivity caused by Q threshold? If other people want to adopt this framework, how they can find a good Q threshold？

Are the data augmentations chosen randomly or on purpose? Why do you choose this augmentation?
limitations: The authors mention real robot experiment and large domain gap as their limitations. These are reasonable. Thank you for being straightforward.
review_title: null

Meta-review: metareview: This paper introduced a simple paradigm for improving sample efficiency of training deep reinforcement learning policies for vision-based control tasks. The idea is to use a 3-stage pipeline: 1) pre-training visual representations on large-scale image datasets, 2) policy training with offline RL, and 3) fine-tuning the policy with online RL. This work received mixed reviews from four reviewers, with one Reject, one Weak Reject, and two Weak Accepts. The reviewers appreciated the demonstrated effectiveness of the proposed approach despite the simplicity of the approach. Meanwhile, they expressed major concerns regarding the limited novelty concerning the burgeoning body of literature on visual pre-training and limited evaluations and ablation studies.

The authors drafted very detailed responses to the reviewers' comments, which clarified many technical issues brought up in the initial reviews. At the end of the discussion period, Reviewer i5GH (who did not engage in the discussions) and Reviewer 1wyj maintained their negative ratings of this paper, while the other two voted Weak Accept.

The AC read the paper, the reviews, and the authors' responses carefully. Reviewer i5GH's main criticisms are 1) the novelty of VRL3 and missing citations and discussions of prior work (MVP, PVR, R3M, etc.) and 2) insufficient comparisons and ablations of key model designs. The AC checked the publication/release dates of the mentioned works and believed they should be considered *concurrent* with this submission. Thus, the technical merit of this work should not be penalized by the existence of these related works. Meanwhile, the authors added the citations and discussions about these works in the revised draft, which addressed the reviewer's comment. In addition, the authors also provided additional ablation studies and clarifications which addressed the second point raised by Reviewer i5GH.

Reviewer 1wyj expressed concerns about the heavy revision during rebuttal and the overclaim and over-selling of the approach. The AC agreed with this reviewer that some language, such as "minimalist", should be toned down in the next reversion of this manuscript.

Taking all these into account, the AC found that the rebuttal has addressed the major issues raised in the reviews. Even though this work does not generate revolutionary ideas, it has shown convincing evidence of a practical approach that improves the learning efficiency of deep reinforcement learning in challenging vision-based control tasks. This work may pave the road for future work to develop more advanced methods. Therefore, the AC thinks that this work has passed the bar of acceptance at NeurIPS, despite the mixed final ratings.
confidence: Less certain
Decision: Accept