Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of how to leverage pre-trained visual representations for control. They show that simply freezing the visual encoder and training a policy head leads to poor performance. Performance can be improved by fine-tuning the visual encoder but this leads to catestrophic forgetting and thus a seperate set of (large) weights must be stored for each task.

The authors propose to use adapters that add a small number of additional parameters to the model and allow it to be adapted to a new task. Only the paremeters of the adapters are learned during fine-tuning, thus the same set of visual encoder weights can be acrossed mutliple tasks.

The proposed method is evaluated across 3 sets of tasks, Metaworld, Kitchen, and RGB block stacking using 3 different backbones. The proposed method is able to (mostly) close the gap to full fine-tuning in all cases. The proposed adapters are also effective in zero-shot sim2real.
strength_and_weaknesses: ## Strengths
- Evaluation of the method itself is comprehensive and the results are meaningful.
- The sim2real results are impressive.

## Weaknesses
- The following is untrue: "While prior work on robotic manipulation has predominantly used frozen pretrained features, we demonstrate that in robotics, unlike in other domains, this approach can fail to reach optimal performance, and that fine-tuning of the full model can lead to significantly better results." In other domains, fine-tuning also improves performance. For example, in ImageNet classification, frozen pre-trained features performance worse than fine-tuning the full model. This is shown by the difference in fine-tuning and the linear probe evaluation in recent SSL papers, for example "Masked Autoencoders Are Scalable Vision Learners, He et al, CVPR 2022".
- While the authors do a good job demonstrating that adapters are effective at closing the gap between fine-tuning and frozen, it is unclear to the reviewer if the proposed adapters are novel. The proposed adapters are very similar to those in "Cross-domain Few-shot Learning with Task-specific Adapters, Li et al, CVPR 2022". It appears as they the exact formulation of the adapters is different however no comparison in performance is provided.
clarity,_quality,_novelty_and_reproducibility: See above for originality. The work is clearly written and reproducible.
summary_of_the_review: My primary concern with this paper is the evaluation of the proposed adapters. While it is clear that they are effective, it is unclear how they compare with similar methods already present in the literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method for finetuning pretrained vision models for robotic manipulation tasks that aims to reduce disruption to the original pretrained visual representation and loss of original representation expressivity. The proposed method injects learned lightweight ‘adapters’ throughout the pretrained architecture at multiple levels, rather than just at the top level. The empirical evaluations show that the approach is able to generally match the success rate of the fully fine-tuned vision representations, but with a fraction of the learnable parameters.
strength_and_weaknesses: Strengths
- Paper is tackling an important and timely challenge of how to best integrate powerful pretrained vision models for control tasks. 
- The experimental setup seems to be thorough and well thought out — decomposing the positioning of the adapter modules into top, middle, bottom portions of the network is interesting, and the authors also conduct a thorough comparison across different vision architectures. 

Weaknesses
- Would be helpful to get a sense of statistical significance by adding error values to the the success rates reported in Tables 1,2,3 and Figure 5.
- The claim ‘lossless adaptation’ made me think that the authors would investigate how much the model’s representations deviated with the adaptors (i.e. looking at performance changes with respect to original classification, detection capabilities, etc.). Perhaps ‘lossless’ is too strong of a term if not actually demonstrated?
- Paper would be strengthened with including further ablations of their architectural design — e.g. impact of not having the skip connections.

Questions
- Beyond the sim2real experiments, have the authors considered investigating more cross-domain transfer — e.g. how well do the representations adapted for MetaWorld transfer to the Franka kitchen domain? Do the trained adapters transfer well or need to be fully retrained?  
clarity,_quality,_novelty_and_reproducibility: The paper is clear to read and well written. To the best of my knowledge, proposing and studying this type of lightweight adaptation of pretrained vision models across multiple model layers and architectures is novel. Code doesn't seem to be provided so reproducibility is unclear.
summary_of_the_review: I recommend an 8. The underlying method is straightforward and simple, and the impacts of various types of adaptation, finetuning, etc. seem thoroughly explored experimentally across a range of different environments and a real world task.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work investigates the use of pre-trained visual representations for robotic imitation learning. First, the authors show empirically that transfer should in general involve fine-tuning, as this is shown to dramatically increase performance over using frozen pre-trained weights. Then, to avoid catastrophic forgetting of the pre-training task, the authors study the performance when fine-tuning using "adapters" -- additional layers with small parameter counts that preserve the original network when initialized -- instead of fine-tuning the original weights. The authors show that using adapters can close most of the gap between pre-trained frozen weights and full fine-tuning. They also show that using adapters at multiple points in the model is important for performance.
strength_and_weaknesses: Strengths

The authors study the important problem of using pre-trained visual representations for robotics. The empirical evaluation is rather extensive, involving: 5 tasks x 2 domains x 3 camera angles = 30 task instances as used by Nair et al. (2022), as well as 5 task instances from RGB-Stacking; 3 popular architectures for visual representation learning; supervised and self-supervised pre-training objectives; and ablations on the use of adapters in various parts of the pre-trained model. The proposed use of adapters appears to result in performant transfer.

Weaknesses

The motivation of the problem that this paper proposes and tackles is not clear. A simpler and more performant way to solve the lossless adaptation problem would be to do full fine-tuning for adaptation while retaining a separate copy of the pre-trained weights. While I do not have in-depth familiarity with the adapter literature, from a cursory skim it does not seem that these works motivate their proposed methods with losslessness, but rather computational efficiency and memory usage during fine-tuning. 
clarity,_quality,_novelty_and_reproducibility: Clarity

The writing is reasonably clear.

Quality

The execution of the work appears sound.

Novelty

The novelty of the work is low. The pre-training architectures, pre-training objectives, transfer methods of fine-tuning and adapters, and robotic imitation learning benchmarks and datasets are all from prior work.

Reproducibility

The authors provide a reasonable level of detail that would facilitate a solid attempt at reproducibility. Code is not provided or promised, however.
summary_of_the_review: From this work, I learned that:
1. Using frozen pre-trained visual representations is a relatively ineffective way of doing transfer to robotic imitation learning.
2. Fine-tuning these representations results in very performant i.i.d. transfer.
3. Using adapters gets us close to full fine-tuning.

Overall, this work applies methods from prior work to obtain results that corroborate those found for other domains (Houlsby et al., 2019; [B]). While the execution of the work is good, the limited novelty and unclear significance (specifically regarding "losslessness") makes me lean towards rejection.

References

[A] Kumar et al., Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022.

[B] Berriel et al., Budget-Aware Adapters for Multi-Domain Learning, ICCV 2019.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an approach that takes pretrained vision modules and fine-tunes them for solving downstream tasks, but importantly without fine-tuning whole parameters by introducing adapters consisting of a small number of parameters. The proposed adapters are designed to be applicable to widely used architectures (ResNet, ViT, NFNet) and the effectiveness of the proposed approach is evaluated on  a wide set of simulator tasks and also sim2real tasks.
strength_and_weaknesses: ## Strengths
- Intuitive approach to utilize pre-trained modules, while it's already known technique in different fields especially NLP.
- Useful observations that show generic pre-trained models might not be sufficient for visual manipulation.
- Performance gain with only introducing a small number of parameters and fine-tuning them instead of whole fine-tuning.

## Weaknesses
- As authors already stated in the related work section, introducing adapter for transfer learning is already known technique and it's not particularly surprising that it also works for robotics tasks (but I think explicitly showing this is still a valuable contribution to the community). It would be more interesting to think of what unique and interesting observations or practical benefits can be achieved by introducing such adapters.
- Following the previous point, more investigation into why we need a loseless adaptation is missing, e.g., what would be the benefit compared to introducing two separate pretrained models and keeping the one while fine-tuning the other one? Or is there an interesting observation that can be made from using adapters for other perception tasks?
- A lot of emphasis is on parameter-efficiency but another important axis would be compute-efficiency. Considered models already seem very large; so it's not clear to me what would be the benefit of being parameter-efficient upon such large models. More discussion on this would be helpful for readers.
- Missing baseline is the performance of training everything from scratch with sufficient compute budgets and varying architecture sizes (as we would not require such a large model for considered downstream tasks).

clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality
The paper reads well, but Figure 3's caption is a bit difficult to parse without reading the main draft.

### Novelty
As far as I'm aware, introducing adapters is already proposed in the context of natural language processing [1, 2, 3, 4], as also discussed by the authors in Related Work. Hence the main novelty of the paper is not in the method. It would be interesting to consider the approach similar to [5].

[1] Houlsby, Neil, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. "Parameter-efficient transfer learning for NLP." In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.

[2] Karimi Mahabadi, Rabeeh, James Henderson, and Sebastian Ruder. "Compacter: Efficient low-rank hypercomplex adapter layers." Advances in Neural Information Processing Systems 34 (2021): 1022-1035.

[3] Mahabadi, Rabeeh Karimi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. "Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks." arXiv preprint arXiv:2106.04489 (2021).

[4] Gao, Peng, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. "Clip-adapter: Better vision-language models with feature adapters." arXiv preprint arXiv:2110.04544 (2021).

[5] Sung, Yi-Lin, Jaemin Cho, and Mohit Bansal. "Lst: Ladder side-tuning for parameter and memory efficient transfer learning." arXiv preprint arXiv:2206.06522 (2022).

### Reproducibility
Source code is not attached.
summary_of_the_review: While the technical novelty of the proposed approach is not high, it is interesting to see that introducing adapters can be also effective for robotics tasks and claims made in the paper are well supported with exhaustive experimental results. Hence I recommend the paper to be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an extensive study of adaptation mechanisms for representation learning in the context of robot manipulation.

The paper offers exhaustive experiments to evaluate the effectiveness of different adaptation schemes for representation learning in the context of robot manipulation. The experiments are impressive and well executed and show interesting results. The paper is addressing a timely challenge of how to best integrate powerful pretrained vision models for control tasks. There were some initial issues regarding novelty of the approach and distinction to related methods. However, this was addressed adequately by the rebuttal of the authors as the paper is considering a different problem domain which also requires different adaptation architectures. I consider most concerns of the reviewers to be resolved adequately and recommend acceptance.
justification_for_why_not_higher_score:  Only small algorithmic novelty.
justification_for_why_not_lower_score: Impressive evaluations.
Decision: Accept: poster