Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a comprehensive analysis of the use of pretrained image models for some major vision tasks. The paper shows the importance of well-placed tunable parameters to bridge the gap between frozen and finetuning settings, and presents an analysis of feature activation to further showcase this.
strengths_and_weaknesses: Strength
* The paper presents an insightful analysis of the properties of frozen pretrained models for downstream tasks
* On detection tasks, the gap between frozen model and finetuning is negligible, while the gap is still there for semantic segmentation and video action recognition.
* The authors also analyze different sizes of base network and different pre-training strategy and report their properties on various downstream tasks. It’s interesting to larger models require less tunable parameter and achieve better performance without increasing trainable parameters.

Weakness:
* The results in Table 5 suggest that for some tasks frozen models are good enough (e.g. detection), but for others fine tuning still provides clear benefits (e.g. semantic segmentation, action recognition). It would be interesting to delve deeper into what causes the gap, e.g.  pretraining methods, model design, task itself, or specific dataset. For example, it’s interesting that the AP mask on COCO shows no gap between frozen vs full ft, but there’s a 2 point gap on ADE20K mIoU. I would expect the trend between AP mask and mIoU to be similar as both involve pixelwise localization. For video action recognition, I’m wondering if using a frozen model pretrained on videos (rather than images) would help bridge the gap with full ft.
* It might be helpful to show the benefits of frozen models by comparing training time or resources used.
* In Table 1, it may be interesting to add MAE (masked autoencoder) style self-supervised learning and see if that shows any difference between frozen and full ft.

questions: See weakness.

limitations: Not adequately. There's fairness risks around using frozen pretrained models depending on the datasets the frozen models are trained on.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies way to use large, pretrained image models for many different tasks. The approach pre-trained a swin transformer model using either supervised learning, contrastive learning or masked image modeling. This model is then frozen, and new layers are added and trained for the specific task. This paper finds that supervised pretraining transfers best and adding more layers for the finetuning is generally beneficial.
strengths_and_weaknesses: The paper is well written, clear and easy to follow. The topic is quite relevant as training large models is becoming common, and finding the right ways to use the pretrained models is quite important.

However, there are a few weaknesses that should be addressed:

(1)  Table 1 results are only on small-ish pretraining datasets of ImageNet-1k and -21k. This could be limiting, as a main benefit of the contrastive training, e.g., CLIP and ALIGN has come from the massive dataset sizes (e.g., hundreds of millions to billions). It has also been shown that large image transformer models (e.g., ViT) benefit more from large supervised pretraining (e.g., billions of samples). These observations make it hard to know if the results in table 1, Fig 4, table 5 will generalize, or if they are specific to this setting. For example, adding experiments using the CLIP pretrained model, to see how that scale of pretraining data effects the results.

(2) The paper doesn't really propose anything new. It is a useful study, however, it is limited by the few settings compared. E.g., adding more layers and BiFPN for object detection, adding more layers for segmentation/action classification. These aren't especially interesting insights. One of the questions this paper sets out to address is:
"What is the key to making the frozen setting work well when the downstream tasks are significantly different from the pretraining task?" 
And I don't think the current experiments really answer that question. Unless the key is simply adding more tuneable layers. Related to that, it would be interesting to compare adding trainable layers vs. making parts of the pre-trained model trainable. E.g., instead of adding 10 new layers, only add 1, but make the last 9 of the pre-trained model trainable.

There are many other options not explored in this work, which limits the insight gained from this work.

Overall, the paper has limited originality and significance due to those weaknesses.
questions: Please address the weaknesses above. Specifically,

(1) Will these results change with different pretraining data scales?
(2) Are these methods really the key to using frozen image models? Are there other methods that could be better?
(3) If this paper is trying to find the key to using frozen models, why are the results lower than finetuned? Is this an inherent limitation of using frozen models? Or has the method to properly utilize pretrained models not been found?
limitations: There was no impact section and the only limitations mentioned were that the approach does not perform as well as full finetuning.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Authors present a relatively comprehensive study of how far frozen pre-trained models can get us for multiple downstream computer vision tasks: object detection, semantic segmentation and action classification. They specifically explore 1) which pretraining task leads to most useful representations (out of supervised, self-supervised and masked image modeling, supervised on In22K is best); 2) How is the performance affected by adding more tunable parameters (it helps significantly, but less so for larger frozen models -- similar to trends observed for large language models) and 3) What is the best results one can achieve with a very large frozen model (Swin-V2-G, 3B parameters). They obtain some good results on all the 3 tasks considered.
strengths_and_weaknesses: ## Strengths

1. [Significance] Impressive results with a frozen backbone: The results reported without finetuning the backbone are quite impressive and to my knowledge not previously reported. For instance 79.6% on Kinetics-400 with a frame-level feature extractor + few temporal/global aggregation blocks is comparable to recent work like TimeSformer, which is finetuned end-to-end. However it is obtained by adding additional tunable parameters on top of the frozen model, so it's not clear how favorably it compares in training compute cost to full finetuning.

2. [Clarity] The paper is well written and easy to follow. The sections are organized well and flow naturally, making the paper an easy and pleasant read.

## Weaknesses

1. [Novelty] The results aren't particularly novel: While they haven't been reported before to my knowledge and hence are interesting for the community to know, they aren't surprising or unexpected. ImageNet-22K models have been well known to generalize better to downstream task; and masked-image-modeling based models have not generalized well without full finetuning. Adding more tunable parameters is expected to improve performance. 

2. [Quality] Analysis of training speed/memory usage: What does training with frozen backbone enable? One important gain would be making state of the art models accessible to users with limited resources. However the paper doesn't show explicit comparison on this axis. How does the training time of frozen + tunable parameter models compare to finetuning the frozen model without tunable parameters? Does it lead to significant training time or GPU memory savings that enable SOTA models to be trained on very small GPUs? Or do the additional trainable parameters somewhat nullify the gains by freezing the backbone?

3. [Clarity] Some parts of the model enhancements are not clear. What is the precise design of the "global" and "temporal" blocks added in Table 4? Also, how is the aggregation over the full video done?

4. [Quality] Additional analysis: While the paper has decent amount of analysis, it is limited to Swin Transformer based architectures; similar analysis on ViTs would make the paper stronger. Also, the set of tasks is somewhat limited -- object detection and semantic segmentation are quite similar (and authors also observe similar trends for both). Paper's impact can be increased by adding few more computer vision tasks, such as vision-language tasks (VQA, captioning etc), 3D/RGBD recognition tasks etc.

## Overall
The paper is generally well written and motivated. The results are interesting however not totally novel or surprising. More analysis of the computational savings would make future versions of the paper stronger. Hence, I am borderline, however leaning towards acceptance given the range of results shown across object detection, segmentation and video classification, with different sized Swin models etc.
questions: I would be interested in seeing more analysis on computational savings (i.e. training time and GPU memory) of adding additional tunable parameters on frozen backbone VS finetuning the frozen backbone. If that improves accessibility of SOTA models to users with limited resources, that would increase the significance of this work.
limitations: N/A
review_title: null

Meta-review: metareview: This paper presents a study of how well pre-trained and frozen large models work across several downstream computer vision tasks. The paper initially received mixed reviews with two of them being borderline accept and one borderline reject. The reviewers shared their concerns about the novelty of the investigation and its impact with some additional questions about the setup. The authors provided a rebuttal that addressed some of the reviewers' concerns. Two out of three reviewers updated their reviews in the post-rebuttal phase. Reviewers generally agree that the paper should be accepted but still have concerns regarding the novelty. Due to the comprehensive empirical analysis, AC recommends acceptance but suggests that the authors are urged to look at reviewers' feedback and incorporate their comments into the camera-ready.
confidence: Less certain
Decision: Accept