Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The goal of the paper is to investigate how to efficiently adapt large-scale pre-trained vision-language models (e.g. CLIP) to downstream tasks. Their paper is based on the observation that while the vision community dominantly uses linear probe as the standard protocol, other approaches such as prompt learning are utilized in language. They compare and analyze several fine-tuning methods -- linear probe, prompt tuning, adapter and compacter networks -- across 12 downstream classification tasks. Focusing on LayerNorm, they further demonstrate combining LayerNorm tuning with existing fine-tuning methods to improve performance. 

main_review: Strengths
* I like the motivation of the paper -- the vision community broadly uses linear probe as the standard protocol for transfer learning, yet the prominence of prompt learning in NLP clearly shows that there can be better methods to transfer knowledge of large-scale pre-trained models in the vision domain. 

Weaknesses
* In Figure 3-4, it seems that LayerNorm performs better with an extremely small margin, which could easily be reversed with initialization or hyperparameter search for each fine-tuning method. This makes me question why the paper is particularly focused on analyzing LayerNorm in the second half of the paper. Also, combining LayerNorm provides minimal improvement. 
* Instead of averaging and phrasing as "low data" vs. "high data" in the paper, it would be more accurate to plot the actual 1-512 shots on the graph. Averaging it makes it hard to grasp the correct trend of each fine-tuning method over the dataset scale. 
* Furthermore, it seems extremely arbitrary to divide the distribution of downstream data as "high and low similarity" with an arbitrary threshold of 55%. 

summary_of_the_review: Overall, this paper seems to have limited contributions for adapting pre-trained vision-language models to downstream tasks. The depth of the analyses is limited and does not provide interesting and novel findings to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates a range of techniques for adapting CLIP to different tasks. They find that only tuning the LayerNorm is effective and further combining it with other adapting approaches delivers better performance. 

main_review: 
Pro:

It is good to know that LayerNorm tuning is quite effective for CLIP and combining it with other approaches gives even better performance.


Con:

1. Limited novelty.

The methods used in this paper are existing methods and the authors do a simple combination and benchmark them with CLIP on various datasets.

The conclusion I could draw is 1) layer norm tuning works well and 2) one can combine it with other methods (essentially have more parameters to tune) to achieve better results. I am not sure if these conclusions are that exciting and sufficient for an accepted paper.


2. Missing full-model fine-tuning.

The authors argue full-model fine-tuning as "inefficient" and exclude it from comparison. However, I think full-model fine-tuning results are important to include.

a. It is important to see where the upper bound is and results of full-model fine-tuning would put the current results in perspective. 

b. The authors need to be more clear about what "inefficient" means. Does it mean fine-tuning time cost? If it refers to the fine-tuning time cost, it is affected by many factors and is not just dependent on parameter count. And under the setting in this paper, I am not sure how "inefficient" full fine-tuning is, compared to the shown approaches. 
   For example, for prompt tuning, even if most parameters are not trainable, gradients still need to be calculated for back-propagation to the input. Thus, the forward/backward computational cost should be the same for prompt tuning and full model fine-tuning. This is similar to layer-norm tuning (depending on the location of the first layer norm layer). 
   The efficiency of prompt tuning over full-model fine-tuning mostly comes from 1) less optimizer overhead and 2) less communication cost of parameters during multi-GPU training. However, CLIP is not a significantly large model by today's standards (the smallest CLIP is ResNet 50) and on the datasets we are adapting to, we often do not need many GPUs to fine-tune the model. I have to suspect that the bottleneck is not on either the optimizer overhead or communication cost when we fine-tune CLIP on those small datasets (please correct me if my assumption is wrong). Thus I am curious to see the actual speedup prompt tuning holds over full fine-tuning.



summary_of_the_review: The paper presents an empirical study of how to fine-tune CLIP for downstream tasks. 

My main concerns are: 1) from my subjective view, I am not sure the conclusions are significant enough; 2) full fine-tuning results are missing.

I recommend adding the full fine-tuning results for a full picture and being more specific about the reason to exclude full fine-tuning. It would be good to provide a compelling reason to favor these methods over full fine-tuning in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper has extensively studied how to adapt the large-scale pre-trained vision-language model CLIP for downstream tasks. Several fine-tuning methods are analyzed across a diverse set of image classification tasks along two spectra. Further, a simple yet effective strategy that combines LayerNorm-tuning with general fine-tuning methods is proposed to improve their performance and benchmark them on few-shot classification tasks. 
main_review: Strengths:

1. The LayerNorm-tuning for adapting CLIP to downstream tasks is shown to be effective.

2. A simple yet effective scheme that combines LayerNorm-tuning with other fine-tuning methods is proposed to obtain competitive performance across the board.

3. A thorough comparison of different adaptation methods is provided in four scenarios across two spectra. 

Weaknesses:

1. The paper title “How to Adapt Your Large-Scale Vision-and-Language Model” is a bit of over-claimed. It should be toned down to “How to Adapt CLIP to Downstream Image Classification Tasks”.

2. The technical novelty of this paper seems very limited. The three methods for fine-tuning new parameters have been extensively studied in existing works. When coming to adapting CLIP, there should be something different. That is, novel fine-tuning methods are needed in adapting CLIP. 

3. The experimental evaluation is far from sufficient. (1) For “LayerNorm-Tuning”, the parameters of all LayerNorm layers are set to be learnable. I wonder if the results could be better when only the last K LayerNorm layers are updated (e.g., K=5). (2) For “Full Model Fine-tuning”, it should also be evaluated in the experiments by only updating the parameters of the last layers of the model. (3) I wonder if the proposed method is effective in other downstream tasks other than image classification. 

summary_of_the_review: This paper had been well written if entitled “How to Adapt CLIP to Downstream Image Classification Tasks”. However, considering the insufficient experimental evaluation and limited technical novelty, I could only give it a score of 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new method (LayerNorm tunning) for finetuning a pre-trained vision-and-language model. While simple, it is shown this method works competitively with other methods (e.g., prompt tuning) in various settings. For the low data & high similarity setting (Fig 3), it shows the best performance across the methods. The authors further claimed LN could be used to boost the performance of general finetuning methods like linear probe or prompt tuning. However, this seems not useful as the performance typically drops when compared to the LN baseline. In addition, the proposed methods are only evaluated in a single model, it is not clear whether they are generalizable.
main_review: - Strengths
    - The proposed LN tuning is simple and seems quite strong compared to other more complex methods.

- Weaknesses:
    - While the authors claim that the proposed method is useful for vision and language models, only a single model (CLIP) is used for these studies. Though the method design is almost model-agnostic, it is not clear whether this approach generalizes to other models.
    - The tasks studied in this work are all image classification tasks, there are no tasks that are normally recognized as vision-and-language tasks, such as text-image retrieval or image captioning [1]. If the authors' intention is to use this method for image classification tasks, it would be better to change the title a bit as "How to adapt your large-scale pre-trained models for visual representations learning", instead of the current confusing one of "vision-and-language model".
    - In the high-data regime, it seems also feasible to finetune all the model parameters, how does this result compare to the considered baselines in this work?
    - The best performance is often achieved by first finetuning the LN parameters, then using another approach like Adapter for a 2nd stage finetuning. This two-stage finetuning process could be cumbersome.
    - After LN finetuning, it seems that additionally finetune with many of the other methods actually hurts performance. For example, under low data & high similarity setting, with only LN finetuning, the performance is 83.74 (see Fig. 3, top-left), while the performance after further finetuned with other methods generally falls (see Fig. 4, top-left), only Compactor shows a minimal gain with a performance of 84.07. This conclusion seems also applicable to other settings in these two figures. This seems to suggest that we should probably only use LN.

- Miscs:
    - Some of the references are not linked, e.g., the 3rd paragraph in Sec 4.5, "Table 4" is not hyper-linked
    - "Normal" setting in Table 1 is not clearly explained. I made an educated guess as "simultaneously tuning them" when evaluating the work.

[1] Chen et al. Microsoft coco captions: Data collection and evaluation server. *arXiv 2015*
summary_of_the_review: The paper proposed a useful technique LN tunning for fine-tuning the pre-training CLIP model for downstream image classification tasks. However, the generalizability of this approach is not demonstrated, and the proposed recipe of combing LN tunning with other fine-tuning methods seems also not useful as it typically shows even lower performance than only using LN tuning. Overall, this paper does not provide enough support for it to be published at ICLR.
review_title: null

Meta-review: This paper investigates several fine-tuning methods for adapting the pretrained vison-language model CLIP to different downstream tasks.  They found that LayerNorm (Ba et al., 2016) is a rather effective and competitive approach and investigated different ways of combining LayerNorm-tuning with other adaptation methods. Reviewers generally agree that the technical novelty (combination of existing techniques) is limited and contribution is marginally significant. While some empirical results look interesting, the overall contribution is incremental. Overall, the paper has done a good job of thorough empirical evaluations, but the overall technical novelty and new empirical findings are not significant enough for publication at this conference.
Decision: Reject