How to Adapt Your Large-Scale Vision-and-Language Model
Authors: Konwoo Kim
Michael Laskin
Igor Mordatch
Deepak Pathak
Venue: ICLR 2022 Submitted (2022)
Keywords: transfer learning
fine-tuning
layernorm
CLIP
prompt-tuning
adaptation
zero-shot
pretraining

Abstract: Pre-training large-scale vision and language models (e.g. CLIP) has shown promising results in representation and transfer learning. We investigate the question of how to efficiently adapt these models to downstream tasks. For image classification, linear probes have been the standard for ease of use and efficiency, while for language, other approaches like prompt tuning have emerged. We analyze several fine-tuning methods across a diverse set of image classification tasks across two spectra investigating the amount and similarity of downstream data to that of pretraining one. We find that just tuning LayerNorm parameters is a surprisingly effective baseline across the board. We further demonstrate a simple yet effective strategy that combines LayerNorm-tuning with general fine-tuning methods to improve their performance and benchmark them on few-shot adaption and distribution shift tasks. Finally, we provide an empirical analysis and recommend general recipes for efficient transfer learning of vision and language models. Website at https://sites.google.com/view/adapt-large-scale-models

Under review as a conference paper at ICLR 2022
HoW TO ADAPT YOUR LARGE-SCALE PRE-TRAINED MODEL FOR DOWNSTREAM IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-training large-scale vision and language models (e.g. CLIP) has shown promis- ing results in representation and transfer learning. We investigate the question of how to efficiently adapt these models to downstream tasks. For image classifica- tion, linear probes have been the standard for ease of use and efficiency, while for language, other approaches like prompt tuning have emerged. We analyze several tine-tuning methods across a diverse set of image classification tasks across two spectra investigating the amount and similarity of downstream data to that of pretraining one. We find that just tuning LayerNorm parameters is a surprisingly effective baseline across the board. We further demonstrate a simple yet effective strategy that combines LayerNorm- -eeettnn- with general fine-tuning methods improve their performance and benchmark them on few-shot adaption and distribu- tion shift tasks. Finally, we provide an empirical analysis and recommend general recipes for efficient transfer learning of vision and language models !
INTRODUCTION
Large-scale deep network models pretrained on ultra large-scale data on the internet, whether text or images, have shown impressive performance recently (Radford et al., 2019; 2021; Brown et al., 2020; Devlin al., 2018; Jia al., 2021). Training such models with billions of parameters on a large internet scale data is expensive and time consuming process often costing over millions of dollars. Hence, replicating such models is not only difficult but also undesirable for every downstream task. Fortunately, the information gathered by these large-scale models using raw internet data seems to transfer well to several downstream tasks with little to no finetuning at all using natural language as a way for zero-shot evaluation (Brown al., 2020; Radford et al., 2021).
While zero-shot transfer performs well, it is generally better to adapt the model itself if there are any labeled examples available for the downstream task. Traditionally, the go-to strategy in the computer vision community has either been to finetune the whole network or an additional MLP layer at the end. With the use of raw language, adaptation techniques such as prompt tuning have surfaced (Li & Liang, 2021; Lester et al., 2021). Alternative methods include new parameters in between the network instead of adding layer at the end (Houlsby et al., 2019; Mahabadi al., 2021). However, it remains unclear as to which approach is preferred under which scenarios.
We ask what are the general guidelines one should adopt while finetuning large-scale pretrained model on downstream datasets. To scope this question, we choose CLIP (Radford et al., 2021) as the base pretrained model and adapt it to several downstream problems. CLIP is iision-and-language model trained on over 400M pairs of image and text descriptions collected off the internet. There are several reasons to choose CLIP for this study. First, CLIP one of the few vision models trained on ultra-large scale, unfiltered and varied raw visual data on the internet. Second, the multi-modal nature of CLIP enables use more general ways of adaptation like using natural language prompts for "zero-shot" transfer to new categories techniques previously popular mostly in NLP.
We find that merely tuning the parameters of LayerNorm (Ba et al., 2016) turns out to be a surprisingly effective approach that is competitive or better than all other adaptation methods across the board. The effectiveness of normalization techniques has been observed by prior work for generalization (Perez al., 2018; Lu et al., 2021) well training from scratch (Frankle al., 2020). Inspired by this,
Wehsite https: //s tes. google e. com /vi ad/atttt 1a large- scall modee1 1

---

Under review as a conference paper at ICLR 2022
learnable prompt
(class)
Text Encoder
"dog"
Tx
4-7
11- Ty
Image Encoder
Linear
Prompt Tuning
Layer Norm Tuning
Adapter
Linear Probe
Illustration of multiple methods for adapting CLIP downstream image classification tasks. Fach labeled approach can used separately for fine-tuning the CLIP model. We analyze a variety fine-tuning methods such as prompt tuning by prepending learnable prompt, tuning Layer Normalization parameters, inserting adapter and compacter modules in-between the Transformer layers, and using linear probe of visual features. The CLIP model can be also used for inference downstream task zero-shot manner.
Figure 1:
we further look into different ways combining LayerNorm-tuniin with other adaptation methods that finetune parameters. We devise effective scheme that first finetunes the CLIP model using only LayerNorm tuning and uses as initialization for adapting new parameters. We evaluate our adaptation techniques across 12 downstream tasks spread along two spectra: size of downstream task dataset as well as the similarity of downstream data to the pretraining data. Across both spectra, we find that our two-stage LayerNorm tuning approach most competitive and show its effectiveness for general-purpose purpose adaptation of CLIP to downstream maggecclassification tasks.
To summarize, our paper's contributions are as follows:
We show the effectiveness of LayerNorm-tunn tunntttunngg for adaptation to downstream tasks. We devise simple yet effective scheme to combine LayerNorm-tuning with other methods of finetuning to obtain competitive performance across the board.
We show thorough comparison of different adaptation methods in four scenarios spectra (amount downstream data and its similarity to pretraining data) studied on numerous downstream classification tasks.
two
We believe our findings will encourage more research and put existing research in perspective of what works best while finetuning large-scale vision-language models to downstream tasks.
2 BACKGROUND: VISION-AND-LANGUAA PRETRAINED MODELS
Vision-andd and-language pre-training methods have recently shown promise on diverse tasks across images and text (Radford et al., 2021: Jia et al., 2021). While many such approaches have emerged, focus on CLIP (Contrastive Language-Iman Image Pre-training), large-scale model with strong zero-shot performance on downstream classification tasks (Radford et al., 2021).
Contrastive Language-Image Pre-training (CLIP) CLIP consists of two parallel encoders for processing images and text, whose outputs are then projected into a shared embedding space. text encoder is a Transformer (Vaswani et al., 2017) following the architecture described in Radford et al. (2019), while the image encoder is a Vision Transformer (ViT) with a patch size of 16 (Dosovitskiy et al., 2020). For our experiments, we utilize the open-sourced pretrained CLIP models.
Training Because the image and text features live in the similarity between any embedded image and text description can be computed. CLIP uses these as prediction probabilities for classifying an image with the correct text caption (or vice versa) across batches. Formally, denote and T as the set of image and text features in a single batch. The prediction probability for the äth image and jth caption the batch is given by
embedding space, the cosine
exp (cos (T;, 1.)T)
PTT I.) exp (cos(T), I,)/T)+ Σxtp exp (cos(Ta, 1./T)
2

---

Under review as a conference paper at ICLR 2022
Linear Probe
Prompt Tuning
Compacter
Layer Norm
Adapter
Full Fine-tuning
769 (# classes)
4096
19,360
65,536
301,344
149, 449,620,737
Text Encoder & Image Encoder
Text Encoder
Text Encoder
Image Encoder
MLP
NLP
Layer Norm
Norm
Multt-tewrree Actention
Norm
Mulh heed-bedddd Attertip
Actention
Laper
Layer Norm
prompe
Layer Tuning
Prompt Tuning
Adapter
Linear Probe
Figure 2: Parameter count and architectures of fine-tuning methods. All of the fine-tuning methods we consider tune only small fraction of the total number of parameters and act in different ways the model. LayerNorm- tuning only trains existing Layer Normalization parameters approaches inject additional parameters which act on different parts activations, and output. Prompt tuning prepends Icarnable prompt to the input text embeddings of classes. Adapter modules are composed of linear down- projeccto,,,,,, non-lincarity, and lincar up-projection, and are inserted inside the Transformer layers of the text encoder after the attention block. Lincar probe directly classifies classes from the output of the image encoder.
all Transformer layers. The remaining
model: the input, intermediate
where 7 is learnable temperature parameter. CLIP is trained with contrastive loss accordingly across 400 million pairs of image and text captions collected online (Radford et al., 2021).
Inference For downstream classification task at test time, CLIP first embeds the textual descrip- tions of all classes. These descriptions may range from phrase like photo of <class>>" to heavily engineered embeddings ensembled over 80 different templates (Radford et al., 2021). Each image then classified using the embedded classes as labels and the prediction probabilities described above. Notably, this inference scheme allows CLIP to be transferred zero-shot to any downstream image classification task. Radford et al. (2021) show that zero-shot CLIP is competitive with a fully supervised ResNet (He et al., 2016) baseline on suite of image classification tasks.
3 METHODOLOGY: FINE-TUNING LARGE-SCALE PRETRAINED MODEL
Although zero-shot CLIP performs well on natural images and general object classification datasets, its performance degrades quickly more abstract tasks from oif-distriiution data. Even simple dataset like MNIST (LeCun, 1998), the zero-shot CLIP model (ViT-B/16) we test attains an accuracy of only 55%. Substantial gains can be achieved by fine-tuning the pre-trained model, but many such strategies have emerged across tasks in vision and language and it's unclear which to use diverse downstream settings. For this reason, we provide extensive study of adaptation approaches. Figure illustrates the fine-tuning methods we consider in context of CLIP while Figure shows more detailed information regarding each approach.
We propose general taxonomy of fine-tuning approaches and consider three major classes: (a) methods which only fine-tune existing parameters, (b) methods which freeze existing parameters and add new parameters, and (c) methods which combine (a) and (b). We first consider two methods in (a) which only fine-tune existing parameters.
3.1 FINE-TUNING EXISTING PARAMETERS
Full Model Fine-tuning The simplest approach to fine-tuning is to train all of the model parameters on the downstream task. However, this unstable and doesn't 't scale well to CLIP-size models with hundreds of millions of parameters. Our empirical results show this behavior well.

---

Under review as a conference paper at ICLR 2022
LayerNorm Tuning Instead of full model fine-tuning for large-scale models, we can tune small subset of chosen parameters when the downstream data is searce. In fact, Frankle et al. (2020) show that just tuning Batch Normalization (loffe & Szegedy, 2015) parameters from random initialization can be highly expressive. In a similar vein, we investigate tuning the parameters of Layer Normalization (LayerNorm) layers (Ba et al., 2016). Unlike Batch Normalization, LayerNorm applies per-element normalization across mini-batches. Given mini batch of inputs æ, LayerNorm transforms this
y
2 + +
where the mean and variance are calculated over the normalized dimensions and 7, 8 are learned parameters. Because the image and text encoders in CLIP share the same underlying Transformer architecture, in LayerNorm Tuning, we fine-tune the Layer Normalization parameters 7, across all layers of both encoders. These parameters are 768-dimensional and 212-dimensional for the image and text encoders respectively.
3.2 FINE-TUNING NEW PARAMETERS
An alternative paradigm is to inject new parameters which can more effectively adapt to downstream tasks. These parameters can act various stages of pre-trained model: on the output, input, or intermediate activations.
Linear Probe The classic method of training linear probe on top of frozen features is example of adding new parameters which act on the model output. Given a pre-trained CLIP model, we discard the text encoder, freeze the image encoder, and learn linear layer top of the image features before they 're projected to the shared embedding space. The linear layer maps the penultimate image features to logits from which class predictions are made. While this simple method is popular and inefficceitt for tasks with higher number of classes and fails to leverage any
effective, it's of the language information contained in CLIP.
Prompt Tuning Alternatively, we can consider adding parameters which act on the model input. Such an approach known as prompt tuning has emerged as a parameter-effiiient fine-tuning method in language (Li & Liang, 2021; Lester et al., 2021). A fixed number of continuous vectors (a "prompt") prepended to the model input and optimized throughout training. Similar to concurrent work by Zhou et al. (2021), we apply prompt tuning to image classification with CLIP. For the model input, we embed the raw text of the classes without a template and prepend a continuous prompt of fixed length. During training, the prompt is learned using cross-entropy loss according the prediction probabilities detailed in Section 3.1. Although prompt tuning can be applied in the same way for transformer-hasee visual encoders, we find that only applying for the text encoder produces better and more stable results.
Prompt tuning is parameter-efficient and removes the need for manual prompt engineering e.g. specifying "a photo of a <class>, a type of flower" for a downstream task on flower classification. Ideally, the learned prompts would contain such domain-specific information. However, prompt tuning suffers from high variance during training and is sensitive to initialization.
Adapter and Compacter Networks The above two approaches inject parameters which act either at the end of the network (linear probe) or at the beginning (prompt tuning). A third option is to inject new parameters for the downstream task within the layers of the network itself. This idea has been popularized as an efficient transfer learning method in language (Houlsby et al., 2019). For mransformer-bsee architectures, a common strategy is to insert a block learnable parameters after feed forward layers or the attention mechanism.
Adapter networks insert learnable adapter blocks after the feed forward layers each Transformer layer (Houlsby et al., 2019). Each block follows a bottleneck architecture and is composed of linear down-prowe poojeproocction, non-linearity, and linear up-projection as shown in Figure 2. However, for architectures with many stacked Transformer layers and larger hidden dimensions, adapter modules are parameter-inefficcie
To alleviate this issue, Mahabadi et al. (2021) introduce compacter modules which follow the same architecture but use low-rank parameters and hypercomplex multiplication to improve parameter efficiency. Specifically, if the down-projection layer maps xr Rm Wa b E Rd where
4

---

Under review as a conference paper at ICLR 2022
Low Data & High Similarity
High Data & High Similarity
00
20
75:01
75.24
CLIP Zero Shot Prots-hhttttee Performance
Amount of Training Data
Low Data & Low Similarity
High Data & Low Similarity
$3.00
50
Full-model mentunnnn Zero-shot
Linear Probe Prompt Tuning
Adapter
Compacter
Figure 3: Comparison of fine-tuning methods across different regimes training data and CLIP zero shot performance. Within each quadrant, results are averaged over all corresponding datascts. LayerNorm tuning is the strongest baseline and performs the best in all regimes. All fine-tuning methods generally provide large benefit over zero-shot CLIP.
We Rmnd b Id are learned parameters and < m, compacter modules represent W
Σ A @ (s,t,)
where A; are global weights shared across Transformer layers and 8;, are local, rank-1 weights. We insert Adapter and Compacter modules across the Transformer layers the text encoder.
3.3 COMBINING LAYERNORM-
WITH FINE-TUNING METHODS
While we find that just LayerNorm tuning by itself is strong baseline, additional benefit is that can be combined with any other fine-tuning method given that the underlying model architecture contains Layer Normalization parameters. The parameters of the alternative method can simply be fine-tuned simultancously with the Layer Normalization parameters. For example, Houlsby et al. (2019) combine LayerNorm tuning with Adapter modules their Adapter network for language tasks.
Fine-tuned LayerNorm Initialization We propose additional approach for combining Layer- Norm Tuning with other fine- tuning methods. We first finetune a CLIP model using only LayerNorm tuning. The weights of this model can then be used initialization for any arbitrary subsequent fine-tuning method. this multi-stage process, we effectively distill the fine-tuned LayerNorm model through the LayerNorm parameters to the secondary fine-tuning method.
4 EXPERIMENTS
Setup The goal of this work is to study transfer learning downstream vision tasks. However, the downstream transfer performance depends on two key factors: the amount of training data present as We aim
well as the distribution of that training data relative what the model was pre-trained investigate transfer learning across both these dimensions. To do so, we create 4 different benchmark suites across these factors: low data and high similarity, low data and low similarity. high data and high similarity, and high data and low similarity. Results along these axes are shown in Figure 3.
Datasets We select a diverse set datasets that Radford et al. (2021) use for zero-shot CLIP evaluation: MNIST (LeCunn 1998). EuroSAT (Helber et al., 2019), CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), Flowers (Nilsback & Zisserman, 2008), DTD (Cimpoi al., 2014), and Food101 (Bossard et al., 2014). We then test on 3 distribution shift datasets from the WILDS benchmark (Koh et al., 2021): FMoW (Christie et al., 2018), Camelyon17 (Bandi et al.. 2018), and iWildCam (Beery et al., 2021). We finally benchmark on few-shot adaptation tasks: MinilmageNet (Vinyals et al., 2016) and CUB (Wah et al., 2011).
image classification datasets. We consider a subset of
5

---

Under review as a conference paper at ICLR 2022
Spectrum 1: Amount of Downstream Data We control for the amount of downstream training data by considering k-shot settings where k samples from each class are made available during the fine-tuning training phase. For the low-data regime, described in Radford et al. (2021) and train with 1, 2, 4, 8, and 16-shots, while for the high-data regime, we train with 256 and 512-shots. For each setting, we evaluate our methods the full tests sets and average our results across all corresponding shots and three random seeds.
follow the few-shot evaluation scheme
Spectrum 2: Distribution of Downstream Data We measure the similarity of downstream data to the one that CLIP was pretrained on by measuring the performance of purely transferring CLIP zero- shot. Our 12 datasets cover a range of image classification tasks including fine-grained classification, distribution shift, and few-shot learning, and they encompass diverse domains and varying downstream distributions. We split them into high and low similarity regimes according to zero-shot accuracy using threshold accuracy of 55%. Under this scheme, MNIST, EuroSAT, DTD, FMoW, and iWildCam fall under the low similarity regime while CIFAR1O, CIFAR 100, Flowers, Camelyon17, and Food101 fall under the high similarity regime. For the high data, low similarity regime and high data, high similarity regime, we exclude DTD and Flowers respectively due to a lack of data.
4.1 EFFECTIVENESS OF INTERMEDIATE WEIGHTS
Low Data
Given this characterization 4 downstream regimes, we evaluate the performance of baseline fine-tuning meth- ods: linear probe, prompt tuning, LayerNorm tuning, adapter networks, and compacter networks. For fair comparison and to isolate the effect of each method, we don't t simultaneously tune LayerNorm parameters in any of the other 4 fine-tuning approaches. Additionally, we
HMoll
iWistam
010
EarotAr
Cameijot1?
ANbT
Foodl01
Rower
OFAR1
compare with a sixth baseline: zero-shot CLIP using the
High Data
prompt engineering detailed in Radford et al. (2021). The addition of strong zero-shot baseline allows us to eval- uate how much our fine-tuning methods help across the different settings. Full-model fine-tuning is included as a seventh baseline for complete, fair comparison.
Our results from Figure show that LayerNorm tuning is a simple but highly effective baseline across all four regimes, performs the best in all four regimes including the most difficult quadrant: the low-data, hard zero-shot regime. Across individual datasets, Figure 4 shows that LayerNorm tuning is consistently among the best as well. This points towards the importance of cross-modal interaction when fine-tuning as LayerNorm tuming the only method which trains parameters in both CLIP encoders. More generally, it suggests that transfer performance on downstream vision tasks can benefit from fine-tuning grounded in language.
CIRARE00
Comeljon17
MINET
Exesdr
CRA810
full- Finesturing
Linear Probe
-roopp Tunng
Compacter
-Tem--tont
Figure 4: Accuracy of baseline fine-tuning methods across datasets in the low and high- data regimes. Datasets ordered along the x axis by average performance of fine-tuning methods. Flowers and Food101 datasets are ommited from the high data regime due to lack of data. We observe that LayerNorm tuning is strong haseline across all regimes and datasets and provides substantial gains over zero-shot CLIP.
Furthermore, the strong performance of LayerNorm tuning as well as adapter and compacter networks in the low- data regime suggests that either fine-tuning or injecting parameters among intermediate layers in a network is key for efficient adaptation. Intuitively, acting only on the model input or output like prompt tuning or linear probe has limited expressivity compared to modifying the intermediate layers themselves, particularly in low-data regimes.
4.2 LEVERAGING LAYERNORM TUNING
The results of the previous section demonstrate that just LayerNorm tuning is a competitive baseline to many of the existing fine-tuning methods. We examine how much LayerNorm tuning can benefit existing methods. We consider two ways of incorporating LayerNorm parameters: by simultaneously tuning them or by applying fine-tuned LayerNorm as initialization described in Section 3.2. For

---

Under review as a conference paper at ICLR 2022
Low-Data Regime
High Data Regime
Type LN Tuning
None
Normal
Initialization
None
Normal
Initialization
Lincar Probe
62.54
63.55
66.61
81.93
83.84
84.36
Prompt Tuning
62.82
80.26
83,69
83.17
Adapter
64.82
66.23
66.63
81.53
83.31
83.63
Compacter
68.15
69.69
68.76
81.75
83.61
83.17
Table 1: Effect of combining LayerNorm tuning with fine-tuning methods. Normal refers to LN tuning by simultaneously tuning LayerNorm parameters with the specified fine-tuning method. For all approaches. the addition of LayerNorm tuning in either form provides significant performance boost. Linear probe and prompt tuning receive the largest benefit when combined with any form of LayerNorm tuning.
each of the remaining fine-tuning methods (linear probe, prompt tuning, adapter networks, compacter networks), we compare the bascline method to these two variants.
We average the results over the corresponding datasets in the low and high-data regimes respectively, As Table shows, incorporating any form of LayerNorm tuning increases performance compared to the normal baselines across all methods and regimes. Because the process of incorporating LayerNorm tuning is method -agoosiagnostic, we recommend this as a simple approach to improve transfer performance.
Across specific fine-tuning methods, linear probe receives the largest benefits from applying Lay- enNorm tuning first before finetuning the linear probe. We posit that this is the case as finetuning LayerNorm first effectively distills the information from the pre-trained text encoder to the Lay- erNorm parameters doesn't 't have access to and provides further evidence towards the benefit of leveraging both vision and text on downstream, unimodal tasks.
the vision side. This is privileged information that a classical linear probe
Finally, we observe that for adapter and compacter networks, using fine-tuned LayerNorm as initial- ization performs equivalently slightly worse compared to simultaneously tuning the LayerNorm parameters across both data regimes. This suggests that LayerNorm tuning and inserting adapter and compacter modules may serve similar roles fine-tuning mechanisms.
4.3 GENERAL RECIPES
From the previous two sections, we' ve seen that just LayerNorm tuning is a surprisingly effective baseline and applying it on top of other fine-tuning methods provides performance gains. We now investigate the question of what the best performing combination of fine-tuning methods across the four regimes are. We follow the experimental setup in Section 4 but compare the performance of our fine-tuning methods when combined with LayerNorm tuning first.
Figure 7 indicates that there is no clear best baseline across all four regimes but linear probe and compacter have the strongest performance when combined with fine-tuned LayerNorm as initialization. Across regimes, we observe similar results to Figure 3. In both low-data regimes, we find that prompt tuning generally perform worse than the remaining fine-tuning methods while compacter performs better. Across both high-data regimes, linear probe performs quite strongly. Notably, these trends hold despite the addition of our initialization scheme. We recommend these, as well as just Layer Norm tuning, as general recipes when selecting fine-tuning method to use downstream task depending on the setting.
4.4 DISTRIBUTION SHIFT
While we test on the WILDS benchmark in our analysis above, benchmark our methods to evaluate how robust they are to distribution shift in downstream tasks. We focus on domain generalization where the train and test distributions come from disjoint domains (Koh et al., 2021). For example, the Camelyon17 dataset contains training and testing images of tumor tissues coming from distinct hospitals. We compare our results to those on the public leaderboard which contains techniques for domain generalization while our models simply trained according empirical risk minimiza- tion objective. For performance metrics, we use average accuracy for FMoW and Camelyon17, and Macro F1 for iWildCam.
7

---

Under review as a conference paper at ICLR 2022
Low Data & Similarity
High Data High Similarity
$4.07
Zero-Shott Shot Performance
20
91:90
5o.07
90.00
24
60
Amount of Training Data +
Low Data Low Similarity
High Low Similarity
57
CLIP Zero
70
33:33
53.40
14
19.10
Linear Probe
Prompt Tuning
Adapter
Compa
Figure 5: Comparison of fine-tuning methods initialized with fine-tuned LayerNorm across different regimes of training data and CLIP zero shot performance. Although there is no clear best combination across all four regimes, we recommend general recipes of using Linear Probe with fine-tuned initialization in the high data regimes and Compacter with fine-tuned initialization the low data regimes.
As shown in Table 2, our methods don't quite match the best reported results, but close the gap from zero-shot CL.P signifi- cantly. We are competitive with other do- main generalization specific approaches on the leaderboard despite the simplicity of our fine-tuning methods. Our results show that our fine-tuning methods are effective in adapting CLIP to difficult downstream tasks and relatively robust to factors in distribution shift.
FMoW
Camelyon17
iWildCam
Zero-shot LayerNorm Tuning Lincar Probe LN as Initialization
19.71
67.46
3.73
47.59
90.47
18.52
48.98
89.98
23.80
Best leaderboard result
55.5
91.6
38.5
Table 2:
Results on the WILDS benchmark. We benchmark our fine-tuning methods three image classification datasets and find that our fine-tuning methods improve upon zero-shot CLIP performance significantly.
4.5 FEW-SHOT LEARNING
Our previous experiments show results in the low-data regime, but we also apply our fine-tuning methods with CLIP to more standard meta fho- tasks evaluate class generalization. In partieular, we consider the setting where given a labeled dataset of base classes, the objective is to identify novel classes only using few samples. Formally, time, in a single episode of a N-way K K-shot few-shot task, we are given a support set with test classes and K samples per class as well as a query set with N test classes and samples per class. We measure the accuracy of classifying the N x Q query images into N classes (Chen et al., 2020).
are given large dataset of B base classes. At test
We adopt our fine-tuning methods this setting in stages. First, we pre-train on the full dataset of the base classes in the same way we fine-tune to any of the prior downstream tasks. Then, we use the image encoder of the fine-tuned model within the Classifier Baseline proposed by Chen et al. (2020). For given few-shot task, we compute representative for each of the N classes by averaging the embeddings of the support examples. We classify each of the N Q query-set examples according to the cosine similarity of their embeddings to the representatives.
Using this approach, we test our fine-tuning methods on MinilmageNet and CUB in the 5-way 1-shot and 5-way 5-shot settings averaged over 600 episodes. Table 4 shows that simply applying zero-shot CL.IP in this way does remarkably well, outperforming the best reported results on the public leaderboards for these tasks.
All our fine-tuning methods provide further improvement with Linear Probe and LayerNorm tuning reaching 6% higher accuracy than the current SOTA on 5-way 1-shot MinilmageNet as shown in Tables and 4. The strong performance of linear probe in this setting is expected as the Classifier Baseline doesn't 't utilize the CLIP text encoder at all. Of the fine-tur fine-tuning methods evaluate. Linear Probe with LayerNorm tuning is the only method which trains parameters solely in the image encoder, so its visual representations transfer the best.
8

---

Under review as a conference paper at ICLR 2022
Mini-ImageNer (1-shot)
Mini-ImigeNNt(--- (5-shot)
CUB (1-sbo)
CUB (5-shot)
Zero-shot
86.30
96.56
Zero-shut
LayerNorm Prompt Tuning
89,22 24
96.46
LayerNomm
98:
97.05
Adapeer + LN as Initialization
91.73
98.20
Adapter LN
91-17
97.39
Prompe Tuning- LN as Intiaaization
92.21
98.20
Lincar Prube+ + LN Best leaderbourd resalt
92.08
97.94
Linear Probe + LN as Initializaiion
93.73
98.50
82.99
91.50
Best leaderhaard result
94.73
96.28
Table 3: Few-shot classification accuracy on Mini- ImageNet. Just zero-shot CLIP performs strongly few-shot adaptation, and fine-tuning methods pro- vide additional performance gains. combination of Linear Probe with LayerNorm tuning performs the best, excceding the current reported SOTA on Mini- MingeemmaeNet.
Table 4: Few-shot classification accuracy on CUB.
Similar to our results on Mini ImageNe,,,,,, zero-shot CL.PP performs strongly hut fine-tuning with LayerNorm on top can produce significant im- provements in accuracy. Linear Probe combined with LayerNorm performs the best again, exceeding the current SOTA in the 5-shot setting.
that
5 RELATED WORK
Large-Scale Transformer-Base Models: Unsupervised pre-training for language typically takes advantage of the sequential nature of text through a self-supervised prediction task. Initially, recurrent neural networks (RNNs) (Hochreiter & Schmidhuber, 1997) were the predominant deep learning architectures for unsupervised language learning and were particularly successful for machine translation (Sutskever al., 2014). In the context of language modeling, RNNs were superseded by attention architectures (Bahdanau al., 2015) and specifically masked self-attention Transformer architectures (Vaswani et al., 2017: Devlin et al., 2018; Radford et al., 2019). Over the last few years, Transformers have produced impressive generalization results through unsupervised pre-training of increasingly larger models on larger datasets (Brown et al., 2020).
Multimodal Learning: Multimodal models (Ngiam et al., 2011) are trained through tasks that leverage multiple data modalities simultancously, such as vision and language. Examples include text to image synthesis (Reed al., 2016) and text descriptions of visual inputs (Krishna al., 2017). Recently, multi-modal architectures have been combined with large-scale unsupervised pre-training achieve impressive text-to-image generation (Ramesh al., 2021) as well learning joint image and language embeddings (Radford et al., 2021). In particular, this work investigates how do adapt a pre-trained CLIP model, which uses noise contrastive estimation and transformers to maximize similarity between images and their text captions, to downstream tasks.
Finetuning Pre-trained Models: With the emergence of large-scale pre-trac pre-trained vision and language models, it's becoming increasingly important to adapt such models efficiently to downstream tasks. For transfer learning in vision, the most common approach is to use linear probe on top of pretrained image features while for language, a variety of fine-tuning approaches have emerged including variations of prompt tuning (Liu et al., 2021; Li & Liang, 2021; Lester et al., 2021; Zhou et al., 2021), adapter and compacter networks (Houlsby et al., 2019; Mahabadi et al., 2021), and multimodal approaches applicable to vision and language (Tsimpoukelli et al., 2021; Shen et al., 2021). We hope our analysis provides insight into some of these methods.
6 DISCUSSION: GENERAL GUIDELINES
Our work analyzes relevant questions in transfer learning of large-scale pretrained vision-and- language model to several downstream classification tasks. We evaluate 5 different fine-tuning baseline methods across 12 total image classification datasets and find that just tuning Layer Normal- ization parameters is a surprisingly effective, parameter-efficient baseline, and propose an effective approach to combine it with other finetuning methods.
We analyze our best pest-performin fine-tuning methods over different settings to find general guidelines. For all of our methods, we combine them with fine-tuned LayerNorm as initialization. For the low-data regime, we recommend using fine-tuning approaches which inject or modify intermediate parameters like LayerNorm tuning, Adapter networks, and Compacter networks. For the high-data regime, we recommend using linear probe or prompt tuning. For generic settings, recommend simply LayerNorm tuning. Code to reproduce the experiments will be made available. We hope that this work will lead to a broader future research in efficient adaptation of large-scale pre-trained models, not just limited to vision-ana and--nnd--an ann-lang models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 1,4 4
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL www.a..ivvvo......... abs////////// 1409. 1409.0473. 0473. 9
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550-555 -5000 2018.
Sara Beery, Arushi Agarwal. Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint 0rXiv:2105.03494, 2021. 5
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446 461. Springer, 2014. 5
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020. 1,9
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseling for few-shot learning. arXiv preprint arXiv:2003.cm 043990, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018. 5
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014. 5
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018. 1,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint UrXiv:2010.11922 2020. 2
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in enns. arXiv preprint arXiv:2003.00/55 2020. 1,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.3
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 183- 11 1997. ISSN 0899-7667. doi: 10.1 1622/ec/n wwteec...1999.99 1997.9.8.1735. URL https:
11 1162//nncoo 1 199999..8 9. 1.8..7.0 1735. 9
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019. 1, 4, 5,
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR. 2015. 4
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint urXiv:2102.05918, 2021. 1,2
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021. 5,
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning
events in videos. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 706-715, IEEE Computer Society, 2017. doi: 10. 10.110//CCV22001..88 URL https:tt////mm /doi. (oooi.ooe org/ /10rg//0..0//. 110. 1109/ R.L09/ICCC ICCVV 2017.22022 83.17.333. 9
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
5
Yann LeCun. The mnist database of handwritten digits. htt:://annn. lecun. com/exdb/mnist// 1998. 3,
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprini arXiv:21044 104.08691, 2021. 1,4,
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 10000000,,,,. 2021. 1,4,9
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv: 13586, 2021. 9
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.0522 2021.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2 2021. 1,4,9
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In JCML, 2011.9 9
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008. 5
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 1, 1,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. 1, 2, 3, 5, 6, 9, 13
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831 PMLR, 2021. URL http: ptwc//doceeiicoooooocoooco.ococc ngs ml r. ww..prss/vlll 399 ramesh21 html. 9

---

Under review as a conference paper at ICLR 2022
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1060-1069. Jorg. 2016. URL http::htpp///oto. v48/reed 6 html. 9
ngs. ml press/
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit assion-and-lanuuaa tasks? arXiv preprint arXiv:2/07.06338, 2021. 9
lya Sutskever, Oriol Vinyals. and Quoe V. Le. Sequence sequence learning with neural networks. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances Neural Information Processing Systems 27: Annual Conference Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104-3112, 2014. URL httpsss/prroooeeeiigg www.iingnnn..u.sssseeeemm neurips. oc/paper/201 h/hash/ al
Neural
Abstract Atsttatcccctaaee-araaaaaa.....t html. 9
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. arXiv preprint arXiv:2 13884, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 2,9
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01134, 2021. 4, 9
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.I DATASET INFORMATION
Dataset
Classes
Train
Validation
Test
MNIST
10
58,000
2,000
10,000
EuroSAT
10
15,000
2,000
10,000
CIFAR-10
10
48,000
2,000
10,000
CIFAR-100
100
48,000
2,000
10,000
DTD
47
1,880
1,880
1,880
Flowers (02
102
1,632
408
6,149
Food-101
101
60,600
15,150
25,250
FMoW
62
76,863
19,915
22,108
iWildCam
182
129,809 809
14,961
42,791
Camelyon17
302,436
34,904
85,054
CUB
200
5,891
2,932
2,965
MinilmageNet
100
38,400
9,600
12,000
Table 5: Dataset information.
Table shows information about the number of classes and the train, validation, and test split for each downstream dataset evaluate. For MinilmageNett the 100 classes are split into 64, 16, and 20 classes for training, validation, and testing respectively. For CUB, the 200 classes are split into 100, 50, and 50 classes for training, validation, and testing respectively.
A.2 CLIP IMPLEMENTATION
We implement our adaptation methods on top of the PyTorch implementation provided by the authors of CLIP htsps: / gi thub. com/ openai/ // CCIPP For the image encoder, Transformer with patch size 16 (ViT-B/16) and initialize all of our models with the corresponding pre-trained weights. For zero-shot CLIP evaluation, we perform the prompt engineering described in Radford et al. (2021) and ensemble the features from the text encoder across 80 templates" ?
Vision
A.3 TRAINING DETAILS
Input Processing For the raw images, we preprocess them with normalization. resizing, and random cropping to size 224 by 224. For the textual descriptions, we encode them using byte pair encoding like Radford et (2021).
Hyperparameters of fine-tuning methods For prompt tuning, we use prompt of 512- dimensional vectors. For adapter networks, we use bottleneck dimension size 24. For compacter networks, the same bottleneck dimension and have 4 global weights of dimension 4 by 4.
Training Procedure For fine-tuning methods, we train with a cross-entropy loss. We use the AdamW optimizer with initial learning rate of 5e-4 and weight decay of 0.02 well cosine annealing scheduler. We evaluate our model on the validation set after every epoch and keep the best-performing checkpoint. In the low-data regime, set of size # of shots except for Flowers where we use min # of shots, 4). In the high-data regime, we train for 50 epochs and use a validation set of size (# of shots) /10. 100. Due data constraints, for CIFAR- 100, we train with 256 and 450-shots (instead of 256 and 5 512-shots) in the high-data regime. For CUB and MinilmageNet, we use the same hyperparameters and train for 100 epochs on the base classes.
train for 100 epochs and use validation
Pttpps: //gi thub. http:///wm/.c.///a////////////ts neeri for_lmageNet, ipynb
13


Reviews:
[{"review_id": "kSGCGEQQFP", "reviewer": "Reviewer_Vfao", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The LayerNorm-tuning for adapting CLIP to downstream tasks is shown to be effective.\n\n2. A simple yet effective scheme that combines LayerNorm-tuning with other fine-tuning methods is proposed to obtain competitive performance across the board.\n\n3. A thorough comparison of different adaptation methods is provided in four scenarios across two spectra. \n\nWeaknesses:\n\n1. The paper title “How to Adapt Your Large-Scale Vision-and-Language Model” is a bit of over-claimed. It should be toned down to “How to Adapt CLIP to Downstream Image Classification Tasks”.\n\n2. The technical novelty of this paper seems very limited. The three methods for fine-tuning new parameters have been extensively studied in existing works. When coming to adapting CLIP, there should be something different. That is, novel fine-tuning methods are needed in adapting CLIP. \n\n3. The experimental evaluation is far from sufficient. (1) For “LayerNorm-Tuning”, the parameters of all LayerNorm layers are set to be learnable. I wonder if the results could be better when only the last K LayerNorm layers are updated (e.g., K=5). (2) For “Full Model Fine-tuning”, it should also be evaluated in the experiments by only updating the parameters of the last layers of the model. (3) I wonder if the proposed method is effective in other downstream tasks other than image classification. \n", "summary_of_the_paper": "This paper has extensively studied how to adapt the large-scale pre-trained vision-language model CLIP for downstream tasks. Several fine-tuning methods are analyzed across a diverse set of image classification tasks along two spectra. Further, a simple yet effective strategy that combines LayerNorm-tuning with general fine-tuning methods is proposed to improve their performance and benchmark them on few-shot classification tasks. ", "summary_of_the_review": "This paper had been well written if entitled “How to Adapt CLIP to Downstream Image Classification Tasks”. However, considering the insufficient experimental evaluation and limited technical novelty, I could only give it a score of 5. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fl9uIW9QTM5", "reviewer": "Reviewer_aQ8A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPro:\n\nIt is good to know that LayerNorm tuning is quite effective for CLIP and combining it with other approaches gives even better performance.\n\n\nCon:\n\n1. Limited novelty.\n\nThe methods used in this paper are existing methods and the authors do a simple combination and benchmark them with CLIP on various datasets.\n\nThe conclusion I could draw is 1) layer norm tuning works well and 2) one can combine it with other methods (essentially have more parameters to tune) to achieve better results. I am not sure if these conclusions are that exciting and sufficient for an accepted paper.\n\n\n2. Missing full-model fine-tuning.\n\nThe authors argue full-model fine-tuning as \"inefficient\" and exclude it from comparison. However, I think full-model fine-tuning results are important to include.\n\na. It is important to see where the upper bound is and results of full-model fine-tuning would put the current results in perspective. \n\nb. The authors need to be more clear about what \"inefficient\" means. Does it mean fine-tuning time cost? If it refers to the fine-tuning time cost, it is affected by many factors and is not just dependent on parameter count. And under the setting in this paper, I am not sure how \"inefficient\" full fine-tuning is, compared to the shown approaches. \n   For example, for prompt tuning, even if most parameters are not trainable, gradients still need to be calculated for back-propagation to the input. Thus, the forward/backward computational cost should be the same for prompt tuning and full model fine-tuning. This is similar to layer-norm tuning (depending on the location of the first layer norm layer). \n   The efficiency of prompt tuning over full-model fine-tuning mostly comes from 1) less optimizer overhead and 2) less communication cost of parameters during multi-GPU training. However, CLIP is not a significantly large model by today's standards (the smallest CLIP is ResNet 50) and on the datasets we are adapting to, we often do not need many GPUs to fine-tune the model. I have to suspect that the bottleneck is not on either the optimizer overhead or communication cost when we fine-tune CLIP on those small datasets (please correct me if my assumption is wrong). Thus I am curious to see the actual speedup prompt tuning holds over full fine-tuning.\n\n\n", "summary_of_the_paper": "The paper investigates a range of techniques for adapting CLIP to different tasks. They find that only tuning the LayerNorm is effective and further combining it with other adapting approaches delivers better performance. \n", "summary_of_the_review": "The paper presents an empirical study of how to fine-tune CLIP for downstream tasks. \n\nMy main concerns are: 1) from my subjective view, I am not sure the conclusions are significant enough; 2) full fine-tuning results are missing.\n\nI recommend adding the full fine-tuning results for a full picture and being more specific about the reason to exclude full fine-tuning. It would be good to provide a compelling reason to favor these methods over full fine-tuning in practice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dWoR-7KWBiD", "reviewer": "Reviewer_Dgds", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths\n    - The proposed LN tuning is simple and seems quite strong compared to other more complex methods.\n\n- Weaknesses:\n    - While the authors claim that the proposed method is useful for vision and language models, only a single model (CLIP) is used for these studies. Though the method design is almost model-agnostic, it is not clear whether this approach generalizes to other models.\n    - The tasks studied in this work are all image classification tasks, there are no tasks that are normally recognized as vision-and-language tasks, such as text-image retrieval or image captioning [1]. If the authors' intention is to use this method for image classification tasks, it would be better to change the title a bit as \"How to adapt your large-scale pre-trained models for visual representations learning\", instead of the current confusing one of \"vision-and-language model\".\n    - In the high-data regime, it seems also feasible to finetune all the model parameters, how does this result compare to the considered baselines in this work?\n    - The best performance is often achieved by first finetuning the LN parameters, then using another approach like Adapter for a 2nd stage finetuning. This two-stage finetuning process could be cumbersome.\n    - After LN finetuning, it seems that additionally finetune with many of the other methods actually hurts performance. For example, under low data & high similarity setting, with only LN finetuning, the performance is 83.74 (see Fig. 3, top-left), while the performance after further finetuned with other methods generally falls (see Fig. 4, top-left), only Compactor shows a minimal gain with a performance of 84.07. This conclusion seems also applicable to other settings in these two figures. This seems to suggest that we should probably only use LN.\n\n- Miscs:\n    - Some of the references are not linked, e.g., the 3rd paragraph in Sec 4.5, \"Table 4\" is not hyper-linked\n    - \"Normal\" setting in Table 1 is not clearly explained. I made an educated guess as \"simultaneously tuning them\" when evaluating the work.\n\n[1] Chen et al. Microsoft coco captions: Data collection and evaluation server. *arXiv 2015*", "summary_of_the_paper": "This paper proposed a new method (LayerNorm tunning) for finetuning a pre-trained vision-and-language model. While simple, it is shown this method works competitively with other methods (e.g., prompt tuning) in various settings. For the low data & high similarity setting (Fig 3), it shows the best performance across the methods. The authors further claimed LN could be used to boost the performance of general finetuning methods like linear probe or prompt tuning. However, this seems not useful as the performance typically drops when compared to the LN baseline. In addition, the proposed methods are only evaluated in a single model, it is not clear whether they are generalizable.", "summary_of_the_review": "The paper proposed a useful technique LN tunning for fine-tuning the pre-training CLIP model for downstream image classification tasks. However, the generalizability of this approach is not demonstrated, and the proposed recipe of combing LN tunning with other fine-tuning methods seems also not useful as it typically shows even lower performance than only using LN tuning. Overall, this paper does not provide enough support for it to be published at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VeEMKGCZ_Pu", "reviewer": "Reviewer_GtTP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* I like the motivation of the paper -- the vision community broadly uses linear probe as the standard protocol for transfer learning, yet the prominence of prompt learning in NLP clearly shows that there can be better methods to transfer knowledge of large-scale pre-trained models in the vision domain. \n\nWeaknesses\n* In Figure 3-4, it seems that LayerNorm performs better with an extremely small margin, which could easily be reversed with initialization or hyperparameter search for each fine-tuning method. This makes me question why the paper is particularly focused on analyzing LayerNorm in the second half of the paper. Also, combining LayerNorm provides minimal improvement. \n* Instead of averaging and phrasing as \"low data\" vs. \"high data\" in the paper, it would be more accurate to plot the actual 1-512 shots on the graph. Averaging it makes it hard to grasp the correct trend of each fine-tuning method over the dataset scale. \n* Furthermore, it seems extremely arbitrary to divide the distribution of downstream data as \"high and low similarity\" with an arbitrary threshold of 55%. \n", "summary_of_the_paper": "The goal of the paper is to investigate how to efficiently adapt large-scale pre-trained vision-language models (e.g. CLIP) to downstream tasks. Their paper is based on the observation that while the vision community dominantly uses linear probe as the standard protocol, other approaches such as prompt learning are utilized in language. They compare and analyze several fine-tuning methods -- linear probe, prompt tuning, adapter and compacter networks -- across 12 downstream classification tasks. Focusing on LayerNorm, they further demonstrate combining LayerNorm tuning with existing fine-tuning methods to improve performance. \n", "summary_of_the_review": "Overall, this paper seems to have limited contributions for adapting pre-trained vision-language models to downstream tasks. The depth of the analyses is limited and does not provide interesting and novel findings to the community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates several fine-tuning methods for adapting the pretrained vison-language model CLIP to different downstream tasks.  They found that LayerNorm (Ba et al., 2016) is a rather effective and competitive approach and investigated different ways of combining LayerNorm-tuning with other adaptation methods. Reviewers generally agree that the technical novelty (combination of existing techniques) is limited and contribution is marginally significant. While some empirical results look interesting, the overall contribution is incremental. Overall, the paper has done a good job of thorough empirical evaluations, but the overall technical novelty and new empirical findings are not significant enough for publication at this conference.
Decision: Reject