How to Adapt Your Large-Scale Vision-and-Language Model

1 INTRODUCTION . Large-scale deep network models pretrained on ultra large-scale data on the internet , whether text or images , have shown impressive performance recently ( Radford et al. , 2019 ; 2021 ; Brown et al. , 2020 ; Devlin et al. , 2018 ; Jia et al. , 2021 ) . Training such models with billions of parameters on a large internet scale data is an expensive and time consuming process often costing over millions of dollars . Hence , replicating such models is not only difficult but also undesirable for every downstream task . Fortunately , the information gathered by these large-scale models using raw internet data seems to transfer well to several downstream tasks with little to no finetuning at all using natural language as a way for zero-shot evaluation ( Brown et al. , 2020 ; Radford et al. , 2021 ) . While zero-shot transfer performs well , it is generally better to adapt the model itself if there are any labeled examples available for the downstream task . Traditionally , the go-to strategy in the computer vision community has either been to finetune the whole network or an additional MLP layer at the end . With the use of raw language , adaptation techniques such as prompt tuning have surfaced ( Li & Liang , 2021 ; Lester et al. , 2021 ) . Alternative methods include new parameters in between the network instead of adding a layer at the end ( Houlsby et al. , 2019 ; Mahabadi et al. , 2021 ) . However , it remains unclear as to which approach is preferred under which scenarios . We ask what are the general guidelines one should adopt while finetuning a large-scale pretrained model on downstream datasets . To scope this question , we choose CLIP ( Radford et al. , 2021 ) as the base pretrained model and adapt it to several downstream problems . CLIP is a vision-and-language model trained on over 400M pairs of image and text descriptions collected off the internet . There are several reasons to choose CLIP for this study . First , CLIP is one of the few vision models trained on ultra-large scale , unfiltered and varied raw visual data on the internet . Second , the multi-modal nature of CLIP enables use of more general ways of adaptation like using natural language prompts for “ zero-shot ” transfer to new categories – techniques previously popular mostly in NLP . We find that merely tuning the parameters of LayerNorm ( Ba et al. , 2016 ) turns out to be a surprisingly effective approach that is competitive or better than all other adaptation methods across the board . The effectiveness of normalization techniques has been observed by prior work for generalization ( Perez et al. , 2018 ; Lu et al. , 2021 ) as well as training from scratch ( Frankle et al. , 2020 ) . Inspired by this , 1Website at https : //sites.google.com/view/adapt-large-scale-models we further look into different ways of combining LayerNorm-tuning with other adaptation methods that finetune new parameters . We devise an effective scheme that first finetunes the CLIP model using only LayerNorm tuning and uses it as initialization for adapting new parameters . We evaluate our adaptation techniques across 12 downstream tasks spread along two spectra : size of downstream task dataset as well as the similarity of downstream data to the pretraining data . Across both spectra , we find that our two-stage LayerNorm tuning approach is most competitive and show its effectiveness for general-purpose adaptation of CLIP to downstream image-classification tasks . To summarize , our paper ’ s contributions are as follows : • We show the effectiveness of LayerNorm-tuning for adaptation to downstream tasks . • We devise a simple yet effective scheme to combine LayerNorm-tuning with other methods of finetuning to obtain competitive performance across the board . • We show a thorough comparison of different adaptation methods in four scenarios across two spectra ( amount of downstream data and its similarity to pretraining data ) studied on numerous downstream classification tasks . We believe our findings will encourage more research and put existing research in perspective of what works best while finetuning large-scale vision-language models to downstream tasks . 2 BACKGROUND : VISION-AND-LANGUAGE PRETRAINED MODELS . Vision-and-language pre-training methods have recently shown promise on diverse tasks across images and text ( Radford et al. , 2021 ; Jia et al. , 2021 ) . While many such approaches have emerged , we focus on CLIP ( Contrastive Language-Image Pre-training ) , a large-scale model with strong zero-shot performance on downstream classification tasks ( Radford et al. , 2021 ) . Contrastive Language-Image Pre-training ( CLIP ) CLIP consists of two parallel encoders for processing images and text , whose outputs are then projected into a shared embedding space . The text encoder is a Transformer ( Vaswani et al. , 2017 ) following the architecture described in Radford et al . ( 2019 ) , while the image encoder is a Vision Transformer ( ViT ) with a patch size of 16 ( Dosovitskiy et al. , 2020 ) . For our experiments , we utilize the open-sourced pretrained CLIP models . Training Because the image and text features live in the same embedding space , the cosine similarity between any embedded image and text description can be computed . CLIP uses these as prediction probabilities for classifying an image with the correct text caption ( or vice versa ) across batches . Formally , denote I and T as the set of image and text features in a single batch . The prediction probability for the ith image and jth caption in the batch is given by p ( Tj | Ii ) = exp ( cos ( Tj , Ii ) /⌧ ) exp ( cos ( Tj , Ii ) /⌧ ) + P k 6=j exp ( cos ( Tk , Ii ) /⌧ ) where ⌧ is a learnable temperature parameter . CLIP is trained with a contrastive loss accordingly across 400 million pairs of image and text captions collected online ( Radford et al. , 2021 ) . Inference For a downstream classification task at test time , CLIP first embeds the textual descriptions of all classes . These descriptions may range from a phrase like “ a photo of a < class > ” to heavily engineered embeddings ensembled over 80 different templates ( Radford et al. , 2021 ) . Each image is then classified using the embedded classes as labels and the prediction probabilities described above . Notably , this inference scheme allows CLIP to be transferred zero-shot to any downstream image classification task . Radford et al . ( 2021 ) show that zero-shot CLIP is competitive with a fully supervised ResNet ( He et al. , 2016 ) baseline on a suite of image classification tasks . 3 METHODOLOGY : FINE-TUNING LARGE-SCALE PRETRAINED MODEL . Although zero-shot CLIP performs well on natural images and general object classification datasets , its performance degrades quickly on more abstract tasks from out-of-distribution data . Even on a simple dataset like MNIST ( LeCun , 1998 ) , the zero-shot CLIP model ( ViT-B/16 ) we test attains an accuracy of only 55 % . Substantial gains can be achieved by fine-tuning the pre-trained model , but many such strategies have emerged across tasks in vision and language and it ’ s unclear which to use on diverse downstream settings . For this reason , we provide an extensive study of adaptation approaches . Figure 1 illustrates the fine-tuning methods we consider in context of CLIP while Figure 2 shows more detailed information regarding each approach . We propose a general taxonomy of fine-tuning approaches and consider three major classes : ( a ) methods which only fine-tune existing parameters , ( b ) methods which freeze existing parameters and add new parameters , and ( c ) methods which combine ( a ) and ( b ) . We first consider two methods in ( a ) which only fine-tune existing parameters . 3.1 FINE-TUNING EXISTING PARAMETERS . Full Model Fine-tuning The simplest approach to fine-tuning is to train all of the model parameters on the downstream task . However , this is unstable and doesn ’ t scale well to CLIP-size models with hundreds of millions of parameters . Our empirical results show this behavior as well . LayerNorm Tuning Instead of full model fine-tuning for large-scale models , we can tune a small subset of chosen parameters when the downstream data is scarce . In fact , Frankle et al . ( 2020 ) show that just tuning Batch Normalization ( Ioffe & Szegedy , 2015 ) parameters from a random initialization can be highly expressive . In a similar vein , we investigate tuning the parameters of Layer Normalization ( LayerNorm ) layers ( Ba et al. , 2016 ) . Unlike Batch Normalization , LayerNorm applies per-element normalization across mini-batches . Given a mini batch of inputs x , LayerNorm transforms this as y = x E [ x ] p Var [ x ] + ✏ · + where the mean and variance are calculated over the normalized dimensions and , are learned parameters . Because the image and text encoders in CLIP share the same underlying Transformer architecture , in LayerNorm Tuning , we fine-tune the Layer Normalization parameters , across all layers of both encoders . These parameters are 768-dimensional and 512-dimensional for the image and text encoders respectively . 3.2 FINE-TUNING NEW PARAMETERS . An alternative paradigm is to inject new parameters which can more effectively adapt to downstream tasks . These new parameters can act at various stages of a pre-trained model : on the output , input , or intermediate activations . Linear Probe The classic method of training a linear probe on top of frozen features is an example of adding new parameters which act on the model output . Given a pre-trained CLIP model , we discard the text encoder , freeze the image encoder , and learn a linear layer on top of the image features before they ’ re projected to the shared embedding space . The linear layer maps the penultimate image features to logits from which class predictions are made . While this simple method is popular and effective , it ’ s parameter-inefficient for tasks with higher number of classes and fails to leverage any of the language information contained in CLIP . Prompt Tuning Alternatively , we can consider adding parameters which act on the model input . Such an approach known as prompt tuning has emerged as a parameter-efficient fine-tuning method in language ( Li & Liang , 2021 ; Lester et al. , 2021 ) . A fixed number of continuous vectors ( a “ prompt ” ) is prepended to the model input and optimized throughout training . Similar to concurrent work by Zhou et al . ( 2021 ) , we apply prompt tuning to image classification with CLIP . For the model input , we embed the raw text of the classes without a template and prepend a continuous prompt of fixed length . During training , the prompt is learned using a cross-entropy loss according to the prediction probabilities detailed in Section 3.1 . Although prompt tuning can be applied in the same way for transformer-based visual encoders , we find that only applying it for the text encoder produces better and more stable results . Prompt tuning is parameter-efficient and removes the need for manual prompt engineering e.g . specifying “ a photo of a < class > , a type of flower ” for a downstream task on flower classification . Ideally , the learned prompts would contain such domain-specific information . However , prompt tuning suffers from high variance during training and is sensitive to initialization . Adapter and Compacter Networks The above two approaches inject parameters which act either at the end of the network ( linear probe ) or at the beginning ( prompt tuning ) . A third option is to inject new parameters for the downstream task within the layers of the network itself . This idea has been popularized as an efficient transfer learning method in language ( Houlsby et al. , 2019 ) . For Transformer-based architectures , a common strategy is to insert a block of learnable parameters after feed forward layers or the attention mechanism . Adapter networks insert learnable adapter blocks after the feed forward layers in each Transformer layer ( Houlsby et al. , 2019 ) . Each block follows a bottleneck architecture and is composed of a linear down-projection , non-linearity , and linear up-projection as shown in Figure 2 . However , for architectures with many stacked Transformer layers and larger hidden dimensions , adapter modules are parameter-inefficient . To alleviate this issue , Mahabadi et al . ( 2021 ) introduce compacter modules which follow the same architecture but use low-rank parameters and hypercomplex multiplication to improve parameter efficiency . Specifically , if the down-projection layer maps x 2 Rm ! Wx + b 2 Rd where W 2 Rm⇥d , b 2 Rd are learned parameters and d ⌧ m , compacter modules represent W as W = nX i=1 Ai ⌦ ( sitTi ) where Ai are global weights shared across Transformer layers and si , ti are local , rank-1 weights . We insert Adapter and Compacter modules across the Transformer layers in the text encoder .

Summary: This paper has extensively studied how to adapt the large-scale pre-trained vision-language model CLIP for downstream tasks. Several fine-tuning methods are analyzed across a diverse set of image classification tasks along two spectra. Further, a simple yet effective strategy that combines LayerNorm-tuning with general fine-tuning methods is proposed to improve their performance and benchmark them on few-shot classification tasks.