How to Adapt Your Large-Scale Vision-and-Language Model

1 INTRODUCTION . Large-scale deep network models pretrained on ultra large-scale data on the internet , whether text or images , have shown impressive performance recently ( Radford et al. , 2019 ; 2021 ; Brown et al. , 2020 ; Devlin et al. , 2018 ; Jia et al. , 2021 ) . Training such models with billions of parameters on a large internet scale data is an expensive and time consuming process often costing over millions of dollars . Hence , replicating such models is not only difficult but also undesirable for every downstream task . Fortunately , the information gathered by these large-scale models using raw internet data seems to transfer well to several downstream tasks with little to no finetuning at all using natural language as a way for zero-shot evaluation ( Brown et al. , 2020 ; Radford et al. , 2021 ) . While zero-shot transfer performs well , it is generally better to adapt the model itself if there are any labeled examples available for the downstream task . Traditionally , the go-to strategy in the computer vision community has either been to finetune the whole network or an additional MLP layer at the end . With the use of raw language , adaptation techniques such as prompt tuning have surfaced ( Li & Liang , 2021 ; Lester et al. , 2021 ) . Alternative methods include new parameters in between the network instead of adding a layer at the end ( Houlsby et al. , 2019 ; Mahabadi et al. , 2021 ) . However , it remains unclear as to which approach is preferred under which scenarios . We ask what are the general guidelines one should adopt while finetuning a large-scale pretrained model on downstream datasets . To scope this question , we choose CLIP ( Radford et al. , 2021 ) as the base pretrained model and adapt it to several downstream problems . CLIP is a vision-and-language model trained on over 400M pairs of image and text descriptions collected off the internet . There are several reasons to choose CLIP for this study . First , CLIP is one of the few vision models trained on ultra-large scale , unfiltered and varied raw visual data on the internet . Second , the multi-modal nature of CLIP enables use of more general ways of adaptation like using natural language prompts for “ zero-shot ” transfer to new categories – techniques previously popular mostly in NLP . We find that merely tuning the parameters of LayerNorm ( Ba et al. , 2016 ) turns out to be a surprisingly effective approach that is competitive or better than all other adaptation methods across the board . The effectiveness of normalization techniques has been observed by prior work for generalization ( Perez et al. , 2018 ; Lu et al. , 2021 ) as well as training from scratch ( Frankle et al. , 2020 ) . Inspired by this , 1Website at https : //sites.google.com/view/adapt-large-scale-models we further look into different ways of combining LayerNorm-tuning with other adaptation methods that finetune new parameters . We devise an effective scheme that first finetunes the CLIP model using only LayerNorm tuning and uses it as initialization for adapting new parameters . We evaluate our adaptation techniques across 12 downstream tasks spread along two spectra : size of downstream task dataset as well as the similarity of downstream data to the pretraining data . Across both spectra , we find that our two-stage LayerNorm tuning approach is most competitive and show its effectiveness for general-purpose adaptation of CLIP to downstream image-classification tasks . To summarize , our paper ’ s contributions are as follows : • We show the effectiveness of LayerNorm-tuning for adaptation to downstream tasks . • We devise a simple yet effective scheme to combine LayerNorm-tuning with other methods of finetuning to obtain competitive performance across the board . • We show a thorough comparison of different adaptation methods in four scenarios across two spectra ( amount of downstream data and its similarity to pretraining data ) studied on numerous downstream classification tasks . We believe our findings will encourage more research and put existing research in perspective of what works best while finetuning large-scale vision-language models to downstream tasks . 2 BACKGROUND : VISION-AND-LANGUAGE PRETRAINED MODELS . Vision-and-language pre-training methods have recently shown promise on diverse tasks across images and text ( Radford et al. , 2021 ; Jia et al. , 2021 ) . While many such approaches have emerged , we focus on CLIP ( Contrastive Language-Image Pre-training ) , a large-scale model with strong zero-shot performance on downstream classification tasks ( Radford et al. , 2021 ) . Contrastive Language-Image Pre-training ( CLIP ) CLIP consists of two parallel encoders for processing images and text , whose outputs are then projected into a shared embedding space . The text encoder is a Transformer ( Vaswani et al. , 2017 ) following the architecture described in Radford et al . ( 2019 ) , while the image encoder is a Vision Transformer ( ViT ) with a patch size of 16 ( Dosovitskiy et al. , 2020 ) . For our experiments , we utilize the open-sourced pretrained CLIP models . Training Because the image and text features live in the same embedding space , the cosine similarity between any embedded image and text description can be computed . CLIP uses these as prediction probabilities for classifying an image with the correct text caption ( or vice versa ) across batches . Formally , denote I and T as the set of image and text features in a single batch . The prediction probability for the ith image and jth caption in the batch is given by p ( Tj | Ii ) = exp ( cos ( Tj , Ii ) /⌧ ) exp ( cos ( Tj , Ii ) /⌧ ) + P k 6=j exp ( cos ( Tk , Ii ) /⌧ ) where ⌧ is a learnable temperature parameter . CLIP is trained with a contrastive loss accordingly across 400 million pairs of image and text captions collected online ( Radford et al. , 2021 ) . Inference For a downstream classification task at test time , CLIP first embeds the textual descriptions of all classes . These descriptions may range from a phrase like “ a photo of a < class > ” to heavily engineered embeddings ensembled over 80 different templates ( Radford et al. , 2021 ) . Each image is then classified using the embedded classes as labels and the prediction probabilities described above . Notably , this inference scheme allows CLIP to be transferred zero-shot to any downstream image classification task . Radford et al . ( 2021 ) show that zero-shot CLIP is competitive with a fully supervised ResNet ( He et al. , 2016 ) baseline on a suite of image classification tasks . 3 METHODOLOGY : FINE-TUNING LARGE-SCALE PRETRAINED MODEL . Although zero-shot CLIP performs well on natural images and general object classification datasets , its performance degrades quickly on more abstract tasks from out-of-distribution data . Even on a simple dataset like MNIST ( LeCun , 1998 ) , the zero-shot CLIP model ( ViT-B/16 ) we test attains an accuracy of only 55 % . Substantial gains can be achieved by fine-tuning the pre-trained model , but many such strategies have emerged across tasks in vision and language and it ’ s unclear which to use on diverse downstream settings . For this reason , we provide an extensive study of adaptation approaches . Figure 1 illustrates the fine-tuning methods we consider in context of CLIP while Figure 2 shows more detailed information regarding each approach . We propose a general taxonomy of fine-tuning approaches and consider three major classes : ( a ) methods which only fine-tune existing parameters , ( b ) methods which freeze existing parameters and add new parameters , and ( c ) methods which combine ( a ) and ( b ) . We first consider two methods in ( a ) which only fine-tune existing parameters . 3.1 FINE-TUNING EXISTING PARAMETERS . Full Model Fine-tuning The simplest approach to fine-tuning is to train all of the model parameters on the downstream task . However , this is unstable and doesn ’ t scale well to CLIP-size models with hundreds of millions of parameters . Our empirical results show this behavior as well . LayerNorm Tuning Instead of full model fine-tuning for large-scale models , we can tune a small subset of chosen parameters when the downstream data is scarce . In fact , Frankle et al . ( 2020 ) show that just tuning Batch Normalization ( Ioffe & Szegedy , 2015 ) parameters from a random initialization can be highly expressive . In a similar vein , we investigate tuning the parameters of Layer Normalization ( LayerNorm ) layers ( Ba et al. , 2016 ) . Unlike Batch Normalization , LayerNorm applies per-element normalization across mini-batches . Given a mini batch of inputs x , LayerNorm transforms this as y = x E [ x ] p Var [ x ] + ✏ · + where the mean and variance are calculated over the normalized dimensions and , are learned parameters . Because the image and text encoders in CLIP share the same underlying Transformer architecture , in LayerNorm Tuning , we fine-tune the Layer Normalization parameters , across all layers of both encoders . These parameters are 768-dimensional and 512-dimensional for the image and text encoders respectively . 3.2 FINE-TUNING NEW PARAMETERS . An alternative paradigm is to inject new parameters which can more effectively adapt to downstream tasks . These new parameters can act at various stages of a pre-trained model : on the output , input , or intermediate activations . Linear Probe The classic method of training a linear probe on top of frozen features is an example of adding new parameters which act on the model output . Given a pre-trained CLIP model , we discard the text encoder , freeze the image encoder , and learn a linear layer on top of the image features before they ’ re projected to the shared embedding space . The linear layer maps the penultimate image features to logits from which class predictions are made . While this simple method is popular and effective , it ’ s parameter-inefficient for tasks with higher number of classes and fails to leverage any of the language information contained in CLIP . Prompt Tuning Alternatively , we can consider adding parameters which act on the model input . Such an approach known as prompt tuning has emerged as a parameter-efficient fine-tuning method in language ( Li & Liang , 2021 ; Lester et al. , 2021 ) . A fixed number of continuous vectors ( a “ prompt ” ) is prepended to the model input and optimized throughout training . Similar to concurrent work by Zhou et al . ( 2021 ) , we apply prompt tuning to image classification with CLIP . For the model input , we embed the raw text of the classes without a template and prepend a continuous prompt of fixed length . During training , the prompt is learned using a cross-entropy loss according to the prediction probabilities detailed in Section 3.1 . Although prompt tuning can be applied in the same way for transformer-based visual encoders , we find that only applying it for the text encoder produces better and more stable results . Prompt tuning is parameter-efficient and removes the need for manual prompt engineering e.g . specifying “ a photo of a < class > , a type of flower ” for a downstream task on flower classification . Ideally , the learned prompts would contain such domain-specific information . However , prompt tuning suffers from high variance during training and is sensitive to initialization . Adapter and Compacter Networks The above two approaches inject parameters which act either at the end of the network ( linear probe ) or at the beginning ( prompt tuning ) . A third option is to inject new parameters for the downstream task within the layers of the network itself . This idea has been popularized as an efficient transfer learning method in language ( Houlsby et al. , 2019 ) . For Transformer-based architectures , a common strategy is to insert a block of learnable parameters after feed forward layers or the attention mechanism . Adapter networks insert learnable adapter blocks after the feed forward layers in each Transformer layer ( Houlsby et al. , 2019 ) . Each block follows a bottleneck architecture and is composed of a linear down-projection , non-linearity , and linear up-projection as shown in Figure 2 . However , for architectures with many stacked Transformer layers and larger hidden dimensions , adapter modules are parameter-inefficient . To alleviate this issue , Mahabadi et al . ( 2021 ) introduce compacter modules which follow the same architecture but use low-rank parameters and hypercomplex multiplication to improve parameter efficiency . Specifically , if the down-projection layer maps x 2 Rm ! Wx + b 2 Rd where W 2 Rm⇥d , b 2 Rd are learned parameters and d ⌧ m , compacter modules represent W as W = nX i=1 Ai ⌦ ( sitTi ) where Ai are global weights shared across Transformer layers and si , ti are local , rank-1 weights . We insert Adapter and Compacter modules across the Transformer layers in the text encoder .

Summary: The goal of the paper is to investigate how to efficiently adapt large-scale pre-trained vision-language models (e.g. CLIP) to downstream tasks. Their paper is based on the observation that while the vision community dominantly uses linear probe as the standard protocol, other approaches such as prompt learning are utilized in language. They compare and analyze several fine-tuning methods -- linear probe, prompt tuning, adapter and compacter networks -- across 12 downstream classification tasks. Focusing on LayerNorm, they further demonstrate combining LayerNorm tuning with existing fine-tuning methods to improve performance.