Learning to Prompt for Vision-Language Models

1 INTRODUCTION . The traditional approach for visual representation learning is to train vision models to predict for a fixed set of object categories using discrete labels ( He et al. , 2016 ; Dosovitskiy et al. , 2021 ) . However , this approach limits visual recognition systems to closed-set visual concepts defined during training , making them unable to handle new categories once deployed in target environments , since additional data are required for learning a new classifier . Recently , vision-language pre-training such as CLIP ( Radford et al. , 2021 ) and ALIGN ( Jia et al. , 2021 ) has emerged as a promising alternative . The main idea is to align images and raw text using two separate encoders—one for each modality . Through large-scale pre-training , vision-language models are allowed to learn open-set visual concepts and can readily be transferred to downstream tasks . In particular , for each new classification task , one can synthesize the classification weights by feeding natural language describing classes of interest to the text encoder , and compare them with image features produced by the image encoder . We observe that for pre-trained vision-language models , the text input , known as prompt , plays a key role in downstream datasets . However , identifying the right prompt is a non-trivial task , which often takes a significant amount of time for words tuning—since a slight change in wording could make a huge difference in performance . For instance , for Caltech101 ( Figure 1 ( a ) , 2nd vs. 3rd prompt ) , adding “ a ” before the class token brings more than 5 % increase in accuracy . Moreover , prompt engineering also requires expertise about the task and ideally the language model ’ s underlying mechanism . This is exemplified in Figure 1 ( b-d ) where adding task-relevant context can lead to significant improvements , i.e. , “ flower ” for Flowers102 , “ texture ” for DTD and “ satellite ” for EuroSAT . Tuning the sentence structure could bring further improvements , e.g. , putting “ a type of flower ” after the class token for Flowers102 , keeping only “ texture ” in the context for DTD , and Describable Textures ( DTD ) EuroSAT a [ CLASS ] . a photo of [ CLASS ] . a photo of a [ CLASS ] . [ V ] 1 [ V ] 2 … [ V ] M [ CLASS ] . 80.77 78.99 84.42 92.00 Flowers102 a photo of a [ CLASS ] . a flower photo of a [ CLASS ] . a photo of a [ CLASS ] , a type of flower . [ V ] 1 [ V ] 2 … [ V ] M [ CLASS ] . 56.68 61.23 62.32 93.22 Caltech101 Prompt Prompt Prompt Prompt Accuracy Accuracy Accuracy Accuracy ( a ) ( b ) adding “ centered ” before “ satellite photo ” for EuroSAT . However , even with extensive tuning , the resulting prompts are by no means guaranteed to be optimal for these downstream tasks . Inspired by recent prompt learning research in NLP ( Shin et al. , 2020 ; Jiang et al. , 2020 ; Zhong et al. , 2021 ) , we propose context optimization ( CoOp ) 1 to automate prompt engineering to allow more efficient and task-specific transfer for pre-trained vision-language models . Specifically , we model a prompt ’ s context using continuous representations which are essentially initialized with random vectors with the same dimension as word embeddings ( see Figure 2 ) . The context could be shared among all classes or designed to be class-specific . During training , we simply minimize the prediction error using the cross-entropy loss with respect to the learnable context vectors , while keeping the pre-trained parameters fixed . The gradients can be back-propagated all the way through the text encoder , distilling the rich knowledge encoded in the parameters for learning task-relevant context . To demonstrate the effectiveness of CoOp , we benchmark on 11 datasets , which cover a diverse set of visual recognition tasks including classification on generic objects , scenes , actions and fine-grained categories , as well as specialized tasks like recognizing textures and satellite imagery . The results show that CoOp can effectively turn pre-trained vision-language models into data-efficient visual learners , requiring as few as one or two shots to beat hand-crafted prompts with a decent margin . The performance can also be further boosted by using more shots , e.g. , at 16 shots the margin over hand-crafted prompts averages at around 17 % and reaches over 50 % for the highest . CoOp also outperforms the linear probe alternative known as a strong few-shot learning baseline ( Tian et al. , 2020 ) , and crucially , demonstrates much stronger robustness to distribution shift . Extensive analysis is also conducted to offer a comprehensive picture on how to apply CoOp in practice . The source code for reproducing the experiments will be released to facilitate future research . 2 METHODOLOGY . 2.1 VISION-LANGUAGE PRE-TRAINING . We briefly introduce vision-language pre-training with a particular focus on CLIP ( Radford et al. , 2021 ) . Our approach is applicable to broader CLIP-like vision-language models . Models CLIP consists of two encoders , one for images and the other for text . The image encoder aims to map high-dimensional images into a low-dimensional embedding space . The architecture of the image encoder can take the form of a CNN like ResNet-50 ( He et al. , 2016 ) or a ViT ( Dosovitskiy et al. , 2021 ) . On the other hand , the text encoder is built on top of a Transformer ( Vaswani et al. , 2017 ) and aims to generate text representations from natural language . 1CoOp is pronounced as /ku : p/ . Specifically , given a sequence of words ( tokens ) , such as “ a photo of a dog. ” , CLIP first converts each one of the token ( including punctuation ) into a lower-cased byte pair encoding ( BPE ) representation ( Sennrich et al. , 2016 ) , which is essentially a unique numeric ID . The vocabulary size in CLIP is 49,152 . To facilitate minibatch processing , each text sequence is encompassed with the [ SOS ] and [ EOS ] tokens and capped at a fixed length of 77 . After that , the IDs are mapped to 512-D word embedding vectors , which are then passed on to the Transformer . Finally , the features at the [ EOS ] token position are layer normalized and further processed by a linear projection layer . Training CLIP is trained to align the two embedding spaces learned for images and text respectively . Specifically , the learning objective is formulated as a contrastive loss . Given a batch of image-text pairs , CLIP maximizes the cosine similarity for matched pairs while minimizes the cosine similarity for all other unmatched pairs . To learn diverse visual concepts that are more transferable to downstream tasks , CLIP ’ s team collects a large training dataset consisting of 400 million image-text pairs . Zero-Shot Inference Since CLIP is pre-trained to predict whether an image matches a textual description , it naturally fits zero-shot recognition . This is achieved by comparing image features with the classification weights synthesized by the text encoder , which takes as input textual descriptions specifying classes of interest . Formally , let f be image features extracted by the image encoder for an image x and { wi } Ki=1 a set of weight vectors generated by the text encoder . K denotes the number of classes and each wi is derived from a prompt that could have the form of “ a photo of a [ CLASS ] . ” where the class token is replaced by the specific class name , such as “ cat ” , “ dog ” or “ car ” . The prediction probability is then computed as p ( y = i|x ) = exp ( < wi , f > /τ ) ∑K j=1 exp ( < wj , f > /τ ) , ( 1 ) where τ is a temperature parameter learned by CLIP and < · , · > denotes cosine similarity . 2.2 CONTEXT OPTIMIZATION . We propose context optimization ( CoOp ) , which avoids manual prompt tuning by modeling context words with continuous vectors that are end-to-end learned from data . An overview is shown in Figure 2 . Specifically , the prompt given to the text encoder g ( · ) is designed with the following form , t = [ V ] 1 [ V ] 2 . . . [ V ] M [ CLASS ] , ( 2 ) where each [ V ] m ( m∈ { 1 , . . . , M } ) is a vector with the same dimension as word embeddings ( i.e. , 512 for CLIP ) , and M is a hyperparameter specifying the number of context tokens . Note that the context here is shared among all classes , which is called unified context and different from classspecific context that is introduced later . By forwarding a prompt t to the text encoder g ( · ) , we can obtain a classification weight vector representing a visual concept . The prediction probability is computed as p ( y = i|x ) = exp ( < g ( ti ) , f > /τ ) ∑K j=1 exp ( < g ( tj ) , f > /τ ) , ( 3 ) where the class token within each prompt ti is replaced by the corresponding word embedding vector ( s ) of the i-th class name . Training is performed to minimize the standard classification loss based on the cross-entropy , and the gradients can be back-propagated all the way through the text encoder g ( · ) , making use of the rich knowledge encoded in the parameters to optimize the context . The design of continuous representations also allows full exploration in the word embedding space , which facilitates the learning of task-relevant context . Other Variants Other than placing the class token at the end of a sequence as in Equation ( 2 ) , we can also put it in the middle like t = [ V ] 1 . . . [ V ] M 2 [ CLASS ] [ V ] M 2 +1 . . . [ V ] M , ( 4 ) which increases flexibility for learning—theoretically , the prompt is allowed to either fill the latter cells with supplementary descriptions or cut off the sentence earlier by using a termination signal such as full stop . Another option is to design class-specific context ( CSC ) where context vectors are independent to each class , i.e. , [ V ] i1 [ V ] i 2 . . . [ V ] i M 6= [ V ] j 1 [ V ] j 2 . . . [ V ] j M for i 6= j and i , j ∈ { 1 , . . . , K } . As an alternative to unified context , we find that CSC is particularly useful for some fine-grained classification tasks . 3 EXPERIMENTS . 3.1 FEW-SHOT LEARNING . Datasets We select 11 publicly available image classification datasets used in CLIP : ImageNet ( Deng et al. , 2009 ) , Caltech101 ( Fei-Fei et al. , 2004 ) , OxfordPets ( Parkhi et al. , 2012 ) , StanfordCars ( Krause et al. , 2013 ) , Flowers102 ( Nilsback & Zisserman , 2008 ) , Food101 ( Bossard et al. , 2014 ) , FGVCAircraft ( Maji et al. , 2013 ) , SUN397 ( Xiao et al. , 2010 ) , DTD ( Cimpoi et al. , 2014 ) , EuroSAT ( Helber et al. , 2019 ) and UCF101 ( Soomro et al. , 2012 ) ( see Appendix A for their statistics ) . These datasets constitute a comprehensive benchmark , which covers a diverse set of vision tasks including classification on generic objects , scenes , actions and fine-grained categories , as well as specialized tasks like recognizing textures and satellite imagery . We follow the few-shot evaluation protocol adopted in CLIP ( Radford et al. , 2021 ) , using 1 , 2 , 4 , 8 and 16 shots for training respectively and deploying models in the full test sets . The average results over three runs are reported for comparison . Training Details CoOp has four versions : positioning the class token in the end or middle ; unified context vs. CSC . Unless otherwise stated , ResNet-50 ( He et al. , 2016 ) is used as the image encoder ’ s backbone and the number of context tokens M is set to 16 . Investigations on other design choices are discussed in Section 3.3 . All models are built on top of the open-sourced CLIP.2 CoOp ’ s context vectors are randomly initialized by drawing from a zero-mean Gaussian distribution with standard deviation equal to 0.02 . Training is done with SGD and an initial learning rate of 0.002 , which is decayed by the cosine annealing rule . The maximum epoch is set to 200 for 16/8 shots , 100 for 4/2 shots , and 50 for 1 shot ( except for ImageNet where the maximum epoch is fixed to 50 ) . To mitigate explosive gradients observed in the early training iterations , we use the warmup trick by fixing the learning rate to 1e−5 during the first epoch . 2https : //github.com/openai/CLIP . Baseline Methods We compare CoOp with two baseline methods . The first is zero-shot CLIP , which is based on hand-crafted prompts . We follow the guideline of prompt engineering introduced by Radford et al . ( 2021 ) . For generic objects and scenes , “ a photo of a [ CLASS ] . ” is adopted . For fine-grained categories , task-relevant context is added like “ a type of pet ” for OxfordPets and “ a type of food ” for Food101 . When it comes to specialized tasks such as recognizing textures in DTD , the prompt is customized as “ [ CLASS ] texture. ” where the class names are adjectives like “ bubbly ” and “ dotted ” . See Appendix A for the details . The second baseline is linear probe CLIP . As suggested by Radford et al . ( 2021 ) and a recent study on few-shot learning ( Tian et al. , 2020 ) , training a linear classifier on top of high-quality pre-trained models ’ features ( like CLIP ) can easily achieve performance that is on a par with that of state-of-the-art few-shot learning methods , which are often much more sophisticated . We follow the same training method used by Radford et al . ( 2021 ) to train linear probe CLIP . Comparison with Hand-Crafted Prompts Figure 3 summarizes the results . Our default model is CLIP+CoOp with the class token positioned in the end . The two different ways of positioning the class token achieve similar performance as their curves highly overlap . From the average performance displayed in the top-left corner , we observe that CLIP+CoOp is a strong few-shot learner , requiring only two shots on average to obtain a decent margin over zero-shot CLIP . Given 16 shots for training , the average gap brought by CoOp can be further increased to around 17 % . Figure 4 ranks the absolute improvements obtained by CoOp at 16 shots over hand-crafted prompts . Huge improvements are observed on specialized tasks namely EuroSAT and DTD where the increase in performance reaches over 50 % and 20 % respectively . The jumps in performance are also significant ( those more than 10 % ) on most fine-grained datasets including Flowers102 , StanfordCars and FGVCAircraft , as well as on scene and action recognition datasets ( SUN397 & UCF101 ) . Since ImageNet is a challenging dataset that contains 1,000 classes , the 5.05 % improvement is also noteworthy . In contrast , the increases on the two fine-grained datasets , OxfordPets and Food101 , are less appealing . By digging into CLIP+CoOp ’ s curves on these two datasets in Figure 3 , we find there is a loss of momentum in performance improvements even with more shots used , seemingly an overfitting problem . A potential solution is to impose higher regularizations like increasing the weight decay . Nonetheless , the overall results are strong enough to serve as evidence of CoOp ’ s capability of learning task-relevant prompts in a data-efficient way . Comparison with Linear Probe CLIP In terms of the overall performance ( Figure 3 , topleft ) , CLIP+CoOp demonstrates clear advantages over linear probe CLIP . The latter requires 4 shots on average to match the zero-shot ’ s performance while CoOp ’ s average gains at 4 shots are already more than 10 % . It is also clear that the gaps in the extreme low-data regime such as one or two shots are much larger , suggesting that CoOp is much more effective than learning a linear classifier from scratch for fewshot learning . We also observe that linear probe CLIP is comparable to CLIP+CoOp on the two specialized tasks ( DTD & EuroSAT ) as well as on a couple of fine-grained datasets ( Flowers102 & FGVCAircraft ) —this is not too surprising as the pre-trained CLIP space has been proved powerful , making the linear probe model a strong competitor . Nevertheless , CoOp ’ s CSC version can beat linear probe CLIP on the aforementioned datasets , and moreover , shows much better potential when more shots become available . Unified vs. Class-Specific Context On average , using unified context leads to better performance . In terms of when to apply CSC and when not to , we have the following suggestions . For generic objects ( ImageNet & Caltech101 ) , scenes ( SUN397 ) and actions ( UCF101 ) , using unified context is clearly better . Unified context also works better on some fine-grained datasets including OxfordPets and Food101 , but on others like StanfordCars , Flowers102 and FGVCAircraft the CSC version is preferred . CSC also yields better performance on the two specialized tasks , DTD and EuroSAT , at 16 shots in particular . However , CSC mostly underperforms unified context in challenging low-data scenarios ( fewer than 8 shots ) , which makes sense because CSC has more parameters than unified context and needs more data for training .

Summary: The authors demonstrate a more efficient form of few-shot learning using CLIP compared to linear probing for image classification: CoOp. Instead of fine-tuning a small linear classifier on the output of CLIP, they propose fine-tuning a number of additional embeddings at the input layer; this modification, in theory, allows CLIP to leverage more computation when adapting to tasks, while still only optimizing a small number of parameters. Experiments across several corpora demonstrate the efficacy of the approach, which generally yields a few accuracy points of gain versus a linear probe trained on the same amount of data.