Learning to Prompt for Vision-Language Models
Authors: Kaiyang Zhou
Jingkang Yang
Chen Change Loy
Ziwei Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: vision-language models
prompt learning
computer vision
transfer learning

Abstract: Vision-language pre-training has recently emerged as a promising alternative for representation learning. It shifts from the tradition of using images and discrete labels for learning a fixed set of weights, seen as visual concepts, to aligning images and raw text for two separate encoders. Such a paradigm benefits from a broader source of supervision and allows zero-shot transfer to downstream tasks since visual concepts can be diametrically generated from natural language, known as prompt. In this paper, we identify that a major challenge of deploying such models in practice is prompt engineering. This is because designing a proper prompt, especially for context words surrounding a class name, requires domain expertise and typically takes a significant amount of time for words tuning since a slight change in wording could have a huge impact on performance. Moreover, different downstream tasks require specific designs, further hampering the efficiency of deployment. To overcome this challenge, we propose a novel approach named \emph{context optimization (CoOp)}. The main idea is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. In this way, the design of task-relevant prompts can be fully automated. Experiments on 11 datasets show that CoOp effectively turns pre-trained vision-language models into data-efficient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin and able to gain significant improvements when using more shots (e.g., at 16 shots the average gain is around 17\% with the highest reaching over 50\%). CoOp also exhibits strong robustness to distribution shift.

Under review as a conference paper at ICLR 2022
LEARNING TO PROMPT FOR VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Vision-language pre-training has recently emerged as a promising alternative for representation learning. It shifts from the tradition of using images and discrete la- bels for learning& fixed set of weights, seen as visual concepts, aligning images and raw text for two separate encoders. Such a paradigm benefits from broader source of supervision and allows zero-shot transfer to downstream tasks since vi- sual concepts can be diametrically generated from natural language, known as prompt. this paper, we identify that a major challenge deploying such mod- els in practice is prompt engincering. This is because designing a proper prompt, especially for context words surrounding a class name, requires domain expertise and typically takes significant amount of time for words tuning since slight change in wording could have a huge impact on performance. Moreover, differ- ent downstream tasks require specific designs, further hampering the efficiency of deployment. To overcome this challenge, we propose a simple approach named context optimization (CoOp). The main idea to model context in prompts us- ing continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. In this way, the design of task-relevant prompts can be fully automated. Experiments on 11 datasets show that CoOp ef- fectively turns pre-trained vision-language models into data-efficient visual learn- ers, requiring decent margin and able to gain significant improvements when using more shots (e.g., at 16 shots the average gain is around 17% with the highest reaching over 50%). CoOp also exhibits strong robustness to distribution shift.
few one two shots to beat hand-crafted prompts with a
INTRODUCTION
The traditional approach for visual representation learning is to train vision models to predict for fixed set of object categories using discrete labels (He et al., 2016; Dosovitskiy et al., 2021). How- ever, this approach limits visual recognition systems to closed-set visual concepts defined during training. making them unable handle categories once deployed in target environments, since additional data are required for learning a new classifier. Recently, vision-language pre- pre-training such CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) has emerged as promising alternative. The main idea is to align images and raw text using two separate encoders one for each modality. Through large-scale pre-training, vision-language models are allowed to learn open-set visual con- cepts and can readily be transferred to downstream tasks. In particular, for each new classification task, one can synthesize the classification weights by feeding natural language describing classes of interest to the text encoder, and compare them with image features produced by the image encoder.
We observe that for pre-trained vision- vision-language models, the text input, known as prompt. plays key role in downstream datasets. However, identifying the right prompt is a non-trivial task, which often takes a significant amount of time for words tuning-since slight change wording could make huge difference in performance. For instance, for Caltech101 (Figure 1(a), 2nd vs. 3rd prompt), adding "a" before the class token brings more than 5% increase in accuracy. Moreover, prompt engineering also requires expertise about the task and ideally the language model's under- lying mechanism. This is exemplified Figure ((b-d) where adding task-relevant context can lead to significant improvements, i.c., "flower" for Flowers102, "texture" for DTD and "satellite" for EuroSAT. Tuning the sentence structure could bring further improvements, e.g., putting "a type of flower" after the class token for Flowers 102, keeping only "texture" in the context for DTD, and

---

Under review as a conference paper at ICLR 2022
Caltech101
Prompt
Accuracy
Rowwer102
Prompt
Accuracy
[CLASS].
80.77
photo of [CLASS).
56.68
photo [CLASS].
78.99
a flower photo of [CLASS).
61.23
[CLASS].
84.42
(CIASS), type flower.
Mi (V] Mi [SL)
92.00
[V] [V] ... (Mju (CLASS).
93.22
(a)
(b)
Describable Textures (DTD)
Prompt
Accuracy
EuroSAT
Prompt
Accuracy
photo of [CLASS].
38.24
photo of [CLASS).
22.30
photo of [CLASS] texture.
37.71
a satellite photo of (CLASS).
31.12
CCASS
centered catallite of (CLASS).
Mv [ Mu [CLASS].
62.55
M [V] Miu (CLASS).
81.60
[c]
(d)
Prompt engineering context optimization (CoOp). The latter uses only 16 shots for learning in these examples.
Figure I:
adding "centered" before "satellite photo" for EuroSAT. However, even with extensive tuning, the resulting prompts are by no means guaranteed to be optimal for these downstream tasks.
Inspired by recent prompt learning research in NLP (Shin al., 2020; Jiang al., 2020; Zhong et al., 2021), we propose context optimization (CoOp)' to automate prompt engineering to allow more efficient and task-specific transfer for pre-trained vision-language models. Specifically, model prompt's context using continuous representations which are essentially initialized with random vectors with the same dimension as word embeddings (see Figure 2). The context could be shared among all classes or designed to be class-specifie. During training, we simply minimize the prediction error using the cross-entropy loss with respect the learnable context vectors, while keeping the pre-trained parameters fixed. The gradients can be back-propagated all the way through the text encoder, distilling the rich knowledge encoded in the parameters for learning task-relevant context.
To demonstrate the effectiveness of CoOp, we benchmark on datasets, which cover diverse set of visual recognition tasks including classification on generic objects, scenes, actions and fine-grained categories, as well as specialized tasks like recognizing textures and satellite imagery. The results show that CoOp can effectively turn pre-trained vision-language models into data-cfficient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin. The performance also be further boosted by using hand-crafted prompts averages at around 17% and reaches over 50% for the highest. CoOp also outperforms the linear probe alternative known strong few-shot learning baseline (Tian al., 2020), and crucially, demonstrates much stronger robustness to distribution shift. Extensive analysis also conducted offer comprehensive picture on how to apply CoOp practice. The source code for reproducing the experiments will be released to facilitate future research.
shots, e.g., 16 shots the margin over
METHODOLOGY
2. VISION-LANGUAGE PRE-TRAINING
We briefly introduce vision- languaeee pre-training with particular focus on CLIP (Radford et al., 2021). Our approach is applicable broader CLiP-like vision-language models.
Models CLIP consists of two encoders, one for images and the other for text. The image encoder aims map high-dimensional images into low-dimensional embedding space. The architecture of the image encoder can take the form of a CNN like ResNet-50 (He et al., 2016) or a ViT (Dosovitskiy al., 2021). On the other hand, the encoder is built top of Transformer (Vaswani al., 2017) and aims to generate text representations from natural language.
'CoOp is pronounced /kup/.
2

---

Under review as a conference paper at ICLR 2022
learnable context
M
[CLASS]
airplane
butterfly
pizza
features
similarity scores
image encoder
image features
maximize the score for the ground-truth class
Figure 2: Overview of context optimization (CoOp).
Specifically, given a sequence of words (tokens), such as "a photo of a dog.". CLIP first converts each one of the token (including punctuation) into lower-cased byte pair encoding (BPE) represen- tation (Sennrich et al., 2016), which is essentially a unique numeric ID. The vocabulary size CLIP is 49,152. To facilitate minibatch processing, each text sequence is encompassed with the [SOS] and [BOS] tokens and capped fixed length of 77. After that, the IDs are mapped to 512-D word embedding vectors, which are then passed on the Transformer. Finally, the features at the [ECS] token position are layer normalized and further processed by a linear projection layer.
Training CLIP trained to align the two embedding spaces learned for images and text respec- tively. Specifically, the learning objective is formulated contrastive loss. Given batch of image-text pairs, CLIP maximizes the cosine similarity for matched pairs while minimizes the co- sine similarity for all other unmatched pairs. To learn diverse visual concepts that more trans- ferable downstream tasks, CLIP's team collects large training dataset consisting of 400 million image-text pairs.
Zero-Shot Inference Since CLIP is pre-trained to predict whether image matches textual de- scription, it naturally fits zero-shot recognition. This is achieved by comparing image features with the classification weights synthesized by the text encoder, which takes input textual descriptions specifying classes of interest. Formally, let f be image features extracted by the image encoder for image æ and {w;) set of weight vectors generated by the text encoder. K denotes the number of classes and each w; is derived from a prompt that could have the form of "a photo of [CLASS]." where the class token is replaced by the specific class name, such as "cat", "dog" or "car". The prediction probability is then computed as
expl< wi f \T) Σ exp(< Wj f /T)
plu i|ær
(1)
where T is temperature parameter learned by CLIP and <,, denotes cosine similarity.
2.2 CONTEXT OPTIMIZATION
We propose context optimization (Co0p), which avoids manual prompt tuning by modeling context words with continuous vectors that are end-to-end learned from data. An overview is shown in Figure 2. Specifically, prompt given to the text encoder g(.) is designed with following form,
M ... M [CLASS].
(2)
where each [m]m (m E (1....,,MM)) is a vector with the same dimension as word embeddings (i.c., 512 for CLIP), and M is hyperparameter specifying the number of context tokens. Note that the

---

Under review as a conference paper at ICLR 2022
context here is shared among all classes, which is called unified context and different from class- specific context that is introduced later.
By forwarding prompt t to the text encoder g(-), can obtain classification weight vector representing a visual concept. The prediction probability is computed as
exp(< g(4), /T)
p(y = (|)) = Σラ=1 exp(< ((1)); f /T)
(3)
where the class token within each prompt t; is replaced by the corresponding word embedding vector(s) of the i-th class name.
Training is performed to minimize the standard classification loss based on the cross-entropy, and the gradients can be proppaaaee all the way through the text encoder g(-), making use of the rich knowledge encoded in the parameters to optimize the context. The design of continuous repre- sentations also allows full exploration in the word embedding space, which facilitates the learning of task-relevant context.
Other Variants Other than placing the class token at the end of a sequence as in Equation (2), we can also put in the middle like
t Mh M [CLASS] V + MMa.
(4)
which increases flexibility for learning theoretically, the prompt is allowed to either fill the latter cells with supplementary descriptions or cut off the sentence earlier by using a termination signal such as full stop.
Another option is to design class-specifer context (CSC) where context vectors are independent to each class, i.e. [V} ... [V] + [V][V] [V] for iĄj and i E [I,...,, K]. As an alter- native to unified context, we find that CSC is particularly useful for some fine-grained classification tasks.
3 EXPERIMENTS
3.1 FEW-SHOT LEARNING
Datasets We select 11 publicly available image classification datasets used in CLIP: Ima- geNet (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012), Stan- fordCars (Krause et al., 2013), Flowers 102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft (Maji al., 2013), SUN397 (Xiao al., 2010), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019) and UCF101 (Soomro et al., 2012) (see Appendix A for their statis- ties). These datasets constitute a comprehensive benchmark, which covers a diverse set of vision tasks including classification on generic objects, scenes, actions and fine-grained categories, as well as specialized tasks like recognizing textures and satellite imagery. We follow the few-shot eval- uation protocol adopted in CLIP (Radford et al., 2021), using 1, 2, 4, 8 and 16 shots for training respectively and deploying models in the full test sets. The average results over three runs are re- ported for comparison.
Training Details CoOp has four versions: positioning the class token in the end or middle; unified context vs. CSC Unless otherwise stated, ResNet-50 (He et al., 2016) is used as the image encoder's backbone and the number of context tokens M is set to 16. Investigations on other design choices are discussed in Section 3.3. All models are built on top of the open-sourced CLIP. CoOp's context vectors are randomly initialized by drawing from zero-mean Gaussian distribution with standard deviation equal to 0.02. Training is done with SGD and an initial learning rate of 0.002, which is decayed by the cosine annealing rule. The maximum epoch is set to 200 for 16/8 shots, 100 for 4/2 shots, and 50 for shot (except for ImageNet where the maximum epoch fixed to 50). To mitigate explosive gradients observed in the early training iterations, we use the warmup trick by fixing the learning rate to le 5 during the first epoch.
httpps:/// //gii thub. com/openat /CLIP.
4

---

Under review as a conference paper at ICLR 2022
Average over 11 datasets
Imagellet
Caltech101
60
Zemyyaa
53
Seure 0N0
CUP + Cady (M-16, cnd
35-
CL e CoOp 14 cndi
CLIP+ cocp IM-35, ondi
CUI Cady (M-16, m:
CJI Cada (M-16, e1d (SO
10-
CL i CoOp 2e mit
CL P CoOp 26 and CSC)
CLi++ Cocp (M-36, mith
CLP Coop (M-36, and CSC)
CJF Cods (4-16, m CSC
CL a Cocp We-16, mit (SC) Linear proze CLP
C.P Coop (Mo36, mid CSC)
33
Liner probe CUP
70
Unner probe CLI
16
Mamber abeded traningg comppts per case
Mamber labeled traverp examates per class
Rumber lazeled train stamptes per dam
OofordPets
StanfordCars
flowers02
10*
CLP
83-
Sture
CLP
e1d)
CeOp
Coop
Colp
42-
50-
24
proce
Skmber ubeded touning comntte
Namber labiied traserg anamzko clas
Rumbar aa brun sammpto per das
Pood101
PCUCAirrraf
SUN397
40
45
75-
30-
25
CJP Cods (M-16, and
20-
CL a Ceop W- end)
CLP Coop (M-36, andi
CuP-- Cods (4-16 m)
CJP CoOs (M-16, and CSC
15
CL Colp 26-14, mit)
CLP Cocp W6-16, and CSC)
CLP Coop (M-26, and CSc
CLP Coop (M-16, mith
32
CJP CoOs (N-16, mi CSC)
Liner probe CUP
10
Liteer przze CuP
CLP CoCp We-11, mit CSC)
25-
C.P Coop (M-26, mid, CSct
Lneer proba Cu?
16
16
16
Mamber abdded truning coupptes per casa
Humber labcie tranerg cxamatis per class
Rumber oa truingg exampte per dass
UC/101
Sooe 10)
Gröp
CLP
ml)
23
40-
CoOp
16
Manbbr of utdded traring courrtee
Number Mobite traserg ceamakk class
Number lazekal trunnng cuamptes per clss
Figure 3: Main results of few-shot learning on the 11 datasets. Overall, CoOp effectively turns CLIP into a strong few-shot learner (solid lines), achieving significant improvements over zero-shot CLIP (stars) and performing favorably against the linear probe alternative (dashed lines). M denotes the context length. "end"' or "mid" means putting the class token in the end or middle. CSC means class-specific context.
Baseline Methods We compare CoOp with two baseline methods. The first is zero-shot CLIP, which is based on hand-crafted prompts. We follow the guideline of prompt engineering introduced by Radford et al. (2021). For generic objects and scenes, "a photo of a [CLASS]." is adopted. For fine-grained categories, task-relevant context is added like "a type of pet" for OxfordPets and "a type of food" for Food101. When it comes to specialized tasks such as recognizing textures in DTD, the prompt is customized as "[CLASS] texture." where the class names are adjectives like "bubbly" and "dotted". See Appendix A for the details. The second baseline is linear probe CLIP. As suggested by Radford et al. (2021) and recent study on few-shot learning (Tian et al., 2020), training a linear classifier on top of high-quality pre-trained models' features (like CLIP) can easily achieve performance that is are often much more sophisticated. We follow the same training method used by Radford et al. (2021) to train linear probe CL.IP.
par with that of state-of-the of-tthe-rt art few-shot learning methods, which
5

---

Under review as a conference paper at ICLR 2022
Comparison with Hand-Crafted Prompts Figure 3 summarizes the results. Our default model CLIP+CoOp with the class token positioned in the end. The two different ways positioning the class token achieve similar performance as their curves highly overlap. From the average per- formance displayed in the top-left comnr, we observe that CLIP+CoOP is strong few-shot learner, requiring only two shots average to obtain decent margin over zero-shot CLIP. Given 16 shots for training, the average gap brought by CoOp can be further increased to around 17%.
Figure ranks the absolute improvements obtained by CoOp 16 shots over hand-crafted prompts. Huge improvements are observed on specialized tasks namely EuroS and DTD where the in- crease performance reaches over 50% and 20% respectively. The jumps in performance are also significant (those more than 10%) on most fine-grained datasets including Flowers 102, Stanford- Cars and FGVCAircraft, as well as on scene and action recognition datasets (SUN397 UCF101). Since ImageNet is challenging dataset that contains 1,000) classes, the 5.05% improvement is also noteworthy. contrast, the increases on the two fine-grained datasets, OxfordPets and Food101, are less appealing. By digging into CLIP+CoOp's curves these two datasets in Figure 3, we find there is a loss of momentum in performance improvements even with more shots used, seemingly an overfitting problem. A potential solution is to impose higher regularizations like increasing the weight decay. Nonetheless, the overall results are strong enough to serve as evidence of CoOp's capability of learning task-relevant prompts in data-efficient way.
Comparison with Linear Probe CLIP In terms of the overall performance (Figure 3, top- left), CLIP+CoOp demonstrates clear advan- tages over linear probe CLIP. The latter requires 4 shots average to match the zero-shot's per- formance while CoOp's s average gains at 4 shots are already more than 10%. is also clear that the gaps in the extreme low-data regime such as one or two shots are much larger, sug- gesting that CoOp is much more effective than learning linear classifier from scratch for few- shot learning. We also observe that linear probe CLIP is comparable to CLIP+CoOp on the two specialized tasks (DTD & EuroSAT) as well as on a couple of fine-graing fine-grained datasets (Flow- is not too
CLIP + CoOp IM-16, and) vs, Zero- Shot CUP
EaroSAT
SuatoroCara
OTD
-21.83
FoVCAicorft
+17.53
ucr15a
12242
SUNB97
+11.2
Caltoca151
17.58
Iageeet
55.00
+3.72
10
20
40
50
Abso ute Improvement (96)
Figure 4: Comparison with hand-crafted prompts.
ers & FGVCAircraft--tth prising as the pre-trained CLIP space has been proved powerful, making the linear probe model a strong competitor. Nevertheless, CoOp's CSC version can beat linear probe CLIP on the aforementioned datasets, and moreover, shows much better potential when more shots become available.
Unified vs. Class-Specific Context On average, using unified context leads better performance. In terms of when to apply CSC and when not to, we have the following suggestions. For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using unified context is clearly better. Unified context also works better on some fine-grained datasets including OxfordPets and Food101, but on others like StanfordCars, Plowers102 and FGVCAireraft the CSC version is preferred. CSC also yields better performance on the two specialized tasks, DTD and EuroSAT, at 16 shots in particular. However, CSC mostly underperforms unified context in challenging low-data scenarios (fewer than 8 shots), which makes sense because CSC has more parameters than unified context and needs more data for training.
3.2 ROBUSTNESS TO DISTRIBUTION SHIFT
Since CoOp requires training on a specific data distribution, it risks learning spurious correlations that are detrimental to generalization in unseen distributions (domains), suggested in recent stud- ies (Taori et al., 2020; Zhou et al., 2021). On the contrary, zero-shot CLIP is not tied to a specific data distribution and has exhibited strong robustness to distribution shift (Radford et al., 2021). In this section, we aim to unveil how robust CoOp is to distribution shift, in comparison to zero-shot CLIP and the linear probe model.

---

Under review as a conference paper at ICLR 2022
Table 1: Evaluation on robustness to distribution shift. M: CoOp's context length.
Source
Target
ImageNet
ImageNetV2
ImageNet-Sketch
ImageNet-A
ImageNet-R
Zero-Shot CLIP
55.41
48.08
31.67
18.63
53.45
Linear Probe CLIP
53.44
43.40
17.63
11.66
32.63
CLIP CoOp (M 16)
60,46
52.17
31.14
19.62
53.31
CLIP CoOp (M =M=8)
60.90
52.53
31.73
19.97
54.34
CLIP CoOp (M= 60.85
53.02
32.99
20.69
55.57
Table Comparison with prompt ensembling.
Table 3: Random vs. manual initialization.
ImageNet
Avg %
Prompt engineering 55.41
[
71.26 71.51
Prompt ensembling
57.81
"a photo of
CoOp
60.46
Datasets The source dataset is ImageNet. The target datasets are ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet- R (Hendrycks et al., 2021a), all of which have compatible class names with ImageNet allowing seamless transfer for the prompts learned by CoOp. ImageNetV2 different sources while following ImageNet's s data collection process. ImageNet-Skeet Sketch contains sketch images belonging to the same 1,000 ImageNet classes. Both ImageNet-A and -R contain 200 classes derived from subset of ImageNet's 1,000 classes. The former consists of real-world adver- sarially filtered images that cause current ImageNet classifiers to produce low results, whereas the latter features rendition of the ImageNet classes in diverse image styles such as paintings, cartoons and sculptures.
reproduced test using
Results Table summarizes the results. is surprising that CLIP+CoOp exhibits stronger robust- ness than zero-shot CLIP to distribution shift, despite exposure to the source dataset. This suggests that the learned prompts are also generalizable. Moreover, it is interesting to see that using fewer context tokens leads to better robustness. More results with different vision backbones are provided in Appendix B. where conclusion remains the same. contrast, linear probe CLIP obtains much worse results on these target datasets, exposing its weakness in domain generalization.
3.3 FURTHER ANALYSIS
Comparison with Prompt Ensembling Radford et al. (2021) have suggested that additional improvements can be obtained by ensembling over multiple zero-shot classifiers generated using different hand-crafted prompts, such as "a photo of the large [CLASS].", "a bad photo of the (CLASS)." and "a origami [CLASS].", which reflect a different scale, view and abstraction re- spectively for an image. We are interested to know whether the prompts learned by CoOp can still maintain advantages when compared with prompt ensembling. For fair comparison, we use the select prompts from Radford et al. (2021), which have been extensively tuned on ImageNet, to construct the ensemble classifier. Table 2 presents the results of prompt engineering (i.e., using a single hand-crafted prompt), prompt ensembling and CoOp, confirming that CoOp still the best performing method. Additional results are provided in Appendix B.2 to show that CoOp also beats prompt ensembling when more advanced vision backbones used. Given the potential of prompt ensembling, future work could investigate how to improve CoOp from the ensembling perspective.
Context Length How many context tokens should used? And is better to have more context tokens? The results in Section 3.2 suggest having shorter context length benefits domain general- ization. Here we study this hyperparameter for source datasets. Specifically, we repeat experiments on the 11 datasets by varying the context length from 4 to 8 to 16. The average results are shown in Figure 5(a), which indicate that having more context tokens leads to better performance and that positioning the class token in the middle gains more momentum with longer context length. To sum

---

Under review as a conference paper at ICLR 2022
Average over 11 datasets
Average over 11 datasets
79.51
CUP + CoOp (M-:66, endi
75
75,02
75.10
72.75
72.0
Scne (8)
Score (50
71.5
65
71.0
70.5
60-
59.45
CLIP CoCp (en6)
57.34
70.0
pww..p.....11. 2112
CLIP CoOp (mic)
55-
55
55.81
16
RN50
RN101
VIT-M/12
VIT-016
Number of contect words
(a) Context length
(b) Vision backbones
Figure 5: Investigations on CoOp's 's context length and various vision backbones.
up, there is golden rule for selecting perfect context length since one needs to balance between performance and robustness to distribution shift. See Appendix B.3 for the dataset settase--ppecficc results.
Vision Backbones Figure 5(b) summarizes the results on the 11 datasets using a variety of vision backbones covering both CNNs and ViTs. The results are expected: the more advanced the back- bone, the better the performance. The gap between CoOp and hand-crafted prompts is significant across all architectures. See Appendix B.4 for the dataset-specific results.
Initialization We compare random initialization with manual initialization. The latter uses the embeddings of photo of a" to initialize the context vectors for the 11 datasets. For fair comparison, we also set the context length to when using random initialization. Table suggests a "good" initialization only brings small improvement. Though further tuning of the initialization words might help, in practice we suggest using the simple random initialization method.
Interpreting the Learned Prompts is difficult because the context vectors are optimized in continuous space. We resort to an indirect way by searching within the vocabulary for words that are closest to the learned vectors based on the Euclidean distance. Note that CLIP (Radford al., 2021) uses the BPE representation (Sennrich et al., 2016) for tokenization, subwords that frequently appear in text, such as "hu" (subsumed by many words like "hug" and "human"). Table 4 shows the searched results somewhat relevant to the tasks, such as "enjoyed" for Food101, "fluffy" and "paw" for OxfordPets, and "pretty" for DTD. But when connecting all the nearest words together, the prompts do not make much sense. We also observe that when using manual initialization (like "a photo of a"), the nearest words for the converged vectors mostly the ones used for initialization. We conjecture that the learned vectors might encode meanings that are beyond the existing vocabulary. Overall, we unable draw any firm conclusion based on the observations because using nearest words to interpret the learned prompts could be inaccurate the semantics of the vectors is not necessarily correlated with the nearest words.
the vocabulary includes
some datasets. We observe that few words
RELATED WORK
Vision-Language Models have recently demonstrated great potential in learning generic visual representations and allowing zero-shot transfer to variety of downstream classification tasks (Rad- ford et al., 2021; Jia et al., 2021: Zhang et al., 2020). To our knowledge, the recent developments in vision-language learning, particularly CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), largely driven by advances in the following three areas: i) text representation learning with Trans- formers (Vaswani al., 2017), ii) large-ming minibachi contrastive representation learning (Chen al., 2020; He et al., 2020; Hénaff et al., 2020), and iii) web-scale training datasets CLIP benefits from 400 million curated image- text pairs while ALIGN exploits 1.8 billion noisy image-text pairs.
The idea of mapping images and text onto a common embedding space has been studied since nearly decade ago (Socher al., 2013; Frome al., 2013; Elhoseiny et al., 2013), but with drastically
8

---

Under review as a conference paper at ICLR 2022
Table The nearest words for each of the 16 context vectors learned by CoOp, with their distances shown in parentheses. N/A means non-Latin characters.
#
ImageNet
Food101
OxfordPets
DTD
UCF101
1
potd (1.7136)
(0.6752)
tosc (2.5952)
boxed (0.9433)
meteorologist (1.5377)
2
that (1.4015)
enjoyed (0,5305)
judge (1.2635)
seed (1.0498)
exe (0,9807)
3
filmed (1.2275)
beh (0.5390)
fluffy (I.6099)
anna (0.8127)
parents (1,0654)
4
fruit (1.4864)
matches (0.5646)
cart (1.3958)
mountain (0,9509)
masterful (0,9528)
5
(1.5863)
nytimes (0.6993)
harlan (2.2948)
eldest (0.7111)
(1.3574)
6
)1.7502)
prou (0.59) (0,5905)
paw (1.3055)
pretty (0.8762)
thof (1.2841)
7
excluded (1.2355)
lower (0.5390)
incase (1.2215)
faces (0.7872)
where (0.9705)
8
cold (1,4654)
N/A
bie (1.5454)
honey (1.8414)
kristen (1.1921)
9
stery (1.6085)
minute (0.5672)
snuggle (1.1578)
series (1.6680)
imam (1.1297)
10
warri (1.3055)
(0.5529)
along (I.8298)
coca (1.5571)
near (0.8942)
11
marvelcomics
(1.5638)
well (0,5659)
enjoyment (2.3495)
moon (1.2775)
tummy (1.4303)
12
(1.7387)
ends (0.6113)
jt (1.3726)
lh (1.0382)
hel (0.7644)
13
N/A
mis (0.5826)
improving (1,3198)
won (0.9314)
boop ( (1.0491)
14
lation (1,5015)
somethin (0.6041)
sTSly (1.6759)
replicd (1.1429)
N/A
15
muh (1.4985)
seminar (0.5274)
asteroid (1.3395)
sent (1,3173)
facial (1.4452)
16
.# (1.9340)
N/A
N/A piedmont (1.5198)
during (1.1755)
different technologies. For text features extraction, early work has mainly utilized pre-trained word vectors (Socher et al., 2013: Frome et al., 2013) or the hand-crafted TF-IDF features (Elhoseiny et al., 2013; Lei Ba et al., 2015). Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai & Johnson, 2021). Our work is orthogonal to recent research in vision-language models, aiming to facilitate the deployment of such models in downstream datasets.
Prompt Learning in NLP Knowledge probing for large pre-trained language models, formally defined by Petroni et al. (2019) as prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li & Liang, 2021: Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020; Liu al., 2021b). The basic idea of knowledge probing is to induce pre-trained language models to generate answers given cloze-style prompts, which can benefit a number of downstream tasks, such as sentiment analysis. Jiang et al. (2020) propose to generate candidate prompts through text mining and paraphrasing, and identify the optimal ones approach,
cloze tests, has recently sparked interest in
that give the highest training accuracy. Shin et al. (2020) introduce a gradient- which searches for tokens with the largest gradient changes in the label likelihood. Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li & Liang, 2021; Lester al., 2021) which optimize continuous vectors in the word embedding space. A drawback of such methods compared to searching discrete tokens is the lack of a clear way to visualize what "words" are learned for the vectors. We refer readers Liu et (2021a) for a comprehensive survey in the topic of prompt learning in NLP.
5 CONCLUSION
We present CoOp, differentiable approach that focuses on continuous prompt learning to facil- itate the deployment of pre-trained vision-language models in downstream datasets. The results on the 11 datasets serve as strong evidence of CoOp's effectiveness data-efficient learning. The learned prompts are proved much more task-relevant than hand-crafted prompts reflected by the huge improvements in performance, as well as stronger in robustness to distribution shift. In terms of limitation, CoOp requires explicit access to the pre-trained model parameters, which might be unavailable when only the APIs of pre-trained models are provided. An interesting future direction thus to investigate "black-box" prompt learning.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffre Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write classifier: using purely textual descriptions. In ICCV, 2013.
Zero-shot learning
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In CVPR-W, 2004.
Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NeurlPS, 2013.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 15723, 2020.
Lluis Gomez, Yash Patel, Marçal Rusiñol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning visual features through embedding images into text topic spaces. In CVPR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of duitof-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lanion-l representation learning with noisy text supervision. ICMI., 2021.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? ACI, 2020.
10

---

Under review as a conference paper at ICLR 2022
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV-W, 2013.
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. ICCV, 2015.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104. 000900081,.... 2021.
Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten. Learning visual n-grams from web data. In ICCV, 2017.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv 2021.
preprint arXiv:2101.000
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint 11X.:22003,10386 2021b.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.514 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models knowledge bases? In EMNLP, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In JCML, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. EMNLP, 2020.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher Manning, and An- drew Y Ng. Zero-shot learning through cross-modal transfer. In NeurlPS, 2013.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12X1v:121..0400 2012.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification.. In NeurIPS, 2020.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In ECCV, 2020.
11

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurIPS, 2019.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.000 00747, 2020.
Zexuan Zhong. Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. In NAACL, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: survey, arXiv preprint arXiv:2 2021.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A DATASETS DETAILS
The detailed statistics the 11 datasets, as well as the four variants of ImageNet, are shown in Table 5. The hand-crafted prompts used for zero-shot CLIP also detailed in the table. For 101, the "BACKGROUND Google" and "Faces casy" classes are discarded. For the video dataset, UCF101, the middle frame of each video is used input the image encoder.
Table 5: Datasets statistics.
Dataset
Classes
Train
Val
Test
Hand-crafted prompt
ImageNet
1,000
1.28M
N/A
50,000
photo of [CLASS]."
Caltech101
100
4,128
1,649
2,465
"a "hoto of [CLASS]."
OxfordPets
37
2,944
736
3,669
"a photo of [CLASS). type of pet."
StanfordCars
196
6,509
1,635
8,041
"a photo of [CLASS]."
Flowers102
4,093
1,633
2,463
photo (CLASS), type of flower."
Food101
101
50,500
20,200
30,300
"a photo of [CLASS], a type food."
FGVCAircraft
100
3,334
3,333
3,333
"a photo of a (CLASS), a type of aircraft." "a photo of [CLASS]."
SUN397
397
15,880
3,970
19,850
DTD
47
2,820
1,128
1,692
"[CLASS] texture."
EuroSAT
13,500
5,400
8,100
centered satellite photo [CLASS]."
UCFI01
101
7,639
1,898
3,783
"a photo of a person doing (CLASSJ."
ImageNetV2
1,000
N/A
10,000
photo of [CLASSS].
ImageNet-Sketch
1,000
N/A
N/A
50,889
"a photo of [CLASS]."
ImageNet-A
200
N/A
N/A
7,500
"a "hhoo of [CLASS]."
ImageNet-R
200
N/A
N/A
30,000
"a photo of [CLASS]."
B ADDITIONAL RESULTS
B.1 ROBUSTNESS EXPERIMENTS
In addition to ResNet-50, further experiment with more advanced architectures including ResNet- 101, ViT-B/32 and ViT-B/16, all which have pre-trained weights available from CLIP's GitHub repository. The results shown in Table 6 where draw the same conclusion the main paper: CoOp offers stronger robustness than hand-crafted prompts and using fewer context tokens benefits domain generalization.
B.2 PROMPT ENGINEERING, PROMPT ENSEMBLING AND COOP
Table provides more comprehensive comparisons covering a variety vision backbones. The observations similar to those discussed in the main paper: prompt ensembling is clearly better than prompt engineering; and CoOp demonstrates consistent advantages over prompt ensembling.
B.3 CONTEXT LENGTH
Figure 6 shows detailed results of using different context lengths for CoOp on each of the 11 datasets. The average performance, displayed the top-left corner, suggests that using more context tokens is better. There three exceptions: ImageNet, OxfordPets, and Food101, the performance is saturated and the improvements diminish when the context length is increased. As discussed in the main paper, selecting proper length needs to balance between performance source datasets and robustness to distribution shift in unseen domains. We suggest practitioners use a validation set to identify the optimal context length for their applications.
13

---

ViT-B/16 Under review as a conference paper at ICLR 2022
Table 6: Comparison with zero-shot CLIP on robustness to distribution shift using different vision backbones. M: CoOp's 's context length.
Source
Target
Method
ImageNet
ImageNetV2
ImageNet- ImageNet-A
ImageNet-R
ResNet-50
Zero-Shot CLIP
48.08
31.67
18.63
53.45
CLIP + CoOp (M = 16)
60.46
52.17
31.14
19.62
53.31
CLIP CoOp (M =
60.85
53.02
32.99
20.69
55.57
ResNet-101
Zero-Shot CLIP CLIP + CoOp (M = 16)
58.72
51.57
36.73
25.11
62.15
64.39
55.00
37.54
26.31
61.73
CLIP + CoOp (M = 4)
63.99
55.45
39.11
27.25
63.58
ViT-B/32
Zero-Shot CLIP
51.98
39.22
27.44
63.79
CLIP + CoOp (M = 16)
64.92
55.90
38.79
28.77
63.45
CLIP + CoOp (M = 4)
64.88
56.21
40.17
29.64
64.60
VIT-B/16
Zero-Shot CLIP
64.71
58.71
44.77
43.37
72.49
CLIP + CoOp (M= = 16)
70.13
62.23
44.82
44.30
72.98
CLIP+ CoOp (M 4)
70.11
62.66
46.27
45.46
74.33
Table Comparison with prompt engineering and prompt ensembling on ImageNet using different vision backbones.
Method
ResNet-50
ResNet-101
ViT-B/32
Prompt engincering
55.41
58.72
59.88
64.71
Prompt ensembling
57.81
60.49
62.01
67.31
CoOp
60.46
64.39
64.92
70.13
B.4 VISION BACKBONES
Figure provides the detailed per-dataset results for various vision backbones. The more advanced the backbone, the better the performance.
14

---

Under review as a conference paper at ICLR 2022
Average over 11 datasets
Imageker
Caltech101
73.8
92.0
125
91.0
76.4
91.6
71.5
177
22A
912-
11.
703
10.0
CUP CoOp
54
Codp
31.3
Coop (mid)
15
36
Numberr conoot warls
Aurbbrr conost werls
Warter of contect words
12.5
71.3
14.0
probe
103
10.0-
+3.0
693
Tear SIaRe CUP
92.5
75-
69.5
00.
74
CLP
CIP Eodp (mvi))
00.0
CUP Colp
DIP Exop (963)
41.5
0P Cada (440
Coda (me))
67.
0
16
Number worts
Number
Foudi01
36-
01.0
520%
$7.0
22-
CLP Cody tend CUP Colpp (m)))
C.17 Cadp lend) D.P Codp (mit)
CUP Co-b (end)
/L.P
00.
Sanser proba
C.." Co0p (m/))
26
16
36
UCF101
63.2
probe
13.0
03.2
150
Stare
Lunar probe
Cx0s lendi CUP CoOp (mid)
CUP CoOp lend) CUP CoOp (ndi)
CUP Collo (and) Cafo (mid)
63.8
16
Nurbier werds
1G
Number contoot wonds
Figure 6: sptaset- results of using different context lengths for CoOp.
15

---

Under review as a conference paper at ICLR 2022
datasets
Cattech101
Zerv shat CUP
70
Zero CUP
7612
Ze1e shut
CUP + CoCp (M-15, and
CLI CoOp (M-15 end)
CoDp (4-15 and)
1522
64-
23.83
32.35
10-
C1.82
70
Score
20:00
E0:24
6C-
36-
55
WT-822
MVD0:
00-
CLP CuOp (Mw3k, undi
trdo (Me38, end
$1.85
Cofls (4
K0.21
75-
72.38
50-
70-
85-
01:57
$4.70
55+
25.40
70-
13.40
50-
22
M950
RML93
0v32
V7-8
RN5O
AW101
MIT WS52
W11-116
9850
NV001
VIT-0/22
Vil
FGVCAircrat
71.5
40-
24.20
36-53
35.48
68
11:04
Store
65.0-
73.29
145
16:27
$3.26
RNI21
VTT 0x32
VI--UE
RA50
RNI.83
VTT-
VIT-816
RAG
RM.61
VTT-
VIT-
DTD
EuroSAT
UCF101
Zarc-ahst
22.68
03
80.5-
Scare
50
65.5
22.72
991
31.53
20.70
$2.5
03.04
61.17
R0:00
BN223
VIT-6122
Vm-311
RA50
VIT-8/22
VIT-326
RAD2
RNC.1
VT.622
VI VTT-36
Figure 7: Results on the 11 datasets using a variety of vision backbones.
16


Reviews:
[{"review_id": "uLmfK84ybVF", "reviewer": "Reviewer_1WJt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1) The paper is generally well-written and easy to follow. The author conducts extensive experiments and ablation studies.\n2) CoOp outperforms CLIP on all 11 diverse datasets, especially showing large improvements in specific domains like texture and satellite images.\n3) CoOp is data-efficient and can boost the classification performance with only a few shots of training data.\n4) CoOp demonstrates better robustness to distribution shift than zero-shot CLIP and linear probe CLIP. \n\nWeakness: \n1) Though effective, the technical novelty of this paper is quite limited. Since the soft prompt tuning approaches (e.g., Prompt Tuning [1] and P-Tuning [2]) have already been proposed in NLP domain. And CoOp adopts a very similar technique.\n[1] Lester, Brian, Rami Al-Rfou, and Noah Constant. “The power of scale for parameter-efficient prompt tuning.” EMNLP 2021.\n[2] Liu, Xiao, et al. “GPT Understands, Too.” *arXiv preprint arXiv:2103.10385* (2021).\n\n2) According to Table 4 in the paper, the nearest neighbor words of the learned context vectors rarely have practical semantic meaning. This casts doubt on using the word “context”. From this perspective, these soft prompts are just more parameters to improve the model capacity.  Hence, can we say that CoOp is just a better version of fine-tuning? In other words, there might be a similar fine-tuning way to utilize additional parameters to get better performance.", "summary_of_the_paper": "The paper proposes context optimization (CoOp) which learns task-aware continuous prompts to improve CLIP in terms of few-shot image classification. By fixing the pretrained backbone, CoOp performs end-to-end learning to update the learnable context vectors for target domain datasets. The simple yet effective approach substantially beats hand-crafted prompts with a large margin. Meanwhile, CoOp also exhibits better robustness to distribution shift than CLIP.", "summary_of_the_review": "The paper achieves better few-shot image classification performance improvements but lacks enough technical novelty and explanations for learned prompts", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mo5m2Y33ekj", "reviewer": "Reviewer_AYDN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's motivation is clear, experiments thorough, and results\nconvincing. In addition to the standard few-shot evaluations, I\nappreciated the authors considering the distribution shift scenario:\nin that case, they show that their model can more effectively leverage\nlabelled data from a different dataset versus the linear probes (even\nthough zero-shot CLIP remains somewhat competitive in that\nregime). Additional experiments about the optimal context length,\nplacement of fine-tune-able token embeddings, and vision backbones\nwhere appreciated.\n\nWhile the CoOp method appears to work well, but I had some concerns\nabout novelty. In particular, this method is more-or-less identical to\nLi and Liang (2021) --- that paper came out on arXiv Jan 1 of this\nyear, and was published at ACL earlier this year. In addition, this\nidea has been \"rediscovered\" in the NLP context several times (which\nare also cited, but perhaps cannot be considered contemporaneous,\ngiven their arXiv dates...). While the authors cite the arXiv version,\nI think that, given the timing, the authors should perhaps not pitch\ntheir method as a \"novel\" approach (as is done in the abstract);\nrather, this seems to be applying prefix-tuning to CLIP.\n\nI had a few technical concerns:\n\n1. there are a lack of details for the linear probe --- how was the\n   regularization parameter chosen?\n   \n2. The baseline for comparison to CoOp was the linear probe, which\n   makes sense. However, I would have also liked to have seen a\n   comparison where all the parameters of CLIP are fine-tuned --- how\n   well does that work for few shot learning?\n\nI also had a few presentation concerns:\n\n1. Figure 1 compares a supervised CoOp method to a zero-shot CLIP\n   baseline. While the CoOp method is \"few shot\", in this figure,\n   there are 16 examples per class provided, which, for some datasets\n   may amount to hundreds or thousands of labelled examples. I would\n   have appreciated Figure 1 comparing to the linear probe.\n\n2. Figure 5b has a similar problem with being potentially misleading:\n   I would recommend including not only zero-shot CLIP, but also\n   linear probe CLIP.", "summary_of_the_paper": "The authors demonstrate a more efficient form of few-shot learning\nusing CLIP compared to linear probing for image classification: CoOp.\nInstead of fine-tuning a small linear classifier on the output of\nCLIP, they propose fine-tuning a number of additional embeddings at\nthe input layer; this modification, in theory, allows CLIP to leverage\nmore computation when adapting to tasks, while still only optimizing a\nsmall number of parameters. Experiments across several corpora\ndemonstrate the efficacy of the approach, which generally yields a\nfew accuracy points of gain versus a linear probe trained on the same\namount of data.", "summary_of_the_review": "Overall: the work is generally clear with thorough and convincing\nexperiments. While there were a few presentation/technical concerns,\nthe main drawback of this work is novelty: the proposed idea is\nidentical to Li and Liang (2021) [arxiv in january], Zhong et\nal. (2021) [arxiv in April], and perhaps Lester et al (2021) [arxiv in\nSep, this one is more recent and I haven't read it yet]. While these\npapers consider only the NLP case and not the vision+language case,\nit's still difficult to call this method \"novel,\" as the authors do in\nthe intro.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iQ0p4tsDGS", "reviewer": "Reviewer_pGwp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper provides a novel approach named CoOp for prompt engineering based on  CLIP. \n2. CLIP-based experiments are sufficient to prove the advantages of CoOp.\n\nWeaknesses:\n1. Only the results based on CLIP are compared, and there are no more experiments of other visual-language models.\n2. It is an improvement of CLIP, and the idea is similar to many existing works such as [1].\n[1] Prefix-Tuning: Optimizing Continuous Prompts for Generation\n", "summary_of_the_paper": "This paper proposes a novel approach named context optimization (CoOp) for prompt engineering of vision-language pre-training models.   The main idea is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. Experiments on 11 datasets show that CoOp effectively turns pre-trained vision-language models into data-efficient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin and able to gain significant improvements when using more shots.", "summary_of_the_review": "This is a somewhat novel but solid work. The comparative experiment based on clip is very sufficient, but it also lacks other important experiments, such as the effect of CoOp on other vision-language models, just as the title of the paper is learning to prompt for vision language models", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CwHrJmanoON", "reviewer": "Reviewer_KMdr", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Extensive experiments\n+ Motivation for the automatic prompt is useful\n\no Investment in Fig 1 is super useful. It is helpful to see how fragile the prompt is affected by word twisting. However, the paper claimed prompt engineering is time-consuming, which I do not disagree with. Yes, human annotators have no way to know whether adding a \"flower\" or changing the sentence order will improve or harm the performance. But the key motivation of the prompt is to reduce the human working load and adding some general description is not super complicated. The task should be down to exploring more robust models and algorithms that can withstand these minor changes in my opinion.\n\n- The biggest concern is about the lack of theoretical contribution. The main content of the methodology is broken down into two sections. The first section is just a review of CLIP which is already well-known. The key content of context optimisation in 2.2 is just half a page. The description is rough.\n\n- It is not super clear why adding extra dimensions (claimed as the same number in that of word embedding) to the class token can help the prompt.\n\n- On page 4 above Eq.3, the concept \"unified context\" and \"class-specific context\" are not well explained.\n\n- paragraph under Eq.3, \"Training is performed ...\" is redundant and not informative.\n\n- Figure 2 is not helpful in understanding the framework and looks very low-quality.", "summary_of_the_paper": "This paper proposes a context optimisation (CoOp) approach that can automate prompt engineering and allow more efficient and task-specific transfer for pretrained vision-language models.", "summary_of_the_review": "Despite extensive evaluation, the paper presents in very low quality and lack of description of key methods and the theory behind it. The overall quality is clearly below the threshold of the ICLR standard.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given the increasing scale of large models (e.g. CLIP), there's an argument that we need better automated techniques for properly utilizing (prompting) these models. Given the success of prompt learning within pure NLP models, the authors apply the same approach to the V+L domain and show that it also is applicable here.  Generally, reviewers felt that the results were clear and thorough, yet technically limited.  The approach is not novel and the result not surprising.  There is a documentary benefit to having this work out in the community for others to reference and extend.
Decision: Reject