Rethinking the Value of Prompt Learning for Vision-Language Models
Authors: Peisong Wang
Weihan Chen
Weixiang Xu
Qinghao Hu
Jian Cheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Prompt Tuning
Visual-Language Pre-training

Abstract: Large-scale visual-language pre-training like CLIP has demonstrated great success in open-set visual concept learning that enables zero-shot transfer to downstream tasks through prompting. To automate prompt engineering, prompt learning is proposed to automatically learn the optimal task-relevant prompts. In this paper, we make some surprising observations that contradict common beliefs about prompts. We observe that even random prompts can achieve pretty good performance for zero-shot recognition. We also find that prompt learning gives comparable or worse performance than directly fine-tuning of the linear classifier. Moreover, prompt learning is no more than parameter-efficient learning, and is a trade-off between optimality and generalization. Our results highlight the need for the rethinking of existing prompt learning, more careful baseline evaluations in future research on prompt learning methods in vision-language models. 

Under review as a conference paper at ICLR 2023
RETHINKING THE VALUE OF PROMPT LEARNING FOR VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale visual-language pre-training like CLIP has demonstrated great suc- cess open-set visual concept learning that enables zero-shot transfer down- stream tasks through prompting. To automate prompt engineering, prompt learn- ing is proposed to automatically learn the optimal task-relevant prompts. In this paper, we make some surprising observations that contradict common beliefs about prompts. We observe that even random prompts can achieve pretty good performance for zero-shot recognition. We also find that prompt learning gives comparable or worse performance than directly fine-tuning of the linear classifier. Moreover, prompt learning is trade-off between optimality and generalization. Our results highlight the need for the rethinking of existing prompt learning, more careful baseline evaluations in future research on prompt learning methods in vision- vision-language models.
than parameter-efficin
learning, and is a
INTRODUCTION
Building a state-of-the-art visual recognition system is one of the core tasks the field of computer vision. Current state-of-the-art visual recognition systems are almost all based Deep Neural Networks (DNNs), which can be roughly divided into two parts: a non-linear feature extractor and linear classifier. For traditional visual recognition, where the class number are fixed and the labels are discretized, the standard practice is to assign each category with weight vector, which optimized maximize the classification accuracy. Take the ResNet for ImageNet classification as an example, the weight vectors for 1000 classes form the weight matrix W E IR. 1000 4096 of the linear classifier (the last fully-connected layer of ResNet), where 4096 the dimension of the features from the feature extractor. This learning paradigm can only learn closed-set visual concepts related to the pre-defined categories, and can not generalize to new classes beyond these closed-set categories.
In contrast to supervised learning with fixed labels a closed-set categories, visual concept learn- ing with the supervision of text has shown great potential. The main inspiration is that language a high level abstraction of human understanding the world, thus it contains rich information and naturally generalize well. One of the representative works is the CL.IP (Contrastive Language- Image Pretraining) (Radford et al.) 2021), which learns joint representations using contrastive learning on large-scale image and text data. Thanks to the rich information and the generality of natural language, the CLIP model can learn diverse and task-agnostic visual-textual representations, which can be generalized to many downstream tasks even under the zero-shot set- ting. This is done by using the names of all classes of a downstream task as the text for textual feature extraction, and conducting classification based on the alignment score of the visual features and the textual features for each class. However, using the class names as the text is deficient due to the lack of context. To this end, the authors of Radford et al. (202T) resort to the technique of prompt taning (Liu et al.) [22 Here the "prompt" is cloze templates which specifies the context about the task at hand. They find that the template "a photo of (CLASS) is a good prompt for image classification. By using elaborate prompt engineering and ensemble, much higher zero-shot performance can be achieved.
vision and language
Prompt engineering has shown greater transferability than the contextless baseline of using class names. The drawback is that the handcrafted prompt tuning requires prior knowledge about the downstream task. Moreover, as pointed out in Zhou et al. (022b),, the performance is very sensitive

---

Under review as a conference paper at ICLR 2023
to slight change in the wording of the prompt template. Thus prompt tuning is non-trivial task. To solve this problem, the authors of Zhou et al. (2022b) bring the concept of prompt learning from natural language processing (NLP) and propose Context Optimization (CoOp) to automate the prompt engineering in vision-language models. More recent works including (Ju et al., 2021] Yao et al. 2022 Zhou al.) (222aa) are continually developed. The core idea of these prompt learning approaches is to treat the embeddings of the words in a prompt as a set of learnable vectors, which are learned through back proppaaapaatinn w.r.t. the downstream task loss.
Prompts can encode context information expressed in natural language about the target tasks, thus they can generalize well and show promising results even in zero-shot. Prompt learning, which au- tomatically optimize the prompts in the same word embedding space of natural language, is believed to have two advantages. First, it believed that prompt learning converge faster and requires fewer training examples than fine-tuning. This is because only the context vectors are updated while the pre-trained parameters of both text encoder and image encoder are fixed. Moreover, during the gradi- ents calculation, the pre-trained knowledge encoded in the text encoder can also be back-propage proppagaed through the network to the context vectors. Therefore, prompt learning is commonly believed to be superior to linear probe, partial fine-tuning even full fine-tuning. Second, it is believed that the learned prompts have strong robustness and generalization ability, as the optimization is conducted in the NLP embedding space, thus the learned prompts expected to provide high generalization ability in the same way as natural language.
In this paper, test these two beliefs by evaluating the prompt tuning/learning performance of CLIP on various downstream tasks. We start from examining the influence of text encoder on the prompts through handcrafted prompts and random prompts and show that the text encoder can indeed provide some regularization on the prompts. To our surprise, we find that even random prompts can still achieve pretty good performance for zero-shot recognition. Then, we compare prompt learning and fine-tuning for closed-set recognition, and observe that prompt learning gives comparable or worse performance than directly fine-tuning the weights of the linear classifier. Last, we examine the generalization ability of the learned prompts, and reveal that prompt learning is no more than parameter-efffcicnt learning, and is a trade-off between optimality and generalization.
RELATED WORKS
Prompt learning originally proposed to transfer knowledge from pre-trained language models to downstream tasks, which has demonstrated great performance in NLP domain Devlin et al. (2018);
Brown al. (2020). A typical example of prompt learning "fillin-the-lankk cloze templates Petroni et al. 2019), which transforms the down-stream task to a format familiar to the pre-trained model. Instead of manually designing prompt templates, later studies focus on automated prompt learning which can be categorized into discrete prompts and continuous prompts Liu et al. 1202 Researchers discover the discrete prompts in a discrete space, e.g. natural language phrases, and most works generate discrete prompts by either gradient- based search Wallace et al. (2019). or prompt mining Jiang et al.] (2020), or prompt generation Gao et al. (2020), etc. Instead of limiting the prompt to human-interpretable natural language domain, continuous prompts in the embedding space of the model are proposed. Several representative methods on continuous prompts learning include prefix tuning & Liang (002I). tuning initialized with discrete prompts [hhon al. 2222 and hard-soft prompt hybrid tuning Liu et al. 22221b)
Motivated by the well performance of prompt learning on NLP, recently researchers begin to apply it into the vision-language models. CLIP Radford et al. (021 uses a manually designed prompt on the text encoder, which enables the zero-shot image classification of vision-language model. To avoid human efforts on prompt design, CoOp Zhou et al. (2226)) proposes a continuous prompts learning method and two implementations that can be applied on different recognition tasks. Yet CoOp /hho et al. (2022b) seems over-fitting the base classes in the training, resulting in inferior performance on unseen classes even within the same dataset. To cure this problem, CoCoOp Zhou et al. (2022a) propose to generate an input-conditional vector for each image by lightweight neural network, which boosts the classifier performance on new classes. Although CoOp and CoCoOp achieve promising improvements, they requires supervised data from the target datasets which may restrict the model scalability. In the contrary, Huang et al. Huang et al. (2022) propose the unsuper- vised prompt learning (UPL.) method which improves transfer performance of CLIP--ike VI. models
2

---

Under review as a conference paper at ICLR 2023
without labeled data. Different from above prompt learning methods which apply the prompts the text encoder of VL model, VPT Jia et al. (2022) uses prompts learning on the image encoder. Specifically, they prepend small amount (less than 1% of model parameters) of trainable parame- ters into the input sequence of transformer layers, and keep the model backbone frozen. Besides the prompt learning for the classification task, there are some studies about transferring knowledge from VL models to other downstream tasks, such video understanding Ju et al. (2021). object detection Du al. (2022), and visual grounding Yao al. (2021).
PROMPT LEARNING BASED ON CLIP
The analysis throughout this paper is based CLIP model (Radford et al. 2021 which consists of an image encoder f()) and a text encoder g(-). The image encoder is usually a ResNet or a ViT, while the text encoder is Transformer. Through contrastive learning, the two encoders trained to transform input images and texts into the same feature space.
The aligned visual-textual feature space makes CLIP to be capable of zero-shot image recognition. Specifically, the input image æ is feed into the image encoder to obtain the visual representation f. Similarly, for each category, the class name (e.g.. "dog"" wrapped in the prompt template (e.g., "a photo of a (CLASS) ") is feed into the text encoder to obtain the textual representation [w: jim where is the class number. Then the prediction probability is as follows:
exp(cos(f. (f. wx)/T)
p(y = i = Σ/= exp(cos(w), wyzr /T)
(1)
where T is the Softmax temperature and cos(,, () denotes cosine similarity.
To ease the prompt engineering process, the concept of prompt learning is proposed. Context Op- timization (CoOp) (Zhou et al.) (022b) is one the earliest works that introduce prompt learning to adapt pre-trained vision-language models to downstream tasks. The key idea of prompt learning to automatically learn the prompt template instead of using a handcrafted template. Specifically, CoCop introduces set of learnable prompt vectors with the following format,
= [v]e-vle-.. ... [M]M H1Ma/OCCLAS |o]CLASS.
where [[v]m, m = 1, I.... M are the set the learnable word embeddings of the prompt template, which are shared for all classes, M is the number of context tokens of the prompt, and (vlCLASS the embedding of the class name. For each class í E 1,.... K, we can obtain the prompt t; according to Eq. ( Then the prediction probability is as follows:
exp(cos(f. gt.)))
ply = i|æ) = Σ/= (f
(3)
With the prediction probability of Eq. (3) and the classification loss of the downstream task, we can optimize the learnable prompt vectors {[0mm m 1..... .M} while frozening the pre-trained weights of the CLIP. Specifically, the gradients w.r.t. the learnable prompt vectors can be back- propagated the way through the text encoder 9(-), at which time, the pre-trained knowledge encoded in the text encoder of CLIP can be distilled to the learnable prompt vectors. In this way, the learned prompts can encode some useful information about the downstream task.
4 TEXT ENCODER AS A REGULARIZATION
In section 3 we show that different prompts would result in different weights for classification, i.e., g(t;), for = 1, K. In this section, examine how different prompts would affect the performance of classification across a wide range of datasets.
4.1 HANDCRAFTED PROMPTS
We start from handerafted prompts. Previous works have shown that good handerafted prompt could greatly improve zero-shot classification accuracy Radford et al. (202T). ). this section, we
evaluate the performance of zero-shot classification with various handcrafted prompt templates.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Various handcrafted prompts.
Prompt Type
Prompt Template
ClassName
"I "(CLASS)" "a photo of a (CLASS) "this is a photo of a (CLASS)."
Basic
Revised
Negative
"this is not photo of a (CLASS)."
Table Results of zero-shot classification with various handcrafted prompts and random prompts across the 11 datasets.
Dataset
ClassName
Basic
Revised
Negative
RandToken
RandEmbed
ImageNet
55.3
58.2
59.2
58.0
50.0=2.7
47.0 47.0+2.8
10
80.9
85.9
87.0
84.6
75.3
74.7
OxfordPets
78.8
83.7
84.7
81.0
73.2=3.6
72.313.4
StanfordCars
54.4
55.6
55.6
49.0
52.6=0.9
16.5+2.0
Flowers102
57.3
60.9
62.8
62.7
51.4=2.6
46. 46.747.6
Food101
73.9
75.3
77.2
75.5
72.8
69 69.0+2.8
FGVCAircraft
15.3
15.7
15.8
15.0
12.2=2.5
10.212.1
SUN397
54.9
58.5
57.7
58.5
47.623.3
45.012.4
DTD
41.1
40.0
42.3
34.1
28.613.4
EuroSAT
28.4
24.2
28.0
36.7
22.4
24.2
UCFI01
56.4
58.3
57.9
57.7
52.5-2.4
47.942.2
First, we want to evaluate the prompts used by current prompt-based methods. The simplest base- line to directly use the class name as the input of the text encoder (ClassName). To improve performance, the authors of CLIP Radford et al. (2021) propose basic prompt template for image [adPoRdfoord
recognition (Basic). The authors of Ju et al. (2021) further revise prompts by adding a "this is" prefix (Revised). Second, we want to examine what will happen if we use the "negative prompt" for the classification task, i.e., by adding a "not" in the prompt template (Negative). For clearness, we summarize the handcrafted templates studied in this section in Table [
Following (Zhou et al. 2022b), we conduct experiments on the 11 image classification datasets used in CLIP which are publicly available, i.e., ImageNet (Deng al.) 2009). Caltech101 (Fii-Fei)
2004), OxfordPets (Parkhi et al. 2012), StanfordCars (Krause et al. 2013). Flowers
(Nilsback
& Zisserman 2008). Food101 (Bossard al., 2014), FGVCAircraft (Maji al.) 2013). SUN397
Xiao et al. 2010). DTD (Cimpoi et al. (2014), EuroSAT (Helber et al. 2019) and UCF101 (Soomro
al.
2012 The results summarized in Table 2
As expected, the results in Table 2 show that the ClassName prompt template achieves the worst performance. On only one dataset (EuroSAT), the ClassName could outperforms the Basic and the Revised prompt. There is a large accuracy improvement using the Basic prompt template introduced in Radfooo et (2021). The accuracy can be further improved when using the Revised template (Ju et al. 2021). which achieves the best accuracy for 8 out of the 11 datasets.
An important finding we want to point out that the relative performance for these three hand- crafted prompts are somewhat consistent across various dataset. Specifically, the Basic template outperforms the ClassName prompt for out of the datasets, and the Revised template outper- forms the Basic template for other 9 out of the 11 datasets. This finding indicates that the handcrafted prompts expressed natural language can generalize well across various datasets. In other word, a prompt template works
dataset has high probability also work another dataset.
Another surprising observation is that the negative prompt can also achieve very good performance. On most of the datasets, the Negative prompt shows quit large improvement over the ClassName template. By comparing with the other handerafted prompts, it even get the best accuracy on out of the 11 datasets.

---

Under review as a conference paper at ICLR 2023
4.2 RANDOM PROMPTS
Then we evaluate what would happen if random prompts are used. Here we consider two kinds of random prompis, namely the random token template and the random embedding template:
Random Token Template prepends some random word IDs which are selected from the 49,152 vocabulary of CLIP.
Random Embedding Template prepends some random embedding vectors after trans- forming token IDs into word embedding.
Similarly, we evaluate the performance across 11 image classification datasets. Note that the selec- tion of the random seed would make large difference in performance. So we run the experiments for 10 times with different random seeds and report the average performance. The results are shown in the last columns of Table 2 To our surprise, the results show that even random prompts still achieve pretty good performance for zero-shot recognition. It seems that the text encoder can indeed provide some regularization the encoding process of the input text with prompts.
4.3 SUMMARY
By now, we have evaluated the performance of different handcrafted prompts and random prompts. In the following list, we summarize our findings, some of which are quit surprising.
1. Handerafted prompts expressed in natural language show great power and generalization ability, and the relative performance for different prompt templates are somewhat consistent across wide range of datasets.
2. The negative prompts, which provide wrong context information for the downstream tasks, can also achieve very good performance.
3. Even random prompts can still achieve pretty good performance for zero-shot recognition.
PROMPT LEARNING V.S. CLASSIFIER FINE-TUNING
In this section, we evaluate the effect of prompt learning for closed-set image classification task, where the training set and testing set are from the same categories. For closed-set classification, widely believed that prompt learning converge faster and requires fewer training examples than fine-tuning. The reasons are two-fold. First, in prompt learning, only the prompt vectors are learned while the pre-trained CLIP model is fixed. Second, as shown in Eq. 3 to optimize the prompt vectors {(v) m: m = 1.... M). the gradients need to be back-propagated all the way through the text encoder g(-). This process allows the knowledge learned by the CLIP model be distilled from the weights to the prompts (Zhou et al. 2022b
In this paper, we challenge this common belief. By comparing Eq, D and Eq. 3 is easy to see that w; g(t,) for 1, .... K, which can be viewed the weights of the last classifier. At inference time, we first need to feed each t; into the text encoder to obtain w, then the generated weights (w,, 1,... K} are used for classification. Our key question is that if we could directly optimize wi, why should we optimize the latent vector t₂? To this end, we conduct comprehensive experiments the 11 image classification datasets to classifier fine-tuning (optimize w:).
if prompt learning (optimize t.)) is superior
5.1 TRAINING DETAILS
For prompt learning, we utilize the Context Optimization (CoOp) method proposed in Zhou et al. (20226), which is one of the earliest works that introduce prompt learning to adapt traitedined vision-language models to downstream tasks. We use the CLIP pre-trained model with ResNet-50 as the image encoder. The number of learnable prompt vectors M is set to 16 and are shared across call categories, which the default setting for in the original paper of Zhou et al. (2022b). During find that the results for prompt learning on downstream tasks are very sensitive
the experiments, hyper-parameters. parameters. For fair comparison, we use the grid search over the training to the choice of hyper- epochs and learning-rate and report the best accuracy for all experiments. Specifically, the number
5

---

Under review as a conference paper at ICLR 2023
Average over 11 datasets
Imagehet
Cahechu01
hame: Merring
horpllantiy Unear 200s C
Promgi Leuring
Lrear pros CaP
Uiear proce CkP
TO-
Rumberel abdke traiing corntees clas
Mumber lodded cuinre clm
Sumberd of londed/oiiing tramtl
Octondhiss
StanforsCas
Flwwesc2
Scon
G.P+ Aang Mariig
Harptuartsy Lnaw ardee LL
Pronge Lauring
Linar prise car
prcbe LLP
Number sbale trainin sconstas dass
Sumber lordeddiiing per
reveurcnn
SUN499
25-
12
68-
OP Aare tearen
Aarep Loartg Lnna ander TLP
scanplas dass
DTD
BunsST
UCP101
GP+ Nomg: Marring
CJl PoYpLuarrig
OUP Prongi Leuring
36
4/1 2
15
15
Comparison between prompt learning (CLIP+ Prompt Learning) and classifier fine-tuning (CLIP+ + Classifier FT) of few-shot learning on the 11 datasets. The results for zero-shot CLIP (stars) and linear probe (dashed lines) are also given.
Figure 1:
of training epoch is set to (50, 100, 200). The starting learning-rate is set to {2e 2, 33 2e 4] 2 2, 2e- 3) for prompt learning and classifier fine-tuning, which are the best choices and {2c- 12 2,
for both cases. Other hyper-parameters are the same as in Zhou et al. (2022b).
5.2 RESULTS
We compare the performance of prompt learning and classifier fine-tuning across all 11 datasets. Following Zhou Zhou et al.) (2022b), we report few-shot learnigg results with 1/2/4/8/16 shots for training while using the original test set for testing. All the results for prompt learning and classifier fine- tuning are the average over runs with different random seeds. We also give the zero-shot results using CLIP model as well as the linear probe results based on CLIP for comparison. The main results are shown in Figure [
From Figure we can see that both the prompt learning and classifier fine-tuning can dramatically outperforms the zero-shot and the linear probe based on CLIP for most of experiments. Compared with prompt learning, the simple classifier fine-tuning cal obtain much higher accuracy, except for the OxfordPets dataset, on which the classifier fine- tuning is slightly inferior to prompt learning. The average results 11 datasets displayed in the top-left corner of Figure I show that classifier

---

Under review as a conference paper at ICLR 2023
Table 3: The optimization time (in minutes) of prompt learning and classifier fine-tuning with vari- ous backbones for 16 shots ImageNet classification.
Method
ResNet-50
ResNet-101
ViT-B/32
ViT-B/16
Prompt Learning 139.5
143.3
136.3
143.1
Classifier Finetuning
13.8
15.9
13.7
14.8
Table 4: Comparison between prompt learning and classifier fine-tuning on robustness to distribution shift using different vision backbones. Bold value indicates the best result.
Source
Target
Method
Average
ImageNet
-V2
-Sketch
-A
-R
ResNet-50
Zero-Shot CLIP
58.18
51.34
33.32
21.65
56.00
44.10
Linear Probe CLIP
55.87
45.97
19.07
12.74
34.86
33.70
Prompt Learning
63.00
55.27
34.03
22.40
55.90
46.12
Classifier Fine-tuning
64.73
56.03
34.13
22.10
58.30
47.06
ResNet-101
Zero-Shot CLIP
61.62
54.81
38.71
28.05
64.38
49.51
Linear Probe CLIP
59.75
50.05
26.80
19.44
47.19
40.65
Prompt Learning
66.53
58.73
40.00
29.00
64.03
51.66
Classifier Fine-tuning
67.50
58.60
40.37
28.90
64.20
51.91
ViT-B/32
Zero-Shot CLIP
62.05
54.79
40.82
29.57
65.99
50.64
Linear Probe CLIP
59.58
49.73
28.06
19.67
47.20
40.85
Prompt Learning
66.80
58.43
40.97
31.30
65.33
52.57
Classifier Fine-tuning
68.10
58.17
41.47
30.63
67.60
53.19
ViT-B/16
Zero-Shot CLIP
66.73
60.83
46.15
47.77
73.96
59.09
Linear Probe CLIP
65.85
56.26
34.77
35.68
58.43
50.20
Prompt Learning
71.97
64.40
47.97
49.97
75.03
61.86
Classifier Fine-tuning
72.97
64.47
47.97
48.50
75.70
61.92
fine-tuning is not vulnerable we think in the few shot setting. On the contrary, for 1 shot to 16 shots, classifier fine-tuning consistently outperforms prompt learning by 1.5% to 2.3%.
Another difference between prompt learning and classifier fine-tuning is about the optimization time. Prompt learning need to propagate the gradients through the text encoder back to the learnable prompt vectors, which is quit time-consuming, By contrast. the classifier fine-tuning optimize the weights of the classifier, thus it is much more efficient. Here we report the optimization time of prompt learning and classifier fine-tuning with various network backbones for 16 shots classification on ImageNet. The results are shown in Table 3 We can observe that classifier fine- tuning about 10x more efficient in speed than prompt learning.
directly
These results have confirmed our assumption about prompt learning for closed-set classification. Specifically, prompt learning can not achieve the goal of sample efficient training as commonly expected. The simple baseline of classifier fine-tuning is much time efficient and achieves much higher accuracy than prompt learning across various datasets.
5.3 ROBUSTNESS TO DISTRIBUTION SHIFTS
Previous works have shown that prompt learning has high domain generalization ability compared with handcrafted prompts. Thus we need to compare the robustness classifier fine-tuning and prompt learning with respect to distribution shifts across domains. To this end and following Zhou

---

Under review as a conference paper at ICLR 2023
et al. 2022b), we use the ImageNet the source domain and use the ImageNetV2 (Recht al. 2019), ImageNet- Sketchh (Wang et al. 2019). ImageNet-A (Hendrycks et al. (021b). and ImageNet- R (Hendrycks et al. 2021a) as the target domain. These datasets have the compatible class names with ImageNet, thus the optimized prompts of prompt learning and the learned weights of classifier fine-tuning can be transfered from ImageNet to these datasets.
We summarize the results in Table As we can see, despite exposure to the source dataset, both prompt learning and classifier fine-tuning outperform the zero-shot and linear probe CLIP for the target datasets, which demonstrates their strong robustness to distribution shift. Moreover, classifier fine-tuning surpasses prompt learning on most models and datasets, verifying the generalization advantage of classifier fine-tuning over prompt learning.
OPTIMALITY-GENERALLZAATT TRADE-OFF
In this section, we examine the generalization ability of prompt learning method. It is usually be- lieved that the learned prompts have strong robustmess and generalization ability, as the optimization is conducted in the NLP embedding space, thus the learned prompts are expected to provide high generalization ability in the same way as natural language. However, as pointed out in ZZou et al. (222aa), the prompt learning method used in Zhou et al. (2022b) fails to learn task-specific context that generalizes well unseen classes. To solve this problem, conditional prompt learning (Con- ditional Context Optimization, CoCoOp) is proposed in Zhou et al. (2022a), in which the prompts are the outputs meta network with the visual features of each image as the inputs. In this paper, want ask the question, why would the prompt learning methods the improved conditional prompt learning methods have strong generalization ability?
Before answering this question, we would like to take one step back and to think where does the generalization ability of a machine learning model comes from. Here we summarize three sources of the generalization ability of machine learning models.
1. Knowledge. Human knowledge is general. As high level abstraction of human knowl- edge, natural language also has strong generalization power. The pre-trained language model and prompt engineering some examples.
2. Inductive bias. During model design, experts can add some biases into the model based on the prior knowledge about the tasks to deal with. An example is the translation invariance of convolutional networks.
3. Diverse training data. The most simple and widely used method to improve generalization is to use large scale diverse data. The CLIP model an example.
Despite that the learnable prompt vectors are optimized in the same word embedding space as NLP, the learned vectors are not natural language. Thus the prompt learning method can not generalize well to new categories that are not seen during training Zhou et al. (2022a). Thus conditional prompt learning is proposed which has shown high generalization ability than prompt learning. The question is where does the generalization power come from? Does it come from the introduced inductive bias or from the diverse training data? The first case (inductive bias) hard to verify. Thus we will examine if the generalization power of conditional prompt learning comes from the optimization process.
Our assumption is that, due to the changed architecture, the improved prompt learning method may be actually trying find a better learned on the source dataset, then they will have poor generalization power on the target dataset. As trade-off, the better the prompt learning fit the source dataset (optimality). the weaker general- ization power it would have.
trade-off. If the prompt vectors are
To verify our assumption, we conduct experiments using the original prompt learning method (CoOp). To find better trade-off between optimality and generalization, we use a very simple method. Specifically, we train the CoOp model for various training epochs and compare the perfor- mance with conditional prompt learning method (we use the CoCoOp method in this paper). We use the same setting as in Zhou et al. (2022a) for all the experiments. The results are summarized in Figure
8

---

Under review as a conference paper at ICLR 2023
Figure 2 shows that by controlling the optimality- generalization trade-off, the original prompt learn- ing method (CoOp) can achieve higher generalization ability than conditional prompt learning method (CoCoOp) for most of the case. It seems that the learned prompts (either learned with or without conditional) are not related with natural languages. They are just some parameters which make parameter-effiiient fine-tuning possible. All need is to find better optimality- generalization trade-off. These results highlight the need for the rethinking of existing prompt learning, and more careful baseline evaluations metries are needed future research on prompt learning methods in vision-language models.
Average ever datasets
Imagehit
Caltechica
Nea www.Ao...o.
Codo
0.F
CLP
CLT
decuress
OutersPets
StarfondCrrs
Tlowerst02
Ortele
Oocete
CLF
Daxe Accurty (1)
Dase Accumt
Dass Acciney (h))
PGVCArouwt
SUN0
Acco ACC PM
Cocode
05.0
165
32
DTD
500
Now AECARDO
0:0g
Cofo Cotade
Cocedd
0LP
(10
20
Figure 2: The trade-off between optimality and generalization.
7 CONCLUSION
This paper rethinks the existing prompt learning, making some surprising observations that con- tradict common belief's about the prompt. First, we find that random prompts without fine-grained design or learning can also perform well in zero-shot recognition. Second, directly fine-tuning the linear classifier exhibits better performance than prompt learning. Moreover, we reveal that prompt learning is just special case of parameter-efficien learning, and is trade-off between optimality and generalization. Our results on 11 datasets highlight the rethinking in this paper can further boost the deployment of pre-trained vision-language models in downstream tasks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446 461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 101118810000555 2018.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
Li Fei-Fei. Learning generative visual models from few training examples. In Workshop Generative-Modee Based Vision, IEEE Proc. CVPR, 2004, 2004.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:20/2. 15723, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222) 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262 15271, 2021b.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language mod- els. arXiv preprint 2rXiv:2204.03644, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, arXiv preprint arXiv:2203. 12119, 2022.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguisties, 8:423-438, 2020.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. arXiv preprint urXiv:21 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00/90, 2021.
10

---

Under review as a conference paper at ICLR 2023
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586, 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, arXiv preprint arXiv:2103. 10385, 2021b.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:/306.51: 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 1909.01066, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021.
Benjamin Recht, Rebecca Roelofs,
Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 12 2012.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do nlp models know numbers? probing numeracy in embeddings. arXiv preprint arXiv:1909.07940 2019.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing, Learning robust global representa- tions by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Lcrge-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-344444 3492. IEEE, 2010.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre- pre-trained vision- vision-language models. arXiv preprint (i.iv:2109.119999 2021.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing [mask]: Learning vs. learning to recall. arXiv preprint 2rXiv:2104.0522. 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- Lasiuu-annguage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 16816-16825 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, (30(9))2337-2344) 2022b.
11


Reviews:
[{"review_id": "clSoLn17p-r", "reviewer": "Reviewer_2dEp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper offers a set of experiments around prompting methods for contrastive vision-language models.\n\nIn particular, the following experiments are presented:\n\n1) Handcrafted & random prompts.\nA number of simple templates are used to do image classification on 11 datasets on top of a CLIP pre-trained model. In addition, two types of random prompts are included in the comparison. The conclusion is that handcrafted prompts do well --some better than others--, even those that oppose our language logic (i.e. \"this is not a photo of\") seem to do a fairly decent job. Random prompts are inferior, but still offer non-trivial (sometimes even strong) zero-shot performance.\n\n2) Prompt learning versus Classifier Fine-tuning.\nMoving away from zero-shot evaluation into the realm of supervised tasks, the paper compares prompt learning (adding learnable parameters for the embeddings of the prompt tokens + finetuning only those) and classifier fine-tuning. The latter, if I understood it correctly, rather than learning the original embeddings for the prompting tokens, directly learns the final representation for each possible class. These will be compared against the final embedding for the input image, and pick the closest class as usual in contrastive learning. Basically, the pre-trained language tower is ignored here. The experiment focuses on a few-shot regime (between 1-shot and 16-shot). It concludes that classifier fine-tuning is almost always superior to prompt learning, and also computationally cheaper / faster.\n\n3) Robustness to Distribution Shifts.\nFollowing up on 2), prompt learning and classifier fine-tuning are evaluated in terms of robustness. Table 4 shows the results of finetuning on ImageNet and then evaluating on other related (out-of-distribution) ImageNet datasets, for a variety of vision towers. While it is not reasonable to expect Zero-shot CLIP to perform as well as finetuned methods, we see that the classifier fine-tuning overall works best (more robust, in this case).\n\n4) Optimality-Generalization Trade-off.\nThis one I didn't fully get. It seems to me one model is trained with CoOp for varying amounts of epochs, and results show that it dominates (in terms of original finetuning and out of distribution evaluation) a new technique (CoCoOp) that uses an auxiliary network.", "summary_of_the_review": "The paper offers four interesting experiments around prompting (handcrafted, learned, finetuning \"final class tokens\", robustness, etc). While promising, I feel the paper still requires some work, including a more cohesive flow, and a bit more structured and unified takeaways.\n", "strength_and_weaknesses": "Overall, the paper seems to be a set of interesting experiments, but with no clear takeaway. While I definitely think there's value in these experiments, I don't think the work is at the publication maturity level required by ICLR at the moment.\n\nQuestions.\n\n- I don't think Linear Probe CLIP is formally defined or explained anywhere. What's the exact method here?\n- In Table 4, what's the amount of few-shot examples per class used for finetuning?\n- I think the classifier finetuning method isn't clearly explained, I was confused during a first reading. Maybe add a diagram or an example.\n- I really struggled to understand Section 6. I think some major re-writing and experiment explanation is needed here.\n- Figure 2 could benefit from a descriptive caption.\n\n- Is the classifier finetuning method a new contribution? Or has this idea been tried elsewhere before? If it's a contribution, it should be made clear (and maybe even re-arrange the story and sections of the paper around it).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CvwV6F87Yky", "reviewer": "Reviewer_4WBY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, prompt learning is reexamined, and several unexpected findings that defy accepted notions of the prompt are presented. First , random prompts without learning or fine-grained design may likewise function effectively in zero-shot recognition. Second, direct linear classifier fine-tuning performs more effectively than prompt learning. Furthermore, prompt learning is essentially a subset of parameter-efficient learning and represents a trade-off between generalization and optimality.  Findings across 11 datasets show that the approach presented in this research can significantly influnce the use of trained vision-language models in subsequent challenges.", "summary_of_the_review": "Overall, this paper provides some valuable hints about prompts but the novelty is limited. Therefore, I give my initial rating as borderline \n", "strength_and_weaknesses": "Strength\n1. The experiments are sufficient and exhaustive.\n2. The results provide the valuable hints that what's the better way to deploy the pretrained vision-language model.\n3. The paper inspire people to think about more effective prompt design.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2RKD6wPOZK4", "reviewer": "Reviewer_pGd5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes the behavior of CLIP towards various compositions of hand-crafted text prompts and puts out some observations. One of the key observations of the paper is the behavior of CLIP to negative prompts, in which the authors insert a “not” word to the prompt, e.g. “this is not a photo of a dog”. The paper shows that the zero-shot classification performance of CLIP is almost the same using negative prompts as using default prompts without the “not” word included. The paper also shows that the use of random tokens appended with the class name as prompts also give considerable performance. In the second part of the paper, the authors propose classifier fine-tuning in which they fine-tune the final layer of the text encoder instead of learning the prompts. The authors argue that this tackles the speed issues of prompt learning by not needing to backpropagate gradients through the whole of the CLIP model and gets better classification performance. Experiments include zero-shot classification using various hand crafted and random prompts, few-shot classification, robustness to distribution shifts and studies on optimality and generalization. Results are shown on 11 datasets.", "summary_of_the_review": "All of my concerns are mentioned in the weaknesses section. I believe it is important to have clarity and proper experimental analysis which the paper lacks. I believe the paper needs to go through multiple iterations of revisions and rethinking on how to properly analyze and convey the findings for being fit for a publication at a venue like ICLR.", "strength_and_weaknesses": "Strengths:\n\n* The observations shown in the paper regarding the performance with various types of handcrafted prompts and their compositional variations is interesting.\n\nWeaknesses/Questions/Suggestions:\n\n* The experiments are only on the CLIP model, while the paper title says Vision and Language models.\n* No parameter count included for classifier fine-tuning experiments. It is very much important to keep a track of the parameter count while proposing an alternative method for prompt tuning. An increase in parameters can increase the classification performance. How many parameters are tuned for classifier fine-tuning? By what number is it larger than CoOp? The parameter count needs to be the same for both the methods for an apples to apples comparison.\n* What is the experimental setup for Figure-2? What are the epoch values?\n* What is the justification for the experiment in Figure-2? Why is CoCoOp not run for multiple epochs?\n* Experimental comparison with different prior works on prompt tuning is missing. Some examples include: Unsupervised Prompt Learning (UPL), Prompt Distribution Learning (ProDA), etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0sIVFi0b3ED", "reviewer": "Reviewer_Tvm8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on analyzing the prompt learning paradigm in CLIP. The authors discuss some observations on the use of various types of hand-crafted prompts including class names, basic prompts (e.g., \"a photo of a {CLASS}\"), negative prompts (e.g., “this is not a photo of a {CLASS}”), and random prompts on 11 benchmark datasets. The highlighted observations include no major drop in performance using negative prompts. Additionally, the paper analyzes classifier fine-tuning as a faster alternative to prompt learning and shows superior performance compared to prompt learning. Experiments on few-shot classification, robustness to distribution shifts, and generalization to unseen classes are included using the CLIP model as the backbone.", "summary_of_the_review": "I think the identified problem is important but I’d like to rate the current submission as a clear rejection due to limited technical contributions and lack of convincing experiments. The paper needs significant changes including new experiments and possibly methodological improvements before being accepted to any major conference.", "strength_and_weaknesses": "**Strengths**:\n\n- The paper is well written and easy to follow.\n- The paper has tried towards analyzing the effect of different compositional handcrafted prompts and studying the performance behavior on several downstream classification datasets.\n\n**Weaknesses**:\n\n- Although the paper states “Vision-Language Models” in its title, the experiments are only performed on CLIP models. It would be great to see similar findings for other vision-language models like DeCLIP, FILIP, CLOOB, CyCLIP, etc. Given that the paper is more of an analysis paper instead of a methodology paper, I would expect authors to verify their claims on other CLIP models like CLIP ViT variants beside CLIP ResNet. Does the size of models effect the conclusions presented in this paper? A more thorough comparison and anlysis should be included in the paper.\n- It is interesting to observe that the four types of handcrafted templates used for zero-shot classification provide accuracies in a similar range, but this requires further extensive analysis to draw concrete conclusions. The {CLASS} token is very much important in classifying the images and is present in all the templates which provides the major information to the text encoder. The negative prompt has all the words the same as the revised prompt except “not”, obtaining accuracies in the similar range is expected as the text encoder is a language model and they lack compositionality which is well studied in NLP [a], and also the way CLIP has been pre-trained using the contrastive loss doesn’t guarantee compositionality. For the random prompts using random tokens and embeddings, the same argument of the presence of the {CLASS} token applies. Thus, a fair comparison between different prompts would be to remove the class token and then compare their performance on the downstream tasks. It might require some modifications at the output as done in \"Learning to Decompose Visual Features with Latent Textual Prompts\".\n- The paper proposes classifier fine-tuning as a faster alternative to prompt learning and shows an average improvement of around 1.5%, but there is no parameter analysis (i.e., number of tunable parameters) provided for this experiment. It is known from the CoOp paper [b] that ensembling improves the performance, I assume that the authors fine-tuned the final projection layer of the text encoder which is of dimension 512x1024 for CLIP-ResNet-50, which has almost 64 times more parameters than learning 16 prompt tokens of 512 dimension. Therefore, it is very much essential to have a fair comparison with CoOp wrt the number of parameters each method uses. A fair comparison would be to use 64 learnable prompts and run CoOp for it. But, I believe from observing the results provided in CoOp [b], only 8 learnable prompts would be able to perform as good as the classifier fine-tuning. This argument applies to all the results in Figure-1 and Table-4.\n- Additionally, the very motivation to learn continuous prompts for large vision and language models was to efficiently adapt the knowledge in the models to downstream tasks, which makes classifier fine-tuning contrasting to the motivation of prompt learning. I would like the authors to discuss on how classifier finetuning is a better alternative to prompt tuning for parameter efficient adaptation of large vision-language models for diverse downstream tasks.\n- It would be good to add the average values across the datasets in table-2.\n- No experimental comparison with other parameter efficient adaptation methods for CLIP, like CLIP-adapter [c], Tip-Adapter [d], UPL [e], PDL [f] etc. has been made in the paper. Authors should compare with these methods to verify the effectiveness of the proposed method over existing methods.\n- Figure-2 is not clear. What are the epoch values corresponding to the points for CoOp?\n- I feel that the section on optimality-generalization trade-off is very subjective and lacks proper experimental evaluation and theoretical support. In the beginning the authors ask the question - “why would the prompt learning methods or the improved conditional prompt learning methods have strong generalization ability?”, but get an answer - “Our assumption is that, due to the changed architecture, the improved prompt learning method maybe actually trying to find a better optimality-generalization trade-off.” I am confused on what it means. Additionally, I am not sure how the experiment of training CoOp for multiple epochs helps support the claims. Isn’t it known that training for a higher number of epochs makes networks susceptible to overfitting and is expected to lose generalizability? Also, why was only CoOp run for multiple epochs and not CoCoOp?\n\n[a] Evaluating Compositionality of Sentence Representation Models: https://aclanthology.org/2020.repl4nlp-1.22.pdf\n\n[b] Learning to Prompt for Vision-Language Models: https://arxiv.org/pdf/2109.01134.pdf\n\n[c] CLIP-Adapter: Better Vision-Language Models with Feature Adapters: https://arxiv.org/pdf/2110.04544.pdf\n\n[d] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling: https://arxiv.org/pdf/2207.09519.pdf\n\n[e] Unsupervised Prompt Learning for Vision-Language Models: https://arxiv.org/pdf/2204.03649.pdf\n\n[f] Prompt Distribution Learning: https://arxiv.org/pdf/2205.03340.pdf\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject