Prompt Tuning with Prompt-aligned Gradient for Vision-Language Models 
Authors: Beier Zhu
Yulei Niu
Yucheng Han
Yue Wu
Hanwang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: prompt tuning
vision-language models
CLIP

Abstract: Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by ``prompt'', e.g., using the model provided similarity measure between an image and the prompt sentence ``$\texttt{a photo of a [CLASS]}$'', as the confidence score of predicting the image is ``$\texttt{[CLASS]}$''. Therefore, prompt shows a great potential for fast adapting the VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed $\texttt{ProGrad}$, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, $\texttt{ProGrad}$ only updates the prompt whose gradient is aligned (or non-conflicting) to the ``general direction'', which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of $\texttt{ProGrad}$ over state-of-the-art prompt tuning methods. Codes are in Appendix.

Under review as a conference paper at ICLR 2023
PROMPT TUNING WITH PROMPT-ALIGNED GRADIENT FOR VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Thanks to the large pre-trained rision-language models (VLMs) like CLIP (Radford et al.] 2021), we can craft a zero-shot classifier by discrete prompt design, e.g., the confidence score of an image being " "[CLASS]" " can be obtained by using the VLM provided similarity between the image and the prompt sentence "a photo of [CLASS]". Furthermore, prompting shows great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the soft prompts with few samples. However, we find a common failure that improper fine-tuning or learning with extremely few-shot samples may even under-perform the zero- shot prediction. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific prompting. In this paper, we present Prompt- aligned Gradient, dubbed P ProGrad to prevent prompt tuning from forgetting the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the general knowledge, which is represented as the optimization direction offered by the pre-defined prompt predictions. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes and theoretical proof are in Appendix.
INTRODUCTION
After seeing and reading countless image-text association pairs, large and deep vision-language models (VI.M) (Radford al. 2021 Jia et al. 202T) can memorize the general knowledge ka.k.a. encyclopedic knowledge) about what visual patterns correspond to what textual sequence and vice versa. Thanks to the powerful language modeling of VLMs, we can establish a communication channel in human-readable natural language, i.e., prompt (Liu et al. 202 la 2022), to query the general knowledge. Prompting bridges the interface gap between the preitrdined and downstream tasks (e.g., regression classification) without the need for additional fine-tuning adaptation. For example, we can craft a concrete prompt-"a photo of [CLASS] ""tto to achieve zero-shot image classification: by using the popular vision-language model CLIP (Radford) itt al. 2021). ). we imput the image to the vision end and the prompt sentence to the language end, then obtain a vision-language similarity as the confidence score of classifying the image "[CLASS]"
Yao et al. 2021 Jin et al.
In practice, the prompt-based zero-shot image classification is not accurate because the hand-crafted prompt may not be the most machine-favorable (e.g., "this is a picture of" could be more grammatically prevailing in VLM training), or not specific to the downstream domain (e.g.. "a photo of person doing" is better in action recognition) (Radford al. (02T). Recently, prompt tuing prefix tuning (Lester et al. [021 Liu et al.) 2021b) Zhou et al. 202T 2022) has been proposed to replace the hand-crafted prompt with a set tunable word embedding vectors, which do not have to be translatable back to human-readable words.
Yet, prompt tuning is still as tricky as conventional fine-tuning: as the training continues, the generalization ability may decrease and even under-perform the zero-shot baseline. As shown in Figure [a&b), the prompt tuning method CoOp (Zhou et al.) (02T) achieves the best results via early stopping, and its accuracies heavily drop by at most 4% when the training continues. Besides, Figure &&d) show that CoOp underperforms zero-shot CLIP without augmentation or enough samples from downstream tasks. To the best of our knowledge, existing methods still rely on the

---

Under review as a conference paper at ICLR 2023
OxfordPets
StanfordCars
OxfordPets
StanfordCars
Accuraty (50)
Proind
Pisient
Caoa no segrornrann
ppach
Shat
Shat
(8)
(c)
(d)
Figure 1: Comparison of sero-shot CLIP, CoOp, and our ProGrad Stanford Cars and OxfordPets datasets. (a)&(b): Given shot training sample, CoOp's performance severely drops and under- performs zero-shot CLIP by large margins when the training continues. (c)& CoOp may fail to improve CLIP without data augmentation or plenty of samples.
Domain-specific Direction (G))
General Knowledge
Parallel Direction
Orthogonal Direction
ProGrad Direction
Label
Image
CUP)
[G]
(C))
(Gremne)
2012 Hyrrdal
2012 Neura
Secian
2005 Aud it Convrrbble
(e)
16]
[el
(e)
Figure 2: Comparisons of Grad-CAM visualization for prompt tuning methods using different
gradient strategies on Stanford Cars Datasets. (b) Model trained with Ga. (c) Zero-shot CLIP model. (d) Model trained with Gi|- (e) Model trained with G.. (f) Model trained with G Prograd.
conventional anti-overfitting techniques such as early stopping and data augmentation (Zhou et al. 2021:2202) Gao et al. 2021b) Qin & Joty| 2022a). which lacks a principled solution to the nature of improper prompt tuning. Furthermore, the Grad-CAM visualization results indicate that the fine-tuned prompt starts to mislead the VLM to forget the general knowledge that the classification should at least focus on the foreground object but not the background. Comparing CoOp (Figure (b)) with zero-shot CLIP (Figure ()), we find that the CoOp model distracts its attention to the background, while CLIP mainly focuses on the foreground object. These results demonstrate the over- fet-iin risk of existing prompt tuning strategies, especially when the number of training samples is extremely limited (e.g., or 2).
To this end, we present novel prompt tuning method called Prompt-aligned Gradient (P (ProGrad) to overcome the improperly biased tuning for CLIP. The principle of ProCrad is to regularize each tuning step not conflict with the general knowledge offered by the original prompt, e.g., the zero-shot CLIP predictions. Specifically, we measure the general knowledge direction G₃ using the gradient of Kulbback-Leibler (KL) divergence between the predictions of the zero-shot prompted CLIP and the few-shot fine-tuned model, which we name as general direction. Similarly, we compute the domain-specific knowledge direction G3 using the gradient of cross-entropy between the ground- truth and the few-shot fine-tuned model, dubbed domain- specific direction. We decompose the specifcc direction Ga into: a vector G orthogonal to the general direction, which denotes the non-conflicting domain-specifi knowledge; and 2) vector G1] parallel to the general direction, which denotes the general knowledge. Note that the first gradient component does NOT override the general direction as any two orthogonal vectors can be transformed into two non-conflicting base vectors. For the second component, it must be one of the two directions: 1) the same of the general direction, which indicates that the update is aligned to the general knowledge, and 2) the opposite of general direction, indicating a conflicting update that should discarded to avoid forgetting. Overall, in each iteration, P ProGrad only updates the parameters in the prompt-aligned direction that has an
2

---

Under review as a conference paper at ICLR 2023
acute angle to the general direction. Compared to CoOp and CLIP, both Gg and G1 (Figure 2 d&e))
help to regularize the model to focus on the foreground, and our ProGrad (Figure 2 f)) can further improve the visual response.
Following CLIP, CoOp and CoCoOp (Zhou et al. 2022). we evaluate our ProGrad under the few-shot learning, domain generalization, base-to-new generalization and cross-dataset transfer settings over 11 image classification benchmark datasets, covering generic object classification, fine-grained image recognition, action classification. In summary, our ProGrad achieves: 1) clear improvement compared to CoOp over all of the 11 datasets; 2) clear improvement on the harmonic mean of base-class and new-class accuracies on all 11 datasets compared to CoOp and CoCoOp, and 3) clear improvement on both the source and target datasets of the domain generalization.
2 RELATED WORK
Fine-tuning for VLMs. It is for the VLM adaptation various downstream tasks. e.g., visual
question answering Kim et al. 2021 Tan & Bansal, 2019). visual grounding (Yao et al. 2021). image retrieval (Lu et al. 2019), semantic segmentation (Rao et al. 202T) and image classification (Zhou et al. [021]] 2022). We focus on image classification task. Conventional "pre-train then fine-tune" paradigm that plugs in an additional classifier on top of visual backbone and trained on downstream data has been hidely-adopted e.g., Linear Probe (Radford et al. 2021). CLIP-Adapter (Gao et al. 2021a> and Tip-Adapter (Zhang et al., 202T> add vision and language feature adapter to boost conventional
results. Recently, NLP community presents novel fine-tuning paradigm named "prompt-based learning", which is formulated as a "fillin--hhe-bland cloze test, and fine- tunes the prompt maximize the ground-truth token (Lester al. 2021 [Lu et al. [022 In CV community, CoOp (Zhou et al. 202T) uses a continuous prompt optimization from downstream data instead of hand-craft design. CoCoOp (Zhou et al.) 2022) further extends CoOp by learning image conditional prompt rather than static one to improve generalization to unseen classes. ProDA (Lu) et al. 2022 adapts VLMs to downstream classification tasks by learning prompt distribution over the output embedding space. VPT (Derakhshani et al.] [2022] introduces variational prompt tuning by combining base learned prompt with a residual vector sampled from a instance-specife underlying distribution. Our proposed P ProGrad follows the line of prompt-bused learning to improve both few-shot classification performance and generalization ability by aligning the gradient to general direction, without model structure modification tuning the pre-trained model parameters.
Knowledge Transfer. Forgetting mitigation by knowledge distillation or memory replay is widely deployed in incremental learning (Liu et al. 2020 Rebuffi et al. 2017 Qin & Joty| [022b] Riemer et al. 2018: Hu et al. 202T). However, prompt- based fine-tuning is fundamentally different from incremental learning: the former assumes that VLMs have already captured all the knowledge needed in downstream tasks and the goal is to compose a domain-specific query, whereas the latter assumes that the knowledge yet to be sufficient. In addition, incremental learning requires old data from memory storage while our prompt-based learning method has no access to the pre-trained al. 2020p projects the gradients from new classes to the
data. For example, OGD (Farajtabar orthogonal direction of the gradients of previous tasks. However, as we have no access to pre-training process, the requirement of OGD to store the gradients of old tasks not possible for prompt tuning. Moreover, OGD alters the gradients of downstream tasks even in non-conflicting scenarios which potentially results in sub-optimal performance for downstream tasks. Another related field that leverages gradient matching to transfer knowledge is domain generalization (Shi et al.) 2022 Rame et al. (02T) and multi-task learning (Sener & Koltun 2018:] Yu et al.) 2020). However, their methods are not directly applicable in prompt tuning whose transfer direction only from general to downstream. In Appendix, we will show how their methods fail in several ablative studies.
METHODOLOGY
In this section, we introduce the preliminary concepts of hand-crafted prompt-based zero-shot inference, prompt-based learning, and present our proposed Prompt-aligned Gradient solution to align the domain knowledge with general knowledge for few-shot generalization.
3

---

Under review as a conference paper at ICLR 2023
G
G
prograd
Text Encoder
[V],
[V]
[
I CCLASS
Mearnable context
label
y
G
text features
W
Wig
(a)
Image Encoder
P
CE Loss
prograd
image fewtures
similantty score
Ga
KL Lass
zero-shot
Ga
G
G prograd
Fixed
P
prediction
Tunable
(b)
(c)
Figure 3: (a) If Ga is aligned with Gg. we set prograd as Go. (b) If Ga conflicts with Gg (i.e., their angle larger than 90%), we set prograd as the projection of Ga on the vertical direction of Gg. (c) Training pipeline of our ProGrad. Only the context vectors are learnable.
3.1 PRELIMINARIES
Contrastive language-image pre-training (CLIP) (Radford et al.) 2021) adopts contrastive language-image pre-training paradigm on tremendous pairs of images with natural language descrip- tions. For contrastive learning, the associated image and sentences while the non-associated pairs are regarded as negative samples. The contrastive objective maximizes the similarity of positive pairs while minimize the similarity of negative pairs.
taken as the positive samples,
Zero-shot transfer inference adapts the pre-trained CLIP model to downstream tasks without fine-tuning the model. Taking image classification an example, zero-shot transfer is enabled by formulating the classification task as an image-text matching problem, where the text is obtained by extending the " [CLASS]" name using template like "a photo of [CLASS] " CLIP (Rad) ford et al. 2021) finds that such simple template narrows the distribution gap to pre-training text inputs. The image-class matching score measured based on the cosine similarity between the image feature ff and the class-cxtended text feature w, for i-th class. The image feature if for image æ is extracted by the image encoder, while the text feature w, for i-th class is obtained by feeding the prompt description into the text encoder. The probability for i-th class is obtained
ui; f >
exp(< w. >/T) === cxp(< Wg. f /T)
Pas (w](x)
where denotes the number of classes, and r is temperature learned by CLIP.
Prompt-based learning further strengths the transferring ability of the CLIP model and avoids prompt engineering by automatically learning the prompt given few samples from the downstream task. Different from the zero-shot transfer that used fixed hand-craft prompt, CoOp (Zhou al. 2021) constructs and fine-tunes a set of M continuous context vectors v (V1, 12 Uss) } as the turnable prompt. Specifically, the prompt t; {V1, V2...., UM. c combines the learnable context vectors e and the class token embedding c,, and is fed to the text encoder 9(.). CoOp optimizes the static context vectors by minimizing the negative ligeeiiol of the ground-truit -rrth token:
exp(< g(t,), > /T)
(ol(v) - lo,p(t;(x), p(t]))) Σ exp(< g(t ff /T)
(2)
where y denotes the one-hot ground-truth annotation and K denotes the number of classes.
3.2 PROMPT-ALIGNER GRADIENT
As we introduced in Section ! CoOp faced a challenge that the transfer performance drops when the number of annotations is very limited (e.g., per class), even underperforms the zero-shot transfer.
4

---

Under review as a conference paper at ICLR 2023
Also, CoOp heavily relies on anti-overfitiing techniques such as early stopping and data augmentation. To overcome the over werr-fiting challenge, we propose an effective and efficient mue-tunnng paradigm ProGrad to align the few-shot downstream knowledge with the large-scale general knowledge.
Motivated by the success of knowledge distillation (Phuong & Lampert 2019 Hinton et al. 2015) in knowledge transfer, leverage the zero-shot CLIP predictions as the general knowledge, and compare the fine-tuned predictions with the general knowledge to regularize the gradient direction. Specifically, we obtain the domain-speciff direction by caleulating the cross-entropy Lee between the model prediction (((()) and the ground-truth according to Eq. (2), and the general knowledge direction based on the Kullback-Leibe (KL) divergence between p(t; (x) and the zero-shot CLIP prediction Pzs (w,wx)
Pas Pen(w; (2) p(t (r)
Lu(V) = Σ
(3)
Pxs (w x))
= We denote the gradients of Lx((v) and (v) V.L( and Ga V. respectively. The relations between Gg and Ga are two-fold. (1) Their angle is smaller than 90% (Figure 3 a)),
which indicates that the optimization direction of few-shot downstream knowledge does not conflict with general knowledge. In this case, we safely set the updated gradient direction G prograd as G.. (2) Their angle is larger than 90% (Figure conflicts with general knowledge. In other words, optimizing the context vectors following Ga will lead to the forgetring of the pre-trained general knowledge. In this case, we project the Ga to the orthogonal direction of Gg to optimize the model for classification, which avoids increasing the KL loss. Our ProGrad strategy is mathematically formulated as:
which indicates that the few-shot downstream knowledge
{ Ga: Ga - x
if Ga GGa 2 otherwise.
G proorrad
G.GGG. Gg. T.P
(4)
Fig (c) illustrates the pipeline of our ProGrad. Instead of updating the context vectors using Ga in CoOp (Zhou et al. 2021). optimize the context vectors using Grograd, which prevent the gradient direction from overfitting to few-shot downstream samples. We further introduce À in Eq. 4 to generalize the formulation, which can flexibly control the strength of general knowledge guidance in applications. In particular, = denotes projecting GJ to the orthogonal direction of Gg (Figure (b)), while setting À 0 makes ProGrad degenerate to CoOp, i.e., CoOp is a special case of our strategy. We include the detailed analysis of À in Appendix.
Generalization Error Analysis. We further theoretically analyze the generalization error of our ProGrad. Here, we provide a sketch proof and include the detailed justification in Appendix. Our ProGrad keeps the optimal value Lai of the pre-trained domain when optimizing the empirical risk on the downstream domain. The model fprograd learned by such update rule can be viewed as optimizing the empirical risk on pre-trained and downstream domains (Yu et al. 2020):
Prograal argmin R (f)+ argmin Ra(f) + R,, JEF
(5)
JEF
where FF the function class, and R(-) and R(-) denote the expected risk and empirical risk. We bound the generalization error of ProGrad by virtue of Rademacher Complexity (Bartlett & Mendelson 2002) and the Theorem 6.2 in (Zhang et al. 2012). The detailed proof is in Appendix.
Theorem Let X {x, J and X
(p) = be two set of ii.dd samples drawn from the downstream domain Da and the pre-trained domain Dp. Then for any O, we have with probability at least -
(d)
(xf+)
1 Ra(Jomgma) S Rid+pi (progna)) + 1= + (F)
+ Ra(F)
(6)
3 In(4/e)
3
/I((4/))
In(4//))
(
Na + 111
2\ 2Na
2V
2Np
2\ 2
where (D, is the integral probability metric (Müller 1997) that measures the difference between the distribution of pre- pre-trained domain and the downstrean domain, Rz(F) and Ri are the Rademacher complexity of F.
5

---

Under review as a conference paper at ICLR 2023
Note that the bound of R(F) is inversely proportional to the number of training samples. The- orem 1 shows that the generalization error Re prograd) is bounded by the empirical training risk Re (Prograa) the two domain gap 7/ (D, P) and the estimation error. The empirical training risk can be minimized to arbitrary small value when using deep models with high capacity. The estimation error that related to Np asymptotically tends to 0 as the sample size Np tends to infinity. Thanks to the large amount of pretrained samples Np. we can approximate the generalization error bound as
1
3 In(4/c)
In(4/c) 1
Ral S +
(FFF)
+
+
(7)
2
2N
2
Na
Similarly, we have the generalization error for CoOp froop as
In(4/e)
In(4/c) 1
Re(fewup) (F) + +3/ 2Na
+
(8)
Na
Under the assumption that the gap between pre-trained and downstream domains r(P, D) is small, the estimation enror bound of R. is at least two times greater than Ra((prognd) Considering that Na typically very small in few-shot setting, our ProGrad model fprograd achieves much lower error bound than conventional fine-tuning model like CoOp fcoop.
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION DETAILS
We follow CLIP, CoOp and CoCoOp to validate the effectiveness our ProGrad on four settings:
(I) sho-shot classification, (2) domain generalization, (3) base-to-new generalization, (4) cross-dataset transfer. We report and discuss the results of datsset transfer in Appendix D.I
Datasets. For evaluations of few-shot learning and base-to-new generalization, we follow CLIP
and CoOp to use 11 image classification datasets, i.e., ImageNet (Deng et al., (2009) and Cal-
tech101 (Fei-Fei et al.; 2004) for generic object classification, OxfordPets (Parkhi et al. 2012).
StanfordCars (Krause et al. (2013), Flowers102 (Nilsback & Zisserman 2008), Food101 (Bossard
(014) and FGVCAircraft (Maji et al. 2013) for fine-grained image recognition, EuroS (Hel
ber et al. (219) for satellite image classification, UCF101 (Soomro et al. 2012) for action classi-
fication, DTD (Cimpoi al.) 2014) for texture classification, and SUN397 (Xiao et al. 2010) for scene recognition. For domain generalization, we use ImageNet as the source dataset and select ImageNetV2 (Recht et al.) 2019), ImageNet-Sketch (Wang et al.) 2019), ImageNet-A (Hendrycks)
et al. 2021b) and ImageNet-R (Hendrycks et al.) (02 as the target datasets.
Training Details. For few-shot learning, following CoOp and CLIP, all models are trained with (1, 2, 4, 8, 16) shots respectively then evaluated on the full test split. For domain generalization and base-to-new generalization, we evaluate 4-shot performance, which justifies the robustness under low-shots condition. All results of learning-bases models are averaged over three random seeds. Unless otherwise stated, we adhere to CoOp to use ResNet-50 (He et al.] (016) as the backbone of image encoder. Following CoOp and CoCoOp, the length of context tokens M is set to 16 for few-shot classificatiin and M = 4 for base-to-new, domain generalization and cross-dataset transfer. We follow the same training epochs, training schedule and the data augmentation settings in CoOp. À is set to by default, except that À is set to 0.8 for 16 shots. Please refer to Appendix for more details.
Baselines. We compare ProGrad with 4 baselines: (1) Zero-shot CLIP (2) Linear probe CLIP (3) CoOp and (4) CoCoOp. Although our method can beat some other fine-tune methods like CLIP- Adapter (Gao al., 2021a), we mainly focus on comparing with prompt- baaed learning methods. The results of other fine-tuning methods are in Appendix.
4.2 FEW-SHOT CLASSIFICATION
Setup. We compare with two zero-shot CLIP models, i.e., CLIP and CLIP++ stand for using single prompt and prompt ensembling respectively (Please refer to Appendix Section C for the templates

---

Under review as a conference paper at ICLR 2023
Average over 11 datasets
ImageNet
Caltech101
OxfordPets
ACTUWTY XC
2
Eedp
Codip
CoOp
X
Coop
Protined
ProGiadd
Prodred
Protieed
Usear probe CUP
Uneer probe CUP
Lieer smoee CUP
Usear probe CUP
Zero shot P
Zero-shat CLP
Zem-shht CUP
Zero stor CLP
16
112
24
112
Shots
per
StanfardCars
Flowers 107
Food101
RGVCAireraft
N
Shins
Shits
Shat
river
SUN397
DTD
EuroSAT
UCF101
24
24
7501
16
CoOp
CoOp
CoCp
CoOp
ProSiod
Prudrad
Prudrod
ProCrad
Linea probe CUP
Lineer CUP
Linerr smoter CUP
Liner probe CUP Zero snot CLP
Zero CLP
X-
Zero-shot CLP
Zero-shot CLP
Shots der cars
Shote per cluss
Stott per dass
Shote per clus
Figure 4: Accuracy (%) of few-shot learning on 11 datasets. The context length M is set to 16.
of single and ensemble prompts). ProGrad and ProGrad++ stand for using single prompt and prompt ensembling as general knowledge to implement ProGrad respectively. Note that we only use the hand-crafted prompt ensembling to generate Gg. which provides more accurate general direction. Therefore, we still optimize a single prompt with 16 learnable tokens for ProGrad+
Table | provides averaged accuracy over 11 datasets. Figure illustrates the detailed com- parisons over 11 datasets. Overall, ProGrad achieves clear advantages over baselines for all the few-shot settings on average performance. Specif- ically, ProGrad outperforms CoOp by 9.5%, 6.9% and 5.1% on FGVCAircraft, EuroSAT and Flowers 102 given shot, and the average im- provement over 11 datasets is 3.2%. These re- sults demonstrate the anti-overfitting ability of our P ProGrad when the training samples are extremely limited. Furthermore, leveraging prompt en- sembling can further explore the potential of P ProGrad. From Table [] with more accurate general knowledge offered by prompt ensembling, CLIP++ improves the zero-shot CLIP from 58.77% 77% to 59.38%; P ProGrad++ increases the accuracy of ProGrad from 74.28% to 75.03% at 16 shots.
Table 1: Averaged accuracy (%) of few-shot learning on 11 datasets. M 16.
fishots
0
1
2
4
8
16
CLIP
58.77
CLIP+ LP
59.38
37.32
48.02
57.27
64.88
70.57
CoOp
59.46
62.37
66.79
70.04
73.49
ProGrad
62.61
64.90
68.45
71.41
74.28
ProGrad++
63.06
65.28
68.71
71.80
75.03
4.3 DOMAIN GENERALIZATION
This setting evaluates the generalization ability of models on a target domain which is different from the source domain. Conventional fine-tuning on limited data from a specific domain may mislead the model to learn spurious correlations or in-distribution patterns, resulting in biased model with under mnder-performanee in unseen domains. In contrast, zero-shot CLIP does not exploit such spurious correlations or patterns, since it is not fine- tuned on that distribution. Since our ProGrad uses the general knowledge from the treitrained domain to regularize the fine-tuning on a specific distribution, our ProGrad is hopefully robust the distribution shift. As shown in Table 2 despite the exposure
7

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluation on robustness distribution shift with different visual backbones. (a) ResNe150
(b) ResNet101
Source
Target
Source
Target
TmageNett
-V2
-Sketch
-A
-R
ImageNet
-V2
-Sketch
-A
-R
CLIP
58.18
51.34
33.32
21.65
56.00
CLIP
61.24
54.82
38.66
28.03
64.34
LP
41.29
33.65
13.09
11.18
26.82
LP
47.01
38,46
19.09
16.33
39.43
CoOp
61.34
53.81
32.83
22.08
54.62
CoOp
63,99
56.99
39.40
29.50
64.04
CoCoOp
61.04
53.71
32.30
22.07
53.60
CoCoOp
63.59
56,98
39.16
29.09
64.14
ProGrad
62.17
54.70
34.40
23.05
56.77
ProGrad
64.98
57.86
40.53
30.13
65.61
(c) ViT-B/32
(d) ViT-B/16
Source
Target
Source
Target
ImageNet
-V2
-Sketch
-A
ImageNet
-V2
-Sketch
-A
-R
CLIP
62.00
54.75
40.82
29.59
66.01
CLIP
66.73
60.84
46.13
47.80
74.01
LP
46.77
39.12
20.32
16.32
39.48
LP
54.70
45,57
28.20
22.47
44.12
CoOp
64.74
56.59
40.03
31.10
64.54
CoOp
69.86
62.83
46,90
48.98
74.55
CoCoOp
64.63
56.59
40.74
30.27
64.12
CoCoOp
70.13
63.05
46.48
49.36
73.80
ProGrad
65.36
57.42
41.73
31.89
66.53
ProGrad
70.45
63.35
48.17
49.45
75.21
prodetion
sndeetone
5000 (81
Sarto (54)
Sonta
(5) Failure analysis EuroSAT
Failurre analysis Oxford Flowers
(c) Failure cases analysis UCF101
Figure 5: Distribution of samples that are mis-classified by ProGrad but correctly classified by CoOp.
the source dataset, ProGrad clearly outperforms Zero-shot CLIP, CoOp and CoCoOp on all target datasets as well as the source dataset with ResNet-based and transformer-based visual backbones.
4.4 BASE-TO-NEW GENERALIZATION
We follow (Zhou et al. 2022 to evaluate the general- ization performance from seen classes to unseen classes, All the classes are equally divided into groups, i.e., base classes and new classes, and all the methods are only trained on base classes and tested on both base classes and novel classes. The harmonic mean of base-class and new-class accuracies reported to evaluate the trade-off. Compared to CoOp and CoCoOp, P ProGrad also gener- alizes well to the new classes. From Table we observed that ProGrad achieves the best average performance in terms of all metrics. In contrast, CoOp and CoCoOp fail new classes: their performance is consistently worse than zero-shot CLIP. These results highlight that ProGrad has better generalizability both seen and unseen classes. Please refer to Appendix for the results of 11 datasets.
Table 3:
Averaged accuracy (%) over 11 datasets for base-to-nee generalization.
Base
New
Harmonic
CLIP CoOp
61.72
65.91
63.64
71.96
61.26
65.58
CoCoOp
72.23
60.77
65.35
ProGrad
73.29
65.96
69.06
4.5 FURTHER ANALYSIS
Failure cases. We further analyze the failure cases where ProGrad models predict incorrectly but CoOp gives right predictions. Specifically, we count the percentage of the failure cases that zero-shot CLIP models also fails in Figure 5 We found that a high proportion of the failure cases are also mis-classifidd by Zero-shot CLIP model (red bar in Figure 5). This observation indicates that the general direction Gg generated by imprecise zero-shot general knowledge is detrimental to model generalization. As the number of samples increases, the downstream knowledge represented by Ga becomes more accurate and unbiased. As expected, observe that the red bar becomes larger.
Conflict of knowledge. ProGrad requires the updated gradient direction be acute to the general knowledge gradient directions. We explore how this constraint helps to defuse the conflicts of domain-
8

---

Under review as a conference paper at ICLR 2023
Care ni ProGmd
StantordCars Proirnd
StanfondCars, Profeedd
Angle (degree)
techio), w/ ProGas Ca techi21, wis ProGred
Stamlandaars wi hectred
Angle (degree)
Caltechus), ProGiad CattachaD1, w/ ProGres
Starfordtars wes PoDreed
Angle (degree)
Stantordtiess Protired
Caltecha)), w/ ProGied Cattach003, we Proiias:
Epoch
Epoch
Epoch
(a) shot
(b) shots
(c) 4 shots
Figure 6: The angles between G. and Gg during training on Caltech101 and StanfordCars.
specific and general knowledge by visualizing the angle between their representative gradients during training (angle between Ga and Gg). As depicted in Figure 66 for the training without Grograd the angle between Ga and Gg converges to 90 degree due to the fact that "all high-ddimensional random vectors are almost always orthogonal to each other" (Cai et al. 2013). Intuitively, without any constraint, the optimization direction GJ is independent to the general direction, and the average angle would be around 90 degree (i.e., orthogonal). In contrast, utilizing prograd during training leads to the result that the angle finally converge to obtuse angle. The reason is that Gpograd intervenes the model to learn the downstream knowledge aligned with the general knowledge and leads to the insufficient learning of downstream knowledge that is incompatible with the general knowledge. As training stabilizes, G. struggles to learn the conflicting knowledge, reflecting an obtuse angle to the Gg. Thanks to ProGrad, we discard such conflicting knowledge to avoid forgetting.
Comparison with conventional knowledge dis- tillation. Since our ProGrad utilizes the gra- dient direction of knowledge distillation loss as regularization, one may wonder whether our ProGrad is indeed conventional knowledge dis- tillation. We answer this question by investigat-
Table 4: Comparison with knowledge distillation. Average accuracy (%) over II datasets.
#shots
2
4
16
CoOp
59.46
62.37
66.79
70.04
73.49
KD, 0.25
61.09
63.01
67.74
70.90
73.39
ing whether simple knowledge distillation (i.e.,
KD, 0.5
61.13
63.36
67.14
70.34
72.68
Luost = Lo + a Lxa) can achieve similar per-
KD, a
61.52
64.07
66.52
70.01
72.01
formance our ProGrad. We repeated the few-
KD, a =
60.98
62.66
64.92
67.78
68.98
shot experiments on 11 datasets with a variety of a and report the average results in Table Overall, ProGrad outperforms KD for various few-shot settings. Although KD with small S promotes CoOp in low-shot (e.g.. 1, 2 and 4 shots), the per- formance drops when number of shots is large (see 8 and 16 shots). These results indicate that our ProGrad works differently from KD and more robust to the number of training samples.
KD, a a 4
59.58
61.76
62.92
65.01
65.42
ProGrad
62.61 64.90 68.45 71.41
74.28
Table 5: Applying ProGrad to cosine classifier.
Average accuracy (%) over 11 datasets.
#shots
4
8
16
Cosine
30.50 43.74
53.33
61.26
65.00
+ ProGrad
32.29 46.14
55.18
62.05
66.47
Applying ProGrad to conventional fine-tune paradigm. We are also interested in whether ProGrad can be applied to the conventional "pre-train then fine-tune" paradigm. Specifically, we plug in an additional cosine classifier on top of the visual backbone and compare the performance by conducting the few-shot experiments. Table 5 shows that conventional fine-tuning can benefit from our ProGrad. The implementation details and the result of each dataset are provided in Appendix.
We also analyze the upper bound performance of ProGrad and effect of the syper-parameter/ X. The results and discussion are presented in Appendix D.3 and Appendix D.2
5 CONCLUSION
In this paper, we pointed out the over-fitting issues of existing prompt tuning methods for few-shot generalization, which heavily relies on early stopping and data augmentation to promote zero- shot inference. We proposed prompt tuning method ProGrad that regularize each tuning step not to conflict with the general knowledge of the hand-crafted prompt. Experiments on few-shot classification, base-to-new generalization and domain generalization over 11 datasets demonstrate the effectiveness and efficiency of our ProGrad ad. In the future, we will explore how to apply ProGrad on other tasks like object detection and segmentation.

---

Under review as a conference paper at ICLR 2023
ETHIC STATEMENT
No human subjects were involved during the research and developments of this work. All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment. Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as diserimiiation/bia/faainnes, privacy, etc.
REPRODUCIBILITY STATEMENT
In this paper, we conduct the experiments three times and report the mean and standard deviation (confident interval at 95%) values alleviate the randomness of the starting seed. In Appendix Section D.4 we provide the full details of our experimental settings.
REFERENCES
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-422, 2002.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Tony Cai, Jianqing Fan, and Tiefeng Jiang. Distributions of angles in random packing on spheres. Journal of Machine Learning Research, 14(21):1837-1864, 2013.
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of vision- languaee models. arXiv preprint arXiv:2210.02390. 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. International Conference on Artificial Intelligence and Statistics, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint a 2021a.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -uiti---iitionutiin generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint orXiv:1503.0253:3 2(7), 2015.
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in insremeactett learning. In CVPR, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resource prompt-based learning for rision-language models. In ACL, 2022.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In ICML, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffiient prompt tuning. In EMNLP, 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10383 2021b.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonies training: Multi-class incremental learning without forgetting. In CVPR, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for asioo-anddlanguuag tasks. NeurIPS, 2019.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1306v:1306.5155 2013.
Alfred Müller. Integral probability metries and their generating classes of functions. Advances in Applied Probability, 29(2):429-443, 1997.
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In ICML, 2019.
Chengwei Qin and Shafiq Joty. Lfpt5: A unified framework for lifelong few-shot language learning based prompt tuning of t5. In ICLR, 2022a.
Chengwei Qin and Shafiq Joty. Continual shot relation learning via embedding space regulariza- tion and data augmentation. In ACI, 2022b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
11

---

Under review as a conference paper at ICLR 2023
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. arXiv preprint arXi:21009.0294, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. arXiv preprint arXiv:21 2021.
Sylvestre-Alvise Rebufñi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. CVPR, 2017.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NewrIPS, 2018.
Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In ICLR, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 1212.002 2012.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNIP-IJCNIPP, 2019.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Shuo Yang. Songhua Wu, Tongliang Liu, and Min Xu. Bridging the gap between few-shot and many- shot learning via distribution calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.1179) 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. NeurIPS, 2020.
Chao Zhang, Lei Zhang, and Jieping Ye. Generalization bounds for domain adaptation. NeurIPS, 2012.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. aip-trt Training-free clip-adapter for better vision- vision-language modeling. arXiv preprint arXiv:2111.03930 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01133 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
12

---

Under review as a conference paper at ICLR 2023
This appendix is organized as follows:
Section
provides the generalization enror analysis for ProGrad.
Section B
provides additional training details.
Section C
lists the adopted hand-crafted prompts.
Section P gives additional experiment results, including the analysis on the effect of the pyper-parameter À (Section D.3): the comparison with other fine-tuning methods (e.g.,
CLIP-Adapter (Gao et al. 2021a), gradient matching method (Yu et al. 2020), knowledge distillation (Hinton et al., 2015) and conventional fine-tune method with cosine classifier) with the confidence interval at 95% on 11 few-shot classification datasets (Section D.4]: and additional detailed results for each dataset of the base-to-ney generalization experiments (see Section D.5)
A JUSTIFICATION FROM GENERALIZATION ERROR
We further analyze the generalization error bound of our ProGrad. We define the expected risk R(-)
and empirical risk R(-) of classifier on domain D
N Σ N
R(f) = -(xx))- R(F)
(((()))))
(9)
where ((f()),Y) denotes the cross-entropy and N is the volume of training data. We are interested in the downstream domain Da and pre-trained domain Pp. respectively.
Let F be function class, the conventional fine-tune model fooop is trained on Da by
feoop
argmin Ra(
(10)
FEF
The zero-shot CLIP model fo is considered to be trained on D, by
f. = argmin
argmin (f)
(11)
fEF
For the implementation of ProGrad, we initialize the model fprggd using the pre-trained model fr We regularize each training step not to increase the KL divergence between the predictions of fprograd and fp- In this way, fprograd can keep the optimal value of the pre-trained domain La when optimizing the empirical risk the downstream domain. The model fprograd learned by ProGrad can be viewed as optimizing the empirical risk on both domains:
Prrggrad argmin (f) argmin R fEF
(12)
feF
Based on Theorem 4.1 (Yang et al. 2021 assuming that the neural network has L layers with parameters matrices Wi,.., WL. and their Frobenius norm are most M1, Mr and the activation functions are I-Lispschitz continuous, rssitve-homogenee of the neural network is the softmax function that predicts classes. Let F be function class with the range [a, b). Distribution such that ||x|| < B. Let x {x!=) N and X (x!+" N, be two set of i.i.d. samples drawn from the downstream domain Dj and the pre-trained domain D Then for any > we have with probability at least 1 E,
and applied element-wise. The output
'The pre-trained dataset includes samples from diverse classes. Here, we only consider the pre-trained data helonging to the classes of downstream task.
13

---

Under review as a conference paper at ICLR 2023
cB (V2log(2)L + 1) 1T M,
FeoSSroor))) S R(d+p) (prograa) + =2 (D.
NN,
cB
V (20og(2)L +
IT M₁
3
(b a) ln)(/()())))
(13)
+
VNa
2\
2Nd
3
a) In(4/e)
(b (n(/e)
1 Na
1
+ 2
+
2N
21
2
N₂
where (D, P) is the integral probability metric (uüleer 1997) that measures the difference between the distribution of pre-trained domain and the downstream domain. The Eq. (13) shows that the generalization error R.a (a((pro)))) is bounded by the empirical training risk R((+p) (Sprogra), the two domain gap (D>( (D, P) and the estimation error that is inversely proportional to number of training samples, Le., Na and Np. The empirical training risk can be minimized to arbitrary small value and the estimation error that related to N₂ asymptotically tends 0 as the sample size N₂ tends to infinity. Thanks to the large amount of pretrained samples Np. we can approximate the generalization error bound for the model learned by ProGrad
cB (Voogg(2) (2) + 1) 11 Mj
1 (s
Ral Re(oorogaaa)
VNa
(14)
(b a) In(4/c)
(b a)² In(//c
+ 2
2Nd
2
2
Nu
Similarly, we have the generalization error for fcoop as
cB (V2log(2)L + [T) Mj +3/ 2Nd
(b -0-aa (n-a)e((4/e)
(b (b-a) (b-a)21n44/c)
Re(feoop) 22
VNa
2
Na
(15)
If the gap between the pre-trained domain Dp and the downstream domain Da is very small, the Y> (D, will tend to O. Under this assumption, the estimation error bound of Roofp)) is least 2 times greater than Ra( frrogrr)))) Considering that in few-shot setting, Na is typically very small, which makes our ProGra ad model fprograd much lower error bound than conventional fine-tuning model farop.
B ADDITIONAL IMPLEMENTATION DETAILS
For ProGrad implementation, first initialize the learnable context vector with the word embeddings of the zero-shot hand-crafted prompt. Concretely, if the context length M is 16 and the hand-crafied prompt is "a photo of a". which only has 4 tokens, we initialize the former 12 context vectors with zeros and the last 4 context vectors with the word embedding "f photo of a". We follow the training settings of CoOp (Zouu al. 202T): All prompt-based models are trained by SGD with an initial learning rate of 0,002 which is decayed by the cosine annealing rule. During the first epoch, we use the warm-up trick by fixing the learning rate to 1 x 10-5 to alleviate the gradient explosion. The training epoch is set to 50 for all shots of experiments of ImageNet dataset. For the rest 10 datasets, the training epoch is set to 50 for shot, 100 for 2/4 shots and 200 for 8/16 shots. We train all prompt-based model with batch size of 32 expect for CoCoOp. As described in (Zhou et al. 2022). CoCoOp consumes a significant amount of GPU memory if the batch size is set larger than one. We set the batch size to following their original setting. Our experiments are conducted on one 2080Ti GPU for all datasets except ImageNet where we train the models on one A100 GPU.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Hand-crafted Prompts.
Dataset
Hand-crafted prompt
OxfordPets
"a type of pet, a photo of a {)."
OxfordFlowers
"a type of flower, photo of a ()."
FGVCAireraft
"a type of aircraft, a photo of a ().
Describable Textures
"a texture of ()."
EuroSAT
"a centered satellite photo of O)."
StanfordCars
"a photo of {)."
Food101
"a type of food, a photo of 0."
SUN397
"a photo a ()."
Caltech101
"a photo of ()."
UCFI01
"a photo a person doing O)."
ImageNet
"a photo of ()."
ImageNetSketch
"a photo a ()."
ImageNetV2
"a photo of ()."
ImageNetA
"a photo of a {)."
ImageNetR
"a photo of {)."
Table 7: Prompt Ensembling Examples for ImageNet.
bad photo of a (1.""a "a photo of many 11." sculpture a 1).
"a photo of the hard to see ()." low resolution photo of the ()."
"a rendering of a (). ().""""""""""""""""""""""""""""" "grafffii of a O.""a bad photo of the ()."
"a cropped photo of the ().""a tattoo of
embraidered {)."
"a photo of hard to see ().""a bright photo of ()."
photo of a clean ().""a photo a dirty ()."
"a dark photo of the ()."aa drawing of a ()."
"a photo of my ()."tthe plastic ()." photo of the ccel ()."
"a close-up photo of a (1.""a black and white photo of the 11."
"a painting of the "I."aa painting of 11."
pixelated photo of the ().""a sculpture of the (}."
bright photo of the ().""a cropped photo of a ()."aa plastic ()."
"a photo of the dirty ().""a jpeg corrupted photo of (I."
"a blurry photo of the ().""a photo of the ().""a good photo of the ()."
"a rendering of the [).""a () in video game.' "a photo of one (t."
"a doodle of ()."aa close-up photo of the (). ().""a phote of ()."
"the origami "th" () a video game.' "a sketch of ().
"a doodle of the 11." origami "!."a low resolution photo of 1)."
"the toy (h."aa rendition of the ()."a photo of the clean O)"
"a photo of large ().""a rendition of ().""a phote of a nice ()."
"a photo of weird "I.""a blurry photo of a ")."aa cartoon 11."
"art of a ()""a sketch of the ()." "aa embroidered ()."
"a pixelated photo of a (). "itap of the ()."
"a jpeg corrupted photo of the "!.""a good photo of a ()."
"a plushie (!."a photo of the nice [}."a, photo of the small 1)."
"a photo of the weird "th" cartoon ()." ()a" of the ()."
"a drawing of the ().""a photo of the large ()."
"a black and white photo of a "th" plushie ()."
"a dark photo of a "itap of 11."
"graffiti of the 11.""a toy (1." "itap of ()."
"a photo of cool ()." photo of small ().""a tattee of the ()."
C HAND-CRAFTED PROMPTS
The prompts for 11 datasets well the ImageNet variants are listed in Table We select the ensemble prompts from CLIP (Radford et al. 202I), examples for ImageNet are shown in Table
15

---

Under review as a conference paper at ICLR 2023
Table 8: Comparison of prompt learning methods in the cross-dataset transfer setting. Prompts are learned from 4-shots ImageNet.
Source
Target
FGVCAireraft
Caltech 101
OxfordPets
StanfordCart
Flowers 102
ImageNer
Food101
SUN397
EuroSAT
UCF101
Average
DTD
CoOp
61.34
84.48
85.99
54.16
60.10
75.48
14.09
57.48
35.32
26.72
57.56
55.70
CoCoOp
61.04
84.73
86.42
52.34
61.24
73.79
13.74
55.94
36.60
23.46
57.97
55.21
ProGrad
62.17
88.30
86.43
55.61
62.69
76.76
15.76
60.16
39.48
24.87
58.70
57.36
D ADDITIONAL EXPERIMENTS
D.1 CROSS-DATASET TRANSFER
All models are trained on ImageNet as source dataset and evaluated on the rest 10 target datasets. The goal of this setting are presented in Table 8 As shown, our ProGrad not only achieves the highest performance on source datasets but also outperforms other baselines 9 of 10 datasets.
demonstrate the potential transfer beyond single dataset. The results
D.2 UPPER BOUND OF PROGRAD
As the regularized general gradient direction G3 is the key to improve the results, we are interested in the opper-bound of performance if we use an oracle direction Gfall instead of the one offered by hand-crafted prompt. To do so, first optimize prompt with plain cross-entropy loss the full dataset to create G/alll and then use such gradient to implement ProGrad. The results are shown in Table The results indicate a more accurate regularization direction Gfull can elicit a stronger ProGrad model.
Table 9: Upper- bound results (%) of 2, 4, 8, and 16 shots of few-shot learning.
(a) Caltech101.
#shots
2
4
8
16
ProGrad
88 68 87, 98
89.99
89,99
92.17
ProGradial
91.08 91.70 91.76
91.84
92.86
(b) DTD.
#shots
2
4
16
ProGrad
46. 14 49.78
54.43 60.69
63.97
ProGraduul
66.11 67.04 67.24 68.46
69.27
(c) Stanford Cars.
#shots
2
4
8
16
ProGrad
58.38 61.81
65.62
69.29
73.75
ProGradaul 70.54 71.57
71.66
72.73
75.27
D.3 EFFECT OF HYPER- PAIEERRRRAMETERR
We further analyze the effect of the pprammtrrrettr À described in Eq. (4) in the main paper. Results shown in Table. 10 As discussed in Section 3.2 in the main paper, a smaller weakens the general knowledge regularization, which results in a inferior performance under low-shot setting for most datasets. However, for DTD in Table T0 using smaller À = 0.9 reduce the general knowledge
16

---

Under review as a conference paper at ICLR 2023
regularization can improve the 16 shots results. One possible reason is that texture images of DTD has large gap with the CLIP pre-traine images that collected from the Internet, stronger regularization from pre-trained knowledge might be detrimental to the fine-tune performance if downstream data is sufficient.
Table 10: Accuracy (%) of 1, 2, 4, 8, and 16 shots training with different À on DTD and OxfordPets.
(a) OxfordPets.
(b) DTD.
A
I shot 2 shots
4 shots
8 shots
16 shots
shot
shots
4 shots
8 shots
16 shots
86.01
82.21
86.63
85.15
87.06
0
44.33
46.22
54.08
59.99
62.89
0.2
87.12
83.16
84.87
84.00
86.15
0.2
43.80
45.21
54.02
60.14
63.61
88.09
83.56
85.50
84.04
86.67
0.4
44.17
47.44
54.32
59.65
63.16
0.7
87.74
84.70
86.93
86.30
87.90
0.7
44.93
47.77
54.92
59.28
63.10
0.9
88.26
86.47
87.52
87.38
88.52
0.9
45.78
48.46
55.44
60.46
64.28
1.0
88.36
86.89
88.04
87.91
89.00
1.0
46.14
49.78
54.43
57.98
61.15
D.4
ADDITIONAL FEW-SHOT CLASSIFICATION RESULTS
In this section, we further provide the detailed few-shot classification results of other learning-bass fine-tuning methods with confidence interval at 95% in Table IT and Table [2]
Cosine. As described in Section 4.5 of the main paper, we plug in an additional cosine classifier on top of the visual backhone and trained on downstream dataset.
CoOp learns the context prompt from data rather than hand-crafted design.
CLIP- CLIP-Adapter learns additional feature adapter to boost conventional fine-tuning results.
Cosine ProGrad employs P ProGrad to the training process of cosine classifier.
CoOp le prompt reg. We further investigate whether simply using the l2 distance between learned prompt vector v and the word embedding vector of hand-crafted prompt Vys as the regularization can improve few-shot performance, i.e., (v) (v) + ai|e U2 12, where select
0.01.
CoOp GM applies gradient matching method Yu et al., 2020) to CoOp, i.e., we not only project the Ga the perpendicular direction of G3 the updated gradient, but also project the Gg to the perpendicular direction of Ga as the updated gradient to fine-tune the model alternately.
CoOp KD. As descrihed in Section 4.5 of the main paper, apply knowledge distillation loss
CoOp, i.e., Lioull = fee Lai
CoOp ProGrad employs ProGrad CoOp.
For all prompt-basee based methods, we set the context length M to 16 except for CoOp l2 prompt reg. The learned length for CoOp
prompt reg needs be equal to the hand-crafted prompt length to compute the l2 norm, e.g., the M has to be 4 if the hand-crafted prompt is "a photo ". According the average results in Table IT] we observe that our CoOp + ProGrad still achieves the best average performance. By comparing the results of 1) Cosine and Cosine + ProGrad; and CoOp and CoOp + ProGrad, we demonstrates both conventional "pre-train then fine-tune" paradigm and prompt tuning paradigm can benefit from our P ProGrad. The gap between CoOp and CoOp + l2 prompt reg demonstrates that directly regularize the learned prompt to be not far away from the hand-crafted prompt has limited improvement. By digging into CoOp + KD and CoOp + GM, we find performance improvement by introducing the general knowledge. However, their performance still umder-perfont our CoOp ProGrad. This is because 1) CoOp KD learns the average knowledge from two domains which still allows the fine-tuned model to learn from the downstream knowledge that conflicts with the general knowledge; 2) CoOp MD additional requires the fine-tuned model to discards the general knowledge that not aligned with the downstream knowledge, the downstream data is limited, the inaccurate estimation of Ga will lead the model focus on biased general knowledge.
D.5 ADDITIONAL RESULTS FOR BASE-TO-NEW GENERALIZATION
Table 13 further presents the results for base-to-new generalization on each of the datasets.
17

---

Under review as a conference paper at ICLR 2023
Table 11: Accuracy (%) with confidence interval at 95% of few-shot learning on datasets (Part 1).
The context length is set 16 for prompt- baset methods. indicates results copied from (Gao et al. 202
#shots per class
Method
2
Cosine
30.50 1.24
43.74
1.37
53.33
1.57
61.26 + 1.45
65,00 2.87
CoOp
59.44 + 1.88
62.31 + 1.40
66.72 - 0.93
70.06 + 0.53
73.48 + 0.39
-daptedapter*
61.45
64.32
67.51
70.78
74.35
Average
Cosine ProGrad
32.29 + 1.12
46.14 1.49
55.18
1.99
62.05 + 0.93
66.47 + 1.69
CoOp + 12 prompt reg
60.84 + 1.16
62.75 + 1.18
66.85 + 0.76
70.08 0.58
72.92 + 0.46
CoOp GM
61.27 + 0.96
63.23 + 0.50
64.59 +.63
66.40
67.12 0:29
CoOp KD
61.52 + 0.99 0.99
64.07 = 0.52
66.52 0.38
70.01 0.31
72.01 0.37
CoOp + ProGrad
62.61 + 0.80
64.90 = 0.86
68.45 + 0.52
71.41 0.49
74.28 + 0.40
Cosine
15.95 + 0.07
26.56
0.30
37.08
0.29
46.18 1 0.19
53.36 0.39
CoOp
57.15 + 1.03
57.25 = 0.43
59.51 + 0.25
61.59 0.17
63.00 = 0.18
ImageNet
CLIP-Adapter*
58.14
58.55
59.41
60.36
61.27
Cosine ProGrad
19.21 + 0.28
31.18 1 0.18
42.59
0.29
51.73
0.18
57.65 +: 0.33
CoOp + l2 prompt reg
57.51 + 0.22
61.27 62.49 0.49
0.49
62.71
0.01
62.88 + 0.09
CoOp GM
60.41 0.17
60.51 00.11:00 0.13
60.75
61.01
61.44
CoOp + KD
60.85 0.22
61.08
0.10
0.07
6
62.05 0.09
CoOp + ProGrad
57.75 +
59.75 + 0.33
61.46 1.46+0000
62.54 0.03
63.45 +
Cosine CoOp
60.76 1.71
73.10 1.01
81.43
87.02 0.60
90.60
87.40 + 0.98
87.92 + 1.12
89.48 0.47
90.25 1 0.18
92.00 + 0.02
Caltech 101
CLIP-Adapted
88.52
89.19
91.04
91.71
93.42
Cosine ProGrad
61.95 1 0.12
75.24 + 0.88
82.98 1 0.38
88.59 1 0.21
91.31 +: 0.19
CoOp la prompt reg
87.04 1 0.61
87.37 + 0.78
88.82 0.40
89.62 0.29
91.67 1 0.26
CoOp GM
89.14 1 0.15
89.37 + 0.26
0.33
89.36 0.31
89.42 0.13
CoOp KD
89.06 0.29
89.71
0.20
0.11
90.09 + 0.30
91.39 0.05
CoOp + ProGrad
88.68 = 0.34
87.98 = 0.69
89.99 0.26
90.83 + 0.07
92.17 = 0.17
Cosine
26.33 0.75
41.60
1.93
$5.29
1.97
66.60 + 0.82
66.84
16.24
CoOp
86.01 + 0.47
82.21
+ 2.12
86.63 + 1.02
85.15 + 1.12
87.06 + 0.88
OxfordPets
CLIP-Adapter*
81.44
81.57
82.69
84.13
85.31
Cosine ProGrad
26.08 + 0.73
40.58
+ 2.01
55.23 + 1.44
66.78 + 1.58
68.96
14.35
CoOp la prompt reg
87.55 + 0.15
82.12 + 2.61
84.93 + 1.77
84.38 + 0.75
86.28 + 0.45
CoOp GM
87,05 + 0.65
87.06 = 0.67
88.45 0.45
88.35 0.15
88.38 + 0.27
CoOp + KD
87.10 1.47
87.40
0.60
88.56
88.77 0.24
89.16 0.16
CoOp ProGrad
88.36 + 0.73
86.89 1 0.42
88.04 1.500
87.91 0.54
89.00 1 0.32
Cosine CoOp
18.96 + 0.34
33.37 + 0.38
47.75 0.38
61.30 + 0.25
71.94 + 0.31
55.68 + 1.23
58.33 + 0.60
63.05 + 0.09
68.37 + 0.25
73.34 + 0.49
StanfordCars
CLIP-Adapterr
56.02
58.24
63.07
67.00
72.83
Cosine ProGrad
21.13 + 0.50
39,44 + 0.83
54.54 0.57
66.47
0.14
73.41 + 0.11
CoOp la prompt reg
55.86 + 0.66
57.69 = 0.51
62.82
0.07
66.63
69.86 + 0.44
CoOp GM
57.37 + 0.36
58.46
0.24
59.72 1.72
6..32
63.87 0.37
CoOp + KD
57.48
1.47
59.09
0.60
61.47 0.19
67.73 0.24
70.48 0.16
CoOp + ProGrad
58.38 + 0.23
61.81 + 0.45
65.62 0.43
69.29 + 0.11
73.46 0.29
Cosine CoOp
51.33 +: 2.77
70.06 + 2.29
82.43 + 1.65
91.74 7774 0.73
95.68 + 0.22
68.13 + 1.74
76.68 + 1.82
86.13 + 0.75
91.74 0.49
94.72 + 0.34
Flowers 102
CI.p--ddapee**
71.97
78.80
85.31
90.69
94.30
Cosine ProGrad
52,08 +: 2.31
70.13 + 1.90
81.09 2.06
91.62 0.41
93.94 +: 0.02
CoOp la prompt reg
71.12 + 0.55
0.55
80.36 + 0.54
86.42 0.33
91.58
0.59
94.25 + 0.38
CoOp GM
67.87 + 0.31
69.09 0.49
71.69
75.76
0.79
78.36 + 0.34
CoOp + KD
68.11 + 1.47
71.02
+ 0.60
76.06
84.53 0.24
88.05 0.16
CoOp ProGrad
73.18 1 0.73
79.77 1 0.65
85.37 1.996
91.64 0.24
94.37 L 0.24
18

---

Under review as a conference paper at ICLR 2023
Table 12: Accuracy (%) with confidence interval at 95% of few-shot learning 11 datasets (Part I.. The context length M is set 16 for prompt-base based methods. indicates results copied from (Gao 202
al.
#shots per class
Method
Cosine
25.32 0.29
41.06
0.29
54.10 1.06
61.88 + 0.33
68,50 + 0.24
CoOp
74.28 + 1.40
72.45 + 1.29
2.07 73.27
71.67 = 0.30
74.68 = 0.03
Foo
CIIP-Adapter+
75.09
75.59
75.92
76.53
76.97
Cosine + ProGrad
27.19 0.15
45.28 + 0.36
58.57 = 1.01
71.25:0.:
75.61 + 0.15
CoOp + 12 prompt reg
73.58
2.20
68.89 + 1.30
71.30 0.49
+
75.64 + 0.33
CoOp GM
76.23 1.51
77.97 0.51
0.10
78.90
79.07 + 0.06
CoOp KD
76.06
1.47
77.59
0.60
7.172
0.24 78.38
78.90 0.16
CoOp ProGrad
76.04 0.54
74.95 = 0.57
75.95 + 0.27
76.65 - 0.23
78.41 - 0.08
Cosine
12.47 1.00
17.75
1.35
22.00 1.50
29.14 1 0.54
36.47 1. 0.18
FGVCAircralt
CoOp
9.71 + 6.09
18.74 = 0.48
21.78 = 0.50
27.55 = 0.06
31.37 = 0.53
CLIP-Adapter*
19.63
22.27
25.62
30.48
38.72
Cosine + ProGrad
12.83 +: 0.48
17.59 + 1.59
19.70 1.62
26.34 + 0.51
31.98 + 0.68
CoOp + la prompt reg
18.01 + 0.44
19.78 + 0.23
2
27.24 0 0.38
30.55 : 0.54
CoOp GM
17.08 + 0.37
19.62
21.07
22.52 = 0.19
CoOp +
17.67 0.45
19.29 0.15
25.55 2.60
= 28.58
CoOp + ProGrad
18.81 0.50
20.47 + 0.90
23.32 0.36
27.02 + 0.67
31.12 +
Cosine CoOp
25.32 0.18
38.13
0.45
56.97 + 0.21
62.84 +
60.30 0.64
59.52 + 0.60
63.33 + 0.39 64.74
65.65 + 0.10
69.14 + 0.11
SUN397
CLIP- ILIP-Adapter*
61.16
62.08
66.88
69.20
Cosine + ProGrad
29.66 1 0.08
45.81 + 0.39
55.92 1 0.35
63.61 1 0.16
67.33 1: 0.25
CoOp la prompt reg
57.64 1 0.33
59.81 1 0.33
64.88 1 0.45
67.66 1 0.16
69.56 1 0.11
CoOp GM
62.73 1 0.35
62.85 1 0.10
63.32 0.21
63.77 0.04
64.47 1 0.27
CoOp KD
62.89 0.40
64.10 0.29
65.83 0.26
67.02 + 0.05
68.32 + 0.19
CoOp + ProGrad
60.54 0.24
63.06 = 0.11
66.39 = 0.43
67.62 = 0.28
69.84 = 0.18
Cosine
27.05 0.83
38.42
0.48
48.44
2.29
58.47 + 0.51
61.88 + 0.38
CoOp
43.77 + 2.12
46.06 + 1.05
53.82 1 0.77
60.06 + 1.18
63.26 + 0.22
CLIP-Adapter*
45.65
50.54
56.43
61.59
66.03
DTD
Cosine + ProGrad
26.95 + 1.38
38.87 + 1.02
48.05 + 3.02
56.24 + 2.81
63.40 + 0.58
CaOp 12 prompt reg
43.74
1.45
45.98 + 2.76
53.25 + 1.55
59.08 + 0.58
62.31 + 1.05
CoOp GM
43.81 + 2.15
47.64 + 0.63
49.17 = 1.52
53.17 - 0.63
54.06 = 0.45
CoOp +
43.01
2.18
49.31
1.10
53.03 1.49
60.26 =
63.14 0.39
CoOp + ProGrad
46.14 = 1.74
49.78 + 1.37
54.43 1 0.86
60.69 1 0.10
63.97 1 0.61
Cosine
37.55 5.27
52.93 5.66
49.81 + 6.23
46.08
11.13
33.30 13.04
CoOp
49.40 + 3.86
62.23 + 4.94
69.49 3.23
76.56 + 1.73
84.05 + 1.05
EuroSAT
CLIP-Adapter*
54.53
63.73
68.33
75.81
82.81
Casine + ProGrad
41.55 + 6.19
51.35 + 5.76
17.64499.0
30.03 + 2.99
33.30 + 1.67
CoOp 12 prompt reg
54.28
5.38
62.60 + 2.77
70.43 10443110 1.81
77.32 + 2.20
83.30 + 1.11
CoOp GM
48.02 1:02::40 4.04
57.12 + 2.03
62.88 + 2.28
68.74 2.18
67.72 1 1.00
CaOp +
49.51 1.12
58.89
1.06
66.79
0.76
74.37 1 0.91
77.87 4 1.74
CoOp + ProGrad
56.32 + 3.04
63.10 + 3.77
72.53 + 1.29
78.04 + 2.45
83.74 + 0.70
Cosine
34.41 0.40
48.21 + 1.00
58.47 + 0.81
68.46 + 0.66
73.64 + 0.32
CoOp
62.03 + 1.13
63.98 + 0.91
67.45 + 0.74
72.11 + 0.29 73.45
75.67 + 0.49
CIJI--dapter**
63.80
66.98
70.07
76.99
UCFI01
Cosine + ProGrad
36.61 +: 0.14
52.11 + 1.43
00.6611.5
69.85 + 0.94
74.27 + 0.30
CoOp l2 prompt reg
62.88 + 0.74
64.43 + 0.71
6..46
72.28 + 0.88
75.77 + 0.29
CoOp GM
64.27 0.48
66.14 + 0.25
66.37
67.91
0.29
68.96 + 0.04
0.04
CoOp +
64.99 0.35
67.29 + 0.46
0.14
71.77
- 0.41
74.15 - 0.55
CoOp + ProGrad
64.55 1 0.50
66.35 1 0.18
69.86.1.. 0.30
73.33 0.65
77.28 1 0.96
19

---

Under review as a conference paper at ICLR 2023
Table 13: Accuracy (%) for the base-to-new generalization evaluation. The context length M 4 for prompt-based methods which are learned from the base classes with 4 shots. H: Harmonic mean.
(a) Average over 11 datasets.
(b) ImageNet.
(c)
Caltech101.
(d) OxfordPets.
Base
New
Base
New
Base
New
H
Basc
New
H
CLIP
61.72
65.91
63.64
64.46
59.99
62.14
90.90
90.72
90.81
85.86
93.85
89.68
CoOp
71.96
61.26
65.49
57.70
61.35
94.38
87.48
90.80
90.31
94.03
92.13
CoCoOp
72.23
60.77
65.35
66.21
58.01
61.84
94.43
87.81
91.00
89.07
91.00
90.02
ProGrad
73.29
65.96
69.06
66.96
60.04
63.23
94.47
90.84
92.46
91.78
94.86
93.29
StanfordCars.
(f)
Flowers 102.
(g) Food101.
(h) FGVCAircraft.
Base
New
Base
New
Base
New
H
Base
New
H
CLIP
55.55
66.35
60.47
64.10
70.92
67.34
81.48
82.15
81.81
17.89
25.13
20.90
CoOp
61.77
62.51
62.14
89.33
62.77
73.73
80.40
81.09
80.74
22.53
20.40
21.41
CoCoOp
61.68
59.98
60.82
88.07
66.26
75.62
79.77
77.68
78.71
22.73
19.40
20.93
ProGrad
63.01
64.32
63.66
88.19
69.38
77.66
83.10
83.57
83.33
22.77
24.24
23.48
(i) SUN397.
(j) DTD.
(k) EuroSAT.
()) UCFI0L.
Base
New
1I
Base
New
H
Base
New
II
Base
New
II
CLIP
66.45
70.17
68.26
49.31
54.35
51.71
39.26
43.62
41.33
63.70
67.71
65.64
CoOp
71.48
65.57
68.40
67.71
43.92
53.28
73.53
40.19
51.97
74.59
58.23
65.40
CoCoOp
71.88
67.10
69.41
63.54
40.78
49.68
83.63
40.95
54.98
73.51
59.55
65.80
ProGrad
73.71
69.78
71.69
66.90
53.06
59.18
79.67
49.99
61.43
75.66
65.52
70.23
20


Reviews:
[{"review_id": "wlx2neNTw7x", "reviewer": "Reviewer_VFpS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work highlights an intriguing problem regarding the tuning of soft prompts in vision and language models. This research asserts that current soft-prompt tuning papers have a tendency to overfit on training data while losing generalization ability on test data. To avoid this, they strive to optimize soft prompts in directions so that they do not clash with general information encoded in vision and language models. This eliminates contradictory directions and helps prevent forgetting. The method proposed in this paper is not novel in general and has been explored in multi-task learning and continual learning. However, this is the first time it is used for soft prompt tuning and I believe it is a appropriate contribution and is aligned with ICLR. In terms of experiments and comparisons, I think it is thorough and authors evaluate their model from different view points. ", "summary_of_the_review": "This paper addresses the issue of reducing the over-fitting problem for downstream adaptation of vision and language models as a whole. I vote for \"marginally above the acceptance threshold,\" unless other reviewers indicate critical issues and I will decrease or increase my score in case it is needed. ", "strength_and_weaknesses": "**Strength:**\n- Paper is clearly written and organized.\n- Theoretical justification and derivation are correct as far as I know unless other reviewers raise important concerns. For me, this paper address the same question as [2,3] did in order to improve generalization. I would suggest authors to elaborate these similar papers in their related work.\n- The idea is novel in the context of soft-prompt tuning, but it has been already explored in multi-task learning and continual learning [1]. It would be even nice if authors discuss a bit more about this prior work or similar ones in the related work. \n- Ablations and experiments are complete in the main body of the paper as well as the appendix part.\n\n**Weakness:**\n\nAccording to my experience working with **CoOp** and **CoCoOp**, the two baselines of this paper, in CoCoOp work, all results are reported on **4 learnable tokens** initialized with \"A photo of a {class}\",  **16 shots** and **ViT/B-16 backbone** on three different random seeds for three different tasks: **base-to-new-generalization**, **cross-dataset transfer learning**, and **domain generalization**. Regarding this information, I would appreciate if the authors answers following questions:\n\n1. Do authors train all baseline models themselves or they adopt the results from the main paper?\n2. Are learnable prompts initialized with \"A photo of a {class}\" or they are trained from scratch?\n3. I would ask authors why the number of shots are different in tasks  **base-to-new-generalization** and **domain generalization** in Table 1 and 2. Are there any specific reasons? Can they provide the performance with same hyper-parameters? I would suggest authors to be consistent with the baselines in terms of hyper-parameters to conclude a fair comparison. To me, Table 1 and Table 2 looks a bit wired as you change the training hyper-parameters. Maybe for 16 shots, CoCoOp performs better. \n4. I noticed that authors did not provide results for cross-dataset transfer learning (Table 2 of CoCoOp paper). I would suggest to report these numbers as well since it makes the paper more complete. \n5. Regarding ProGrad++, where authors use the idea of prompt ensemble, would it be possible to clarify how the prompt design changes in terms of number of learnable tokens. To me, since the input prompts changes, the length of learnable prompt would change accordingly and it is a bit unclear how many learnable parameters exist. \n\n\n\n[1]. Farajtabar et al., Orthogonal Gradient Descent for Continual Learning, 2019\n\n[2]. Lu et al. Prompt Distribution Learning, CVPR 2022\n\n[3]. Derakhshani et al., Variational Prompt Tuning Improves Generalization of Vision-Language Models, Arxiv 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sSm91GtNBY", "reviewer": "Reviewer_edmg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses VLM few-shot learning with prompt tuning. It addressed the overfitting of the learned prompts by regularizing the direction of gradients. The direction is enforced not in the opposite direction of the gradient computed with the human-selected prompt. The paper provides a theoretical justification to show its regularized gradients lead to a smaller error bound. The experiments in few-shot image classification, domain generalization, and unseen class classification show that the proposed method achieves better accuracy in all benchmarks.", "summary_of_the_review": "The method is well-motivated and described. Its experiments cover a wide range of settings and show improvements. Overall it looks like a well-executed paper.", "strength_and_weaknesses": "- Strength\n1. The proposed method looks novel and makes much sense. Using a generalizable prompt to regularize the learning of a domain-specific prompt is a simple and effective strategy to avoid overfitting in few-shot learning.\n2. The experiment is comprehensive. It includes three aspects: standard few-shot image classification, domain generalization, and unseen class classification. The comparison to KD (Table 4) and the cosine classifier (Table 5) directly support its claims that its regularized gradient direction is the key to improving the results.\n- Weaknesses\n1. The improvement of ProGrad looks marginally better and may be affected by the choice of hyperparameters. What is the upper bound result this approach can get? How about simulating the upper-bound case using many training data to get an optimized prompt instead of using the man-made prompt to create G_g?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WRLpxuRRiT", "reviewer": "Reviewer_7YcA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "### Problem\nThe paper tackles the problem of prompt tuning for large scale vision-language models. This is a very recent direction.\n\n### Proposed method\nThe method is built on top of (i) zero-shot CLIP with hand-crafted prompts and (ii) CoOP which proposed to fine-tune context vectors as tunable prompts. The method proposes a modification on CoOP where the gradients are slightly modified.\n\n### Experimental validation\nAuthors perform experiments on commonly used datasets.", "summary_of_the_review": "I am voting for rejection because:\n1. The delta over CoOp is very incremental.\n2. The method seems like a heuristic without any proper technical motivation. Even more heuristics have been added into experiment section which are missing from method section. It is not enough for a conference paper.\n3. The improvement over CoOP is small and standard deviation has not been reported. This makes it difficult to draw a conclusion.\n\n\n======================= \nPOST REBUTTAL\n=======================\nAfter carefully going through the authors comments and the other reviews, I have decided to keep my rating the same. My reasons remain the same, which have not been addressed in the authors response:\n1. The method is an incremental extension of CoOP. The fact that the authors have done serious re-branding is not so convincing. If it is a simple extension, saying so is something I would have preferred.\n2. Standard deviation is missing from tables. It is good Machine Learning practise to have this. Despite me asking for this, the authors instead of suggesting that 'we will add this to the final version....', the authors chose to disagree with it. \n3. I felt that the introduction is not well written. Authors 'disagree with it, saying that other reviewers did not feel this way, so my claim is wrong'. There are multiple reviewers to get different opinions. \n4. I still feel that the motivation of the method is heuristics.\n\nWhile these factors might be small alone, but all of them combined together make it a weak paper in the current format. These are changes that the authors can do to make the paper stronger. But the authors did not even accept these weaknesses, let alone proposing to work on them in the final version. Therefore, it is a reject from my side.", "strength_and_weaknesses": "###STRENGTHS\n1. Wide range of experiments\n\n### WEAKNESSES\n1. Very incremental contribution\n- The delta over CoOP is very small. Infact the method is a very simple extension of CoOP. I don't know why the authors made a whole new name.\n2. Results not convincing\n- The reason is that the improvement over CoOP seems very negligible in more important datasets like ImageNet and Table2. Can you aadd a table which shows results on individual datasets?\n- Some tables have ProGrad and others have ProGrad++. If ProGrad++ is also your method, add it into the method section. \n- Standard deviation is missing. So it is very difficult to draw any conclusion. 0.27 improvement on ViT-B in Table 2 can easily go away with multiple runs.\n3. Not properly motivated\n- The method does not have a sound technical motivation. It seems like a heuristic. \n- Phenomenon shown in Fig.2 is not enough. It could be a cherry-picked exception.\n- Starting of sec3.2 should motivate the challenge with previous methods. However, it says that it has been discussed in Introduction. Can you discuss it properly?\n4. Poor Writing\n- Although the method section is decently written, the introduction is hard to follow.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NGOsv4J0-U", "reviewer": "Reviewer_Wdn9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses the challenge of prompt engineering under few-shot setting. The key idea is to treat conventional zero-shot CLIP transfer inference as general knowledge, and enforce that the learned prompt will not diverge much from such general knowledge. In implementation, the gradient at each optimization step will be determined according to the divergence of two gradient direction. Comprehensive experiments are conducted to validate the effectiveness of the proposed idea.", "summary_of_the_review": "Overall I would regard this is an interesting albert simple work toward more robust prompt engineering, under the usual setting of few-shot samples. It will be a good addition to the conference.", "strength_and_weaknesses": "Strength:\n\n- The work addressed some key challenging in prompt engineering for vision-language models.\n- The idea is computationally simple and empirically validated to bring superior performance.\n\nWeakness:\n\n- The proof of generalization bound is not much informative. It is a direct extension of the theoretical results in prior works (e.g. Zhang and Ye's 2012 paper).\n- More insights are required for the readers fully understanding the empirical superiority of the proposed idea. In specific,  the \"general knowledge\" induced by standard zero-shot CLIP transfer does not provide ground truth. The gradient-calculating rule as proposed in Eq. 4 is essentially a fusion of two kinds of cues: one is from zero-shot extension, and the other is data-driven. The trick is to cleverly choose the way for the fusion. In the practice of multi-source fusion, the final results tend to be improved given that all sources are informative and complementary to one another. Section 4.5 provides some analysis on the failure case, which is unfortunately somewhat superficial. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7UylN_q7n4K", "reviewer": "Reviewer_6JR9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propses a method for prompt tuning that is less sensitive to overfitting compared to the CoOp approach.  The main idea is simple and effective, and is based on the observation that CoOp starts overftting when the training is not stoped early,  outpefroming Zero-shot CLIP. The authors propose to requalarize the CoOp objective via preserving the Zero-shot CLIP prediction, similar to learning without forgetting[1]. Moreover, they propse to modify the CoOp gradient in the direction of the regulariztion term gradient during the training. The proposed methods outperofrmrs not only CoOp approach but also more adnvced approaches like CoCoOp on the standrd zero-shot learning benchmarks.\n\n[1] Zhizhong Li and Derek Hoiem, Learning without Forgetting, 2016", "summary_of_the_review": "Overall, the idea of this paper is simple, effective, and executed well. More comparisons with [1] would be interesting to include. I'd be happy to discuss this matter during the rebuttal.", "strength_and_weaknesses": "Strengths:\n\n- The paper is based on a simple and effective idea and is executed well.\n- The paper is accurately and clearly written.\n- The performance is satisfactory.\n\nWeaknesses :\n- I see a lot of similarities between the proposed method with [1]. If I am not mistaken, the auxiliary loss function prevents forgetting the general knowledge is the same as [1]. However, this paper's gradient projection approach and theoretical analyses are new. While the authors limit their application to zero-shot learning, I believe the contributions of this paper hold in a general setting where some previous knowledge has to be preserved for training on a new task.\n\nA discussion on the differences with [1] and direct comparisons with their method would be interesting and strengthen the paper, in my opinion. A quick experiment that can be done during the rebuttal is to show the performance of the proposed method without the gradient projection, i.e. adding the KL loss to CoOp and optimizing with the standard SGD. A similar experimental setup as [1] and director comparisons with them would be even more interesting.\n\n[1] Zhizhong Li and Derek Hoiem, Learning without Forgetting, 2016", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject