Variational Prompt Tuning Improves Generalization of Vision-Language Models
Authors: Mohammad Mahdi Derakhshani
Enrique Sanchez
Adrian Bulat
Victor Guilherme Turrisi da Costa
Cees G. M. Snoek
Georgios Tzimiropoulos
Brais Martinez
Venue: Submitted to ICLR 2023 (2023)

Abstract: Prompt tuning provides an efficient mechanism to adapt large vision-language models to downstream tasks by treating part of the input language prompts as learnable parameters while freezing the rest of the model. Existing works for prompt tuning are however prone to damaging the generalization capabilities of the foundation models, because the learned prompts lack the capacity of covering certain concepts within the language model. To avoid such limitation, we propose a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling. This results in a more complete and richer transfer of the information captured by the language model, providing better generalization capabilities for downstream tasks. The resulting algorithm relies on a simple yet powerful variational framework that can be directly integrated with other developments. We show our approach is seamlessly integrated into both standard and conditional prompt learning frameworks, improving the performance on both cases considerably, especially with regards to preserving the generalization capability of the original model. Our method provides the current state-of-the-art for prompt learning, surpassing CoCoOp by 1.6% average Top-1 accuracy on the standard benchmark. Remarkably, it even surpasses the original CLIP model in terms of generalization to new classes. Implementation code will be released.

Under review as a conference paper at ICLR 2023
VARIATIONAL PROMPT TUNING IMPROVES GENERAL- IZATION OF VISION-LANGUAN MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Prompt tuning provides an efficient mechanism to adapt large vision-language models to downstream tasks by treating part of the input language prompts as learnable parameters while freezing the rest of the model. Existing works for prompt tuning are however prone to damaging the generalization capabilities of the foundation models, because the learned prompts lack the capacity of covering certain concepts within the language model. To avoid such limitation, we pro- pose probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochas- tic sampling. This results in more complete and richer transfer of the informa- tion captured by the language model, providing better generalization capabilities for downstream tasks. The resulting algorithm relies on simple yet powerful variational framework that can be directly integrated with other developments. We show our approach is seamlessly integrated into both standard and conditional prompt learning frameworks, improving the performance on both cases consider- ably, especially with regards to preserving the generalization capability of the orig- inal model. Our method provides the current state-of -fate-of-the-art for prompt learning, surpassing CoCoOp by 1.6% average Top-1 accuracy on the standard benchmark. Remarkably, it even surpasses the original CLIP model in terms of generalization to new classes. Implementation code will released.
INTRODUCTION
In a continuous quest for better pre-training strategies, models based on image and language super- vision have set impressive milestones, with CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and Flamingo (Alayrac et al., 2022) being leading examples. Contrastively trained vision-language models consist of image and text encoders that align semantically-related concepts in joint embed- ding space. Such models offer impressive zero-shot image classification by using the text encoder to generate classifier weights from arbitrarily newly defined category classes without relying any visual data. In particular, the class name is used within a handcrafted prompl template and then tok- enized and encoded into the shared embedding space to generate new classifier weights. Rather than manually defining prompts, Zhou et al. (2022b) and Lester et al. (2021) proposed that prompts can be instead optimized in a data-driven manner through pack-propagation by maximizing cross-entropy loss on the downstream task. However, despite the performance improvement on downstream tasks, prompt learning negatively affects the generalization capability of the vision- languaee model.
While subsequent works have focused on how to bridge the generalization gap, e.g. Zhou et al. (2022a); Zhu et al. (2022), in practice the generalization power of the foundation model is signif- icantly degraded. Our work tackles this same problem, as it seeks to improve downstream perfor- mance but without degrading the generalization capability of the original model.
To do so, we propose data-driven method for directly learning the underlying distribution within the prompt space associated to the target concept. In particular, we frame prompt tuning as a variational inference problem, where base learned prompt is combined with residual vector sampled from the instance-specific underlying distribution. This formulation provides two advantages. First, investigates the prompt space more thoroughly and results in more informative use of the language space, leading to better generalization. Second, it enables us to boost performance by capturing the uncertainty information in fine-grained classification problems. The resulting approach is orthogonal

---

Under review as a conference paper at ICLR 2023
to standard prompt learning approaches, being effective when combined with both standard (Zhou et al., 2022b) and conditional (Zhou et al., 2022a) approaches. In fact, when combined with the conditional approach, our method maintains the gains on the seen classes provided by the conditional method while simultaneously matching or even surpassing the generalization capability on unseen classes of the original vision-language model.
In summary, our contributions in this paper are as follows:
We propose a variational framework that is capable of capturing the general or instance specific distribution within the prompt space. Since generalization is obtained through transfer from the language space, we obtain better generalization capability.
2. We show that the proposed approach is orthogonal to recent developments, and can be successfully combined with both standard and conditional prompt learning variants.
3. We empirically show that our proposed method improves performance and provides better generalization. leading to state-of-the-an accuracy in 24 out of 28 standard benchmarks set forth by prior work, surpassing CoCoOp by 1.6% average Top-1 1 accuracy.
RELATED WORKS
Prompt learning in NLP. Prompt learning was originally proposed within the NLP domain, follow- ing the appearance of foundation models such as GPT-3 (Brown et al., 2020). Early prompt learning methods constructed prompts by combining words in the language space such that the model would perform better on downstream evaluation (Shin et al., 2020; Jiang et al., 2020). Subsequent meth- ods, e.g. Li & Liang (2021); Lester et al. (2021), prepend a set of learnable prompts to the input of frozen model and optimize through kuck-propagation, which allows better flexibility than ing existing words, at the cost of leading to prompts that do not correspond to an actual phrase. Instead, Hc et al. (2022) focus on multi-task scenario and use a HyperNetwork to conditionally generate task-specific and layer-specific prompts that are pre-pended to the values and keys inside the self-attention layers of the frozen model. Within the NLP domain, prompt learning has also been shown to work better than in-context learning (Liu et al., 2022).
Prompting in Vision and Language models. Research on prompt learning for vision-language models have been largely inspired by prior work within NLP. Similar to e.g. Li & Liang (2021), CoOp (Zhou et al., 2022b) proposes a prompt learning method that optimizes unified or class spe- cific prompts in the continuous space through back proekk-ropggaionn While CoOp obtains good accuracy on downstream tasks, it negatively affects the generalization ability to new unseen classes. Co- CoOp (Zhou et al., 2022a) extends CoOp and partially bridges the generalization gap by generating instance- speccficc prompt residuals through a conditioning mechanism dependent on the visual data. ProGrad (Zhu et al., 2022) shares the same goal CoCoOp of bridging the generalization gap, but instead proposes to match the gradient of the prompt to the general knowledge of the CLIP model to prevent prompt tuming from forgetting the general knowledge learned from the foundation model. Alternative directions consist of test-time prompt tuning (Shu et al., 2022), where consistency across multiple views is used as the supervisory signal, and unsupervised prompt learning (Huang al., 2022), where a pseudo-labelling strategy is proposed instead to obtain the labels needed to drive the prompt learning. Perhaps the most similar work to ours is Lu et al. (2022). In this work, the authors use an ensemble of prompts and model their distribution within the language embedding space, with optimization seeking to minimize the negative log-likelihood with respect to the cor- responding visual embedding. Unlike ours, their method relies on hand-crafted rules to define the prompt ensemble, thus still relying on the effectiveness of hand-crafted designs. The number of learnable prompts is also pre-defined, potentially offering sub-optimal coverage of an NLP concept. Finally, it is not clear how to apply their strategy within the context of conditional prompt learning- We believe that modelling the input prompt space rather than relying fixed number of templates is a more powerful and flexible approach. We provide empirical evidence of the superiority of our approach in the experiments.
While beyond our current scope, it is worth noting that prompt learning has been applied to a wider range of problems and scenarios, which highlights its power and flexibility. Among them are impor- tant topics such as unsupervised domain adaptation (Ge et al., 2022), multi-label classification (Sun al., 2022), video classification (Ju et al., 2022), object detection (Du al., 2022; Feng al., 2022)
2

---

Under review as a conference paper at ICLR 2023
x(x)
Learnable Prompts
Fold(x):uu)
f(x)
"Cat"
(x(((x))
"Dog"
Probabilistic Sampling
Input Image
"Tree"
(O,,
[(x)
gt)
"Sun"
Cosine Similurity Score
Figure I: Overview of variational prompt tuning. For cach input image x, we use image features f(x) to infer the mean µ(x) and standard deviation [xx) of the residual distribution using metanet Tg. The prompts to generate the classifier weights are constructed by summing up learnable prompts p and residual samples from the residual distribution. The obtained prompts are fed through text encoder gt), and the classifier weights are estimated. Finally, the cosine similarity scores are computed between the image features (x) and the classifier weights.
or pixel-level labelling (Rao et al., 2022). Finally, prompt learning as a means to adapt pre-trdined models has also been applied purely vision models (Jia al., 2022: Sandler al., 2022) providing similar performance to fine tuning the whole model but with great parameter efficiency.
3 METHOD
3.1 BACKGROUND
Contrastive Language-Image Pretraining (CLIP) (Radford al., 2021) consists of image en- coder fxx and text encoder g(t), each producing a d-dimensional (L2 normalized) embedding from arbitrary image E x and word embeddings E RLxe with L representing the text length and the embedding dimension Both encoders are trained together using a con- trastive loss from large-scale dataset composed of paired images and captions. Once trained, CLIP can be used for zero-shot C--lass image classification by generating each of the classi- fier weights w. as the d-dimeeniional text encoding g(tz). Here t, results from adding the class- specific word embedding e, pre-defined prompt p E 1xx i.c., w.=g(te) with t. {p, e. The prompt p is manually crafted to capture the semantic meaning of the downstream task, c.g., t "An image of (class)". The probability of image being classified ef(x)
{1...CC
thus defined as p(y]x)= E5 e/(e)"
Context Optimization (CoOp) (Zhou et al., 2022b) provides learned alternative to manually defining prompts. CoOp learns a fixed prompt from a few annotated samples. The prompt is de- signed as learnable embedding matrix sification error through the frozen CLIP model. Specifically, for set of N annotated meta-training samples (xt,
R l.xe which is updated via back-propagating the clas-
yi 1.V the prompt p is obtained by minimizing the cross-entropy loss,
p arg min ExminExx.... P
Ex., log p(y: | xx, p)].
Note that this approach, while resembling that of common meta-learning approaches, can still be deployed in a zero-shot scenario provided that for new classes the classification weights will be given by the text encoder. Although this approach generalizes to new tasks with few training iterations, learning fixed prompt is sensitive to domain shifts between the annotated samples and the test set.
In CLIP the word embedding is learned together with the text encoder. A tokenizer is used to convert the text into one-hot vectors, tokens, that can be directly mapped into the word emheddings. For the sake of clarity refer indistinctly words and word emheddings.
3

---

Under review as a conference paper at ICLR 2023
Conditional Prompt Learning (CoCoOp) (Zhou et al., 2022a) attempts to overcome domain shifts by learning an instance-spectio specific continuous prompt that is conditioned on the input image. To case the training of a conditional prompt generator, CoCoOp defines each conditional token in residual way, with a task-specific, learnable set of tokens and a residual vector that is conditioned on the input image. Assuming p to be composed of I. learnable tokens p=[p1, P2, Pr), the residual vector (x))=<<(((
E IR is produced by a small neural network T6 with as input the image features f(x). The new prompt is then computed pxx)=(Pi + r(x), P2 + r(x), ... Pr + r(x)] The training now comprises learning the task sask-specific prompt p and the parameters @ of the neural network a Defining the context-specific text embedding tx(x)= (p(x), cz), and p(y) x) as:
e((x)=a(t(xx)) (x))
p(y|x)
(2)
Ee s((x)))(tt(x)))
the learning formulated as:
p o" = arg min Rx,,9 log X1, P, 6)]
(3)
p.
While CoCoOp achieves state-of-the-ar results in large variety of downstream tasks, it is still prone to the domain shift problem, considering that provides deterministic residual vector from the image features f(x) which are expected to comain-sppciffie
Prompt Distribution Learning (ProDA) (Lu et al., 2022) is work concurrent to CoCoOp that fo- cuses learning a distribution of prompts that generalize to broader set of tasks. ProDA proposes to learn a collection of prompts ( | that can be used to subsequently generate an a pos- teriori distribution of the classifier weights for each of the target classes. For given mini-batch of K sampled prompts P& ~ P. the classifier weights We are sampled from the posterior distri- bution N (Pw..c: Ewic), with mean Wwi. and covariance Ewi. computed from the collection with tkec ee). The objective is now formulated as:
{Wk.c ((tt.e))==1:.Ccc
P = arg min Ex,, [- log PwinN(pwwo Ew P(y [1, wi)]-
P
Computing VEw; x, wi)] is intractable and an upper bound to Eq. 4 is derived. During inference, the classifier weights are set those given by the predictive mean w. Pwi.c computed across the collection of learned prompts P. While showing promising results compared to CoOp, how to combine it with the conditional prompt learning framework of CoCoOp unclear.
3.2 VARIATIONAL PROMPT TUNING
In this paper, we propose to model the input prompt space in probabilistic manner, priori, instance-specific distribution. In particular, we define a distribution py over the prompts p that is instance-specific, i.e. p (x). To this end, assume that p be split into fixed set of prompts p. and an instance specific residual vector r that act as a latent variable over p. The snstance-specific prompt is then defined as:
(x) = [P1 + Tyy, P2 +r,, Pz. + ryl, I, (x)..
(5)
where (x(x) refers to the real posterior distribution over conditioned on the observed features x. Denoting the class-specific input as (x) e.), the marginal likelihood p(y/x) is:
p(y\x) = / e(x) alt altt,, (x))
(x()) ((t))
(6)
el(x)*
Solving Eq. 3 with the marginal likelihood defined as in Eq. 6 is intractable, as it requires computing p.y (rlx)p,(x)). (x). Instead, resort to deriving lower bound, by introducing variational posterior distribution (x) from which the residual r., can be sampled. The variational bound is defined as:
logp(ylx) >
(logg/ollseeee p(v|x, r)] Dkt. (r(2)) (1/2))),,(r))
with p(y(x,r) x (x)) (x)), where the dependency on r comes through the definition of tz The variational posterior distribution ne plays role akin to the metanet in CoCoOp. We thus refer to it as metanet in the following to align terminology. Following standard variational optimization practices (Kingma& & Welling, 2014; Gordon et al., 2019), we define ne Gaussian distribution
4

---

Under review as a conference paper at ICLR 2023
conditioned on the input image features x, r(x) ~ (N((x), E(x)). with μ and Σ parameterized by two linear layers placed on top of the metanet T (see Figure 1). The prior p. (r(r) is defined as (0, I), and we make use of the reparameterization trick to generate Monte-Carlo samples from To to maximize the right side of Eq. 7. The optimization of Eq. 7 comprises learning the prompt embeddings {p:) L as well as the parameters of the metanet z and the linear layers parameterizing j and Σ. Note that this adds little complexity it requires learning p and The given that μ and Σ are defined as two linear layers on top of Tt.
At test time, K residuals are sampled from the conditional distribution TT. (x), which used to generate K different prompts per class [n=[Pı + rk: P2 + Tk. ... Pr. + +x). Each prompt is prepended to the class-specific embedding to generate a series of K separate classifier weights wk,e- We then compute p(y-c (1/(()/))) Σ p(y=c x, Wk.e) and select غ= arg max, p(y=c) x) as the predicted class. It is worth noting that because the posterior distribution is generated by the text encoder, it is not expected that for K - 20, (1/K) Es p(y=c) x, we.k) - p(y=clx, g( (((xx)), ee3), meaning that sampling at inference time remains relevant. We study the dependency on the number of samples in the ablations.
Inference.
Notably, our framework can be also used with the unconditional setting of CoOp. by simply remov- ing the dependency of the input image from the latent distribution. In such scenario, we keep fixed set of prompt embeddings and learn global latent distribution p- over the residual vectors r, as r ~ [(,,, E), where and Σ are parameterized by two learnable vectors. In this case, Py is general distribution learned during training with no dependency on the input sample x. We show that CoOp with our proposed approach improves generalization new classes.
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL. SETUP
We follow the exact experimental setup of CoCoOp (Zhou et al., 2022a), currently the sfate-of-the- art for prompt tuning for vasion-language models. We describe the setup in the following.
Three tasks and fifteen datasets. We evaluate variational prompt tuning for three different tasks: base-to-new generalization, cross-dataset transfer, and cross- generalization. For base- to-new generalization and cross-dataset transfer tasks, we rely on the same 11 image recognition datasets as Zhou et al. (2022b:a). These include generic image classification datasets (ImageNet by (Deng et al., 2009) and Caltech101 by (Fei-Fei et al., 2004)), fine-grained classification datasets (OxfordPets by (Parkhi et al., 2012), StanfordCars by (Krause et al., 2013), Flowers by (Nilsback & Zisserman, 2008), Food101 by (Bossard et al., 2014) and FGVCAircraft by (Maji et al., 2013)) 30)3),, scene recognition (SUN397 by (Xiao et al., 2010)), action recognition (UCF101 by (Soomro al., 2012)), texture classification (DTD by (Cimpoi et al., 2014)), and satellite imagery recognition (Eu- roSAT by (Helber et al., 2019)). For the cross-domain generalization task, we train our model on ImageNet and report on ImageNetV2 (Recht et al.. 2019), ImageNNe-SSkecch (Wang et al., 2019), ImageNet-A (Hendrycks al., 2021b), and ImageNet-R (Hendrycks et al., 2021a).
Evaluation metrics. We report average accuracy and harmonic mean H=2 x (base x new)/ /(base + new) (Xian et al., 2017) for the base-to-new generalization tasks. For cross-dataset transfer learning and domain adaptation, we provide average accuracy results.
Baselines. We compare against zero-shot CLIP (Radford et al., 2021), CoOp (Zhou et al., 2022b), CoCoOp (Zhou et al., 2022a), and ProDA (Lu et al., 2022). For zero-shot CLIP, CoOp, CoCoOp, all results are adopted from (Zhou et al., 2022a), and we reproduce all results for ProDA.
Implementation details. Our variational prompt tuming contains three sub-networks: an image encoder f(x), a text encoder g(t), and a metanet T.. The image encoder f(x) and text encoder gt)) are a ViT-B/16 (Dosovitskiy et al., ) and transformer (Vaswani et al., 2017), which are initialized with CLIP's pre-trained weights and kept frozen during training. as in Zhou et al. (2022b;a), The metanet π. consists of two linear layers followed by ELU activation function as trunk and two linear heads on top to estimate the μ and Σ of the residual distribution. For each task and dataset, optimize the number of samples K and epochs. Other hyper-parameters well as the training pipeline in terms of few-shot task definitions are identical to Zhou et al. (2022b;a) (see table 6 and in the appendix). Implementation code will be released.
5

---

Under review as a conference paper at ICLR 2023
Average
|1169
+1.60
ImageNet
H0.83
-0.28
01
-0.90
+0.60
OxfordPets
-0.31
10.39
StanfordCars
+2.09
+1.06
02
-0.34
00.59
Food1O1
-5.83
-0.58
FGVCAireaatt
+2.80
+6.95
SUN397
3:27
+0.24
DID EuroSAT
-3.9
+2.42
-1.06
-6.50
UCFI01
16.47
(a) A(H) H(Coop+VPPT) H(CoOp) (b) (H) = H[CCCoO--VeT)
Figure 2: Relative enhancement of variational prompt tuning over CoOp and CoCoOp in terms of harmonic mean over 11 datasets for 3 distinct random seeds. Variational prompt tuning improves the harmonic mean for all baselines other than OxfordPets for CoOp and Flowers 102 for CoCoOp.
4.2 -ASE-TO-NEWW GENERALIZATION
Setup. We report the few-shot generalization of our method 11 datasets for three different random seeds. Each dataset is divided into two disjoint subsets: base classes and new classes. We train our method on base classes and evaluate on both base and new classes. For fair comparison, we follow Zhou et al. (2022a;a) in terms of dataset split and number of shots (see Table 6 appendix.)
Results. From the results in Table the lack of generalization capability of the CoOp approach is evidenced by the considerable discrepancy between the base and new classes' accuracy. This is expected as CoOp only observes a small number of training samples to adapt the CLIP model for downstream tasks, resulting in overfitting. Adding variational prompt tuning to CoOp reduces over- fitting to base classes, improving the performance of new classes by 11.54% and the harmonic mean by 1.7% across all datasets, at the expense of base classes accuracy drop by 10.7%. Figure 2(a) depicts the relative improvement of our proposed strategy when compared to CoOp in terms of the harmonic mean, where observe improvement in 10 out of 11 datasets. The limited gener- alization capability of CoOp is mitigated by CoCoOp by exploiting cnsdatoe-condiitonal prompts, which improves the accuracy new classes from 63.22% 72.23%. Nonetheless, augmenting CoCoOp with variational prompts still improves performance on the new classes and the harmonic mean by 3.25% and 1.6%, respectively, with a small decrease of 0.37% in base accuracy. Note here however that training for longer increases the performance on base classes and lowers it on new classes. Our setting optimizes harmonic mean, but simple tweak the training scheduler on EuroSAT is enough to surpass CoCoOp on base class accuracy, see Table 8 on appendix A.2.2. Fig- ure 2(b) shows per-dataset relative harmonic mean improvement. We observe an improvement in 10 out of 11 datasets. Note that the drop in the average base accuracy for CoCoOp+VPT is negligible compared with 300++VPT. Moreover, our best performing model CoCoOp+ VPT performs better than ProDA (Lu et al., 2022) in terms of new classes accuracy and harmonic mean by %2.64 64 and %8.78. We also considered ensemble learning the soft prompt from different ways of initialization, this does not lead to improvement (see Appendix A.2.3). It is also worth noting that CoOp+VPT and CoCoOp+VPT excel at CLIP zero-shot performance base, new and harmonic accuracies.
4.3 CROSS- TRANSFER LEARNING
Setup. For cross-dataset transfer learning, the model is trained on source dataset (ImageNet) and then assessed on 10 distinct target datasets. This experiment tries to determine how effectively our methods generalizes transfer beyond the scope single dataset.
As reported in Table 2, CoOp+ has drop in performance in ImageNet by 1.76% while outperforming the target dataset on average by 1.63%. Moreover, our proposed method leads to an increase 9 out of 10 target datasets, with small drop in accuracy of 0.03% in C11 Note that on target datasets such as FGVCAircraft and DTD, our proposed method achieves an improve- ment of more than 3%. Similarly to CoOp, augmenting CoCoOp with our method still leads to an
Results.

---

Under review as a conference paper at ICLR 2023
Base-to-new generalization comparison between the state-of-the-art and variational prompt tuning. We average our accuracy over three random seeds. Our proposed model is trained on a few-shot training set (base) and then evaluated on held-out classes (new). As shown, CoOp and CoCoOp overfit on base classes and do not provide good generalization on new classes. However, our model provides better generalization performance on new classes as well as harmonic mean.
Table 1:
Dataset
CLIP
CoOp
+VPT
CoCoOp
+VPT
ProDA
Base
69.34
82.66
71.98
-10.71
80.47
80.10
-00.37
81.56
Average
New
74.22
63.22
74.76
+11.54
71.69
74.94
+03.25
72.30
71.69
71.65
73.34
01.69
75.83
77.43
-1..60
76.65
Basc
72.43
76.14
74.73
-01.41
75.98
76.00
+00.02
75.40
ImageNet
New
68. 14
67.88
70.60
+02.72
70.43
70.93
+00.50
70.23
H
70.21
71.77
72.60
00.83
73.10
73.37
00.27
72.72
Basc
96.84
98.00
95.47
02.53
97.96
98.00
+00.04
98.27
1a
New
94.00
89.81
93.80
+03.99
93,81
94.93
+01.12
93.23
H
95.39
93.72
94.62
00.90
95.84
96.44
00.60
95.68
Base
91.17
93.67
90.77
02.90
95.20
95.67
00.47
95.43
OxfordPets
New
97.26
95.29
97.83
02.54
97.69
98.00
00.31
97.83
H 94.11
94.47
94.16
00.31
96.43
96.82
00.39
96.62
Base
63.37
78.12
65.27
12.85
70.49
72.93
02.44
74.70
StanfordCars
New
74.89
60.40
75.97
+15.57
73.59
73.23
00.36
71.20
H
68.65
68.12
70.21
02.09
72.01
73.07
01.06
72.91
Base 72.08
97.60
72.97
24.63
94.87
95.70
00.83
97.70
12owers102
New
77.80
59.67
75.90
16.23
71.75
70.40
01.35
68.68
II
74.83
74.06
74.40
00.34
81.71
81.12
00.59
80.66
Base
90.10
88.33
90.37
02.04
90.70
91.03
00.33
90.30
Food101
New
91.22
82.26
91.67
09.41
91.29
92.13
00.84
88.57
H
90.65
85.18
91.01
05.83
90.99
91.57
00.58
89.43
Base
27.19
40.44
29.57
10.87
33.41
34.40
00.99
36,90
FGVCAircraft
New
36.29
22.30
33.80
11.50
23.71
35.00
11.29
34.13
H
31.08
28.74
31.54
02.80
27.74
34.69
06.95
35.46
Base
69.36
80.60
73.77
06.83
79.74
79.17
00.57
78.67
SUN397
New
75.35
65.89
77.90
12.01
76.86
77.87
01.01
76.93
II
72.23
72.50
75.77
03.27
78.27
78.51
00.24
77.79
Base
53.24
79.44
$7.67
21.77
77.01
75.30
01.71
80.67
DTD
New
59.90
41.18
58.70
+ 17.52
56.00
60.80
04.80
56.48
H
56.37
54.24
58.18
03.94
64.85
67.27
02.42
66.44
Base
56.48
92.19
67.97
24.22
87.49
80,30
07..99
83,90
EuroSAT
New
64.05
54.74
71.63
+ +16.89
60.04
75,30
+15.26
66.00
H 60.02
68.69
69.75
01.06
71.21
77.71
06.55
73.88
Base 70,53
84.69
73.23
11.46
82.33
82,53
000200
85.23
UCF101
New 77.50
56.05
74.63
+ 18.58
73.45
75.77
02.32
71.97
H 73.85
67.45
73.92
06.47
77.64
79.00
01.36
78.04
overall performance enhancement of 0. 06.66% on out of 10 target datasets, showing its effectiveness for cross-dataset transfer learning. In addition, unlike CoCoOp which has better performance in ImageNet-like datasets such as Caltech and OxfordPets, our proposed method exhibits improve- ment dissimilar datasets (e.g. FGVCAircraft, DTD, and EuroSAT), demonstrating its capacity to capture the unique characteristics of each dataset.
4.4 CROSS-DOMAIN GENERALIZATION
Setup. Lastly, We examine variational prompt tuning through the lens of distribution shift and robustness. We train our proposed model on the source dataset (ImageNet) for three different random seeds, and assess on ImageNetV2, ImageNet-Sketch. ImageNet-A, and ImageNet-R. Prior work such as CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a) demonstrate empirically that learning soft-prompt improves the model's resilience against distribution shift and adversarial

---

Under review as a conference paper at ICLR 2023
Cross-dataset transfer learning comparison between the state-of-the-art and our varia- tional prompt tuning in terms of average accuracy on three different random seeds. Following (Zhou al., 2022a), our proposed model is trained a source dataset and evaluated on target datasets. As shown, variational prompt tuning performs better than other baselines 16 out of 20 datasets, al- though it loses performance on the source dataset.
Table 2:
Source
Target
RG PGVCAireraft
Stanford Cars
Caltech101
OxfordPets
Flowerst02
ImageNet
Food101
SUN397
EuroSAT
UCP101
DTD
Methods
CoOp
71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
+VPT
69.73
93.67
89.27
65.50
70.20
86.27
22.13
66.57
46,93
47.43
67.2
65.51
A
-.78
-0.03
+0.13
+0.99
+1.49
-0.97
+3.66
+2.42
+5.01
+1.04
+0.65
+1.63
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
+VPT
70.70
93.67
90.63
65.00
70.90
86.30
24.93
67.47
46.
45.87
68.67
65,95
A
-0.32
-0.76
+0.49
- 0.32
0..88
-0224
+1.99
+0.11
+0.37
+0.50
+0,46
+0.16
attack. Following their experiments, we are also interested determining if treating prompts in variational manner maintain or improve the performance.
Results. As reported in Table 3 our method enhances the accuracy of CoOp on ImageNet-Sketch, ImageNet-A, and ImageNet-R by 0.88%, 0. 0.70%, and 2.19% while degrading the performance on ImageNet and ImageNetV2 by 1.78%, by 1.03%. However, on CoCoOp, adding VPT, while losing the performance source dataset similar to CoOp, consistently improves the accuracy all target datasets which highlight the effectiveness of our proposed method.
Table 3:
Cross-domain Generalization comparison between the state-of-ther tfet--rrthh-rtttt and variational prompt tuning in terms of average accuracy three different random seeds. Our proposed model is trained on source dataset and evaluated on target classes. Variational prompt tuning outperforms alternative baselines on the target datasets while losing performance the source dataset.
Source
Target
Methods
Learnable
ImageNet
ImageNetV2
ImageNet-Sketch
ImageNet-A
ImageNet-R
CLIP
X
66.73
60.83
46.15
47.77
73.96
CoOp
/
71.51
64.20
47.99
49.71
75.21
+VPT
/
69.73
63.17
48.87
50.77
77.40
A
-1.78
-1.03
+0.88
-0.70
+2.19
CoCoOp
/
71.02
64.07
48.75
50.63
76.18
+VPT
V
70.70
64.23
49.20
51.33
77.00
A
-0.32
+0.16
+0.45
-0.70
+0.82
4.5 ABLATIONS
Effectiveness of the posterior distribution Ts- We first ablate the effectiveness the variational posterior distribution. To do this, we consider sampling one residual vector from the uniform dis- tribution (0, 1), normal distribution (0, normal distribution N (f(x),,0) 0), normal distribution M(((),, [x)),, and report the new class accuracy for CoCoOp+VPT for one random seed in Ta- ble 4. Except for the EuroSAT dataset, sample from the normal distribution N (x)xx), obtains the best-performingce
comparison with alternatives. showing that the mean of the normal dis- tribution u(x) is the most effective sample. In addition, we find that drawing one sample from N(((x), E(x)) yields superior results compared to drawing one sample from uniform distribution (0, 1) and normal distribution (0, I), further demonstrating the efficacy of our proposed method in capturing the underlying distribution the prompt space. We also ablate increasing the number of samples from the normal distribution [(u(x), Σ(x)) understand the informativeness of the learned variational distribution. is shown that enlarging the number of samples further improves the model performance they capture the prompt space appropriately.
Prompt initialization. We investigate the effectiveness of the prompt initialization on the new class accuracy for CoCoOp+VV VPT for one random seed. We consider variants. In the first one, ini-
8

---

Under review as a conference paper at ICLR 2023
Table 4: Effectiveness of the posterior distribution. The informative posterior distribution N [)(x), Σ(x)) outperforms the two uninformative distributions (0, 1) and (0, 1) by a large mar- gin for all datasets. Increasing the number of samples further improves results.
Methods
Samples DTD
Flowers102
EuroSAT
FGVCAircraft
UCF101
2400, 1)
33.20
45.30
54.20
10.50
55.70
((O, 1)
26.60)
36.10
50,00
07.70
48.80
N(u(x).))
I
59.80
73.50
59,90
34.10
76.50
M(((x), I(x))
1
56.40
72.30
64.50
33.00
75.60
M(u(x), E(x))
2
60.00
73,90
67.40
33.90
76.20
M(n(x), Σ(x))
5
62.20
74.00
71.00
34.20
76.60
M(((x), Σ(x))
10
61.60
73.50
73.60
34.40
77.00
tialize the context tokens randomly using a normal distribution, whereas in second one we initialize the context tokens with "An image of a {class)". Table 5 summarizes this ablation. Comparing the two variants demonstrates that an appropriately initialized prompt consistently outperforms randomly initialized prompt, highlighting the necessity for further research the prompt space. We will leave it open for future research direction.
Table 5: Prompt initialization. Initializing the context tokens with an appropriate prompt "An image of (class)" improves the performance compared random tokens.
Method
DTD
Flowers102
EuroSAT
FGVCAireraft UCF101
Random
74.10
67.23
75.10 77.90
34.30
81.30
"An image of {class)"
74.80
70.05
35.50
82.50
Number of Monte-Carlo samples. When approximat- ing the log-likelihood of input data, the number of Monte Carlo samples an important hyperparameter. Generally, large number of samples should lead to better approx- imation and better classification accuracy. We ablate this new accuracy for CoCoOp+VPT by varying the number of Monte Carlo samples at inference time. We show results for varying number of samples in Figure 3 for DTD, Flowers 102, EuroSAT, FGVCAir- craft, and UCF101. Increasing the Monte Carlo sam- ples from 1 to 10 consistently improves the new accu- racy, afterwards the accuracy saturates. Hence, we rec- ommend evaluating variational prompt tuning on a larger number of Monte Carlo samples for better model accu- racy. To further aid the interpretation and understanding of the learned prompts we provide a further ablation on the variational distribution of our best performing model, CoCoOp+ VPT in Appendix A.2.4 and A.2.5
Rower UCF101
RGVCAkcat
70
Accimey
30
hyperparameter
Number of MC samples
Figure 3: Effectiveness of Monte Carlo samples on new accuracy. As demonstrated, increasing the number of Monte Carlo samples boosts perfor- mance initially but reaches plateau af- ter 10 samples for all datasets.
CONCLUSION
In this paper, we introduce variational prompt tuning, probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling. By doing so, we are able to generate adaptive context tokens for each data point. This formulation leads to better generalization capabilities in terms of the new accuracy and harmonic mean for downstream tasks. We show that it can seamlessly integrated into both standard and conditional prompt learning frameworks, considerably improving the performance in both cases. We conduct extensive experiments on 15 datasets and demonstrate the benefits a variational formulation in learning data-driven prompts. Our method provides the current state-of- the-art for prompt learning, and constitutes, to the best of our knowledge, the first method for CLIP adaptation that fully maintains the generalization capability to new classes of the original model. Finally, prompt distribution can be obtained using other ways of density estimation, e.g. normalizing flows, energy-based models diffusion models, and we will leave them open for future research.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our method has the potential to affect applications that often need rapid adaptation, such as medical imaging, astronomical imaging, and autonomous driving. Because of this, there may be negative so- cietal consequences associated with the adoption of our technology. For example, a lack of fairness with models trained on insufficient data, regulatory compliance, patient privacy in medical imaging, and more general biases encoded into large vision-language models such as CLIP.
REPRODUCIBILITY STATEMENT
Details on benchmarks, metrics, and the implementation of variational prompt tuning in terms of architecture and training are contained Section 4. We also list of all hyperparameters Table and 7. Lastly, we will release the source code, scripts to reproduce the results and evaluating the performance of the model at: https: / /ttpttgttbb./////////////ttttttt <redatteddtddd>//// / rrdacctdd>.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
lean-Baptiste Alayrac et al. Flamingo: a visual language model for few-shot learning. Advances on Neural Information Processing Systems, 2022.
Lukas Bossard, Matthieu Guillaumin, and I Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances on Neural Information Processing Systems, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- voccbuaayyy object detection with vision-language model. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2004.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. PromptDet: Towards open-vocabulary detection using uncurated images. European Conference Computer Vision, 2022.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv preprint arXiv:2202.06633 2022.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019.
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. HyperPrompt: Prompt-based task- conditioning of transformers. In International Conference on Machine Learning, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-dist generalization. In Proceedings of the IEEF/CVF International Conference Computer Vision, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021b.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision- laniuuel mod- els. arXiv preprint 2rXiv:2204.06649, 2022.
11

---

Under review as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision Laionnlanguage representation learning with noisy text supervision. International Conference Machine Learning, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference on Computer Vision, 2022.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Trans. of the Association for Computational Linguistics, 2020.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-languant models for efficient video understanding. European Conference on Computer Vision, 2022.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graines categorization. In Proceedings of the IEEE international conference on computer vision work- shops, 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. Conference on Empirical Methods in Natural Language Processing, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1xX1::2010001900 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205.00 05638, 2022.
Yuming Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515. 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics Image Processing, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guide dense prediction with context-aware prompting. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980. 2020.
Manli Shu, Weili Nie. De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision- langualanguage models. Advances Neural Information Processing Systems, 2022.
12

---

Under review as a conference paper at ICLR 2023
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2012.
Ximeng Sun, Ping Hu, and Kate Saenko. DualCoOp: Fast adaptation to multi-label recognition with limited annotations. arXiv preprint arXiv:2206.0955: 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances on Neural Informa- tion Processing Systems, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances Neural Information Processing Systems, 2019.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, 2010.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- -siounlanguage models. In IEEE Conference Computer Vision and Pattern Recognition, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal Computer Vision, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning, arXiv preprint arXiv:2205.14800 2022.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 HYPERPARAMETERS
In this section, we provide the detailed hyperparameter settings in Tables 6 and that are used to generate results in the main paper for each dataset. There are two sets of hyperameters. The first are shared among the two variants of variational prompt tuning CoOp+ and CoCoOp+VPT (See Table 6). The second correspond to datasct-specific parameters that are optimized per dataset (See Table 7).
Table 6: Shared hyperparameters used to generate all results in the main paper.
Hyperparameters
Values
Batch Size
Input Size
224 224 "Bicubic"
Input Interpolation Method
Input Mean
00.48145466 0.4578275, 0.40821073) 0.22682294,, 0.26130258, 0.27577711]
Input STD
Transformation
("random resized crop", "random flip", "normalize")
Optimizer
SGD
Learning Rate
-
LR Scheduler
"cosine"
Warmup Epoch
Warmup Type
"Constant"
Warmup LR
1e-5
Backbone
ViT-B/16
Prompt Length
Prompt Initialization
"a photo of a {class}" 16
Number of Shots
Table 7: Dataset-specific hyperparameters used to generate all results in the main paper.
PGVCAircraft
Stanford Cars
Caltech101
OxfordPets
Flowers102
ImageNet
Food101
SUN397
EuroSAT
UCF101
DTD
Hyperparameters
Number of Monte Carlo Samples
10
20
40
20
10
20
10
20
40
20
Number of Epochs
10
20
20
40
40
20
10
10
10
60
20
14

---

Under review as a conference paper at ICLR 2023
A.2
MORE ABLATIONS
A.2.1 VISION ENCODER ALTERNATIVES.
All previous experiments benefit from VIT-B/16 as the vision encoder's backbone following (Zhou al.. 2022b:a; Lu et al., 2022). For completeness, in Figures 5, and 6, we replace this vision encoder with Resnet50 and Resner100 and examine its impact on base accuracy, new accuracy and harmonic mean for one random seed. The visual transformer outperforms the Resnet alternatives across all 10 datasets for accuracy and harmonic mean and in 9 out of 10 datasets for base accuracy. Hence, we suggest training and evaluating variational prompt tuning on visual transformer for better model performance.
10M
Accuracy
25-
Caltech101
DID
buroSAT
PGVCAriaatt
Flowers
Foodlo1
OxfordPeis
StanfordCars
Sun397
UCH101
Datasets
Figure 4: Ablation of different vision encoder backbones with respect base accuracy. A more over parameteriee model leads to better performance across all datasets except EuroSAT.
10
Accuracy
Caltech101
DTD
EuroSAT
PGVCArisaat
Flowers112
Food10]
OxfondPeis
StanfordCars
Sun397
UCF10)
Datasets
Figure Ablation of different vision encoder backbones with respect new accuracy. A more over-parameterized model leads to better generalization performance across all datasets.
Resnet101
VIT-8/16
10
Accuracy
Caltsch101
DTD
DuroSAT
PGVCArizaat
Flowers1
Food10]
OxfordPeis
StanfordCaas
Sun197
UCF10]
Datasets
Figure 6: Ablation of different vision encoder backbones with respect to harmonic mean. more over peretttaeeter model leads to better performance across all datasets.
15

---

Under review as a conference paper at ICLR 2023
A.2.2 BASE AND NEW CLASSES ACCURACY TRADE-OFF
In this section, we ablate whether there trade-off between base and new classes accuracy. To do so, we train our proposed method on EuroSAT dataset for 60 and 80 epochs, and compare them with in Table 8. As shown, training ET for 20 more epochs raises the performance on base classes by +5.37. This alone improves the average performance on base classes across the 11 datasets by +0.49, resulting 80.58 (now +0.11 over CoCoOp). This change slightly affects average harmonic mean, reducing it by 0.17% (now - 1..4% better than CoCoOp). Consequently, we indeed observe a trade-off between performance on base and on new classes. Training for longer will increase the performance on base classes and lower it new classes.
Table 8: Base and new classes accuracy trade-off for EuroSAT dataset. As reported, We indeed observe trade-off between performance on base and on new classes. Training for longer will increase the performance on base classes and lower on new classes.
Dataset
CoCoOp
CoCoOp+VPT (60) CoCoOp+VPT (80)
Base
80.47
80.10 (-0.37)
80.58 (+0.11)
Average
New
71.66 71.69
74.94 (+3.25)
74.22 (+2.53)
H
75.83
77.43 (+1.60)
77.26 (+1.43)
Base
87.49
80.30 (-7.19)
85.67 (-1.82
EuroSAT
New
60.04
75.30 (+15.26)
67.30 (+7.39)
H
71.21
77.71 (+66550)
75.46 (+4.25)
A.2.3 ENSEMBLE BASELINES
Variational prompt tuming can be considered efficient ensemble approach as it generates sev- eral samples from the prompt space and uses them in conjunction. Nonetheless, we compared Co- CoOp+ VPT against modified CoCoOp that naively uses an ensemble of prompts. We implement this by initializing several Icarnable prompts per class using M hand-crafted templates provided in the CLIP codcbase and fine-tuning them together. The final text feature representation is computed as simple average of the ensemble of prompts for each class as in Section 3.1.4 of (Radford et al., 2021). For a fair comparison, the number of ensembles, M, equates to the number of Monte Carlo samples K per dataset (see Table 7). In Table 9, when comparing CoCoOp and CoCoOp + Ens, the ensemble counterpart, we can see that adding an ensemble of prompts does not necessarily lead to an improvement in all datasets. For instance, on DTD and UCF101 datasets, CoCoOp performs better, while on Flowers 102, AT, and FGVCAircraft datasets, CoCoOp + Ens has a higher harmonic mean. Moreover, when comparing CoCoOp + Ens against CoCoOp+VPT we can that our method performs better in terms of new classes accuracy and harmonic mean in all datasets other than 120 with average increase of 1. 1.94% in harmonic mean, whereas the drop in Flower (2 is only 0.66%.
A.2.4 FACTOR OF VARIATION ANALYSIS
Here, we analyze whether prompts that are sampled at the distributional modes correlate with fea- tures that characterize any known factors of variation (e.g., sub-domains) in dataset. We perform this experiment on our best-performing model, CoCoOp+VPT.
First, we randomly select three classes C1, C22 and from the Flowers dataset (Columbine, Passion Flower and Cyclamen, for reference). Then, for each class, we compute image features by forwarding the respective input images x into the image encoder of the CLIP. Next, apply K-means to group the image features into Q = cluster centroids. Note that each cluster centroid
hhtps///itthh.cco/opeaa//..P
16

---

Under review as a conference paper at ICLR 2023
Table 9: Comparison between CoCoOp, CoCoOp Ensembling, and CoCoOp+VPT. As shown, variational prompt tuning performs better than other alternatives on 4 out of 5 datasets in terms of new accuracy and harmonic mean.
Dataset
CoCoOp
CoCoOp+Ens
CoCoOp+VPT
Base
77.01
77.03
75.30
DTD
New
56.00
52.17
60.80
H
64.85
62.21
67.27
Base
94.87
94.33
95.70
Flowers 102
New
71.75
72.17
70.40
H
81.71
81.78
81.12
Base
87.49
86.60
80.30
EuroSAT
New
60.04
66.10
75.30
H
71.21
74.97
77.71
Base
33.41
34.93
34.40
FGVCAircraft
New
23.71
32.63
35.00
H
27.74
33.74
34.69
Base
82.33
81.20
82.53
UCFI01
New 73.45
73.90
75.77
H
77.64
77.38
79.00
assumed to capture some factors visual variation within each class. Afterwards, we treat the cluster assignments pseudo-labels of those images. Note that this is done separately for each class, Additionally, for an input image x, we generate K = 21 different prompts. These prompts fed into the clip's text encoder, which generates K different weights WI, W2: ww K per image instance. We now have a set of K weights for Q pseudo-labels across three classes, which are visualized each row of Figure 7. In each row, on the left side, visualize how prompt samples are distributed across the five different clusters, while on the right side, we provide the top-3 most representative samples the individual clusters (i.e. the images that produce feature representations that are the closets to the centroids). From this figure, it can be seen that there is high intersection between the distribution of all sub classes, representing shared knowledge. However, some particular expressed in regions without intersection in the distribution. Consequently,
visual differences we believe that there is a correlation between the modes of the variational distribution and visual particularities of subsets of the same class.
17

---

Under review as a conference paper at ICLR 2023
Columbine
diatte
chanee
<haner
(b) Passion Flower
etecter
ekanee
dleatr
(c) Cyclamen
Factor of variation analysis. For three different classes, the contours intersect region that represents shared knowledge related to their corresponding class, while diverging bit and expressing a specific factor of variation. Consequently, we believe that there is a correlation between the modes of the variational distribution and characteristics that adequately represent all known drivers of variation in a dataset (see the text for more details).
Figure 7:
A.2.5 INTERPRETATION OF VARIATIONAL DISTRIBUTION
We now provide an intuition of how prompt samples are distributed within the variational distribu- tion. We perform two experiments on our best-performing model, CoCoOp+VPT.
For an input image and its corresponding target y, K residuals sampled from the conditional distribution no (x), which are used to generate K different prompts per class P+=[pi Fk: P2 + Tk.... Pr. + ru]. We also construct new prompt based the mean of variational distribution μx() named as mean prompt = +(x), P2 +xx), Pr. +x()]. These prompts are sorted based on their distance to the mean of variational distribution p(x), where for any and j (i & j). d(f(x), < d(u(x),ry) ry). The K prompts and mean prompt
Experiment 1:
18

---

Under review as a conference paper at ICLR 2023
prepended to the class-specific embedding generate series of K + 1 separate classifier weights Wx(x),es Wi.es W2,e. ... W.,e- For each class y, we compute the cosine similarity of the image encoding f(x) and W W1,1 W2,ys w K.y and average across all samples per class. In Figure 8, we can see that the mean prompt Pε(x) is the most similar classifier weight the image encoding m and, as we move further away from the mean prompt, the cosine similarity decreases
Experiment 2: Given weights Wu(x)ar Way W2,y W,,y from Experiment define new weight as w/-=Wu(x), - Ej=1 Wayy where wy is the cumulative sum of weights regarding prompt Pi to P^ and mean prompt Pμ(x)) The cosine similarity between the image encoding f(x) and all (w/JEE j=1 are computed and visualized in Figure As shown, summing up classifier's weights together increases the cosine similarity with the maximum similarity obtained when all
samples are combined
From these two experiments, believe that prompt samples are well-distributed inside the prompt space such that the prompt distribution provides adequate coverage of the underlying distribution for downstream tasks.
19

---

Under review as a conference paper at ICLR 2023
Class Label: Pasture Land
Cosine Simillarity
8.10-
DUNS
500
EJAN
204
P
Ps
Ps
Po
Pr
Ps Pu
Pio Pi Pi2
Pis
Pie
P1s
Pio
Pir
Pis
Pio
Pro
Number of Samples
Class Label: Permanent Crop Land
Cosine Similarity
0.16-
0.12-
0.12
5:30
5.00
P1
P2
P5
P5
P6
P7
Ps
Pio Pu P12
Pis
Pie
P15
Pio
P17
P18
P19
P20
Number of Samples
Class Label: Residential Buildings
Cosine Similariy
5.00
2.06
6.44
8.14
2.02
DUNI
2.00
Pia
Pie Pio
Pir
Pis
Number of Samples
Class Label: River
Cosine Similarity
6.12
EUR
SUNO
2.00
Patr
P1
P2
ps
p.
Pr
P6
P7
Ps
Po
P1o
Pu P12
Pis
Pie
P15
Pio
P17
P18
Pio
P20
Number of Samples
Class Label: Sea or Lake
Casine Sumilarity
0.30
5.14
2.17
8.10
Pais) P1
P2
P5
P6
Po
Pio Pi1 P12
Pia Pie
Pis
Pio
Pir
Pis
P19
P20
Number of Samples
Figure 8: Interpretation of variational distribution for EuroSAT dataset. Experiment shown, the mean prompt Pą(x) is the most similar classifier weight to the image embedding, and as move further away from the mean prompt, the cosine similarity scores decreases. Experiment As shown, suming classifier's weights increases the cosine similarity, where the maximum similarity is obtained when all classifier weights combined.
As
20


Reviews:
[{"review_id": "XkRLPiRDoe", "reviewer": "Reviewer_ANZ7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed to leverage variational inference to learn the underlying distribution of the prompts with or without image feature as the condition. The effectiveness of the proposed method is verified on several benchmarks.", "summary_of_the_review": "Overall, the reviewer thinks this paper solves an incremental problem following a rather standard approach without providing in-depth analysis and insights on why solving this problem.", "strength_and_weaknesses": "Pros:\n1. The proposed idea is intuitive and straightforward.\n2. The improvement is shown by combing the proposed method with two popular methods for visual prompt tuning.\n\nCons:\n1. It is unclear why it is needed to learn the distribution of prompts. Unlike where variational inference was usually used in computer vision, e.g., VAE, the possible distribution of the prompts is actually not that untractable. \na. Simple baselines like ensemble learning the soft prompt from different ways of initialization should be provided. It is possible that when ensembling several prompts learned from different initializations, the performance of the model is already somewhat achieved. Because the possible way of introducing an image in caption is rather limited. Probably a few main ways of the prompt initialization can already cover.\nb. Any interpretation of the learned/sampled prompts could be provided? This may help to understand why the learning process is necessary.\nc. Although the authors claim on the improvement of generalization to new classes, the performance gain on cross-dataset setting is still quite limited. Therefore, the model may still learns pretty domain specific distribution of prompts.\n\n2. In Table1, more than half pf the results sacrifice the base class accuracy for improvement of the average. Therefore, it is also important to provide simple baselines like early stopping or adding some common regularization terms on the prompts.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "QyIKlD5G1Nm", "reviewer": "Reviewer_mkLi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes a probabilistic framework for modeling the underlying distribution of prompts, which acts as an additive module on top of existing prompt-learning approaches, and seeks to enable improved domain generalization.", "summary_of_the_review": "Missing some intuition regarding the prompt distribution, beyond empirical results", "strength_and_weaknesses": "(Strengths)\n\nThe manuscript proposes a simple, yet principled approach.\n\nThe manuscript is well-written and easy to follow.\n\nThe manuscript provides numerous experiments, against several strong baselines.\n\n(Weaknesses)\n\nSection 4. The phrase 'Domain Generalization' is slightly misleading, as the section seeks to highlight properties of unseen generalization through zero-shot transfer. The discussion could either be reframed under 'Cross-domain Generalization'.\n\nSection 4. I would like to see further analyses/illustrations of the prompt distribution obtained by VPT. Do prompts that are sampled at the distributional modes correlate with features that well-characterize any known factors of variation (e.g., sub-domains) in the dataset(s)? What further evidence can be provided to show that the prompt distribution approaches reasonable coverage of the underlying distribution? Can equivalent results be obtained through other forms of density estimation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EZfMc_CaC4E", "reviewer": "Reviewer_SMRu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an instance-specific variational prompt-tuning framework for image-language downstream tasks. Here, the experiments are performed with a frozen CLIP model while fine-tuning the parameters used for prompt creation. It combines two ideas from existing publications on prompt-tuning with learnable prompt embeddings:\n\n1. Adding a residual vector derived from input image features to the learnable prompt embeddings\n2. A variational framework for prompt tuning, which this work implements by assuming a Gaussian distribution on residual vectors that are then added to the prompt embeddings.\n\nImplementation of this framework is discussed in both the conditional (instance-specific) setting, as described above, and the unconditional setting of the previous work CoOpOp. Evaluation is performed on 15 image recognition and classification datasets via experiments on the tasks of base-to-new generalization, cross-dataset transfer, and domain generalization.", "summary_of_the_review": "Borderline accept. The paper presents useful results about the implementation of simple changes to existing systems and the resultant performance improvement in certain scenarios. However, since existing ideas are heavily used and combined, there is an overall lack of novelty, so I would hesitate from a strong accept.", "strength_and_weaknesses": "Pros\n\n- Increase in generalization performance against CoCoOp and ProDA is shown empirically for most datasets, with state-of-the-art results in prompt learning for image recognition tasks with generalization requirements.\n- Either of the two mechanisms described - instance specific (conditional prompt learning) and instance independent (standard prompt learning), can be implemented as a simple change to the existing methods with a boost in generalization performance.\n- This boost enables higher-performance for few-shot and zero-shot learning for downstream image-text tasks, possibly increasing resilience against distribution shift and adversarial attacks.\n- Ablation studies provide evidence of the usefulness of the proposed mechanism.\n\nCons\n\n- While the proposed model is simple to implement and integrate with existing systems, and has shown improved performance, especially for usage with different domains, there is a concern of lack of enough novelty since the ideas of residual vector and variational prompt learning were already introduced in previous publications.\n- The increase in generalization performance comes at a cost to performance of in-domain/seen samples.\n- A minor con is that prompt initialization is still manually specified.\n\nMinor Edits\n\n- More details can be specified about implementation in the zero-shot learning scenarios.\n- More clear details can be added about the implementation of this method in the unconditional setting of CoOp.\n- Can discuss ensembling interpretation of sampling", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "63MDm2rMJB", "reviewer": "Reviewer_iM4z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: \nThe authors propose a variational prompt generator conditioned on an input instance. This is learned by adding a residual (which is input conditioned) to a fixed set of learnt prompts. During inference, multiple residuals can be sampled to generate different prompts. The authors show that the proposed approach surpasses existing methods on 16/20 datasets for classification\n", "summary_of_the_review": "I think the paper has merits, but technical novelty is limited. I am on the fence about the paper because it has strong empirical results but the approach combines existing ideas in unsurprising ways. I will look at the author response and read other reviews to decide my final rating.\n\nUpdate after rebuttal: I think the authors took effort in addressing all my concern and most of other revierwer's concerns during the rebuttal. I am still on the fence due to limited novelty and tradeoff in performance between base and new classes.  But I think the new analysis makes the paper stronger. To reflect this I am updating my score to 6 (marginally above acceptance threshold). ", "strength_and_weaknesses": "\nStrengths:\n- The method is intuitive and show strong empirical results on a bunch of tasks. The ablations are reasonable too! \nThe model maintains generalization capability for both new  and old classes. \n\nWeaknesses:\n- The paper combines two existing ideas in prompt learning (the residual addition, and the variational formulation). This limits the technical novelty of the paper a little bit. \n- The whole motivation of the paper is about preserving the generalization capability for downstream tasks, yet we see that the model perfoms worse on base classes than existing methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject