M$^3$PL: Identifying and Exploiting View Bias of Prompt Learning
Authors: Chujie Zhao
Tianren Zhang
Guanyu Chen
Yizhou Jiang
Feng Chen
Venue: Accepted by TMLR ()

Abstract: Prompt learning is an effective means of fine-tuning multi-modal foundation models such as CLIP. Despite existing success, the inner mechanism of multi-modal prompt learning has not been well understood. In this work, we identify an inductive bias of multi-modal prompt learning, which we refer to as view bias, that the learned prompts may extract only a partial subset of useful features (views) and ignore others. This bias can undermine the model's generalization ability, particularly under distribution shifts. We further observe that independently trained prompts have distinct view biases, contrary to the existing belief that they may converge to similar local optima due to having the same cross-modal representation matching objective. Based on our observations, we propose Multi-modal Matching Multi-Prompt Learning (M$^3$PL), which incorporates multiple paired prompts and a cross-modal contrastive regularizer that facilitates the prompt pairs to encapsulate a broader spectrum of views. Extensive experiments show that M$^3$PL effectively boosts the model's generalization capability, achieving state-of-the-art performance under various distribution shifts.

Published in Transactions on Machine Learning Research (09/2024)
PL Identifying and Exploiting View Bias of Prompt Learning
Chujie * Department of Automation Thinghua University, Beijing, China
http::/j22mmmii..ssinnnggg en..co
Tianren Zhang
sbanghr222mmi3..tsinnuuue
Department of Automation
Tsinghua Universily, Beijing, China
Guanyu Chen
chen-gy2280miis.sinnnn edu.ccm
Deportment of Automation Tsinghua University, Beijing, China
Yizhou Jiang
jiangs::00@maiis.ssiinggae. edw..c
Department of Automation Tsinghua University, Beijing, China
Feng Chen
chenfeng@mmiil.tinnghuu edu.ccc
Department of Automation
Tsinghua Universily, Beijing, Chvnna
Reviewed on OpenReview: https: // ottp:///eeprreee....mmmm net/ forum? id= 2rn.IBBm19
Abstract
Prompt learning is an effective means of fine-tuning multi-modal foundation models such CLIP. Despite existing success, the inner mechanism of multi-modal prompt learning has not been well understood. In this work, we identify inductive bias of multi-modal prompt learning, which we refer to as view bias, that the learned prompts may extract only a partial subset of useful features (views) and ignore others. This bias can undermine the model's generalization ability, particularly under distribution shifts. We further observe that independently trained prompts have distinct view biases, contrary to the existing belief that they may converge similar local optima due to having the same cross- cross- modal representation matching objective. Based Prompt Learning (MPPL), which incorporates multiple paired prompts and cross-modal contrastive regularizer that facilitates the prompt pairs to encapsulate a broader spectrum of views. Extensive experiments show that M effectively boosts the model's generalization capability, achieving stato-of-tho-art performance under various distribution shifts.
observations, we propose Multi-modal Matching Multi-
Introduction
Recent advancements Vision-Languago pre-trained Models (VLMs) such as CLIP (Radford et al. 2021 and ALIGN (Jia et al. 2021 have demonstrated impressive open-vocabiee vocabulary generalizen seneralization capabilities across various downst tasks [Li et al. 2022bb Ramesh al. 2022 Tevet al.) 2022). However, the large scale of VLMs and the scarcity of high-quality training data often make fine-tuning the entire model costly. In response, prompt learning, which appends additional, learnable continuous vectors (prompts)
Equaal contribution "Corresponding author

---

Published in Transactions on Machine Learning Research (09/2024)
All learnable features
73.00 78:19
CoOp
CoCodp
Lake
Prompt
Prompt 1
Predictive features
Prompt
00.40
Residential Buildings
Prompt
Prodictive features for 000 tast
MaPLe
Performanca//
MPPU)o)))
Permanent Crop Land
predictive features
Industrial Buildings
93
Highway or Road
Prompt
Forest
Annual Crop Land
Predictive
10
20
60
50
0OO test
Prompt
Proportion predicted misclassified
ImageNet(ID)
EuroSAT((OO)
Images(%)
Figure 1: (Left) Zero-shot performance of CT.IP ViT-B/16 on EuroSAT after few-shot fine-tuning on Ima- geNet. Existing prompt learning methods compromise the original generalization capability of CLIP. (Mid- dle) The distribution of predicted labels for misclassified images on EuroSAT of three independently trained prompls with nearly identical ID test accuracies. Different prompls exhibit distinct predictive distributions. (Right) An illustration of view bias of different prompts. Each prompt only learns a partial set of OOD- useful features and thus can only solve certain OOD tasks but not all of them.
to VLMs while keeping pre-trained weights intact, has emerged as an efficient alternative for fine-tuning VLMs [Zouu et al. 2022b Khattak et al. 2023b Lu et al. 2022 Khattak et al. 2023a Zhou et al. 2022a
Although previous prompt learning methods [Zhou al. 2022b a) have significantly enhancod the in- distribution (ID) performance of the fine-tuned models, their improvements in out-of-distribution (OOD) settings are still limited. In particular, datasets where both image and text exhibit substantial distri- bution shifts, existing prompt-base methods may even underperform zero-shot CLIP. As an example, in the EuroSAT (Helber et al. 2019) satellite dataset, existing methods reduce the OOD accuracy of CLIP by 0.07% to 7.40% after few-shot fine-tuning on ImageNet (Deng et al. 2009). as shown in Figure [ (left).
Why do existing prompt learning methods reduce OOD robustness? To answer this question, is necessary analyze what is actually learned by prompts and how it relates to generalization. In existing work, it is believed that what is learned by prompts is roughly uniquely determined by the training data and the prompt learning objective (Chen et al.) 2023). However, through an empirical study of the mistakes made by independently trained prompts in OOD settings, we challenge this belief. In particular, we observe that prompts with nearly identical ID accuracies can make very different OOD mistakes. For example, as illustrated in Figure | (middle), a set of learned prompts with almost the same ID performance exhibits dist inct incorrect image predictions. This phenomenon implies that prompts optimized under the same conditions may converge to different local optima, where the model use different features for prediction. As will be detailed in Section similar phenomena also manifest in many datascts with distribution shifts.
To investigate the inner mechanism of the above phenomenon, features of multi-modal prompts. However, the existing analysis is also limited in this direction: to our knowledge, the most relevant work is by Oymak et al. (2023). which studies uni-modal instead of multi-modal prompt learning. Moreover, they focus on analyzing the roles of the attention mechanism in prompt learning without characterizing the learned features of the prompts. To overcome this limitation, we theoretically analyze multi-modal prompt learning under a structured feature model. Compared to the work by Oymak et al. 2023), our analysis draws inspiration from recent studies the feature learning process of neural networks [Allen-Zhu & Li 2023 Shah et al. 2020) and analyzes the interaction between prompts and inputs in different modalitics. Through our analysis, we show that (1) prompt learning can be viewed as feature selection process that selects pre-trained features to match visual and textual representations on downstreem tasks, and (2) due to the multi-solution nature of the feature selection schemes, prompts may only select a subset of useful features (views), which we term as view bias. For ID data, since the features useful for prediction are often redundant (Guyon Elisscefff 2003), view bias does not impact test performance much and may even mitigate overfitting. However, in OOD scenarios where not all features useful in ID data are
need to first characterize the learned
2

---

Published in Transactions on Machine Learning Research (09/2024)
still predictive, view bias can lead to the lack of predictive features, thus limiting the generalization ability. This is consistent with our empirical observation that prompts with different OOD mistakes can still achieve similar ID accuracies. Please see Figure | (right) for an illustration.
Based on the analysis, we propose a principled Multi-modal Matching Multi-Prompt Learning (MDPL) method to mitigate the adverse effect of view bias in OOD generalization. The main idea of M is to aggregate different views from multiple independently optimized multi-modal prompt pairs. Leveraging the fact that independently trained prompt pairs tend to have different view biases, M can obtain diverse and rich collection of useful views through aggregation, hence improving generalization under distribution shifts. However, ensembling multiple prompt pairs may also lead to view redundancy that can harm generalization. To mitigate this problem, we further introduce a cross-modal contrastive regularizer to facilitate distinct views for different prompts, which also enhances the model's OOD robustness. Empirically, on the dross-dataset setting with distribution shifts in both visual and textual domains, MSPL achieves a 3.5x increase over previous methods in OOD accuracy gains over zero-shot CLIP, and significantly outperforms prior methods in terms of a complementary OOD performance measure named effective robusiness ralio.
In summary, our main contributions are three-fold:
We identify a failure mode in existing prompt learning methods under large distribution shifts caused by view bias, which provides new insights into analyzing the generalization of prompt learning.
We theoretically analyze multi-modal attention- based prompt learning, which explains the view bias phenomenon and lays a foundation for future analysis.
We propose a theoretically grounded and minimally constrained prompt learning framework, M which achieves state-of-the-arl performance in average test accuracy and effective robust ratio across three common OOD generalization settings.
2 Related Work
Vision-language pre-trained models and downstream task adaptation. Vision-Language pre- trained Models (VLMs) have achieved remarkable performance in few-shot and zero-shot recognition tasks by leveraging large-scale image-texx text paired training data to align vision and text representations (Li et al,,
2021 Jia et al. 2021 Radford et al., 2021 Kim et al., 2021). ). Supported by the expressive power of language, VLMs gain an understanding of open-world visual concepts, enabling them to adapt to various applications,
including object detection and segmentation (Li et al. 2022a Xu et al. 2022 Gu et al. 2022 Li et al. 2022b), image generation (Ramesh et al. 2022 Patashnik et al.) 2021], action recognition (eeeet et al. 2022 Wang et al. 2, etc. While VLMs provide gencralizable representations, how to efficiently adapt them to downstream tasks remains an important challenge. Prior work has proposed parameter-efficiee tun- ing methods based on CLIP, including adapter- based (Gao et al.] 2023) Zhang et al. 2022) and prompt- rompttbbased methods (Zhou et al.) 2022b a). Our work introduces multi-modal multi-prompt learning framework that,
while maintaining parameter-efficieng during adaptation, enhances the robustness of the adapted models.
Prompt learning. Prompt learning originated in the NLP domain. Early methods used expert knowledge to manually construct prompts, also known as prompt engineering (Brown et al. 2020 Petroni et al.] 2019)
Later, Jiang et al. (2020); Shin et al. (0200) proposed to automatically search for templates, and Li &
Liang (2021) y Tsimpoukelli et al. (2021) : Liu et al. (2023); Lester et al. (2021) extended the search to the continuous representation space. Recently, prompt learning has been introduced to vision tasks. JJa et al. (2022) incorporated learnable prompts in vision models. CoOp (Zhou et al.) 2022b) and CoCoOp (Zhou et al. 2022al add a learnable single prompt in the language branch of CLIP. MaPLe (Khattak et al. 2023a) extends
this approach to both vision and language branches. PromptSRC (Khattak et al. 2023b) incorporates sclf- regularization into the prompt learning process. ProDA (Lu et al. 2022) and PLOT (Chen el al. 2023)
learn multiple prompts only in the language branch; ProDA assumes a Gaussian distribution for prompts, while PLOT employs a two-stage optimization strategy basod on local features and optimal transport theory.
Unlike these methods, 2L does not require modifying the objective or assuming a parameter distribution,
enabling the learning of diverse prompts in simpler and minimally constrained manner. Wang et al. (023)
3

---

Published in Transactions on Machine Learning Research (09/2024)
attempts to utilize multiple soft text prompts and fine-tune linear classifier on them, while our M adopts the CoOp paradigm of optimizing the prefix of the prompt. On the theoretical side, there is little work analyzing prompt learning, even outside the multi-modal setting. The recent work by Oymak et al. (2028) analyzes the role of attention in prompt learning. However, their analysis focuses on the single-modal setting and considers a simplified attention model where learnable tokens are only appended to queries but not keys and values, which deviates from the multi-modal prompt learning practice.
3 Preliminaries
This section briefly reviews the prompt learning framework based on CLIP, which our method built upon. Empirically, we can also apply our method other image-text pre-trained backbones such as SigLIP (Zaii
et al. 2023 in similar fashion. Here we only focus on CLIP for simplicity.
CLIP architecture. CLIP comprises both an image encoder and text encoder and performs zero-shot classification by matching the visual representation with different textual representations corresponding to different labels. Our implementation is based on CLIP with Vision Transformer (ViT) (Dosovitskiy et al. 2021) as its image encoder. Concretely, denote CLIP's image encoder as f and text encoder as 9, with parameters denoted by Of and Og. respeclively. Both encoders consist of multi-head self-aliention layers. In the vision branch, the input image X is initially divided into N fixed-size patches {æI...., Ø Next, this patch sequence is embedded as tokens [21,..... 2N and concatenated with a lcarnable classification token z00 to form the input sequence Zo- = { 2 ,,....,,........................ =%) of the first multi-head self-altention layer. Similarly, we denote Z, = (z zi..... } as the input sequence for the (i - 1)-th layer. Finally, the classification token <Li from the output of the L-th transformer layer is mapped to d-dimensional vector in CLIP's aligned representation space, serving as the visual representation v = f(Zo: 0y) E R In the language branch, the serve as input. This input is
label is concatenated with a fixed template, such as "a photo of (label)", then tokenized and embedded form the input text token sequence Wo = {wgos: wf...., w wp. whosh (assuming the template has K tokens), where wi represents the token corresponding to the class label. Similar to the vision side, the whos from the output of the I-thh transformer layer is mapped to a d- dimensional vector, serving as the textual representation = gWWo; 0g)) E Rd.
CLIP for image classification. For classification, assuming set of C' candidate class labels {ui,... yc). the probability of a basdd model predicting the label as U% is then given by:
p(ù = y/ X) exp(cos(v. U, tx)/T)
(1)
exp(cos(v, (v, t/)
where cos().)) denotes the cosine similarity, 7 a temperature parameter, v is the output of the input image, and to represents the textual representation corresponding to label yk-
Prompt learning based on CLIP, We employ the basic method of Independent Hision-Language Prompt- ing (IVLP) (Rasheed et al. 2023) to elucidate the fundamental principles of prompt learning. At the input layer, N, and N₂ learnable tokens serve as visual and textual prompts, denoted by p. and pp: respectively. In the vision branch, p is concatenated directly with Zo. while in the language branch, pf. replaces the cor- responding tokens in Wo, resulting in new input sequences Zo- = (pp, Zo} and Wo {wSos pl, wy whoss) Given prompt. depth J, prompls will be added to the first J layers of the transformer. At the i-th layer, the input sequences are Z = {pp-1 1p-- and W = (wioss P. i-1 w whols) Note that the output tokens at the positions of the previous layer's prompts are replaced with new learnable tokens added in the subsequent layer. Ultimately, we obtain the visual and textual representation denoted by = [Zo f,, (o') and f = g( (W; 0g. (oi) respectively. During training, the pre-trained parame- ters 0, and 0g are frozen and only the learnable prompts are optimized.
4 Empirical Evidence of View Bias
This section details the experimental settings and main observations in our empirical study in Section I aaddd presents additional evidence of the view bias problem in more datasets.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Input
Prompt
Prompt
Promot
Prompt 4
Prompt 5
Prompt
Prompt 7
Prompt 8
Source
ImageNiet
Stanford Floerrs02 EuroSAT
Target
Figure 2: Visualization of attention heatmaps of different prompts on test images in ImageNet and three different target datasets. Different prompts focus varying regions of the images, with significant differences in the attention distribution. All prompts are learned using identical training data.
Table 1: Relative average Jenson-Shannon (JS) divergence of the predicted label distributions of different prompts. A larger value indicates greater disparity. We also report the relative average JS divergence (compared to the JS divergence on the EuroSAT dataset without M between the aggregated prompts trained with different initializations using our proposed MSPL. method.
Caltech
Oxford
Stanford
Flowers
Food101
FGVC
Aireraft
SUN397
EuroSAT
UCF101
Amerage
101
Pets
Cars
102
DTD
Average JS Divergence
1.043
0.520
0.590
0.850
0.254
1.470
0.358
0.516
1.000
0.579
0.718
(PuL(
0.364
0.091
0.079
0.155
0.031
0.303
0.055
0.083
0.117
0.144
0.142
A
-65%
-83%
-87%
-82%
-88%
-79%
-85%
-84%
-88%
-75%
-80%
Rasheed al. 2023 as described in Section Following standard experimental settings (Zhou et al. 2022b Rasheed et al, 2023 we train CLIP ViT-B/16 (Radford et. al. 2021 on ImageNet in a few-shot Tashion, by randomly sampling 16 images per class in training. Under identical training conditions (using the same few-shot training data on ImageNet and hyperparameters), we independently optimize a set of prompt pairs with differences only in their random Gaussian initialization.
Experimental settings. For the prompt learning method, we employ the baseline IVLP
Main results. As illustrated in Figure I (middle), although all trained prompts achieve nearly identical ImageNet (ID) test accuracy, their label prediction distributions of misclassified images on the EuroSAT dataset (OOD) exhibit significant differences. For instance, the first prompt tends to misclassify samples as "Highway or Road" whereas the third prompt tends to categorize them as "Permanent Crop Land." This phenomenon implies that prompts optimized under identical conditions can converge to different local optima, resulting in the divergence in their prediction distributions when significant distribution shifts occur.
The ubiquity of view bias. To show that view bias also exists in datasets other than EuroSAT, for every dataset, we compute its Kenson-Shannon (JS) divergence (relative EuroSAT) between independently- trained prompts' predicted label distributions for misclassified images. As shown in Table 1 the average
5

---

Published in Transactions on Machine Learning Research (09/2024)
JS divergence most datasets is comparable to EuroSAT, implying the generality of the observation in Figure 1 As another piece of empirical evidence of view bias, we present attention heatmaps of different, independentlyy-trained prompts on the source dataset ImageNet and three distinct target datasets including EuroSAT, Flowers102 (Nilsback & Zisserman 2008), and StanfordCars (Krause et al., 2013). As shown in Figure 2 the heatmaps of different prompls indeed capture diverse range of views. Concretely, different prompts exhibit significantly varied attention distributions in both TD and OOD scenarios. For example, on the ImageNet dataset, various prompts focus on different aspects: some on the whole object, some on specific parts of the object, and others on the background areas which may serve potential cues for classification. Similarly, in the StanfordCars dataset, different prompts highlight different parts of the cars, such as headlights, wheels, and windshields, while some capture more background information.
Discussion on possible extensions to larger VLMs. While our main empirical investigations are re- stricted to CLIP, we expect similar findings may also be obtained larger models. In particular, a large number of existing VLMs such as LLaVa (Liu et al.] 2024), MiniGPT-4 (Zhu et al. (023) and Instruct- BLIP (Dai al., 2023) directly leverage CLIP and its derivatives image encoders. It has been shown that fine-tuning those image encoders can lead to worse OOD performance (Karamcheti et al. 2024 similar to our empirical observation that the OOD accuracy can drop after prompt learning. We thus envision that our results may be extended to those VLMs, but leave rigorous investigations as future work due to computation constraints. Empirically, we also apply our method to SigL.P (Zhai et al. 2023 and observe similar performance gains as to CLIP (see Section 6.7).
5 Analysis and Methodology
In this section, we thcoretically analyze multi-modal prompt learning with the representation matching objective and characterize the features learned by multi-modal prompls. First, by investigating the role of the coftmax-attention mechanism in prompt learning, we show that the representation matching objective can be decomposed into complementary terms that isolate the feature selection effect of visual and textual prompts (Section 5.1 Then, we analyze the innate multi-soluttion nature prompt learning under a linear feature model and further relate this to view bias and OOD generalization failure (Section 5.2p. Motivated by our analysis, we then introduce the M framework, showing that the view bias of single prompt pair can be mitigated by aggregating the output of multiple prompt pairs (Section 5.3), and further propose cross-modal contrastive regularizer to facilitate the learning of more diverse views different prompt pairs (Section 5.4
5.1 Prompt Learning as Feature Selection
Self-attention model. We begin our analysis by introducing model of single-head self-attention, which serves as a primary building block of transformers. Concretely, let Zi = (zo...., E (N++1)xdo be the input sequence of the self-attention layer with žy being the representation token (Zela on the vision branch and wBos on the language branch). The output of the layer is then defined as
Zout o (Z Wo W 2 Z W,
(2)
where We E Rdaxm, Wx E Rd and W E RRdo are model weights, and ø is softmax nonlinearity that acts row-wise when taking into a matrix as input. We consider the case where the weights We E dd WK E Rdomm and W E Rdx have boen pre-trained and keep frozen during prompt learning. The final representation v E Rd is then mapped from the representation token in Zout, given by its first row:
T (zī We W 2ii Z W.
Multi-modal prompt learning. For simplicity, we consider appending single learnable prompt token p E x) be the new input. The new representation for Zin
p E dd to the raw input Zini let Z,m =
classification is then given by
p W
6 [6 WoWWW
2 W M
(4)
Zin W
6

---

Published in Transactions on Machine Learning Research (09/2024)
Our main observation here is that ö is weighted mirture of the raw input Zi and the prompt p. In other words, B takes the form of
i my + w) p.
(5)
where the weighting coefficient 7) is obtained by expanding and reweighting the original ceitmax-attentio map in Eq. (3, with its concrete form detailed in Appendix B.1 In multi-modal prompt learning, both vision branch and language branch have their learnable prompts. To avoid confusion, in what follows we shall use v()) and ((t) to denote visual and textual representations, respectively. For other parameters, we will use subscripts "v" and "t" to denote if they belong to vision branch or language branch.
Feature selection effect of prompts. We then introduce the common representation matching objective in multi-modal prompt learning. For a C-way classification problem with training distribution D, multi- modal prompt learning aims minimize
LCE Eg (Z log cxp(sim( i, b,)))
(6)
E exp(sim [.)
where for every label y E fl....,CO C) t, denotes the textual representations of y. and sim(.,): R x R is a similarity measure. In practice, sim(...) is often the cosine similarity as in Eq. ( In our analysis, we assume sim(..- to be the inner product (:,)). Note that inner- anner-product and cosine similarity are equivalent if we normalize the atpossentations before calculating the loss. In practice, normalizing the representations often results in comparable classification performance to using unnormalized representations (Radford all 2021 We consider binary classification setting with y E {-1,1} and + 1. This allows us to derive cleaner form of the loss function that reveals the role of multi-modal prompts, which is formally shown by Proposition !
Proposition 1 (Objective decomposition). Under the condilions stated above, we have
LCE = EZ log (1 + exp { no (n.v n (W, Pt, w) + (1- no) pt- m.tty W. po)
(2)
+ (1 (1-m)) (1t. Tt, (W, Pc, W pa) + mobne -y st n. 2)
(3)
(4)
where denotes the weighting coefficient in the vision branch, and mhy denotes the weighting coefficient in the language brunch for class E f 11,1]
Proof. The complete proofs of Proposition aand the following propositions are deferred to Appendix B
Remarks. Proposition shows that the multi-modal prompt learning objective can be decomposed into terms that reflect the similarity between (1) the textual prompt and the visual representation, (2) the visual prompt and the textual representation, (3) visual and textual prompts, and (4) visual and textual representations. In particular, the first two terms can be viewed as a feature selection mechanism that allows the model to emphasize the task-related features in both visual and textual representations by adjusting pa and p.. This also justifies the ad advantage of meull--mdal prompls as it makes the model expressible enough to accommodate distribution shifts in both vision and text domains, which we empirically verify in Secion 6.6
5.2 View Bias and O0D Generalization
Multi-solution property of prompt learning. Given Proposition I our key insight multi-modal prompt learning is that minimizing Lee can lead to multiple representation matching schemes that give similar training risks, resulting in the observed view bias of different prompls. As an example, given input Zi from a class E 1), we assume that each input token 2; for i .O, is a linear combination of set of orthogonal, unit-norm features f E (I.....,] 1) with each feature f, E Rd. Similar assumptions are common in analyzing the feature learning process of neural networks, and prior work has shown that it capture many practical feature learning characteristics (Allen-Zhu & Li 2023 Zhang et al. 2024). For
7

---

Published in Transactions on Machine Learning Research (09/2024)
simplicity, we assume that there are no "uscless" features, ie., every feature is correlated with the label on D. Then, due to Eq. 3), we can write v as also combination of those features: = -== 8g fy for some random variable 3 E IK depending on distribution 1 and pre-trained weights. Hence, by Eq. 7 we have that matching Ps and v by pushing W. Pt along the direction of any feature ft that correlates with the label y can reduce the training risk fCE. We note that the crux of the above argument exploits the inner-product term between multi-modal representations. Our analysis can thus be extended to other loss functions with inner-product terms such as the sigmoid loss used by SigLIP (Zhai et al., 2023
Given the multi-solution nature of the objective, how the finally learned prompt Ps correlates with each feature f: cannot be uniquely determined by the training distribution D and the loss function LCE, but is also determined by the concrete process of optimization, where the inductive biases of the neural network and gradient descent play a critical role. We will detail this next.
View bias in feature Icarning. Recently, several works (Allen-Zhu & Li 2023 Zhang & Bottou 2023
show that independent neural networks with the same architecture trained by gradient descent. can converge to different optima that each extracts only a subset of useful features. This phenomenon also relates to the simplicity bias of neural networks (Shah et al. 2020), iee, neural networks may prefer "simple" solutions, such as using only a subset of useful features for classification, over "complex" solutions, such as using all useful features. In our prompt learning setting, such a bias corresponds to (fj, W, pt) * 0 holding for only j E S with S being a subset of (I...../} (}, which we name view bias. Empirically, we show that such a view bias indeed holds in practice in Section | Yet, rigorously proving it is challenging due to the requirement of analyzing fine-gre -haigra gradient descent dynamics, which we leave as future work.
While view bias may benefit in-distribution generalization by serving as implicit regularization, Proposi- tion 2 formally shows that it can adversely harm generalization under certain distribution shifts.
Proposition 2 (View bias can harm generalization). Under the conditions stated above, consider a test distribution D' satisfying 1 = ED1 for every j E S and Epin-=13 # ED'Ig=-18 for every j J S with S' C [L,,...,4 \ SS. Assume that the weighting coefficients satisfy nt,1 = ft.-1. We then have > (v, t.y]
Prizm.e) (W Eg) > (v, = Pr(zzm.)
(8)
Remarks. Proposition 2 reflects a distribution shift scenario where only a feature subset S' remains useful in the test distribution D'. In an extreme case, if this useful subset S does not overlap with the feature subset S extracted by prompt learning, then prompt learning would essentially lead to no improvement in test accuracy since no additional useful fcature is properly conditioned during prompting. To make matters worse, when the learned features S have spurious correlat ions with labels (Simon) 1954; Schölkopp el al. 2021 or contain large noise, reer-neliance on those features by prompt learning may even decrease distributional robustness. This is consistent with our empirical observations that in some cases, prompt learning does not improve the performance of CLIP under large distribution shifts and sometimes even decreases it.
5.3 PL Multiple Prompt Pairs and View Aggregation
Motivated by the above analysis, this section proposes M that aims to mitigale the intrinsic flaw of view bias in prompt learning by introducing multiple, paired multimodal prompts and aggregating their views.
Incorporating multiple prompt pairs. Specifically, building upon the vanilla prompt learning approach in Section 3 for each layer in the first J layers of CLIP's vision branch, we introduce M sets of learnable prompts, denoted as P=,1 = {p', 1/ ..... Po,M = {p' (pr 1/=0 Symmetrically, in the language branch, we also add M sets of learnable prompts in the first J layers denoted by Pt. for i E (1,..., M). For each i, we treat the visual prompt set pa, and the textual prompt set pt,i as a prompt pair. The input sequences for the i-th prompt pair and the j-th layer in the vision and language branches are then given by
ž = (p Z'}, w! = { {wsos: pi., wg, weost-
(9)
Paired representation matching with multiple prompts. In the forward process, we obtain visual and textual representations for all prompt pairs, given by é. f( s(ZP; O,: Po.)) and i, = g(W, 0g Pt.)) for
8

---

Published in Transactions on Machine Learning Research (09/2024)
Visual Representations with Multiple Prompts
Contrastive Regularization
A Batch of Images
1
V1
1
1/2
12
Image Encoder
Random Sampling
concat
:
:
13
Pairing Visual and Textual Prompts
Visual Prompts
Paired Representation Matching
1
Textual Prompts
A Batch of Labels
71
22
yz
"bagel"
concat
Text Encoder
y
22
yz
Random Sampling
Textual Representations with Multiple Prompts
Figure 3: The M framework. We introduce multiple paired visual and textual prompts and jointly optimize each prompt pair using representation matching (Section 5.3 Meanwhile, randomly sample from the multiple prompt representations corresponding to each example for contrastive regularization to further enhance the learning of more diverse prompts (Section 5.4). We use superscripts to denote the indices of prompt pairs in the figure for visual clarity.
the i-th prompt pair. During training, we sum the representation matching loss for every prompt pair:
M
exp(cos(o, fy,:)/T) Ey= exp(cos((;; 6; {))))
Lmatchh Exx.) - log
(10)
1=1
where for each y C), by denotes the textual representation corresponding the label y for the i-th prompt pair. During inference, average the prediction logits obtained from all prompt pairs.
Exploiting view bias by aggregating different views. The key intuition of our approach is that we empirically observe in Figures ! and 2 independently trained prompts tend to have distinct view biases. Hence, aggregating them naturally results in a richer collection of useful features. Formally, Proposition 3 demonstrates that if independently optimized prompts extract independent feature subsets, then aggregating them by simply averaging their representation matching scores can benefit OOD generalization.
Proposition (Effectiveness aggregating multiple views). Under the same conditions as in Proposition consider M prompls that each independently extracts feature subset S., (1,..., M) with |SP]| = s and the elements in each S; uniffrrmly drawn from {I....,.}. We then have
M5
Pr Pr(zunz/)-D
Σ
(0, Eyya)
Σ (v, 8 (ns)
> Pr (v, b-y))
(11)
with probability al least 1 e @ (-IS')
Remarks. Proposition 3 assumees a scenario where different prompts learn independent views, while only some of them remain useful OOD data. Since we cannot determine which views are useful solely basod on ID data, simply aggregating all of them seems be fair approach as adopted by M Yet, such aggre- gation may also induce redundant views, which is indeed observed in experiments (see Appendix C.4.1
9

---

Published in Transactions on Machine Learning Research (09/2024)
Thus, it would be more efficient to actively incentivize different prompts to learn more diverse views. In the next section, we propose cross-modal contrastive regularization to achieve this goal.
5.4 Cross-Modal Contrastive Regularization
To further enhance the diversity of the learned views of different prompts, we introduce a cross-modal contrastive regularization penalty. The main idea to mazimize the representation difference between different prompt pairs while matching the representations in the same prompt pair. Concretely, given batch of examples "r",."")... (XB y))), for every example, we randomly sample prompt pair (())) P where r(i) E [l...., 3 denotes the prompt pair index for the i-th example in the batch. We then calculate the cross-model contrastive regularization penalty by
exp(cos(@i.r0)) (0)) log exp(cos(Ui.r()) E ((),)(1//T)) Σ k= exp(cco((Uk.r(1) by (1)rr))/7))
Coontrast Σ
log B Lk=1
(12)
exp(cos( (i)) Eg (())
k=
Our overall training objective then given by:
Lattch + ACeontrast,
(13)
where A > 0 is the balancing coefficient.
Explanation on contrastive regularization. Contrastive loss aims to pull positive examples together and push negative examples apart. Here, we treat the visual and textual represcntations of the same imput with the same prompl pair positive examples and all other cases negative examples. In other words, representations with different prompts would become negative examples and are thus pushed apart, even for the same input. This relates to the effect of "class collision" that have been observed in the contrastive learning literature (Goyal et al. 2023). However, instead of mitigating this effect, we actively leverage it to encourage different prompt pairs to learn more diverse views. We empirically verify this in Secion 6.6
6 Experiments
6.1 Protocols for Evaluating Generalization Performance
In prompt learning, the average accuracy OOD test sets is commonly used to evaluate a model's gen- eralization performance (Zhou et al., 2022a Khattak et al. 2023a b). However, Taori et al. (2020) points out that OOD accuracy is insufficient to reflect the accuracy drop under distribution shifts after fine-tuning, And Miller et al. (2021) finds through large-scale experiments that there a strong correlation between a model's OOD and ID performance, suggesting that improvements in OOD accuracy cannot be entirely attributed to the fine-tuning methods. Instcad, may simply be due to better fit on the ID distribution. Therefore, to comprehensively evaluate the generalization performance of prompt learning methods, we pro- pose the effective robustness ratio, inspired by Taori et al. (020)) complementary metric to average OOD accuracy. Its expression is follows:
p(f) = aCCoodlF) - acCood (fo)
(14)
acCza()) acCra(fo)
where fo is the zero-shot CLIP, f is the fine-tuned model, and acCood()) denotes the average OOD accuracy.
Discussion on effective robustness ratio. This metric measures the relative accuracy drop under dis- tribution shifts for the fine-tuned model compared to the pre-trained CLIP. Generally, p(f) < 0% indicates that the model has overfitted to the ID distribution. For p(f) E (0%, 100%), the larger p(f), the smaller the compromise of fine-tuning methods on the generalization ability of CLIP and the greater the generalization ability. In particular, when the ID and OOD distributions are nearly identical, p(f) approaches 100%.
10

---

Published in Transactions on Machine Learning Research (09/2024)
6.2 Experimental Settings
Base-to-new generalization. This setting validates the model's capacity to generalize unseen classes during fine-tuning. We equally divided the classes of cach datasct into new and base scctions. After training base classes, the model is directly zero-shot lested on new classes.
Cross-dataset generalization. To verify the generalization performance of our method when both the vision and language modalities distributions shift during testing, we fine-tune the model on ImageNet and then conduct zero-shot testing directly on other downstream datasets.
Domain generalization. Unlike the previous settings, which exhibit significant distribution shifts in both vision and language modalities, DG shows distribution shifts only in the vision modality and is not the main focus of our method. Nonetheless, our proposed M still achieves comparable performance in the DG setting, slightly surpassing previous prompt-based algorithms. Details are provided in Appendix C.3.3
Implementation details. Following MaPLe (Khatiak et al. 2023a), employ the ViT-B/1t 16 based CLIP as the backbone. We use a few-shot setting that samples 16 shots per class and report the results averaged over three runs. For M we use a normal distribution with a mean of zero to randomly initialize the prompts, and increase the variance with the number of prompts (M) to ensure diversity. In our experiments, we set M to 8. Due the use of Ceontrast, we use larger batch size while reducing the training iterations compensate for the computation overhead. Since ProDA lacks an official implementation, we report the results in Derakhshani et al. 2023) for the Baso-to-New sctting. For other baselines, reproduce the results based on the provided rapeerparaeeeerss. Please refer to Appendix C.1 for additional training details.
Datasets. For cross-dataset generalization and from base-to-new generalization settings, we follow the protocols of Zhou et al. [2022a]] b): Khattak et al. (2023a) and consider 11 recognition datasets, including
ImageNet (Deng et al. 2009) and Caltech101 (Fei-Fei Fei et al. 2004) for generic recognition, OxfordPets PPrrkh
ct al. 2012), Stanford (Krause et al. 2013)) Flowers102 (Nilsback & Zissorman 2008), Food101 (Bossard
et al. 2014 and FGVCAireraft (Maji et al. 2013 for fine-grained classification, SUN397 (Xiao
2010
for scene classification, DTD (Cimpoi et al.
2014] for texture recognition, EuroSAT (Helber et al.
2019 for
satellite image recognition, and UCF101 (Soomro et al.
2012) for action recognition
Baselines. We use zero-shot CLIP (Radford et al.
2021
CoOp Zhou et al. 2022b), CoCoOp (Zhou et al.
2022a), ProDA (Lu et al. 2022 MaPLe (Khattak et al.
2023a and PromptSRC (Khattak et al. 2023b
6.3 Fase-to-New Generalization
In the generalization from basc to new classes, shifts in both modalities occur due to partial observations during fine-tuning. In Table 2 M demonstrates superior performance across all average metrics on 11 datasets, comprising base and new class test accuracy, harmonic mean accuracy, and effective robustness ratio. In tests on new classes, M consistently outperforms the state-of-the-art. PromptSRC in 9/11 datascts, improving the average accuracy by 1.03% without compromising base class accuracy. It worth mentioning that on the larger-scale dataset ImageNet, MSPL surpasses PromptSRC by 1.13% in zero-shot new class test accuracy. Full results detailed in Appendix C.3.1
6.4 DattaeD Generalization
Table 3 illustrates that M substantially improves both the average zero-shot test accuracy and the effective robustness ratio in the cross-dataset generalization setting with shifts in both vision and language modalities. Compared to zero-shot CLIP, existing methods only achieve modest increase of 0.61% in average accuracy, whereas M rcalizes substantial improvement of 2.16%. Even excluding the superior performance on the EuroSAT datasel, where accuracy increased by 9.12% compared to zero-shot CLIP, M still demonstrates an average accuracy gain of 1.39%. Against the state-of-the-art PromptSRC, M excels in 8/10 target datascts, boosting the effective robustness ratio by 2.8 times without markedly affecting ID performance. These results highlight the exceptional robustness of our framework in handling distribution shifts. Full results detailed in Appendix C.3.2
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Comparison with previous methods in base-to-new generalization. All baselines are reproduced with reported parameters. HM and p(f) refer to harmonic mean and effective robustness ratio, respectively.
Dataset
Zero-shot CLIP
CoOp
CoCoOp
ProDA
MaPLe
PrompISRC M'PL(Ous))
Base
69.48
82.29
80.49
81.56
82.00
84.18
84.90
Average
New
74.28
68.78
72.04
72.30
74.88
75.76
76.81
11 datasets
HM
71.80
74.93
76.03
76.65
78.28
79.75
80.65
p(f)
-43%
-00%
-16%
5%
10%
16%
Base
72.37
76.47
75.93
75.40
76.87
77.80
77.72
ImageNet
New
68.10
67.50
70.13
70.23
70.73
70.60
71.73
HM
70.17
71.71
72.91
72.72
73.67
74.03
74.60
Base
97.22
98.10
97.80
98.27
97.93
98.10
98.45
Caltech101
New
94.21
93.20
93.00
93.23
95.23
94.03
94.25
HM
95.69
95.59
95.34
95.68
96.56
96.02
96.30
Base
91.23
94.53
95.08
95.43
95.60
95.33
95.85
PetordPets
New
97.20
95.80
97.73
97.83
97.27
98.15
HM
94.12
95.16
96.36
96.62
96.79
96.29
96.99
Base
63.69
75.60
70. 73
74.70
72.40
78.13
79.07
StanfordCars
New
74.92
70.03
72.50
71.20
73.67
75.37
74.03
HM
68.85
72.71
71.60
72.91
73.03
76.73
76.47
Base
71.70
97.53
94.43
97.70
96.10
98.17
98.17
Flowers102
New
77.52
71.30
70.63
68.68
72.87
77.37
75.72
HM
74.50
82.38
80.81
80.66
82.89
86.54
85.50
Base
90.07
89.50
90.57
90.30
90.83
90.63
90.85
Food101
New
91.17
88.90
91.27
88,57
92.03
91.47
92.08
HM
90.62
89.20
90.92
89.43
91.43
91.05
91.46
Base
27.55
38.67
35.33
36.90
36.17
42.27
42.88
FGVCAircraft
New
35.93
29.80
31.07
34.13
34.87
37.43
39.11
HM
31.19
33.66
33.06
35.46
35.51
39.70
40.91
Base
69.38
81.20
79.37
78.67
80.97
82.77
82.84
SUN397
New
75.58
70.43
76.23
76.93
78.30
78.50
78.85
HM
72.35
75.43
77.77
77.79
79.61
80.58
80.80
Base
53.13
79.67
76.93
80.67
80.47
82.97
83.91
DTD
New
60.27
49.37
54.67
56.48
58.40
59.57
61.75
HM
56.48
60.96
63.92
66.44
67.68
69.35
71.14
Base
56.98
88.97
87.10
83.90
91.07
92.70
96.72
ET
New
63.74
56.00
61.87
72.83
73.17
77.94
HM
60.17
68.74
72.35
73.88
80.94
81.79
86.32
Base
70.99
85.00
82.13
85.23
83.57
87.10
87.40
UCF101
New
78.47
64.20
73.30
71.97
76.73
78.57
81.29
HM
74.54
73.15
77.46
78.04
80.00
82.62
84.23
Table 3: Comparison with previous methods in the cross-datasel generalization. All baselines reproduced
with reported parameters. M shows significant improvement in the effective robustness ratio.
Source
Target
ImageNet
Caltech
Oxford
Stanford
Flowers 102
Food101
FGVC
Aircraft
SUN397
EaroSAT
UCF101
Average
Pets
Cars
DTD
101
Zero-shot CLIP
66.68
93.31
89.10
65.51
70.73
85.88
24.66
62.60
44.09
48.40
67.59
65.19
CoOp
71.63
93.73
88.27
63.63
68.70
85.63
19.27
64.60
41.63
48.33
67.37
64.12
-22%
CoCoOp
71.20
94.43
90.60
64.83
71.03
86.13
23.23
67.20
45.87
41.00
68.50
65.28
2%
MaPLe
70.40
93.53
90.03
64.90
71.93
85.97
23.77
66.67
45.03
44.87
67.47
65.42
6%
PromptSRC
71.37
93.37
90.30
65.70
70.43
86.47
23.57
67.43
45.83
45.43
69.50
65.80
13%
M M'PL(Our)
71.11
93.94
90.88
66.35
72.26
86.56
24.23
67.65
45.61
57.52
68.45
67.35
49%
12

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Ablation on each component M framework in the cross-dataset generalization setting.
Method
ID Acc.
Average OOD Acc.
p(f)
Zero-shot CLIP
66.68
65.19
Independent V-L Prompting (IVLP)
71.16
65.22
1%
+ textual
71.00
66,15
22%
+ visual multi-prompts (w/o matching)
70.99
66.64
34%
+ visual multi-prompts
70.95
66.87
39%
Loontrast (w/o collision)
70.96
66.82
38%
Coontrast
71.11
67.35
49%
6.5 Performance Analysis
While the empirical evidence in Section implies the prevalence of view bias in prompt learning, our proposed M algorithm, which leverages view bias, yields varying degrees of improvement across different datasets in the cross-dataset generalization setting. This section provides an in-depth analysis of this phenomenon.
Theoretical interpretation. As shown by Proposition minimizing the representation matching objective can be viewed as implementing a feature selection mechanism for both visual and textual pre-trained features. Hence, the degree of improvement of ML on specific dataset depends not only prompt learning but also the overall quality and adaptability of CLIP's pre-trained features and features that are learnable in downstream ID data. For example, if pre-trained features are not predictive or the downstream ID data lacks predictive features under distribution shifts, prompt. learning may not improve the OOD performance much.
Empirically, examine the quality and adaptability of pre-trained and ID features, we design two com- plementary metrics. (1) Informativeness: the generalization potential of CLIP's pre-trained features on specific target dataset, measured by the average performance of zero-shot CLIP and the linear probe CLIP's features this dataset. (2) Transferability: the distributional similarity between target datasets and ImageNet, measured by the average of the cosine similarity between visual and textual representations of examples from the two datasets. We then examine the linear relationship between those metrics and M performance gains compared to zero-shot CLIP. More details are provided in Appendix D
Results. On the EuroSAT dataset, where CLIP's pre-trained features' capability and the visual-textual joint distribution similarity are both high, only MdPI fully realizes the above theoretical potential. Conversely, the FGVCAircraft dataset, both metrics are lower, resulting in poor prompt lcarning performance. Nev- ertheless, MSPL still performs best among existing prompt-based methods. Performance on other dalasets can also be explained by these two metrics. For a detailed analysis, refer to Appendix D.3
Reduction in JS Divergence. Morcover, we report in Table the relative average JS divergence between the predicted label distributions of aggregated prompts trained with different random seeds and initializa- tions using MLL Comparing it to the results without M we observe that the average JS divergence significantly decreases in all datascts, with average reduetion rate of 80%. This further demonstrates the effectiveness of MLL in mitigating the view bias problem.
6.6 Ablation Study
Effectiveness of M As shown in Table the baseline IVLP, impaired by view bias, shows negligible improvement in term of OOD accuracy over zero-shot CLIP. In contrast, integrating multiple prompts significantly enhances OOD accuracy (rows 2-4), supporting our theoretical analysis in Section 5.3 The incorporation of visual prompts further improves model performance (rows 3-4), corroborating our analysis of the multi-modal prompt. learning objective in Section 5.1 The introduction of the matching design in Cmulti (row 4) proves more effective than scenarios where interplay exists among different prompt pairs (row 3). In addition, without Lcontrasts the OOD accuracy of M already 1.68% higher than CLIP, indicating that it effectively exploits view bias and enhances generalization under distribution shifts. Please refer A Appendix C.4.1 for ablation study the number of prompts.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Cross-modal contrastive regularization. In Table 4 further delincate the contributions of proposed cross-modal contrastive regularization. Compared to scenarios with class collisions (row 6), avoiding collisions (row 5) does not enhance generalization performance with the addition of the Lcontrast This observation aligns with our discussion in Section 5.4 indicating that our proposed regularization objective effectively steers different prompts to learn diverse views, thereby improving OOD performance. We also conducted additional ablation studies on the impact of the and batch size in Appendix C.4.2
67.00
66.50
Average Accuracy
66.25
Accuracy
71.00
66.00
70.75
70.50
65.50
70.25
ID
65.25
Avg. OOD Acc.
70.00
65.00
0
10
14
16
18
The number of prompts (M)
Figure 4: Ablation on the number of prompts in the cross-dataset generalization setting. The left vertical axis represents the ID) test accuracy on ImageNet, and the right vertical axis indicates the average zero-shot OOD test accuracy across target datasets. The trends of ID accuracy and average OOD accuracy with the number of prompts M (A 0) are depicted by curves with circular and square markers, respectively.
The number of prompts. Based on the ID and OOD performance with different numbers of prompts on the held-out validation set, set the number of prompts (M) to 8 in the experiments achieve optimal OOD accuracy while maintaining satisfactory ID accuracy. Figure 5 presents the ID and OOD test accuracy curves as M varies in the cross-dataset generalization setting. With an increase in M, ID accuracy initially increases slightly but then decreases due to feature redundancy. In contrast, OOD accuracy significantly improves first and then gradually declines. This behavior is attributed to the lack of view diversity when M is small, and feature redundancy becomes dominant and causes performance degradation when M large.
Computational overhead. MaPLe sets the prompt length for both visual and language branches half that of uni-modal prompt learning methods like CoOp, resulting in total floating-point operations (FLOPs) comparable to CoOp (Khattak et al. 2023a). Following the same prompt. length and depth settings MaPLe, the FLOPs of MDPL are approximately Mx those of CoOp. Since M requires learning multiple prompt pairs, it incurs a higher time cost. To address this, we reduce the number of training iterations, ensuring that the total training time for M is comparable to that of PromptSRC. Table presents a practical comparison between our M&PL and previous prompt learning methods in terms of training time and inference speed. The training time for all methods measured under the base-to-new generalization setting, using single Nvidia A100 GPU on the SUN397 dataset for complete training. Compared to the state-of-the-arr PromptSRC, our M model reduces training time by approximately 5% when A 8, but at the expense of 44% decrease in inference speed. Notably, as shown in Figure 5 with M = 4, M already achieves significant improvements over existing methods. Therefore, using fewer prompts can be an option, which sacrifices part of generalization performance for a large increase in computational efficiency.
Comparison to temporal prompt aggregation in PromptSRC. Please refer to Appendix C.4.3
Prompt length and depth. Please refer to Appendix C.4.4
14

---

Published in Transactions on Machine Learning Research (09/2024)
CoOp
CoCoOp
MaPLe
PromptSRC
M (Ours, = 8)
Training time (min)
8.52
56.23
6.17
22.85
21.77
Inference time (images/s)
323.1
19.0
328.2
329.7
183.4
Table 5: The practical training and inference time comparison with previous methods (M = 8).
6.7 More Results on SigLIP
We extend experiments to the state-of-the-art vision-language pre-training model, SigL.P Zhai et al. 2023), which uses pairwise sigmoid loss. The results demonstrate that M similarly enhances the generalization performance of fine-tuned SigLIP, validating the scalability and universality of our approach as robust prompt learning method for large multi-modal models. For detailed experimental results, please refer to Appendix C.5
7 Discussion
Limitations. ML adopts a straightforward aggregation stratogy averaging different prompts' logit scores. While being simple and empirically effective, this design choice may lead to suboptimal generalization on specific OOD tasks due to feature redundancy. Additionally, our experiments are currently limited CLIP/SigLA ViT-B/16 and visual recognition tasks, although we expect that our results may also hold for other backbones well.
Future work. The currently rapidly evolving test-time prompt tuning methods (Shu et al. 2022 could potentially serve as an effective means to filter the optimal prompts learned by MPPL. Furthermore, our method has the potential to extend to larger-scale VLMs and more diverse tasks. We hope that M a theoretically grounded, highly scalable, and minimally constrained framework, will establish itself as a universal bascline of regularized prompt learning methods and facilitate future rescarch this domain.
Broader Impact Statement
This work is devoted to developing more robust ways to fine-tune VLMs. Therefore, it may benefit the broad research area building machine learning models that are robust, generalizable, and trustworthy. However, it may also inherit the negative socictal impact of the original VLMs, such as potential misuse cases, biasod output, and privacy and security concerns.
Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under STI 2030-Major Projects 20212D0200300, and in part by the National Natural Science Foundation of China under Grant 62176133.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. In ICLR, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In ECCV, pp. 446 461, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncc- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Grelchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, volume 33, pp. 1977-1901, 2020.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT: Prompt learning with optimal transport for vision-language models. In TCLR, 2023.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, June 2014.
Wenliang Dai, Jumnan Li, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: To- wards general-purpose vision- rision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Lí, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248 255, 2009.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G. Turrisi da Costa, Cees G.M. Snock, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for imago-language model generalization. In ICCV, pp. 15237 15246, 2023.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, pp. 178 178-178,11 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongleng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- language models with feature adapters. 1JCV, pp. 1-15, 2023.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetume like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, pp.
June 2023.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
Isabelle Guyon and André ElisseefT. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157 1182, 2003.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 12(7):2217 2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robust A critical analysis of out-of-distribution generalization. In ICCV, pp. 8340 8349, October 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, pp. 15262-15271, June 2021b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, volume 139, pp. 4904-4916, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pp. 709-727, 2022.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423 438, 2020.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. arXiv preprint (xXiv:2102.07866 2024.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pp. 10113-19122 June 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. Tn ICCV, pp. 15190-15200, October 2023b.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and -adgaauaagg transformer without convolution or region supervision. In ICML, volume 139, pp. 5583 059-5594, 2021.
Jonathan Krauso, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops,
June 2013.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficiett prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045- 3059, 2021.
Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In ICLR, 2022a.
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrIPS, volume 34, pp. 9694-9705, 2021.
Liunian Harold Li, Pengchman Zhang, Haotian Zhang, Jianwei Yang, Chumynan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In CVPR, pp. 10965-10975, 2022b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anreua Meeling of the Associalion for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 1582-4597, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. CVPR, pp. 5206 5215, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthow Blaschko, and Andrea Vedaldi. Fino-grained visual classification of aircraft. arXiv preprint orXiv:1066.00 5151, 2013.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between out-of-distribution and in-distribution generalization. In ICML, volume 139, pp. 7721-7735, 2021.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722 729, 2008.
Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis. On the role of attention in prompt-tuning. In ICML, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, pp. 3498-3505, 2012.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-On Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, pp. 2085-2084 2021.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlaa Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pp. 8748-8788 8448-776,, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint orXiv2220.001220 2022.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fino-tuned clip models are efficient video learners. In CVPR, pp. 6545-6554, June 2023.
Benjamin Recht. Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In ICML, volume 97, pp. 5389 5400, 2019.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612-634, 2021.
Harshay Shah, Kaustav Tamuly, and Aditi Raghunathan. The pitfalls of simplicity bias in neural networks. In NewrIPS, 2020.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Fric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, 2020.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuming for zero-shot generalization in vision-language models. In NewrIPS, volume 35, pp. 19774-1289, 2022.
Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Asso- ciation, 49(267):467-44) 1954.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.4402, 2012.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mca- suring robustness to natural distribution shifts image classification. In NewrIPS, volume 33, pp. 18583 18599, 2020.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In ECCV, pp. 358-374, 2022.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. In NewrIPS, volume 34, pp. 200-212, 2021.
Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alex Schwing, and Heng Ji. Learning decompose visual features with latent textual prompts. In ICLR, 2023.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global representations by penalizing local predictive power. In NewrIPS, volume 32, 2019.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.088 08472, 2021.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba, Sun database: Large- scale scene recognition from abbey to zoo. In CVPR, pp. 2010.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pp. 18134-18144, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, Sigmoid loss for language image pre-training. In ICCV, pp- 11975-11986, October 2023.
Jianyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. ICLR, 2023.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV, pp. 493 510, 2022.
Tianren Zhang, Chujie Zhao, Guanyu Chen, Yizhou Jiang, and Feng Chen. Feature contamination: Neural networks learn uncorrelated features and fail to generalize. In ICML, 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In CVPR, pp. 16816 16825,,,,, June 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337 2348, 2022b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXim:2344,0059: 2023.
19

---

Published in Transactions on Machine Learning Research (09/2024)
A Appendix
In the appendix, provide additional supplementary information, including proofs of the thcoretical deriva- tions presented in the main text, further implementation details, comprehensive experimental results, addi- tional ablation studies, and visualization outcomes. The specific organization of the appendix sections is as follows:
Proofs of Theoretical Results (Appendix
B
Proof of Eq. (5) (Appendix
B.1
Proof of proposition
(Appendix
B.2
Proof of proposition
Appendix
B.3
Proof of proposition [Appendix
B.4
Additional Experiment Details (Appendix
Implementation details (Appendix C.1
Datasets (Appendix C.2
Full results (Appendix C.3
Additional ablation study (Appendix C.4
Supplementary Experiments (Appendix D
Linear Probe Experiment (Appendix D.1]
Datasets Representation Similarity Experiment (Appendix D.2
Multivariate Linear Regression (Appendix D.3
B Proofs of Theoretical Results
B.1 Proof of Eq. (
exp(2n Wo W(,
exp(2] WoW
Proof. Define
Vi (O,...,N), ap Σ exp(=] Wo W x;) and the weighting
21=0 exp(=o We WoW
Σn exp(20 WoW
coefficient exp(2,] Wo W p)++ exp( ( WoW
Applying the definition of a; to Eq. (3) gives
W
Σ
0,24
(15)
Plugging the above equations into Eq. 4
then gives
N 8 = W
na;2: + W 7app
(16)
k=0
na (1 11) W p
as desired.
B.2 Proof of Proposition| 1
Proof. Applying Eq. 5 to both vision and language branches gives
E -= (1 - nu.) n.) WT P.
(17)
and
7, by + ne. Pt, Vy E (-1,1).
(18)
20

---

Published in Transactions on Machine Learning Research (09/2024)
Recall that the cross-entropy loss with sim(:.-) being the inner product, C = 2 and can be written
exp E,,)) LCE EiZi,.) logg exp((0: (Ü, ty) + exp( (Ü, t-y))_ B(Zm. log (1 + exp((O, (_g) (0, b)
(19)
where
(ö, (g) (0, [ng
= (nov + (1 - W. Pv. Pu: :),ytt)) + (m v + (1 Th:) W. Pu. (Th. - - We P:)
(20)
-oo(ne.-utty me u) t/ (Nt. 7t, -w,-y/ (v, W pc)
+ (1 (1-7)) (Mt. - bhy W po) + (1 - T/) (Tr. B, -. Po. W, pr).
Finally, plugging Eq.
20 into Eq.
19 completes the proof.
B.3 Proof of Proposition
Proof. Applying Eq. the language branch gives
ty = nh + (1 - (1.21 W, Pt, Vy (-1,-1,
(21)
Combining the above equation and
Σ Byff, Vy [-1,1}
(22)
yields that for every
(w, = Σ t,
(23)
Σ
By (nr (fs, ty) (1- (f,, W. p.))
5=1
Since (f,. W.T p.) = -0 for every j $ S, we have for every y that
(0, Ey = nrs
Σ 1=1
B, + (1 Thy
Σ
3 W, pc).
(24)
jES
Since for every E S, the random variable Bi satisfics Ep 33
E E fy-yy=11. we have for every y that
B B(zn..)~ (v. Eny) = n1 E Z.n wn D' Σ B byy + Ci
(25)
= nt. E (v, tyy + Ct,
where Cj is a constant that does not depend on Using the assumption that 17,11 = n1,2) we then have
Pr(z.ny> ((v, Egy > (v,t Pr(2mn.))D' (n, byy + C1 m, (n.-y(u,ta)) Ci) -Pr(z..))>D ((v > (0, t-)t-))]
(26)
This completes the proof.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Remark 1. In Proposition 2 make the assumption that the coefficients R,, and f2 are equal in order to simplify our analysis. However, in practice, this assumption may not hold since those coefficients also depend on the pre-trained attention weights as well as the learned prompts. We posit that this discrepancy may lead to the scenarios where single-prompt learning can also harm OOD generalization due to the overfitting of "Ity lo the training distribution, as we empirically observed the experiments.
Remark 2. In our analysis, the learnable prompts in the vision and language branches have a symmetric structure. Therefore, the analysis in Proposition can directly extended to the setting where both the vision and language branches have learnable prompts.
2
B.4 Proof of Proposition
3
Proof. Akin to the proof of Proposition 2 we have for every that
(v, Ey) = n1.
Bylf,, ty) + (1 - m.y) Bo W, Pt).
(27)
JES
Proposition 2 indicates that for every i, if the i-th prompt extracts a feature subset St, E fl....,M with S; E S' Ø, then
(r [(v, ty,s) (v, (u, (v.)] Pr(zuns)~ (V,, ty.i) > (v, t t-y.))]
(28)
for this prompt. Conversely, if n'' # Ø, then the second term in the RHS of Eq. (27 also depends on the label y, which leads to better accuracy since it brings more expressibility by incorporating more predictive features (note that By also depends on the prompts). We thus have
Pr Pr(Zaa))> [(v, Egyt > (v, f E-p Pr.zun))) ty.i) (v, t y.y)
(29)
for this prompt. Therefore, as long as at least one prompt satisfies St n + Ø, we must have
M
Pr (r D
E(v, > Ev E
E v.n
(30)
i=1
Pr(zma))= ((v, ty) (v, twyl
We then formally characterize the above probability (Denoting the event that the incquality 30 holds by E). Since here work with the simple case that different prompts extract independent feature subsets that are uniformly drawn from (1....,I). the probability that at least one prompt extracts features in S' is given by the union bound over M Bernoulli distributions, each with failure probability p G-1s4
Pr(E) =
p MM
C Ci-1st
(31)
=
Cf
Consider the case where I is sufficiently large, we have Cf = ((1)) and Ci-ss" (((-))))))) Plugging them into Eq. (3 gives the desired result.
C Additional Experiment Details
C.1 Implementation Details
Base-to-new generalization. Following the settings of MaPLe, we the prompt depth J to 9, and the length of both visual and textual prompts to 2. Due the significant impact of class collision the
22

---

Published in Transactions on Machine Learning Research (09/2024)
effectiveness of our proposed cross-modal contrastive regularization objective, and its frequency being related to the ratio of batch size to the number of classes in dataset, we employ varying batch sizes across different datasets to maintain this ratio around 0.6 (e.g., 32 for DTD (Cimpoi et al. 2014). 64 for UCF1101 (Soomro ctt al. 2012)); ). We utilize an SGD optimizer with a learning rate of 2.5c-3, weight decay of 5e-4, and training for 30 epochs (for a few datasets prone lo overfiuting, the training was limited 20 epochs). The number of prompts, M, is set to 8, with balance coefficient, X, of 1.0 (and 1.2 for EuroSAT). For computing the effective robustness ratio, we use the average zero-shot test accuracy of new classes across 11 datasets acCood (+), and the average test accuracy of base classes acC,a(-).
Cross-datasct generalization. Following the settings of MaPLe (Khattak et al. 2023a we sct the prompt depth J to 3, with both visual and textual prompts having a length of 2. Due to the incorporation of a cross-modal contrastive regularization objective, we utilize a larger batch size of 512. To compensate for the additional time expenditure, all the models are trained for only 50 epochs (1550 iterations, compared MaPLc's 20,000 iterations). The model optimized using SGD with a learning rate of 2.5c-3 and a weight decay of 5e-4. We set the number of prompis M to 8 and the balancing coelficient À to 1.0. For caleulating the effective robustness ratio (p(f)), use the average zero-shot test accuracy on the target dataset acCood and the test accuracy on ImageNet as acc,(())
Domain generalization. Same as the cross-datasel generalization, we set the prompt depth, J, to be 3, with both visual and textual prompts having a length of 2. We employ the SGD optimizer with a learning rate of 2.5 x 10-3, weight decay of 10-4, and a batch size of 512, training the model for 50 epochs. The number of prompts, M, is set 8, and the balancing parameter, X, is 0.1. For the effective robustness ratio, we the average zero-shot test accuracy the target dataset as aCCood()) and the test accuracy on ImageNet as aceia(-).
Reproducibility. We provide publicly the source code of M which contains the configuration files we used, to ensure the reliability and reproducibility of our experimental results. All experiments are conducted on NVIDIA A100 GPUs.
C.2 Datasets
ImageNett (Deng al., 2009): The ImageNet dataset contains over 14 million high-resolution images, manually annotated, and categorized into 1000 classes. It is widely used for image classification and object detection tasks.
Caltech101 FFei-Fei et al., 2004): The Caltech101 dataset includes 101 object calegories and 1 background category, with 9k images. The number of images per category ranges from 40 to 800, with an average of about 50 images per category.
OxfordPets (Parkhi et al. 2012): The OxfordPets dataset comprises 7349 images of cats and dogs, divided into 37 categories. Each category contains approximately 200 images, suitable for pet recognition and classification tasks.
The StanfordCars dataset consists of 16,185 images of cars, categorized into 196 classes. Each class represents a specific car model and manufacturing year, primarily used for car classification and recoguition tasks.
StanfordCars Krause et al. 2013):
The Flowers102 dataset includes 8189 images of flowers, catego- rized into 102 classes. The number images per category ranges from 40 258, with an average of about 80.
Flowers102 Nilsback & Zisserman 2008):
Food101 [Bossard et al.
2014h: The Food101 dataset comprises 101,000 images of 101 food categories.
Each category contains 1000 images, used for food recognition and classification tasks.
FGVCAircraft Maji et al.
2013): The FGVCAircraft dataset contains 10,000 images of aircraft, catego- rized into 100 classes with 100 images per category, most of which are airplanes.
SUN397 Xiao al. 2010): The SUN397 dataset includes 108,753 images of scenes, categorized into 397 classes. The number of images per category ranges from 100 2000, with average of about 300.
23

---

Published in Transactions on Machine Learning Research (09/2024)
DTD Cimpoi et al. 2014;:
The Describable Textures Dataset (DTD) comprises 5640 images of textures, divided into 47 categories. Each category contains 120 images, used for texture recognition and classification tasks.
EuroSAT Helber et al., 2019 : The EuroSAT dataset contains 27,000 satellite images, categorized into 10 3000 images, primarily used for geospatial classification tasks.
classes. Each class contains 2000
UCF101 (Soomro al. 2012: The UCF101 dataset includes 13,320 video clips categorized into 101 action classes. Each class represents specific sport action, mainly used for action recognition tasks.
C.3
Full Results
In this section, we report the average accuracy and standard deviation from three runs with three different random seeds in three generalization benchmarks. It is important to note that baselines are reproduced using the official configuration file parameters on the same random seeds and hardware as our experiments, ensuring fairness in comparison.
C.3.1 Base-to-New Generalization
The full experimental results in the base-to-new generalization setting are shown in Table 6
Please note that, due to the absence of an official implementation for ProDA (Lu et al. (002)), we report only the results provided by Derakhshani et al. (2023 in Table of the main text, and do not include the full results in the appendix.
C.3.2
Cross-Dataset Generalization
The full experimental results in the cross-dataset generalization setting shown in Table 7
C.3.3 Domain Generalization
Datasets. For DG, we use four ImageNet-derrved datasets with different domain shifts: ImageNet (Recht ctt al. 2019), ImageNet-Sketch WWagg et al. 2019), ImageNet-A (Hendrycks et al. 2021b), and ImageNet- R (Hendrycks al. 2021a).
The full experimental results in the domain generalization setting are shown in Table Unlike the previous two settings, in the DG sctting, only the visual modality experiences shifts. Although cxisting methods have achieved commendable results in this scenario, M still attains enhancements in both the average target dataset accuracy and the effective robustness ratio.
C.4
Additional Ablation Study
C.4.1 Effectiveness of M
Figure 5 presents the variation curves of both in-distribution (ID) and out-of-distribution (OOD) accuracy with the changing number of prompts (M) in the cross-dataset setting. The ID accuracy initially increases slightly with an increase in M and then decreases, aligning with our analysis in Section [ This trend is attributed to the view bias bias of prompts. When M is relatively small, the aggregation of useful features from different views enhances ID test accuracy. However, M further increases, redundant features exacerbate overfitting. In contrast, the OOD accuracy significantly rises before gradually decreasing. This is because when M is too small, the insufficient variety of views leads to inadequate coverage of OOD predictive features, leading to a rapid improvement in OOD performance as A increases. But with larger M, the dominance of redundant and irrelevant features from the introduced views deteriorates the performance. Considering the trends in both ID and OOD changes, we opt for M 8 trade off performance with computational cost.
24

---

Published in Transactions on Machine Learning Research (09/2024)
Dataset
Zero-shot CLIP
CoOp
CoCoOp
MaPL.e
PromptSRC
MSPT.(Ours)
A
Base
69.48
82.29
80.49
82.00
84.18
84.90
+0.72
Average on 11 datasets
New
74.28
68.78
72.04
74.88
75.76
76.81
+1.05
HM
71.80
74.93
76.03
78.28
79.75
80.65
1..90
p(
-43%
-20%
5%
16%
66
Base
72.37:10.00
76.47:0.17
75.93:10.24
76.871:0.05
77.80+0.000
77.7210.08
-0.08
ImageNet
New
68.10:10.00
67.50:+0.29
70.13:+0.33
70.73+0..3 333
70.60+0.00
71.73:+0.04
+1.13
HM
70.17
71.71
72.91
73.67
74.03
74.60
+0.57
Base
97 17.22+00.00
98. 10+0.00 00
97 800 08
97 131
98. 10+0, 16
98.45=00.5
0.35
Caltech101
New
94.21+0.00
03.20=00.0 11
93.00=:0.2 .99
95.23++0225
94.03+0.11
94.26=0.22
00.2
HM
95.69
95.59
95.34
96.56
96.02
96.30
+00.88
Base
91. 91.23:10.00
94.53:1.336
95.03:10..0 40
95.60+10.22
95. 95.3310.00 009
95.85
+0.52
OxfordPets
New
97.20:00.00
97.73
98.00:10.33 366
97.27
98.15-10.16
+0.83
HM
94.12
95.16
96.36
96.79
96.29
96.99
+0.70
Base
63. $3.69440000
1..60+1.13
70.73+0.77
72.40++0.22 .99
78.1330.25
79.07=0.62
StanfordCars
New
74.92.10.00
70.03:10.62
72.50:10.80
73.67.10.60
75.37 1.0333
74.03
-1.34
HM
68.85
72.71
71.60
73.03
76.73
76.47
-0.26
Base
71:00:00000
97.53:10.09
94.43:10.66 666
96.101:0222
98.17::0.05
98.17:10.20
+0.00
Flowers102
New
77.52:10.00
7..00:1.87
70.63:10.9
72.8710.98
17.3710.17
75.72:10.93
-1.65
HM
74.50
82.38
80.81
86.54
85.50
-1.04
Base
90.07.1..000
89.50.10.44
90.57
90.83.10.05
90.63110.12
90.8510.11
+0.22
1ood101
New
91.17:10.00
88.90110.4 45
91 91.27:1:047
92.03:10.12
91.47:10.12
92.0810.28
+0.61
HM
90.62
89.20
90.92
91.43
91.05
91.46
+0.41
Base
27. 27.55=+0000
38.67400.1 12
35.33+:0.8 .97
36.17=0.00
42.27=0.64
42.88+0.84
+0.61
FGVCAireratt
New
35.93++0.00
29.80+00.45 43
31.07+0.221
34.87=1.68
37.43+0.73
39.11=0.32
+2.56
HM
31.19
33.66
33.06
35.51
39.70
40.91
11.21
Base
69.38.10.00
81.20 11.20.10088
79.37:10.44
80.97
82.77-10.09
82.8410.22
+0.07
SUN397
New
75.58:10.00
70.43:11.65
76.23:10.58
78.30+00.41
78.50+0.57
78.85-10.24
+0.35
HM
72.35
75.43
77.77
79.61
80.58
80.80
+0.22
Base
53. 33.134:0.00
79.67:00.54
76.93+0.88
80.47+1.33
82.9740.90
83.91+0.29
0..9
DTD
New
60.27440.00
49.3743.56
54.67++4.49
58.40:+0.66
50.5743.22
1..7
12.18
HM
56.48
60.96
63.92
67.68
69.35
71.14
+1.79
Base
56. 36.9810000
88.97++1.07 U7
87.10+00.70
91.0743.76
92.70+0.99
96.72:10.39
+4..22
AT
New
13.74100.00
56.00+3.35
61.87:11.47
72.83433.20
73.17:33200
77.94+1..7
+4.777
HM
60.17
68.74
72.35
80.94
81.79
86.32
+4.53
Base
70.99
28
82.13.10.17
83.57 1.0.8
87.1010.22
87.40.10.30
+0.30
UCF101
New 78.47:10.00
64.20.13.7
73.30.10.85
76.73.11.53
78.5711.55
81.2911077
+2.72
HM
74.54
73.15
77.46
80.00
82.62
84.23
+1.61
Table 6: Full results in the base-to-new generalization setting. All baseline results are reproduced using the reported parameters. The harmonic mcan of the base and new class test accuracy is denoted HM. Improvements over PromptSRC are in blue.
Source
Target
POYCADER
StannorrdCaree
Calteeoh101
Flowors122
Imagen
0ood101
BUNN899
EurGAA
LOPTOO
Averaggo
DTD
Zero-abet CLIP 66 68..88.00000 00 93.31 10000 30.10:00.00
65.51
00000000000 70. 70.715500.00 00 85. 15.8822.00024
0..00.....0000.0 0.00 82. 6002
0.00000000 DO 46, 20.00 48 30:100000 87.59:20.00
85.19
CoOp
71.68-0.179 008-00.179.77777 93.70 88.2 38.2770..5 0.552
63.63++1.96
68. 70+00.92
85.63-0.12 19.27 1.19 64.60+
20 41.69
40.02 35-00.40
84.12
222%
CoCoOp
71.201:0 00 20.0000000
90.802.0..0.. 0.008 0.00
64.83100.65
71 01.0331..88 ST 86. 86.1320.00 05 23. 23
87
07.20000
10.62
001.3.22
68.5020.57
85
2%
70.401:0.11
23.50
23.77.0.55
07
07.47.2..37
PromptRc
71.37+00.09
0. 90.90++0..0
65.70++0.4 0.96
70. 70.45.+0. 24 80.47 0.000 29.57 0.0.0
07.45++0 0.
45. 1------00 0..12
47 60. 500.00+0.644
65.50
13%
71.114:0 02 03.94.0..11 90.00.0.00 0. 1eee0.00000 0066636-1111 3...3.5--0044 447...4...2 20..0..1666...... 1ww.1666666.00 0.000.0 24.23 24..28.00.45 67.06: 0. 0G 65.61.44 + 0..3777. 67.62 1. 08266686 0.8.4552.3333 03.307.55 39%
Table 7: Full results in the cross-dataset generalization setting. All baseline results are reproduced using the reported parameters. Since CL.IP is directly zero-shot tested without training, standard devia- tion reported as zero.
25

---

Published in Transactions on Machine Learning Research (09/2024)
72.00
67.00
71.75
66.75
71.50
66.50
Accuracy %6
71.25
06.25
71.00
00.00
70.25
AcC.
Avg. O0D AcC.
70.00
65.00
10 12
34
16
The number prompts (M)
Figure 5: The ablation experiments on the number of prompts in the cross-dataset generaliza- tion setting. The left vertical axis represents the ID test accuracy on ImageNet, and the right vertical axis indicates the average zero-shot OOD test accuracy across target datascts. The trends of ID accuracy and average OOD accuracy with the number of prompls M (A 0) are depicted by curves with circular and square markers, respectively.
71.4
67.4
ID Accuracy
71.0
67.0
70.6
ACC. 00D Acc.
66.6
The balancing coefficient (1)
Figure 6: The ablation experiments on the balancing coefficient À in the cross-datasct general- ization setting. The left vertical axis represents the ID test accuracy ImageNet, and the right vertical axis indicates the average zero-shot OOD test accuracy across target datasets. The trends of ID accuracy and average OOD accuracy with the balancing coefficient À (M 8) are depicted by curves with circular and square markers, respectively.
26

---

Published in Transactions on Machine Learning Research (09/2024)
Source
Target
ImageNet
ImageNetV2
ImageNet-Sketch
ImageNet-A
ImageNet-RR
Average
p(f)
Zero-shat CLIP
66.68
60.911.0.00
46.09
47.76.00.0
73.97.10.00
57.18
CoOp
71.6
64.27=0.17
07.93=0.29
50.3710.25
75.33+0.21
59.48
46%
CoCoOp
71.20=0.00
64.27=0.25
48.67:0.25
50.73+0.11
16.104
59.94
61%
MaPLe
70.40=0.14
63.73=0.12
48.60=0.16
50.20+0.37
76.57=0.12
70%
PromptSRC
71.37=0.09
64.43=0.05
49.53+0005
50.77+0.26
77.77=0.05
73%
MPPL(Ours)
70.9510.07
64.4910.00
49.60.10.10
51.4710.09
77.4010.07
60.74
83%
Table 8: Full results the domain generalization setting (A=
= 0.1). All baseline results are repro-
duced using the reported parameters. Since CLIP is directly zero-shot tested without training, its standard deviation is reported zero.
C.4.2 Cross-modal Contrastive Regularization
Figure presents the variation curves of ID and OOD accuracy in the cross-dataset generalization setting function of the balancing coefficient A. Both ID and OOD accuracies initially increase and then decrease with the rising values of X. We set =1.0 to optimally balance Lmulti and Ccontrast-
Batch size
X
ID Acc.
Average OOD Acc.
p(f)
512
0.0
70.95
66.87
0.39
128
1.0
71.02
67.15
0.45
256
1.0
71.11
67.25
0.47
512
1.0
71.11
67.35
0.49
Table 9: The ablation experiments on the impact of batch size on the cross-modal contrastive regularization objective the cross-dataset generalization settings (M = =8). Note that under different batch size settings, we control the same iteration for training.
Table g presents the ablation study results on the impact of batch size on the cross-modal contrastive regularization objective in the cross-dataset generalization setting. We observe that with À = 1.0, the average OOD test accuracy on the target datasets increases as the batch size increases. This trend is attributed to a lower class collision rate when the batch size smaller, which in turn reduces the performance of the cross- modal castiastive regularization. However, it is still evident that
with batch size of 128 (leasible single Nvidia A100 GPU), there is significant improvement over the baseline that does not utilize Ccontrast: further demonstrating the effectiveness of our proposed regularization objective.
C.4.3 Comparison to Temporal Prompt Aggregation in PromptSRC
Method
ID Acc.
Average OOD Acc.
p(f)
Independent V-L Prompting (TVLP)
71.16
65.22
0.01
IVLP + GPA of PromptSRC
72.10
65.51
(+0.22 29)
0.06
IVLP M (w/o Lcontruss)
70.95
66.87
(+ (..55
0.39
Table 10: Ablation on aggregation strategy in the cross-datased setting.
Rather than use multiple prompt pairs and aggregate their results, PromptSRC (Khattak et al. 2023b employs Gaussian weighted prompt aggregation (GPA), which temporally aggregates the results of a single prompt pair across its training trajectory. Here we compare the effectiveness of the two techniques in Table 10. As shown the table, GPA yields little improvement due to the same view obtained from single optimization trajectory, which consistent with our analysis that a single optimization trajectory may fail to capture broad range of views.
27

---

Published in Transactions on Machine Learning Research (09/2024)
C.4.4 Prompt Length and Depth
67.0
71.0
66.9
ID Accuracy
70.8
00.7
Acc. OOD Acc.
The Prompt Length
Figure 7: The ablation experiments on the prompt length in the cross-dataset generalization setting (M= 8, A= = = 3). The left vertical axis represents the ID test accuracy on ImageNet, and the right vertical axis indicates the average zero-shot OOD test accuracy across target datascts. The trends of accuracy and average OOD accuracy with the prompt length are depicted by curves with circular and square markers, respectively.
Figure 7 displays the results of ablation experiments prompt length in cross-dataset generalization setting. The results indicate that both ID and OOD test accuracies generally exhibit. an initial increase followed by a decrease. Consequently, we select a prompt length of to trade off the performance between and OOD scenarios.
71.6
67.0
ID Acc.
66.6
ID Accuracy %
60.4
70.8
66.2
70.6
66.0
65.8
The Prompt Depth ())
Figure 8: The ablation experiments on the prompt depth in the cross-dataset generalization setting (M = 8, X = 0, and prompt length 2). The left vertical axis represents the ID test accuracy on ImageNet, and the right vertical axis indicates the average zero-shot OOD test accuracy across target datasets. The trends of ID accuracy and average OOD accuracy with the prompt depth J are depicted by curves with circular and square markers, respectively.
Figure S presents the ablation study results regarding the depth of prompts J in the cross-dataset general- ization. It is observed that both ID and OOD test accuracics generally follow an initial increase followed by decrease, with OOD test accuracy being more significantly influenced by J. We opted for to trade off the performance between ID and OOD settings.
28

---

Published in Transactions on Machine Learning Research (09/2024)
C.5 Full Results SigLIP
Our experimental design in the main lext primarily focuses CLIP. In this section, report additional results on the latest state-of-the-art image-text pre-trained model, SigLIP (Zhai et al.) 2023), to further validate our method's scalability and generalization ability. Specifically, we use the pre-trained SigLIP-B/ 16 as the backbone and use the same hyperparameters as in our CLIP experiments. We evaluate the performance of the zero-shot SigL.P model, Independent. V-T, Prompting (TVLP), and M under the standard Base- to-New generalization setting, as shown Table [1 The results demonstrate that compared to the baseline method IVLP, PL achieves significant improvement in out-of-distributioo (OOD) performance (18.03%
while maintaining in-distribution (ID) performance, corroborating the theoretical analysis in Section 5.2
Table 11: Results extending the MÒPI, approach to the SigLIP (Zhai et al. 2023 generalization. HM and p(f) refer to harmonic mean and effective robustness ratio, respectively.
backbone in base-to-new
Dataset
Zero-shot SigLIP
SigLIP + IVLP
SigLIP M²PL((Ous)
Base
77.75
87.00
86.84
Average on datasets
New
81.23
78.75
82.61
HM
79.45
82.67
84.67
p()
-27%
15%
Base
79.60
81.26
81.70
ImageNet
New
76.90
76.09
78.32
HM
78.23
78.59
79.97
Base
98.50
98.90
98.71
Caltech101
New
97.80
97.38
98.36
HM
98.15
98.13
98.53
Base
96.50
95.27
96.97
OxfordPets
New
98.10
98.55
98.60
HM
97.29
96.88
97.78
Base
85.00
91.58
89.63
StanfordCars
New
96.20
95.67
96.56
HM
90.25
93.58
92.97
Base
91.00
99.43
98.67
Flowers102
New
86.00
81.06
36.24
HM
88.43
89.31
92.04
Base
92.30
91.57
93.24
Food101
New
93.60
92.38
94.15
HM
92.95
91.97
93.69
Base
29.20
48.44
41.24
RGVCAircraft
40.80
37.19
46.07
HM
34.04
42.08
43.52
Base
75.30
82.69
83.75
SUN397
New
78.60
76.68
81.23
76.91
79.57
82.47
Base
75.60
88.19
88.43
DTD
New
73.60
67.27
73.55
HM
74.59
76.32
80.31
Base
56.40
93.27 03.21
96.50
EuroSAT
New
68.00
68.61
71.51
HM
61.66
79.06
82.15
Base
75.90
86.45
86.35
UCF101
New
83.90
75.39
84.10
HM
79.70
80.54
85.21
Supplementary Experiments
To further elucidate the performance gain of MIP in zero-shot cross-dataset genoralization across datascts with varying distributions, we conduct two supplementary experiments: (1) A Few-shot Linear Probe exper-
iment based the same protocol in Radford et al. 2021 Zhou al. (002bb) to evaluate the generalization
29

---

Published in Transactions on Machine Learning Research (09/2024)
potential of CLIP's pre-trained features each target dataset, as detailed in Appendix D.1 (2) An as- sessment, of the cosine similarity between average visual and textual representations across target dataset categories and those from ImageNet categories to measure the information gain through few-shot prompt learning on ImageNet, presented in Appendix D.2 For simplicity, we limit our experiments to datasets with vero-shot CLIP's 8 accuracy below 85%. We assume that zero-shot CLIP performs above this threshold, its pre-trained features are already generally sufficient for generalizing to the target dataset, so the influence of view bias is relatively minor.
Based on the metrics from the above experiments, use simple multivariate linear regression model to interpret M relative improvements over zero-shot CLIP across different datascts, with specific results detailed in Appendix D.3
D.1 Linear Probe Experiment
Method
StanfordCars
Flowers102
FGVCAircraft
SUN397
DTD
EuroSAT
UCF101
Zero-shot CLIP Linear Probe CLIP
65.51
70.73
24.66
62.60
44.09
48.40
67
80.60
97.28
83.30
73.15
70.15
86.33
82.66
Average Performance
73.06
84.01
35.49
67.88
57.12
67.37
75.13
Table 12: Few-shot linear probe performance (%) on the target datasets.
Experimental settings. We adhere to the few-shot linear probe setup in Zhou et al. (2022b), sampling 16 instances per class and reporting the average results across three random sceds. Consistent with the cross-dataset generalization setting discussed in Section 6 we employ ViT-B/16 as the backbone for CLIP.
Results Table 12 reveal that on the FGVCAircraft dataset, both zero-shot CLIP and linear probe CLIP demonstrate notably low performance, indicating the inadequacy of CLIP's pre-trained features for this dataset. Conversely, the significant improvement with linear probe CLIP on the EuroSAT dataset highlights the generalization potential of CLIP's pre-trained features this distribution.
In practical prompt learning scenarios, samples from the target dataset distribution are unavailable. There- fore, we evaluate the generalization potential of CLIP's pre-trained features on each dataset by averaging the performance of zero-shot CLIP and few-shot linear probe CLIP.
D.2 Datasets Representation Similarity Experiment
StanfordCars
Flowers102
FGVCAircraft
SUN397
DTD
EuroSAT
UCF101
Visual Similarity
0.3289
0.3433
0.2781
0.4359
0.4097
0.5103
0.2924
Textual Similarity
0.1253
0.1910
0.2017
0.2407
0.2196
0.3014
0.1993
Average Similarity
0.2271
0.2672
0.2399
0.3383
0.3147
0.4059
0.2459
Table 13: Estimated similarity between the target dataset and the ImageNett distribution. The similarity refers to the minimum pairwise cosine similarity between category representations of the target datasets and ImageNet.
Experimental settings. Since prompt learning adapts to downstream tasks through few-shot learning using frozen CLIP pre-trained features, we measure the similarity between target datasets and ImageNet using representations from the vision and text encoders of zero-shot CLIP. Specifically, for visual similarity, we caleulate the pair-wise cosine similarity between the average representation of test images from each category in the target datasct and the average representation of few-shot images from each category in ImageNet, selecting the minimum value as the measure of visual similarity. For lextual similarity, we use the fixed template "a photo of label" as input, compute the pair-wise cosine similarity between text representations of cach category the target datasct and ImageNet, and again select the minimum value as the measure of textual similarity. Ultimately, the average of visual and textual similarities taken as the estimated similarity between the distributions of the target dataset and TeageNet.
30

---

Published in Transactions on Machine Learning Research (09/2024)
The results in Table 13 demonstrate that the FGVCAircraft dataset exhibits low similarity with ImageNet, aligning with observations from the experiment where prompt-based fine-tuning algorithms generally under- perform zero-shot CLIP on FGVCAircraft in cross-dataset generalization settings. Conversely, the EuroSAT dataset shows higher similarity to the ImageNet distribution, which partially explains the differing perfor- mance of M on these datasets.
D.3 Multivariate Linear Regression
StanfordCars Flowers102
FGVCAírcraft
SUN397
DTD
EuroSAT
UCF101
Average Performance
0.7306
0.8401
0.3541
0.6788
0.5712
0.6737
0.7513
Average Similarity
0.2271
0.2672
0.2399
0.3383
0.3147
0.4059
0.2459
Performance Gain (%)
0.84
1.53
-0.43
5.05
1.52
9.12
0.86
Table 14: Two metrics and performance gains of M compared to zero-shot CLIP.
In this section, multivariate linear regression to explain the performance improvements of our proposed M model (relative to zero-shot CLIP), based on two metrics derived from the previous sections. The first metric measures the generalization polential of CLIP's original pre-trained features on the target datasel, indicated by the average performance zero-shot CLIP and few-shot linear probe CLIP. The second metric, the average cosine similarity of textual and visual representations, estimates the distribution similarity between the target dataset and ImageNet. We utilize these two metrics as independent variables in simple multivariate linear model, with the performance gain of M&PL the dependent variable.
Y = =0 Xo a0 X1 +
(32)
where a; the regression coefficient and B is the intercept, Xu represents the average performance (infor- mativeness), X, the average similarity (transferability), and Y the performance gain of M
The fitting results in Multiple R of 0.952 and R² of 0.906, indicating a strong fit and demonstrating the interpretability of our method regarding the performance the target datasct. Furthermore, both coelficients og 3.647 and 47.429 are positive, suggesting that the performance improvement of M a given target dataset positively correlates with both the generalization potential of CLIP's pre-trained features on that datasct and the dataset's 's similarity to ImageNet.
As shown in Table 32 the EuroSAT and SUN397 datasets exhibit high average performance and average similarity metrics, which correlate with their significant performance enhancements. Conversely, the FGV- CAireraft dataset shows lower values in these metrics, resulting in the poorest performance of M The StanfordCars and UCF101 datasets, while having high average performance, similarity, limiting their gains to less than 1%. In contrast, the DTD and Flowers datasets benefit from higher average similarity and average performance, respectively, achieving improvements exceeding 1.5%.
constrained by low average
31


Reviews:
[{"review_id": "ru2oCee2jI", "reviewer": "Reviewer_k5Sg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper studies an interesting yet often overlooked problem; the authors describe this as the \"view bias\", which manifests that a single fixed prompt can only help the model select part of the features.\n\n2. The theoretical analysis seems correct, the proof and the results seem novel to me, and the implications from Proposition 2 and Proposition 3 seem interesting and non-trivial.\n\n3. The proposed method showcases significant improvements over OOD scenarios.\n\n**Weaknesses:**\n\n1. Proposition 2 is somewhat hard to grasp; based on my understanding, the definition of \"generalization\" for general-purpose pre-trained models such as CLIP is not as clearly defined as a supervised learning setup. I recommend authors to \n\n2. The problem setup seems to differ from my understanding of prompt tuning in CLIP, based on the original setup proposed in Coop [1]; only the text branch prompt is learnable. Do authors adapt to a more trending setting? \n\n3. What approaches are undertaken to make sure that different prompt branches are trained *independently*? This does not seem feasible unless (a) those learnable prompts are defined over different hypothesis classes and (b) those prompts are trained with different data. \n\n[1] Learning to Prompt for Vision-Language Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the limitations of multi-modal prompt learning, particularly the tendency of prompts to capture only a subset of features, termed \"view bias.\" This bias often undermines model generalization, especially under distribution shifts. To address this, the authors introduce Multi-modal Matching Multi-Prompt Learning (M3PL), a new framework that uses multiple prompt pairs and a cross-modal contrastive regularizer, thereby enhancing model generalization when facing OOD data. Extensive experiments demonstrate that M3PL significantly improves OOD performance, offering a robust solution to the identified limitations of existing prompt learning techniques in vision-language models", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please make changes in accord with aforementioned weaknesses.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "610lbSuTr3", "reviewer": "Reviewer_dPz4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The identified problem is interesting.\n2. The proposed solution is straight-forward.\n3. The result is convincing.\n\nCons:\n1. The proposed method achieves a much more significant improvement on EuroSat compared to on other datasets. This raise a natural question: does the substantial improvement really comes from the proposed theory?\n\na. It is necessary to further examine other datasets with rather large domain shift.\n\nb. It is important to further examine the EuroSat dataset and the resulted model on it. Is it possible that this dataset has some hidden shortcut? For example, the dataset has a spatial prior between image regions and classes?\n\n2. The current experiment is only done with one backbone. It is necessary to check the behavior when scaling up to larger and better backbones.\n\n3. The proposed method is a generalized form of [a]. But the effect on using multiple text prompts is different from [a]. It is worth examination and discussion for such discrepancy.\n\n[a] Learning to Decompose Visual Features with Latent Textual Prompts", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose Multi-modal Matching Multi-Prompt Learning (M3PL) to improve the generalization ability of prompt tuning. The key idea is to leverage multiple paired prompts and a cross-modal contrastive regularization that facilitates the prompt pairs to comprehensively utilize the features. The effectiveness is verified on several standard prompt tuning datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check weakness for details.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gmnPs1066B", "reviewer": "Reviewer_TUPP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n1. This paper is well-written and easy to understand. I like that the paper looks into a specific problem in existing prompt learning methods and use the analysis to propose a new method. \n \n2. The authors discover and analyze an intriguing problem of existing multi-modal prompt learning methods, where the learned prompts capture a limited set of features. This phenomenon should be of interest for the research community.  \n\n3. The proposed method is intuitive and technically sound. The authors have demonstrated the effectiveness of the proposed method M3PL on many datasets.  \n\n\nWeaknesses:\n1. The use of multiple prompts in both the image and text encoder can make the proposed approach computationally heavy. However, the paper does not have discussions on this practical perspective. (It is merely discussed with one line in Section 6.6.)  \n\n2. It is not clear to me how to choose the number of prompts M in the proposed methods. I couldn’t find any mentions of this in the main text, e.g., how is M=8 chosen empirically in the experiments?  \n\n3. I wonder if the proposed method actually solves or alleviates the view bias problem. Is there a way to test this, similar to the analysis in Figure 1 and Figure 2?  \n\n4. The authors only test on one CLIP pretrained model. It would be good to show the results on other variants of CLIP (e.g. SigLIP [1]) to demonstrate the generalization ability of the proposed approach.  \n\n5. There are no results on ImageNet correuption datasets (e.g. ImageNet-C, ImageNet-R), which are representative datasets for measuring OOD robustness of CLIP models [2,3].\n\n[1] Sigmoid loss for language image pre-training. Zhai et al, 2023.   \n[2] Robust fine-tuning of zero-shot models. Wortsman et al, 2021.  \n[3] Finetune like you pretrain: improved finetuning of zero-shot vision models. Goyal et al, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies the problem of view bias in multi-modal prompt learning, where the learned prompts represent a partial subset of useful features. The existence of view bias in learned prompts leads to varying behaviors under distribution shifts. The authors showed theoretical analysis into the view bias problem and connect it to the multi-solution nature of prompt learning. Based on these observations, the authors proposed a new method M3PL (Multi-modal matching multi-prompt learning) to encourage learning diverse views of features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses for details. But there are two specific points that I would prefer to be addressed:  \n\n1. Can the authors modify section 4 into a self-contained section? Currently it is very short and I guess the initial evidence refers to the Figure 1 described in the introduction. This seems confusing for paper presentation. It would be good to dedicate this section to formally investigate the view bias problem. You can give an overview of Figure 1 in the introduction but make sure to defer the technical details and discussions in the main text.  \n\n2. More discussions on the computational cost should be put in the main text. In the experiment section, the authors should discuss how the number of prompts 8 is chosen. Also it would be good to discuss how does the use of more prompts in M3PL affect the computational efficiency.", "broader_impact_concerns": "I do not have concerns regarding the broader impacts of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BHbijG9yyF", "reviewer": "Reviewer_bZhj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-  The paper is well-organized and centered on the OOD problem of prompt learning, equipped with both theoretical analysis and strong empirical validation.\n- The theoretical analysis in Sections 5.1-5.2 is interesting, extending prompt learning from single-modal [1] to multi-modal. \n- The proposed methods are well-motivated and achieve strong performance.\n- The experimental results are comprehensive, with a number of datasets and baselines.\n- The code is provided in the supplementary Material.\n\nWeaknesses:\n- The proposed method might be time-consuming as it requires learning lots of different prompts in numerous layers.\n- Proposition 1 is a trivial result in my view. It is straightforward to see that the objective can be decomposed into these four terms: text prompt-vision prompt, vision representation-vision representation, text prompt-vision representation, and vision prompt-text representation. However, the so-called feature selection is too broad as it does not clarify how these weights select the features. Why do different prompts select different features?\n- The considered multimodal model is only CLIP-based classifier and does not cover vision language models.\n- The theoretical analysis should cover a more practical setting. For example, 1) inner-product is considered as sim() while in practice cosine similarity is used; the difficulty and possible ways to extend this should be discussed. 2) The implications of the assumptions in Section 5.2 should be remarked (do they hold in practice?)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper first poses an interesting phenomenon that existing prompt learning methods fail to generalize well under large distribution shifts. It then provides a theoretical analysis of multi-modal attention-based prompt learning and highlights the view bias in prompt learning. Finally, the paper proposes a new prompt-learning pipeline that introduces multiple, paired multimodal prompts and achieves state-of-the-art performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be beneficial to see the efficiency of the proposed method, e.g., adding running time in Table 2.\n- Include a discussion on the difficulty of addressing (removing) the assumptions in the theoretical part.\n- Provide a brief introduction of each dataset used in the appendix.\n- Provide more details on the experimental setup, e.g., the hyperparameters set up in prompt learning, and how to create different prompts.\n- Minor request: Is it possible to show a similar phenomenon on large vision language model? ( it would be great to see the result or just  a brief discussion on them.)\n- Provide a discussion on the broader impact of the paper.", "broader_impact_concerns": "Broader impact concerns are missing; please provide a discussion on it.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Even though the authors have explicitly mentioned the limitation "our experiments are currently limited to CLIP ViT-B/16", I would encourage them to consider extending their method beyond this CLIP ViT-B/16 model. If this is not possible, the paper seems ready to be accepted, but I do think the paper would become stronger with an experiment extending beyond this mdoel.
Decision: Accept (poster)