Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
Authors: Debarshi Brahma
Anuska Roy
Soma Biswas
Venue: Accepted by TMLR ()

Abstract: Recently, Vision-Language foundation models like CLIP and ALIGN, which are pre-trained
on large-scale data have shown remarkable zero-shot generalization to diverse datasets with
different classes and even domains. In this work, we take a step further and analyze whether
these models can be adapted to target datasets having very different distributions and
classes compared to what these models have been trained on, using only a few labeled
examples from the target dataset. In such scenarios, finetuning large pretrained models is
challenging due to problems of overfitting as well as loss of generalization, and has not been
well explored in prior literature. Since, the pre-training data of such models are unavailable,
it is difficult to comprehend the performance on various downstream datasets. First, we try
to answer the question: Given a target dataset with a few labelled examples, can we estimate
whether further fine-tuning can enhance the performance compared to zero-shot evaluation?
by analyzing the common vision-language embedding space. Based on the analysis, we
propose a novel prompt-tuning method, PromptMargin for adapting such large-scale VLMs
directly on the few target samples. PromptMargin effectively tunes the text as well as visual
prompts for this task, and has two main modules: 1) Firstly, we use a selective augmentation
strategy to complement the few training samples in each task; 2) Additionally, to ensure
robust training in the presence of unfamiliar class names, we increase the inter-class margin
for improved class discrimination using a novel Multimodal Margin Regularizer. Extensive
experiments and analysis across fifteen target benchmark datasets, with varying degrees of
distribution shifts from natural images, shows the effectiveness of the proposed framework
over the existing state-of-the-art approaches applied to this setting.

Published in Transactions on Machine Learning Research (01/2025)
Prompt Tuning Vision Language Models with Margin Reg- ularizer for Few-Shot Learning under Distribution Shifts
Debarshi Brahma
deharshsb@iisc.oo
Indian Institute of Science, Bangalore
Anuska Roy
anuskovoy@@maiil.com
Indian Institute of Science, Bangalore
Soma Biswas
somabiswas www.:aowaiiiiicc..oooo
Indian Institute of Science, Bangalore
Reviewed OpenReview: https: opppreve/pperevieuu. net/ forum? id= 2nNgtPhaW7
Abstract
Recently, Vision-Language foundation models like CLIP and ALIGN, which are pre-trained on large-scale data have shown remarkable zero-shot generalization to diverse datasets with different classes and even domains. In this work, take step further and analyze whether these models can be adapted to target datasets having very different distributions and classes compared to what these models have been trained on, using only few labeled examples from the target dataset. In such scenarios, finetuning large pretrained models challenging due to problems overfitting as well as loss of gencralization, and has not been well explored unavailable, it difficult to comprehend the performance on various downstream datasets. First, we try to answer the question: Given a target dataset with few labelled ezamples, can estimate whether further fine-tuning can enhance the performance compared to zero- shot evaluation? by analyzing the common vision-language embedding space. Based on the analysis, we propose & novel prompt-tuning method, PromptMaryin for adapting such large-scale VLMs directly on the fow target samples. PromptMargin effectively tunes the text as well as visual prompts for this task, and has two main modules: 1) Firstly, we use a selective augmentation strategy to complement the few training samples in each task; 2) Additionally, to ensure robust training in the presence of unfamiliar class names, we increase the inter-class margin for improved class discrimination using novel Multimodal Margin Regularizer. Extensive experiments and analysis across fifteen target benchmark datasets, with varying degrees of distribution shifts from natural images, shows the effectiveness of the proposed framework over the existing state-of-the-ard approaches applied to this setting.
prior literature. Since, the pre-training data of such models are
Code:
1 Introduction
With the rapid advancement of deep learning models, it is possible to achieve very high performance for tasks like classification, etc., where large amounts of training samples can be collected and annotated. However, in roal-world scenarios, the difficulty in curating huge amounts of labelled data has led to rescarch in Few-Shot Learning (FSL), where a model trained on a large datasel can be (ransferred to a downstream task having few labelled samples from unseen categories. Furthermore, the target distribution can be very different from the source distribution, making the problem even more challenging. Recently, vision-language foundation models like CLIP have shown remarkable generalization capabilities in the zero-shot scenar-
ios Radford et al. 2021 Efficient finetuning techniques like Prompt learning (Zhou al. 2022b a) have

---

Published in Transactions on Machine Learning Research (01/2025)
been successful in generalizing these models to new classes new domains separately, but adapting such multimodal models to few samples having both novel categories and domains simultaneously is relatively less explored. Foundation models like CLIP have been pretrained on large web-scale data, which is not public. Hence, it is unclear as to which classes and domains does this training data encompass, and what is really in-distributton (ID) and out-of-distribution (OOD) for such models. In this work, we first explore whether we can estimate CLIP's performance on a given target dataset from the text and image embeddings in the joint representation space, even before finetuning the model using the data. Based on this observation, we aim to enhance the performance of CLIP such datasets using prompt learning. For this, we propose novel prompt-learning framework, termed Prompt.Margin in completely source-free setting. This implies that, unlike standard frameworks of few-shot learning, where the pre-trained model is meta-trained on a source domain (e.g,, ImageNet. (Russakovsky et al. 2015)) before it is fine-tuned on the downstream dataset, we utilize a more practical setting of directly adapting the original pre-trained CLIP model on the few target samples. PromptMargin has two main modules, namely (i) Selective Augmentation and (ii) Multimodal Margin Regularizer, to handle the different challenges in this setting. To address the challenge of availability of few samples from the target data, we use selective augmentation strategy to increase the mumber of training samples. Prompt tuning generally relies on the class names of the new categories, which may not be available for the target dataset or the names may not be meaningful to the CLIP model (e.g., names of rare diseases, etc. which may not. have been seen during training)- In order to learn discriminative classi- fiers even in such scenarios, we propose a novel Multimodal Margin Regularizer (MMReg), which enforces a consistent separation between the class-wise embedding vectors in the joint vision-languase representation space. Extensive experiments on benchmark datasets, namely BSCDFSL (Guo et al. (020), and Meta- dataset (Triantafíllou et al. 2019), show that the proposed framework performs favourably compared to the state-of-the-art, even though it is fine-tuned in a completely source-free manner. Our contributions can be summarized as follows:
1. We first empirically explore CLIP's zero-shot performance on few target datasets with limited labels, by investigating the image and text feature distances in the multimodal representation space. This provides insights into when CLIP's performance is sufficient for some downstream task, and how to enhance its performance in other cases.
2. We propose a novel prompt learning framework PromptMargin where we adapt CLIP directly to a few-shot setting with different classes as well as distribution shifts, without training it on some source dataset. To the best of our knowledge, this the first work which addresses this task using vision-language models (VLMs).
3. Towards this goal, we introduce a novel Multimodal Margin Regularizer (MMReg), which jointly steers the image and text category embeddings to uniform separation, thereby improving the prompt- learning performance.
Our framework performs favourably over zero-shot, meta-training, and baseline prompt-learning mothods across fifteen benchmark datasets in 1-shot and 5-shot settings.
Next, we discuss the related work in litcrature followed by the details of the proposed framework and experimental evaluation.
2
Related Works
Here, we briefly discuss the related work on prompt learning and OOD few-shot learning.
Prompt Learning for Vision-Languago Model (VLM): The recent emergence of foundation VLM's like CLIP (Radford el al., 2021), and ALIGN (Jia et al., 2021), has changed the landscape of deep learning. These models are trained on abundant web-scale data, where they align the image-text representations contrastive manner, exhibiting remarkable zero-shot performance. Though such models generalize well to most cases, leveraging the knowledge learned by these models for downstream tasks is challenging task. Recently, prompt learning has emerged an effective choice for finetuning these
2

---

Published in Transactions on Machine Learning Research (01/2025)
large-scale models to downstream tasks, where few additional trainable parameters are added to the input branches. Prompt learning in VLM's was first explored by CoOp (Zhou et al. 2022b), where the handcrafted prefix of the text input was replaced by a few trainable parameters, to be finetuned for a classification task. CoCoOp (Zhou et a 2022a) addressed the reduced genoralization of CoOp in certain cases by conditioning the lext. prompts on the image embeddings. MaPLe (Khattak et al. 2023 first introduced the concept of multimodal prompt learning, where a coupling function was utilized to enable mutual synergy between the textual and visual prompts. This approach enabled joint training of both the prompts in the CLIP representation space, and demonstrated improved performance over unimodal prompting approaches. This state-of-the-art model serves as the baseline of our proposed framework, where we learn prompts in both the text and vision encoder branches.
Few-Shot Learning: Few-Shot Learning (FSL) aims to transfer a trained model to novel calegory data when a few number of samples are available from each class (Snell et al.) 2017 Sun et al., 2019 Xu et al.) [202]) Existing litcrature provides two approaches to this problem, namely meta-learning and transfer-learning. Meta-learning based approaches typically aim to simulate few-shot tasks on a source dataset, and then transfers this learner to the test domain tasks (Ravi & Larochelle] 2016 Finn et al. 2017). The conventional transfer learning approaches are explored by methods like BSCDFSL (Guo et al, 2020), BSR (Liu et al., 2020), and NSAE (Liang et al. 2021). where the models are first trained on a source dataset. like ImageNet, before finetuning them on the larget datasets. Recently, the prominence of foundation models saw the emergence of a variety of methods for adapting them to the few-shot learning task. For instance, FDAlign (Song et al. 2024) and Wise-FT (Wortsman et al., 2022). ; finetunes the entire CLIP model parameters on source data using regularization techniques avoid loss of rich representations of the original model. Wise-FT utilizes weight interpolations between the zero-shot and the fine-tuned models while training, to enhance the performance. FD-Align aims to maintain the spurious correlations intact before and after finetuning, by minimizing the divergence between the predictions of the two models. Training free approaches like Tip-Adapter (Zhang et al. 2021) constructs weights from a key-value cache model from the few-shot training set, to adapt the CLIP model without any backpropagation. An intermediate approach of paaaameter-ellicient finetuning is adopted by CoOp (Zhou el al. 2022b), MaPLe (Khattak al., 2023), which first trains the prompts on a source dataset before transferring them to the target datasct discussed carlier.
Though prompl learning based upprouches have been ulilized in VLMs for FSL, to the best of our knowledge, no work in literature addresses the additional significant distribution shift problem in this context. Inspired by these advances, our proposed PromptMargin aims to address this task using an effective regularization technique for prompt learning, in a completely source-free manner.
3 Analyzing the CLIP Representation Space for target datasets
Foundation models like CLIP have been pretrained on a large corpus of web-scale data, which is not publicly known, and hence may span wide variety of classes and domain data, ranging from standard academic datasets to specialized datasets. Recent papers like Udandarao al. (2024) have studied the possible relationship of zero-shot performance of CLIP with the concept frequencies in pretraining datasets. A pertinent question in this scenario is, Given a target dataset with a few labelled ezamples, can we estimate whether further fine-tuning can enhance the performance compared to zero-shot evaluation? The zero-shot generalizability of the model depends upon both the distribution and semantic difference of the target dataset from the CLIP training datasct. Since the source dataset is unavailable and only few samples of the target dataset are provided, directly estimating distribution difference and also understanding whether the target classes seen not is not straightforward. Here, propose simple method to estimate the extent of the distribution shifts of some of the target datascts, by relating their inter-class mean image and text embedding distances in the CLIP representation space.
We consider some representative datasets from BSCDFSL (Guo et al. 2020 and MetaDataset (Triantafillou ct al. 2019) benchmarks to illustrate this point. We estimate the distribution and semantic difference using the mean inter-class L2 distances of both text and image features from the frozen zero-shot CLIP model. The class text features obtained by passing "A photo of [CLASS]" through the zero-shot model is denoted
3

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: The mean inter-class text and image embedding distances along with the estimated combined
(semantic and distribution) differences for some target datasets. *** denotes that pseudo class names were used for that particular dataset, due unavailability. We replace my with small value (0.1) for such cases.
Datasct
EuroSAT
ISIC
Omniglot
Quickdraw
Plantac"
Traffic Signs"
MSCOCO"
Aircraft
mini-ImageNNtt
We
0.588
0.561
0.679
0.716
0.100
0.100
0.100
0.700
0.860
my
0.627
0.582
0.420
0.520
0.727
0.583
1.010
0.770
1.000
dif[(mp.my)
1,296
1.500
1.854
1.320
9.376
9.715
8.990
0.730
0.153
28-CLIP (%)
47.70
22.40
28.14
61.54
26.54
12.68
18.61
80.98
99.21
MaPLe (%)
75.46
31.96
77.82
72.54
55.34
56.45
53.09
79.76
99.17
by X2r and the image feature prototypes are denoted by Xav. The inter-class mean distances my and my are computed as follows:
2
mT C2-C
18 - -Xer, 12, Vj (2, C}
(1)
my - K2J
2
Σ
Nxv, - Xev, 11,, Vj E ( C}
(2)
Here, C' is the total number of classes. approximated as follows:
The combined estimated distribution and semantic difference is
P (MTT, mv)
(3)
M&T
my
When the target dataset D Dtorget has significant semantic and distribution difference with the original training data, the CLIP model will not be able distinguish between the image and text embeddings of the target dataset. Thus, the values of mr and my will be smaller and the difference f will be larger and vico-versa. Since, the extracted feature vectors normalized between 0 and 1, subtract 2 offset from diss to set its minimun value to zero, while preserving the relative differences.
Table shows the difference along with the accuracy of zero-shot CLIP (ZS-CLIP) and MaPLe (Khattak ct al. 2023), where both the image and text encoders were finc-tuned using the few available labeled target large, L.e. the
samples using prompt learning. We observe that for the first four datasets, d classes are not well separated, and thus there is scope for improvement over zero-shot CL.PP accuracy. This is consistent with the improvement obtained with MaPLe. Similarly, ZS-CLIP performs very poorly when used instead of original names due to their unavailability (Plantae, Traffic Signs
placcholder classnames and MSCOCO datasets), thus implying the possibility of significant. performance improvement, as can be seen in MaPLe. On the other hand, for mini-ImageN ImageNet and Aircraft, since diff is low (between 0 and 1), the classes are already well separated in the latent space, and fine-tuning using few samples can adversely affect the model, thereby justifying the drop in accuracy of MaPLe. We also compute the Pearson correlation between the ZS-CI.IP performance and the d metric which is 0.7713, and shows a strong negative correlation. This conforms with the empirical analysis above, suggesting that an increase in the diff Durree metric results in reduced zero-shot performance of CLIP.
This analysis illustrates that the relative separations between image and text features for the different classes in the CLIP representation space is crucial in explaining the zero-shot performance on the respective downstream datasets. We address this issue by introducing a simple and effective regularization framework for prompt tuning, where we guide the image and text features to separate out in the feature space. We now formally give the problem definition and describe our proposed approach in detail.
4 Problem Definition and Background
We address the problem of transferring a model trained on a large source dataset a target domain containing very few labeled training examples and having significantly different data distribution. For this,
4

---

Published in Transactions on Machine Learning Research (01/2025)
we consider the N-way k-shot episodic setting, where random tasks/episodes T are sampled, comprising of support set S, and a query set Q. Both S and Q contains N classes randomly selected from among all the novel categories of the target dataset. For the N-way k-shot setting, k samples are drawn from each of these N sampled classes to create the support set. Additionally, q samples are also drawn from the same classes to create the query set. The support and query sets are given by S = {(,,, Nx Q = {X,, ve)) Thus, the objective is to classify the query set samples in each task, when we are provided with few support set samples from the target dataset.
Prompt Learning: Since PromptMaryin is based on prompt learning, we briefly describe it here for completion. Prompt learning is an efficient and popular method of finetuning large-scale models like CLIP to downstream tasks, where a set of learnable vectors are appended to either the textual branch (Zhou et al. a)) or the visual branch (Jia et al. 2022), or both (Khattak et al. 2023). For our mothod, we use multimodal prompt learning framework MaPLe (Khattak ct al. 2023) as our bascline.
Let us denote the CLIP text encoder as Ss and the image encoder as fe- The input image X T RRCHHxW is broken up into M patches {el, e2...., eM } and appended with the CLS token eCLS before passing it through the image encoder. Similarly, the text input, which is typically of the form "a photo of a /CLASS)", is embedded into the tokenized format {tsos, t1, t2...., C.. tgos} before passing it through the text encoder, where f1, 12,.. denotes the token embeddings, Isos and LEOS denotes the Start-of- Sentence tokens respectively and G denotes the kth classname. However, for multimodal prompt learning, we append both the text and visual inputs with learnable prompts. Specifically, let the T learnable textual prompts be denoted as Og (0t, Ot., O and the V learnable visual prompts be denoted as 0g (8., (0.y....,00 }. In our sctting, similar to KKattak al. (2023), , we project the textual prompts to visual prompls by a function F, ie., 0,- F(O.). Then, the O2 and 0, are respectively appended to the text and vision inputs as follows: Xr = ftsos, B2, t1, tg.... Cx2 tyos} and Xv = {ECLS, e2. eMr, before passing them through the text and image encoders. The final text and image embedding vectors can be written as Xy = (((r) and Xy = (XVv). Apart from adding learnable prompt parameters to the input only (termed as shallow prompling), we also add such learnable prompts after every transformer block of the encoders (deep prompting) (Khattak et al.] (202).. The final prediction is taken as the cosine similarity between the image and text embedding vectors. Finally, the multimodal prompts are jointly trained on downstream classification lask. Now we describe the proposed framework in detail.
and End-of-
5 Proposed PromptMargin Framework
The proposed PromptMargin works in completely source-free setting, i.c. we directly finetune the CLIP model on the small number of samples provided in the support set of the target datasct. We do not perform any mela-training on a separale sowrce domain like meli--maaeeNel as the eristing stale-of-the-art approaches (Song et al., (002)). For this, we utilize prompt learning in both the textual and the visual branches, similar to Khattak et al. (2023) as described in the previous section. Given the support and query set from a randomily sampled episode, we train the prompls on the few samples available in the support set, and evaluate the model performance on the query set. In particular, suppose (X,y) E S, where X E BXxCxHX denotes the images in the support set, and y E {ci, (2,, ] denotes the classname texts. We append the textual and visual prompts to the classname texts and images respectively, and pass it through the CTIIP encoders (fr and fo). Let the final text and vision embedding vectors obtained be denoted as Xy and Xy (Sec. 4.. Next, the prompts are learned through a cross-entropy loss objective, while the encoder parameters are kept frozen. The objective function can be writlen as follows:
LCE argmin Exx,)~ L(sim(Xr,Xv). (0.0.)
(4)
where, sim(.) denotes the cosine similarity. In this work, we aim to address the two important issues of this problem: (i) less data in the support set, (ii) unknovu/specciaizedd categories in the target domains. To address data scarcity, we use a Selective Augmentation strategy, where we only select the image
augmentations, whose embeddings are close enough to the respective text embeddings in the joint represen- tation space. In order to address the second challenge, we propose a Multimodal Margin Regularizer
5

---

Published in Transactions on Machine Learning Research (01/202))
Image Embed
Xy
Without MMReg
With MMReg
CE
X
"cat", "oxx. "fish"
Labels
Word Embed
X1
Target Dataset Support Set
Joint Embedding Space
trainable
X
frozen
Select Augmentations
added prompts
Figure 1: An overview of our proposed PromptMargin framework. A randomly sampled episode from the target dataset is considered. The support set images along with their augmentations are passed through the CLIP image encoder, and their labels are passed though the CLIP text encoder. The selective augmentation strategy selects sugmentations based on the embedding vectors. The Max-Margin Regularizer (MMReg) enforces the class-wis image prototypes and the text embeddings to uniformly separate out.
(MMReg) which uniformly separates the class-wise image and text prototypes in the joint feature space, thereby enforcing inter-class variability. The proposed framework is illustrated in Fig. I We now describe the two modules in detail.
5.1 Selective Augmentation
We use selective augmentation strategy increase the handful number of support set samples the target dataset. For example, in the 5-way, I-shot setting, have only 5 images (one image from each class) for the model to train on. Naturally, training the prompts such less number of data samples cause overfitting, hence, reducing the accuracy the query set. To address this problem, we first take combination of different augmentations of the support set images like HorizontalFlip, RandomRotation, ColorJitter, elc. However, instead of considering all the augmentations, we elficiently select only a few of the above augmented images, which works equally good, or better than taking all the augmentations, with reduction in training time.
Let us consider the support set images X origs and the respective augmented versions Xaug: The total support samples classnames (Xr), with which append the learnable textual prompts and pass them through the frozen CLIP text encoder fi, to obtain five text embeddings. Similarly, we append learnable visual prompts to X* and pass them through the CLIP image encoder fe, and get the image embeddings. In the vision-language multimodal space, the LcE tries to train the prompts such that the respective class text embeddings and the image prototypes (mean of the class-wise image embeddings) come closer. For the selective augmentation strategy, we choose subset of Xs, whose cosine similarity with the corresponding class text embedding in the feature space is higher, i.c.,
then be written as X i Xavg)- Consider the 5-way setting. We have five
X (fmm ((0;; s.t., X X
Here, topr(.) function takes the r top values of its argument and X is the r selected images from the set of all images X Unlike scenarios where large number of training samples may be present and strong augmentations may be more beneficial, in this application with few one example per class, it is important that the augmentations are trained with class representative examples, which aids the model training.

---

Published in Transactions on Machine Learning Research (01/2025)
Effectiveness the selective augmentation module. We generate different augmentations and select 15 examples based on the proposed strategy. We report accuracies and training time for the BSCDFSL benchmark after training on the same 20 sampled episodes.
Table 2:
ISIC
Disease
X-Ray
Augmentations
Accuracy
Time (mins.)
Accuracy
(mins.)
Accuracy
Time (mins.)
Accusary
Time (mins.)
20 augs
79.39
22.20
33.60
22.50
81.46
21.88
22.90
22.11
augg
18:57
30 aaga
79.20
30.03
33.20
30.45
80.27
29.97
20.60
30.06
augg
45 aags
78.26
41.61
33.86
42.21
80.13
41.83
22.33
42.01
sunge
Omniglot
(d)
Plant Disease
(b)
(c)
(d)
Figure 2: Effectiveness of the MMReg. All the heatmaps represent inter-class L2 distances bet ween embeddings for a representative episode the 5-way setting. Darker hues represent higher values and vice versa. In the images, the text features are followed by image features from left to right. (a) represents initial text and image feature distances, (b) represents the text and image embedding distances without MMReg, (c) represents the text and image distances with only text regularization, while (d) represents the text and image embeddings with MMReg. We observe in (b) that there is significant difference between the interclass distance of the image and text embeddings, implying that their embeddings are not that similar.
Using only text regularization part separates the text but not the image features. In contrast, the maps are very similar for (d), which justifics the usefulncss the MMReg.
We perform simple experiment to justify this selection strategy. Here, we initially generate differrnt number of augmenlations for each support image and then select 15 examples based the proposed selection stralegy. Table 2 shows the accuracies and training time for the four datasets in the BSCDFSL benchmark (Guo et al, 2020 We note that if the difference in the number of original and selected augmentations is low, the time and accuracy differences are not significantly impacted, and it may be feasible to consider all the original augmentations instead of a selection strategy. However, as the difference increases, it worth considering this strategy due to time and accuracy considerations. Only for PlantDisease, we observe a slight decrease in performance, which upon analyzing the augmented images, we feel can be attributod to the quality of the initial augmentations. In conclusion, our proposed stralegy reduces the training time by 40% while maintaining or even improving the accuracies consistently across the four datasets in majority of the cases.
5.2 Multimodal Margin Regularization
When deploying the model on a downstream task, the target dataset can contain examples of novel categories which are unseen and very different from what was encountered during pretraining. These data can be from specialized domains, e.g., EuroSAT (Helber al.) (019)) which contains satellite images, Chest X-Ray (Wang al. 2017), containing medical X-Ray images, plant disease data (Mohanty et al. 2016), etc. Since

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Effect of prompt tuning with text regularization R(Xx) vs MMReg.
Method
Omniglot
Plantac
ISIC
Plant Discase
MaPLe + sel. augs
85.49
64.69
33.54
82.66
MaPLe + sel. augs + text reg
85.76
64.65
33.44
82.54
MaPLe + sel. augs + MMReg MaagintMarginn)
87.01
66.13
33.90
83.48
the CLIP model may not have seen similar data during training, it may not be able to generalize to these classes and bring their text and image embeddings close in the multimodal latent space using few labeled training examples. In addition, generalization of CLIP significantly depends on the presence of meaningful class names of the unscen categories. But often, such class names may not be provided, or even they are available, they may not be semantically meaningful in the CLIP space. We have observed this in Section which remonstrales a clear correspondence between zero-shot performance and the joint. feature space aliguments of the text and image embeddings. Thus, while finetuning with less amount of support set data, the model may not be able to generalize and discriminate between these classes. Now, we describe the proposed Mullimodal Margin Regularizer (MMReg), which tries to simultaneously improve the inter-class discrimination and bring the inter-moda embeddings closer.
Our proposed MMReg inspired from Hayat al. (2019), where the regularizer addressos the training data imbalance problem in classification lasks by unilormly spreading out the classifier weights in the feature space. In our context, we aim to spread out the text embeddings in the feature space to avoid confusion between the distinct classes. The regularization term can be expressed as follows:
R(XT)= N2 N
E(1x7 - X7,112 uo) Vj E (2,3...,
(6)
Here, N denotes the number of classes in each episode, in a N-way k-shot setting. Each classname text, appended to learnable prompts is passed through the text encoder to obtain the classwise text embedding vectors Xy. The mean distance between these text embeddings denoted by 44 and can be written as:
2 N N
ΣIXT - X1 Vj E (2, KCJ
(7)
This regularizer trains the prompts in such a way that the text representations are uniformly separated by distance of (te Since, the text prompts are coupled to the visual prompts through function F(-), it is expected that this regularization term will also guide the visual representations to separate out. However, because of the presence of few training samples, we empirically observe that there is a lack of consistent separation between the image embedding prototypes compared to their textual counterparts. We consider four datasets to illustrate this in Table B where we observe that simply separating text features reduces the performance prompt luning (MaPLe). We can also visualize this from the L2 distance heatmaps shown Fig. [ where the visual features are not affected by the text regularizer. Hence, we add another regularization term to the loss function, to separate out the image prototypes as follows:
2 R(Xv)= N2 - N
Σ(I - Xiv -,) Vj E (2.3...., N}
(8)
13
where, Ny are the prototypes (means) of the class image embeddings obtained from the visual encoder, i.e., X.e)) Pt is the same mean distance from Equ. 7 This additional regularization term enforces the class-wise image protolypes to be equally separated by the same mean distance as the text representation embeddings. We observe Table 3 that the proposed MMReg improves performance in all the cases. We also this in Fig. 2 where the classname reprosentations and the image prototypes better separated in the feature space. Thus, the final loss function for training the prompts is given as:
Liotal LCE R(Xx) R(Xv)
(9)
8

---

Published in Transactions on Machine Learning Research (01/2025)
Minimizing this objective function ensures that the class text embeddings come close to the respective class image protolypes, and at the same time uniformly spreads out both of them in the joint vision-language representation space.
Inference. Once we train the joint VLM prompts on the support set image-text pairs of a sampled episode, we freeze the prompt parameters for the inference phase. Next, we feed the query set image-text pairs of that episode to our frozen model, and take the highest probability class as the final prediction.
6 Experimental Evaluation
Here, we describe the results of extensive experiments performed to evaluate the performance of the proposed approach. First, describe the datasets used.
6.1 Dataset Description and Experimental Protocol
The proposed PromplMargin works in a source-free setting, i.e. the model is directly fine-tuned on the episodic support set of the target datasets. Thus, unlike many of the prior approaches, we do not require a source domain like ImageNet for pre-training our model. We conduct experiments on fifteen target bench- mark datasets with varying distribution shifts, namely, EuroSAT (Helber et al. 2019). ISIC (Codella et al.
2019
Plant Disease (Mohanty et al. 2016), Chest X-Ray (Wang et al. 2017] (from BSCDFST. (Guo et al. Omniglot (Lake et al. (2015), Traffic Signs (Houben et al.) 2013). 3 MSCOCO (Lin et al.) 2014], Tex-
2020
tures
Cimpoi al.) 2014), CUB (Wah et al. 2011 Quickdraw (Jongejan al., 2016), Aireraft (Maji al.,
2013
VGG Flower NNilsbck & Zisserman 2008). Fungi (Schroeder & Cui 2018), mini Mini-ImageNet (Vinyals
ct al.
2016
(from Metadataset (Triantafillou et al. 2019) ). and the iNaturalist Plantae dataset (Van Horn
et al.
These datasets contain images ranging across different styles and categories, including medical images, satellite images, handwrilten character images, etc.
2018
As in the existing literature, we consider two settings for the experiments, namely, 5-way 1-shot and 5-way 5-shot, where one and five images from each of the five classes are randomly sampled in each episode for training. Following standard protocol (Guo et. al.) 2020 Liu et al., 2020 Liang et al.) 2027), we also take 15 query images from the same set of classes, and evaluate the model on 600 episodes, and report the average accuracies and 95% confidence intervals.
Implementation Details: We use the CLIP VIT-B/16 backbone for all our experiments, similar to MaPLe Khattak et al. 2023). For the learnable text prompts, we initialize the vectors with the standard text prompt "A photo of The function F()) taken as lincar layer which projects the text prompts to visual prompts. The vision and text prompt lengths are set as 2. For deep prompting, we introduce learnable prompts before every transformer block upto depth of 9. For the 1-shot setting, we generate multiple augmentations, out of which we selectively choose 15 augnentations
discussed. Similarly, for the 5-shot setting, we consider 3 selective augmentations, such that there are 15 examples per class. We jointly train the prompts on the support set images to minimize the final loss function, with SGD optimizer for 150 epochs with a learning rate of 0.01 and momentum of 0.9. Following Liang et al. (2021), we keep the above hyperparamotors same across all datasets and settings. All the experiments are performed on a single NVIDIA RTX A5000 GPU. Now we report the results of experimental evaluation.
6.2 Comparison to the state-of tf-th--rtt methods
We compare our method with the most recent CLIP-based methods for all the fifteen datasets and report the results in Table 4 This includes full finetuned methods well prompt-tuning methods of CLIP, We also explicitly mention the backbones as well as different training procedures followed by the different approaches, which should be considered while comparing them.
A recent state-of the-art approach, FDAlign (Song et al. 2024), was the first to investigate CLIP's gen- cralization capability to the few-shot learning datasets with domain differences under similar settings. It proposes a CLIP finetuning technique, which meta-trained on the minilmageNee dataset before adapting to the target datasets. Wise-FT (Wortsman et al. 2022 was originally proposed for the domain generaliza-

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: The performances (accuracies) of CLIP-based methods Wise-FT, FDAlign, MaPLe and Ours (PromptMargin) on all the target datasets for both the 5-way 1-shot and 5-shot settings. Wise-FT and FD-Align has been trained on a source dataset while MaPLe and PromptMargin is directly trained on the few target samples. Highost values are marked in bold.
Datasets
5-way 1-shot MaPLe
5-way 5-shot MaPLe
WiSE-FT
FD-Aligu
PromptMargin
WiSE-FT
FD-Align
PromptMargin
EuroSAT
63.99 + 0.39
60.39 + 0.43
75.46
0.19
78. 95 + 0.19
80.96
0.19
77.25 0.16
89.55 + 0.30
91.40 0.10
ISIC
29.40 + 0.34
28.84
+ 0.44
31.96
0.15
33.90 + 0.15
39.54
+ 0.40
38.91 = 0.44
46.92 + 0.14
46 88 0.16
Plant Disease
75.06 + 0.33
75.13
+ 0.33
79.38
0.22
83.48
+ 0.19
91.78
+ 0.31
91.84
= 0.19
93.32 + 0.11
94.31 0.10
ChestX
22.27 + 0:28
22.31
0.17
21.30
0.10
21.51
0.10
0.14 25.08
24.95 = 0.15
23.29 + 0.10
23.92 + 0.09
INaturalist Plantae
55.34
66.13 + 0.27
83.07 + 0.26
85.36 0.19
Omniglot
83.56 +: 0.25
83.81 + 0.25
77.82 0.29
87.01 + 0.22
95.26 + 0.09
94.81
0.19
96.23 + 0.30
96.37 0.13
Traffic Signs
60.84 +: 0.29
67.32
+ 0.26
56.45
0.24
G7 24.24:. + 0.24
78.11 + 0.24
73.39
0.29
85.21 + 0.19
87. 0.16
MSCOCO
07.28 + 0.32
69.16
0.28
53.09
0.24
56.12 0.24
81.08 + 0.35
81.37
+ 0.24
75.13 + 0.22
78.68 + 0.23
Textures
63.55 + 0.19
66.05 + 0.12
79.28
+ 0.18
78.99 0.20
83.31 + 0.31
83.60
= 0.34
88.45 0.14
88.71 0.15
CUB
81.16 + 0.71
82.38 + 0.60
96.96
0.22
96.97 + 0.22
93.41 + 0.32
93.87 =
97.65 + 0.00
97.12 + 0.06
Quickdraw
62.54 + 0.50
64.49 + 0.58
72.54
0.22
74.84 + 0.30
82.78 +
82.78
85.08 +
85.21 0.13
Aircraft
62.64 + 0.62
63.45 + 0.65
79.76 +
77.21
77 66 +
78.21
87.56 + 0.21
86.53 +
VGG Flower
94.16 1: 0:255
93.50
0:24
98.24 1:
97.
99.06
98.95
99.23
99. 27 0.02
Fungi
53.10 1: 0:277
53.83
0.30
58.55
0.27
61.16 1: 0.24
73.25 28 1:
73.69 0.14
79.69
80.91 0.18
Mini-test
93.55 + 0.17
95.04 + 0.16
99.17 + 0.08
98.85 0.08
98.44 + 0.06
98.52 = 0.07
99.39 + 0.02
99.19 + 0.02
Average
65.26
65.40
69.02
72.00
78.55
78.01
81.92
82.76
tion lask, where the CLIP model was finetuned with parameter weight interpolations, which was adapted to the given setting. We compare with both of these methods, and include their performance accuracies as reported in Song et al. (2024). Additionally, for prompt lcarning mothods we report MaPLe (Khattak et al., 2023), which is a Dision-language prompt tuning method, which serves as another strong baseline for our method. We adapted this in our setting, where we finetune it directly on the few samples from the sampled episodes of the target datasets.
Hence, we primarily explore whether CLIP can be deployed using prompt learning very few samples the target dataset without any meta-training on large-scale dataset like ImageNet. Here, in addition to the challenge of robustly learning prompts with few samples, for many datasets, the class names are either not provided or are not quile meaningful for CLIP to generulize, tsus making prompl-tunin even more challenging.
For the 5-way 1-shot setting, we observe that both full finetuning methods (FDAlign and Wise-FT) perform poorly compared to the parameter efficient finetuning methods. Our proposed method outperforms the baseline method MaPLe in eleven out of fifteen datasets, achieving average accuracy of 72% compared to MaPLc's 69.02%. In some cases, where original classnames were not present or are semantically not meaningful, e.g., Plantae, Plant. Disease, Traffic Signs, we achieve absolute accuracy gains of 10.79%, 4.10%, 10.79% respectively over MaPLe, highlighting the fact that our method gives significant improvement even when original classname texts are not present in the datasets. This is discussed in details further in the following section. However, our method exhibits slight decrements on certain datasets (like IaaeNet and Aircraft), which can be attributed to the dataset distributions being not shifted in the CLIP space, discussed later. For the 5-way 5-shot setting, we observe similar trend, where PromptMargin outperforms MaPLe in twelve out of fisteen dalasets, achieving an average accuracy of 82.76%.
Our method aims to highlight that large-scale VLMs like CLIP can be efficiently transferred to out-of- distribution datascts with few-shot samples, without any access to source datascts, and can still provide a relatively close performance to meta-trained and finetuned methods.
6.3 Ablation Studies
Here, we analyze the effectiveness of the two proposed modules PromptMargin, and summarize the results in Table 3 For analysis, we consider four representative datasets from across all the benchmarks and report accuracies for the 5-way 1 s-ohot setting using the same hyperparameters from Sec. 6.1 for 600 episodes. Since
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Ablation study; Both selective augmentation and MMReg are important.
Method
Omniglot
Plantae
ISIC
Plant Disease
MaPLe (baseline)
77.82
55.34
31.96
79.38
MaPLe + MMReg
78.46
58.04
32.49
79.59
MaPLe + 15 sel augs
85.49
64.69
33.54
82.66
MaPLe + 15 sel augs + MMReg (PromptMargin)
87.01
66.13
33.90
83.48
Table 6: Effect of the proposed regularizatinn terms (MMReg) in separating out the interclass joint features in the CLIP representation space with only a single example per class.
Dataset
EuroSAT
ISIC
Omniglot
Plant Disease
Quickdraw
Plantae
Traffic Signs
MSCOCO
mini-IuageNet
mr
0.588
0.561
0.679
0.770
0.716
0.100
0.100
0,100
0.860
my
0.627
0.582
0.420
0,540
0.520
0.727
0.583
1.010
1.010
MaPLe (96)
75.46
31.96
77.82
79.38
72.54
55.34
56.45
53.09
99.17
MaPLe + MMReg (%)
75.81
32.49
78.46
79.59
73.02
58.04
59.24
51.90
98.99
our framework is built upon MaPLe (Khattak et al. 2023), we report its accuracies as the baseline method. As illustrated in the table, both the proposed modules improve the baseline method significantly.
We had proposed the MMReg module based on the observations of the feature aliguments in the joint CLIP vision-language space. Now, we see in Table B how this simple regularization term is effective in guiding the vision language features to separate out even for a single example per class. As noted in Section
3 when the inter-class text and image embedding distances (my and my) were low, prompt-learning (MaPLe) had improved the performance, but additionally incorporating our regularizer further improves the class discriminations and results in better accuracy. Notably, we observe that when the image feature separation (mv) is low, and placeholder classnames have been used, the improvement with our MMReg module is significant (+2.7 and +2.79 for Plantae and Traffic Signs respectively). Although in MSCOCO, pscudo-classnames have been used, the image features are extremely well separated, resulting in the MMReg module slightly decreasing the performance. However, utilizing only the text regularization term R(Xr) improves the leanpt- accuracy by +1.28, hence conforming with our proposed notion of separating out closely situated embeddings in the representation space. For mini- Main--mmaeNNet, since both modalities are well separatod, prompt-learning, even with our regularizer, on the few samples does not improve results, and can adversely affect the latent space alignment. We also illustrate some qualitative results in Fig. 3 for visualization of some of the different datasets.
Scope for future work: Although the two modules of our proposed framework demonstrate good perfor- mance for most of the datasets, in few cases, it failed to outperform MaPLe. As for the first module, a lack of carefully chosen augmentation strategies may hurt the generalization more than it improves. As an example, in Fig. 4 we illustrate some poor augmentations generated for the Aircraft dataset, where our method fails to improve over MaPLe. Similarly, in some cases, where the features in the latent space are already well separated, further finetuning with MMReg may adversely affect the CLIP space, hence reducing the performance, Nevertheless, this work may serve as strong baseline for robust prompt learning techniques of foundation models like CLIP for such challenging and real-world settings.
7 Conclusion
Largo-scale vision-language models like CLIP are emerging as popular choice due to their powerful zero- shot generalization capabilities. Prompt learning is an efficient technique to transfer CLIP-like models to downstream datasets with few samples. However, to the best of our knowledge, there has been no work where prompt learning has been utilized for classification tasks where the datascts simultancously contain few samples as well as shift in distribution from natural images. In this work, we explore the possibility of learning only few prompt parameters the target datasets, in completely source-free manner. Extensive
11

---

Published in Transactions on Machine Learning Research (01/2025)
paintbrush
cactus
cactus
headphones
Industrial
Highway
SeaLake
Pasture
Vascular
Benign Keratosis
Melanocytic Nevi
Actinic Keratosis
Exidia pithya
Cortinarius anomalus
Russula claroflava
Cortinarius balteatocumatilis
Lesions
Figure 3: Some qualitative results across different datasets. From top left, samples are shown from Quickdraw [Jongejan et al. 2016). EuroSAT (Helber et al., 2019). ISIC (Codella et al., 2019 and Fungi Schrooder & Cuil 2018) datascts. Green and red denote correct and incorreet predictions respectively.
(b)
Figure 4: Some instances of poor augmentations generated for two support set images, leading to reduced generalization. (a) and (b) represents images from two classes of the Aircraft dataset, namely "Hawk T1" and "Boeing 747-400".
experiments on standard benchmark datasets highlight the efficacy of our proposed approach over state-of- the-art methods.
References
Mehdi Cherti, Romain Beaunont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2829, 2023.
Mircca. Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamod, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 1806-3614 2014.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gulman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902, 10X3:3990,.0368, 2019.
Chelsca Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1133 PMLR, 2017.
Yumhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot. learning.
In Computer Vision
12

---

Published in Transactions on Machine Learning Research (01/2025)
ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 124-141. Springer, 2020.
Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Gaussian affinity for max- margin class imbalanced learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6469 6479, 2019.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222 2019.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. The 2013 international joint conference on neural networks (IJCNN), pp. 1-8. Teee, 2013.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Tim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(2018):4, 2016.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 19113-19122 2023.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332 1338, 2015.
Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross- domain few-shot learning via noise-enhanced supervised autoencoder. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9424-9434, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 18th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Parl V 13, pp. 740 755. Springer, 2014.
Bingyu Liu, Zhen Zhao, Zhenpeng Li, Jianan Jiang, Yuhong Guo, and Jieping Ye. Feature transformation ensemble model with batch spectral regularization for cross-domain few-shot classification. arXie preprint wwXiv2200500843, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint orXiv:1306.515. 2013.
Sharada P Mohanty, David Hughes, and Marcel Salathé. Using deep learning for image- based plant disease detection. Frontiers in plant science, 7:1419, 2016.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sizth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamcla Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International conference on learning represenlations, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2523 2015.
Brigit Schrooder and Yin Cui. Fgvex fungi classification challenge 2018. Available online: github. com//viipediaa/fee hatpaa/fwwoooo_g___o_o_ fungi_come comp (accessed on 14 July 2021), 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Kun Song, Huimin Ma, Bochao Zou, Huishuai Zhang, and Weiran Huang. Fd-align: Feature discrimina- tion alignment for fine-tuning pre-trained models in few-shot learning. Advances in Newral Information Processing Systems, 36, 2024.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 403-412, 2019.
Eleni Trriantffillo,, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, al. Meta-datasct: A dataset of datasets for learning to learn from few examples. arXie preprinl XXix:1903.03066 2019.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No' zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint 2Xxx:2444404125, 2024.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1769-8778, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on ceakly-supervised classification and local- ization of common thorax diseases. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2097-2106, 2017. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. 7959 7971, 2022.
Chengming Xu, Yanwei Fu, Chen Liu, Chengjie Wang, Jilin Li, Feiyue Huang, Li Zhang, and Xiangyang Xue. Learning dynamic alignment via meta-filter for fow-shot learning. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognition, pp. 5182 5191, 2021.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling, orXiv preprinl 3rXiv:2111.03999 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition, pp. 16816-16825 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for langunelanguage models. International Journal of Computer Vision, 130(9):2337-22)) 2348, 2022b.
14

---

Published in Transactions on Machine Learning Research (01/2025)
A Choice of the number of Selective Augmentations
In general, generating more augmentations can improve performance by mitigating overfitting in few-shot learning, but these can also include bad augmentations. The proposed Selective Augmentation module aims to efficiently select subset of the generated augmentations, removing worse augmentations in the feature space, and hence maintaining a competitive performance while reducing the training time significantly discussed in Section 5.1 Here, we report a sensitivity analysis on the number of selective augmentations in Table for two datasets, EuroSAT and ISIC. As expected, for very few augmentations (say, 5), due to less number of examples, the results are general lower. After around 10 augmentations, the performance stabilizes and does not vary siguificantly with the number of augmentations. We have chosen 15 augmen- tations in all our experiments to achieve good performance in reasonable training time, since for few-shot setting, it is not possible to tune hyperparamcters for individual datascts. Dynamically choosing the number of selective augmentations in an episodic manner can be an interesting direction for future work.
B Analysis of Selective Augmentations
We first randomly generate large number of augmentations using standard techniques like RandomHori- zontalFlip, ResizeRpeszzeCrro.. ColorJitter, etc. 'he initially generated augmented images random mixture of the various augmentation strategies and produces variety of augmentations which differ across episodes as well datasets. The selective augmentation module then selects 15 augmentations from these samples based on their cosine similarity with the corresponding class text embeddings in the feature space. To understand which augmentations are being chosen implicitly, we perform an experiment which instead of taking random augmentations, we fix the number of augmentations of each type. E.g. we generate 6 images by RandomCropping, followed by images by ColorJitter and so on, and then analyze which are being discarded by this proposed module. We observe that in general, the augmentations generated by Random GandomCropping discarded, where the region of interest is being removed, followed by ColorJitter, which sometimes blurs or blackens the images. Analysis on EuroSAT and ISIC datasets is illustrated in Fig. 5
Table 7: Sensitivity analysis on selective augmentations for the same 20 sampled episodes.
EuroSAT
ISIC
Augmentations
Accuracy (%)
Accuracy (%)
augs
79.39
33.60
5 sel. augs
75.53
28.13
10 sel. augs
81.47
34.47
15 sel. augs
78.26
34.40
30 augs
79.20
33.20
sel. augs
76.53
27.07
10 sel. augs
77.47
36.27
15 sel. augs
81.93
35.47
20 sel. augs
79.53
33.86
45 augs
78.26
33.86
5 sel. augs
79.47
26.73
10 sel. augs
79.53
34.40
15 sel. augs
78.47
35.33
20 sel. augs
79.40
35.40
augs
78.73
33.26
5 sel. augs
75.13
25.86
10 sel. augs
77.67
29.80
15 sel. augs
79.73
33.66
20 sel. augs
80.33
28.73
15

---

Published in Transactions on Machine Learning Research (01/2025)
Figure 5: Visualizations of the augmentations being selected by the proposed Selective Augmentation module for EuroSAT and ISIC with 20 initial augmentations. Each row corresponds a particular class. The red discarded. We observe that augmentations where the region of interest
borders denote the oncs which are removed or darkened, have fairly more chance of getting discarded. However, in some cases, more augmentations may be needed to be removed than only five.
Table 8: Effect of strong vs weak augmentations.
EuroSAT
ISIC
Dataset
Accuracy (%)
Accuracy (%)
20 standard augs
79.39
33.60
20 strong augs
77.93
34.93
30 standard augs
79.20
33.20
30 strong augs
77.20
32.33
standard angs
78.26
33.86
45 strong augs
77.60
30.67
C Weak vs strong augmentations
Strong augmentations are intense transformations of the images like AutoAugment and CutOut, whereas weak sugmentations are simple transformations like Flip and Shift. In our work, we have used standard augmentations like RandomCrop and HorizontalFlip. which can be attributed to weak augmentations, from which we have then selected subset based the feature space similarities. To understand the effect of strong augmentations instcad of standard augmentations (as originally used), we consider two benchmark datasets, EuroSAT and ISIC and apply RandomErasing (which removes a lile from the image similar to CutOut), and AutoAugment. The results Table suggest that, in general, such aggressive augmentation strategics do not help in classification as distorts and removes crucial information from the images, thereby hurting the generalization. In very few instances, such strong augmentations can crop the background and focus the region of interest. in datasets like ISIC. Some visualizat ions shown in Figs. aadd g
16

---

Published in Transactions on Machine Learning Research (01/2025)
Table 9: Experiments on OpenCLIP B/16.
Dataset
Omniglot
Quickdraw EuroSAT
Plant Disease
ChestXX
Zero-shot (%)
27.79
66.49
48.93
24.65
20.08
MaPLe (%)
80.61
72.21
71.60
77.50
21.09
PrrgitMargin (%)
89.44
77.08
73.28
82.26
22.75
Table 10: Effect of MMReg when classnames are not available.
Dataset
Plantae
ISIC
ChestX
MaPLe (%) (with classnames)
31.96
21.30
MaPLe (%) (with pseudo classnames)
55.34
28.32
20.19
MaPLe + MMReg (%)
58.04
30.24
20.89
Experiments on an alternate VLM
We have reported results for CLIP ViT-B/16 in the main text for fair comparison to other prompt tun- ing methods. Here, we consider an alternate VLM, OpenCLIP B/16 (Cherti al.) 2023), which has been pretrained different dataset. The results some benchmark datasets (with the same hyperparame- ters) are reported in Table g We observe that although MaPLe improves over the zero-shot performance, PromptMargin outperforms MaPLe in all the datasets, further justifying the effectiveness of the proposed framework.
E t-SNE Visualizations
The t-SNE visualizations of the image embeddings after training with our proposed MMReg module is shown in Fig. 0 for two benchmark datasets, Omniglot and Plant Disease. We observe that training with the MMReg regularizer uniformly separates and more compactly clusters the few-shot image examples in the feature space.
F Effect of MMReg in the absence of classnames
Since in many specialized datasets (e.g., medical datasets like ISIC and Chest X-Ray), meaningful classnames have been used, the model may still rely on them for proper classification in the CL.PP space. However, often classnames may not be available for such datasets in practical settings. To emulate such scenario where classnames may not be available, classnames (like C1, C2, C3, elc.), and see the effect of the MMReg module them (Table 10 We observe that while performance of MaPLe (prompt tuning) drops because the class semantics are now missing, our MMReg module helps them separate out in the multimodal representation space, thereby improving the performance both the datasets, which conforms with our motivation for this proposed module.
replace the classnames of these specialized datascts with pseudo
G Sensitivity analysis
In Table 11 we vary the weighting coefficients (a and (3) of the two terms of the MMReg module (R(Xr) and R(xv) respectively for two benchmark datasets, EuroSAT and ISIC. In the main text, we have used a 1 and B 1 for all experiments, since hyperparameter tuning not possible for sew-shot learning. But observe that the performance is quite stable for wide range of parameters.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Omniglot
1:3
(a)
(b)
(c)
Plant Disease
(a)
(b)
Figure 6: Visualizations of the image embeddings using t-SNE for the 5-way setting. (a) denotes the image features initially, (b) trained with MaPLe and (c) with the proposed MMReg. We see that MMReg more compactly clusters and uniformly separates the embeddings the feature space compared to normal training.
Example episade classes where strong augmentations can be better than weak augmentations
Example episode classes where weak augmentations can be better than strong augmentations
Figure 7: Visualizations of strong augmentations and weak augmentations for the ISIC dataset. Each block corresponds to particular class. The first and the second rows in cach block corresponds to strong and weak augmentations respectively. In some instances, the strong augmentatioons can be better than weak augmentations and vice versa.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Weak augmentations better than strong augmentations
Figure 8: Visualizations of strong augmentations and weak augmentations for the EuroSAT dataset. Here, strong augmentations are always worse than weak augmentations since the region of interest is not confined to specific region.
Dataset
Weighting Coefficients
Accuracy
B
1
76.33 0.095
EuroSAT
1
2
77.61 0.019
2
77.49 0.018
1
32.93
ISIC
1
2
32.77+
2
1
32.88 0.063
Table 11: Hyperparameter sensitivity analysis for (for R(Xr)) and B (for R(Xv)).
19


Reviews:
[{"review_id": "OeD6J4CwOU", "reviewer": "Reviewer_EUMS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Experiments were extensive on 15 datasets with varying distribution shifts to demonstrate the robustness of PromptMargin, which showed improved performance over existing methods in most cases. \n\n- The selective augmentation strategy is practical, as it optimizes training time while maintaining or even improving model accuracy, which is essential for few-shot scenarios.\n\n- The ablation studies provide insight into the importance of each module within PromptMargin, helping to justify the effectiveness of each component.\n\n### Weaknesses\n- In cases where target data is closely aligned with the source distribution (e.g., mini-ImageNet), PromptMargin sometimes underperforms compared to baseline methods.\n- The proposed approach’s reliance on multiple augmentations during training may increase computational overhead in some cases", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes PromptMargin to adapt large-scale VLMs for few-shot learning under distribution shifts.\n\n- The authors provides a detailed analysis on CLIP’s embedding space to determine conditions under which it performs well on a target dataset and when fine-tuning is beneficial.\n\n- Moreover, the prompt-learning framework is proposed to directly adapt VLMs to few-shot tasks without pre-training on a large source dataset. The method is \"source-free\", meaning that it relies only on target data samples.\n\n- To improve class separation and generalization, the authors further propose a regularizer to ensure that text and image embeddings maintain a consistent separation in the embedding space.\n\n- The empirical results are presented on 15 different datasets under various few-shot settings, showing superior performance to existing baselines in most cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include additional qualitative or visual analyses (e.g., t-SNE) to illustrate how the MMReg affects class separability in embedding space. This would provide valuable insights into the internal mechanics of the framework and support claims about improved class discrimination.\n- A detailed hyperparameter sensitivity analysis would be beneficial. For example, a brief exploration of the sensitivity of the model's performance to key hyperparameters including number of augmentations and MMReg separation margin could help further understanding the method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yhJ843CT9K", "reviewer": "Reviewer_DSBP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: please refer to the contributions\n\nWeaknesses: \n\n1. The motivation of the proposed methods is not strong enough. While I agree that improving the inter-class variability and selecting only the most relevant augmentations based on their similarity is intuitively helpful, it is widely discussed in traditional domain adaptation literatures and this paper does a simple adaptation in the context of CLIP. I fail to find new insights in the existing submission. Are there any evidences that the mentioned issues (and the corresponding solutions) are especially important for CLIP?\n\n2. While the MMReg aims to improve generalization by separating class embeddings, the model’s ability to handle highly specialized or unfamiliar domains (e.g., medical images) still relies on the presence of meaningful class names, which may not always be available.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Empirical Analysis: Explored CLIP's zero-shot performance on various target datasets by analyzing image and text feature distances in multimodal space.\n2. PromptMargin Framework: Introduced a novel prompt learning framework that adapts CLIP for few-shot tasks with distribution shifts, without requiring training on a source dataset.\n3. Multimodal Margin Regularizer (MMReg): Developed MMReg to improve prompt learning by ensuring uniform separation of image and text category embeddings.\n4. Performance Evaluation: Demonstrated superior performance across 15 benchmark datasets in 1-shot and 5-shot settings, compared to zero-shot, meta-training, and baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please refer to weakness.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0CdufW12c9", "reviewer": "Reviewer_8apX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper is well-written. The motivation is very clear from the abstract. The method is clear. The experiment show that the PromptMargin framework indeed improves over the baseline methods. \n- In section 3, the authors analyzed the mean distances between inter-class prototypes and found that when the target dataset has higher differences show the lowest performance with vanilla CLIP. On the other hand, if the differences are low, then the model performance after fine-tuning vanilla CLIP can also drop. This is an interesting insight. This adds to the strength of the authors’ claims.\n- The ablation studies show that selective augmentation boosts the performance the most. This is an interesting insight. \n\n### Weaknesses\n- While selective augmentation is a neat trick to boost performance, the paper does not provide a clear understanding of why it works.There are several unanswered questions: (a) why did the authors choose 15 augmentations as opposed to 20?, (b) are the 15 augmentations same across the datasets?, and (c) how is this related to the weak augmentations commonly used in self training [a]. A more detailed analysis and experiments are required to understand the effect of selective data augmentation. \n\n- The work is limited only to CLIP. It will be very interesting to readers if the same conclusions from the experiments hold true for other vision-language models. \n\n### Minor \n- In Section 3, the authors claim they estimate the extent of the distribution shift. While the results empirically demonstrate their claim, I would like to see the Pearson correlation between CLIP performance and the distribution and semantic differences metrics.\n\n- Tables 1 and 2 are a bit difficult to understand at first glance because the first three rows are different from the last two rows, and the units are not comparable. The authors could separate the tables.\n\n- Table 4 bolds results from row 2 and row 4. However, only row 4 has the best accuracy, and row 3 has the second-best. Please correct the table accordingly. \n\n[a] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. NeurIPS 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a multimodal prompt tuning method for few-shot learning. First, the authors show that the mean distance between the image and the text embeddings from the CLIP model is a simple but unique approach to determining whether fine-tuning CLIP improves or hurts model performance. Then, they introduce the PromptMargin framework, a multi-modal prompt tuning method that includes selective data augmentation and multi-modal margin regularization. \nThe results for few-shot adaptation (1-shot and 5-shot) show improved performance, on average, compared to strong baselines on 15 datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates few-shot learning via prompt tuning in CLIP models where the few-shot tasks are out-of-distribution. Unlike past work that has tackled such OOD tasks with CLIP by first finetuning the model on a larger dataset before the few-shot adaptation, the method proposed in this work is "source-free" and directly performs that adaptation. They first investigate, by inspecting distances in embedding space, whether we can estimate the performance of zero-shot CLIP on a new task. They then propose a framework for improved source-free adaptation by enhancing prompt tuning with selective data augmentation as well as a regularizer designed to encourage uniform separation of image and text category embeddings. 

The reviewers found that the paper is well-written and the motivation is clear (Reviewer 8apX), the experimental investigation is extensive, and the ablations studies insightful (Reviewers EUMS and 8apX). Reviewers brought up weaknesses with respect to computational overhead, limitations of the method in terms of not improving in all scenarios over baselines (improvements are seen clearly when there is a large distribution shift but not when the source and target domains are similar) (Reviewer EUMS). The authors have clarified the strengths and limitations of their work and discussed computational overhead (which is no worse than competing methods). Reviewer DSBP brought up the limitation of reliance on "meaningful class names". During the rebuttal, the authors addressed this by running additional experiments without the semantically-meaningful names. The authors also ran experiments with another model during the rebuttal, in response to Reviewer 8apX's feedback that the experimental investigation is limited to CLIP.

Based on the above, I recommend the paper to be accepted to TMLR; the reviewers found the claims backed by sufficient evidence, and the findings of interest to the community.
Decision: Accept (poster)