XPL: A Cross-Model framework for Semi-Supervised Prompt Learning in Vision-Language Models
Authors: Omprakash Chakraborty
Aadarsh Sahoo
Rameswar Panda
Abir Das
Venue: Accepted by TMLR ()

Abstract: Prompt learning, which focuses on learning soft prompts, has emerged as a promising approach for efficiently adapting pretrained vision-language models (VLMs) to multiple downstream tasks. While prior works have shown promising performances on common benchmarks, they typically rely on labeled data samples only. This greatly discredits the information gain from the vast collection of otherwise unlabeled samples available in the wild. To mitigate this, we propose a simple yet efficient cross-model framework to leverage on the unlabeled samples achieving significant gain in model performance. Specifically, we employ a semi-supervised prompt learning approach which makes the learned prompts invariant to the different views of a given unlabeled sample. The multiple views are obtained using different augmentations on the images as well as by varying the lengths of visual and text prompts attached to these samples. Experimenting with this simple yet surprisingly effective approach over a large number of benchmark datasets, we observe a considerable improvement in the quality of soft prompts thereby making an immense gain in image classification performance. Interestingly, our approach also benefits from out-of-domain unlabeled images highlighting the robustness and generalization capabilities.

Published in Transactions on Machine Learning Research (06/2024)
XPL: A Cross-Model framework for Semi-Supervised Prompt Learning in Vision-Language Models
Omprakash Chakraborty IIT Kharagpur
opekgp@hitkgp.occ ac.
Aadarsh Sahoo
aadarsh@ibm.
MIT-IBM Watson AI Lab
Rameswar Panda
rpanda@ibm.com
MIT-IBM Watson AT Lab
Abir Das
abir@cse. üikkp. oc. in
IIT Kharagpur
Reviewed on OpenReview: https: // openreview. net/ forum? id= oaAZv30D6M
Abstract
Prompt learning, which focuses on learning soft prompts, has emerged as a promising approach for efficiently adapting pretrained vision-language models (VLMs) to multiple downstream tasks. While prior works have shown promising performances benchmarks, they typically rely labeled data samples only. This greatly discredits the information gain from the vast collection of otherwise unlabeled samples available in the wild. To mitigate this, we propose a simple yet efficient cross-model framework to leverage on the unlabeled samples achieving significant gain in model performance. Specifically, we employ a semi-supervisod prompt learning approach which makes the learned prompts invariant the different views of a given unlabeled sample. The multiple views are obtained using different augmentations the images well as by varying the lengths of visual and text prompts attached to these samples. Experimenting with this simple yet surprisingly effective approach over a large number of benchmark datasets, we observe considerable improvement in the quality of soft prompts thereby making an immense gain in image classification performance. Interestingly, our approach also benefits from out-of-domain unlabeled images highlighting the robustness and generalization capabilities.
common
1
Introduction
Recently vision-language models (VLMs) (Jia et al. 2021 Li al. 2022
2021 Radford al.) 2021 Wu et al.
2021 Fini et al.) 2023 Jiang al. 2023 have shown encouraging progress number of downstream tasks. These models are initially trained on large-scale data to aligu language and vision modalities. Such paradigm allows zero-shot transfer to downstream tasks since one can synthesize natural language description known as prompt the new class (c.g., photo of class name') to be fed to the text encoder and compare the generated lext features with visual features. However, the non-trivial task of choosing the best hand-crafted prompts is difficult, requiring a lot of time and domaan-specific heuristics. This has led prompt learning (Lu) et al. 2022 Zhou al. 2022b a Wang al. 2023). It aims use soft prompts that are learned using labeled samples from downstream tasks, keeping the pretrained model frozen. These approaches have demonstrated comparable performance Lo full fine-tuning though learning only few parameters and are known to adapt to
new tasks quickly (He et al. 2022
Project page: http:://evirr.itthbb..oom wttphhub..bo/poojttts/////

---

Published in Transactions on Machine Learning Research (06/2024)
Performance Gap:
Performance Gap:
100
Accs Accis
ACCs
-1
-15
0% (0.l
Proportion labeled 3/%
18%
Category
Category
Proportion labeled data
(a) EuroSAT
(b) CropDiseases
(c) EuroSAT
(d) CropDiseases
Category-wise performance gap between two models leveraging same amount of labeled and unlabeled data but with different number of learnable prompts (8 and 16 textual and visual prompts) on EuroSAT and CropDiseases respectively. Acos and Aceie denote the accuracy with and 16 length prompts respectively showing the complimentary knowledge acquired by the two models. ( dd: comparison
Figure 1: a b):
of XPL with the conventional text-only prompt learning CoOp (Zhou et. al. 2022b) trained using different. percentages of labeled training data on the same datasets. With only 1% of labeled data, XPL surpasses the fully supervised CoOp (shown with red dotted line). CoOp with same amount of labeled data fail to reach the accuracy of XPL.
To the best of our knowledge, prompt learning has thus far relied only supervised approaches, which makes it critically dependent on heavily curated data requiring tedious human labeling effort. This motivates us to look beyond traditional supervised prompt learning in order to not only minimize the annotation effort but also to improve the performance on downstream tasks in extremely low labeled data regime. Semi-supervised Learning (SSL) has shown promising results in visual scene understanding. Among these, self- solf-training or pscudolabeling (Arazo al. 2020) uses confident predictions of unlaboled samples as true label for further training. Consisteney regularization (Bachman et al. 2014) transforms unlabeled samples to different views and forces the model to learn invariant representations. However, in low-labeled data regime, the learned representations tend to lack enough discriminative power for downstream tasks. To handle this issue, works like (Xu et al. 2022) employes not single, but multiple models towards cross-model representation learning leveraging the complementary representations from these different models. Although these approaches have shown promising results, the strength semi-supervised learning has not been harnessed in prompt learning for large VLMs. In this work, we show that cemi-supervised prompt learning not only exploits the unlabeled data present in hand but also helps learn richer representations without additional manual labeling.
While prompt learning is efficient and quick adaptation paradigm, their low capacity may not allow single prompt learning model to achieve best performances in all. To better exploit multiple prompt learners, we present a semi-sun ceni-supervised approach based on the complementary representations at the model level. We observe that two models leveraging unlabeled data but with different number of Icarnable prompts exhibit markedly different category-wise performance (ref. Figure la and b This indicates that the two models learn complimentary knowledge and thus can complement in providing semi-supervision to each other. To this end, we introduce our semi-supervised Cross-model Prompt Learning (XPL) approach that relies on the invariance of the learned prompts to different views of unlabeled data. Given a pretrained VLM, create set of augmented versions of the unlabeled data and pass them via two pathways (known as the primary and the auziliary pathways) each having a different length of soft prompts associated to them. Then, given an unlabeled image, we bring a confident prediction from the auxiliary network as the pseudo-label for the primary and vice versa facilitating a greater engagement of unlabeled images. To the best of our knowledge, XPL is one of the first works semi-supervised prompt learning VLMs. We evaluate our approach on different image classification tasks in 15 standard datasets from diverse categories including Aerial, Medical, Natural, Illustrative, Texture, Symbolic and Structured images. We focus on learning prompts at significantly low labeled data regime, which includes the conventional few-shot classification settings as well settings involving various proportions of labeled training data. Figure Ic and 4 show that using only 1% training data with labels and rest as unlabeled data, XPL superseeds the performance of the supervised text-only prompt learning approach CoOp (Zhou et al. 2022b) that uses 100% training data with labels in the benchmark datascts of EuroSAT (Helber et al. 2019) and CropDiscases (Mohanty et al. (0166) respectively. XPL also shown to be consistently better than CoOp that uses the same amount of labeled data as ours showing the advantage of multimodal, semi-supervised and model approach for prompt learning.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2
Related Works
Vision Language Models (VLMs). Development of VLMs employing single-stream (Su et al. 2019
2020b Li et al. 2019 2020] or dual-stream (Tan & Bansal 2019 Goel al. 2022 Jia al. Radford et al. 2021 Jiang al. 2023) paradigms have progressed significantly. The prevailing dual-stream paradigm, which separates the image encoder and text encoder, forms the backbone of our approach. By enabling zero-shot transfer to range of downstream tasks, works like CLIP (Radford et al., 2021) and ALIGN (Jia et al. (021]) have substantially changed computer vision lately. Few methods have learned transferable features using additional supervision (Li et al.) 2021 Mu et al. 2021
Chen al.
2021
Li ct al. 2022 2021
finer- gher-grained interactions Yao et al., 2021), modern Hopfield networks (Fürst et al., 2021). optimal transport distillation (Wu al.) (021), cycle consistency (Goel et al. 2022), and hierarchical feature alignment (Gao et al. 2022). However, these use supervised training only. Ours one of the first works that goes beyond the supervised setting and learns prompts leveraging unlabeled data alongside few labeled samples.
Prompt Learning. There have been numerous studios on prompt tuning (Huang et al. 2022 Zhou et al. 2022b) for effective adaption of VLMs. CoOp (Zhou et al.) 2022b), à well-known prompt tuning framework k
draws its inspiration from NLP [Lster et al. 2021 Zhong et al. 2021 Zhou et al. 2022c Sordoni et al. 2024
and uses cross-entropy loss to learn prompt vectors. UPL (Huang et al., 2022) proposes an unsupervised
prompt learning framework without necessitating any annotations of the target dataset, while, ProDA (Lu et al. 2022) learns various prompls from data to manage the variation of visual representations. Some approaches like CLIP-Adapto (Gao et al.) 2021 and Tip-Adapter (Zhang et al. 2021) adjust VLMs by training additional adapter networks using labeled data. In (Shu et al. 2022), a framework for test-time prompt tuning is also proposed that does not require training data or annotations. These methods outperform hand-crafted prompts in a reasonable variety of ways, but they frequently have low generalizability when there are changes in the data distribution.
Sume-Super Learning. Semi-supervised learning (SSL) comprises of several techniques CChapelle et
al.] 2009) to utilize unlabeled data for considerably reducing the dependency on annotations. Many efficient approaches have been proposed over time. For instance, self-training with pseudo-labels (Arazo
ct al. 2020 Grandvalet & Bengio 2005 Lee 2013), contrastive learning (Singh et al. 2021) and consistency
regularization (Bachman et al.j 2014 Berthelot et al.) 2019a by Miyato et al. 2018) have shown to significantly enhance the performance over their supervisod counterparts. Another current trend for SSL is the use of self-supervised learning techniques like rotation prediction (Gidaris et al. 2018), discriminative image transformation (Dosovitskiy et al. [0144] etc. Recently, several semi-supervised approaches have been proposed which work both multi-modal (Alwassel et al. 2020) and cross-model settings. (Xu al. 2022) considers two video models with different architectures to generate pseudo-labels that are used to train each other in cross-teaching fashion. Although semi-supervised image classification has made great strides, SST for prompt. learning is still new and understudied issue.
3 Methodology
Using pretrained vision-language model e.g., CLIP (Radford et al.] 2001); the aim of our proposod approach is to learn prompts in a semi-supervised setting for efficient and generalizable adaption of the model to various downstream tasks.
3.1
Background
Revisiting Vision-Languse Models. We build our approach on top of pre-trained VLM, CLIP (Radford et al. 2021). that combines a text encoder and an image encoder. Specifically, we adopt a vision transformer (ViT) Dosovitskiy et al. 2020 based CLIP model, which is consistent with current prompt learning techniques (Zhou et al. 2022bla). As explained below, CLIP encodes an image alongside an associated text. description. The image encoder takes image I, splits it into M fixed-size patches and embeds them into patch embeddings ef, where p p=1, 1,..., denotes spatial locations. We denote the collection of embeddings E, (eglp- = (1,..., M)} as input to the (i + 1)th layer T.41 of the vision encoder. Together with an extra
3

---

Published in Transactions on Machine Learning Research (06/2024)
learnable classification token ([CLS]), the vision encoder can be compactly written as,
Xx Ej = (1. Vi = 1, 1 (1 is # of layers)
(1)
where x, E Rd denote [CLS] embedding at Li+i's input space. Similarly, words from the text descriptions are sent to the text encoder to produce text embedding w E Rd. CLJP uses a contrastive loss during training to find combined embedding space for the two modalities. For a mini-batch of image-text pairs, CLIP maximizes the cosine similarity for each image with the matched text while minimizing the cosine similarities with all other unmatched texts.
Once the two encoders are trained, recognition can be performed by finding the similarity between an image and its textual description in the joint embedding space. In place of only the classnames a more informative natural language class description or prompts for these classnames can be used. Some of such carefully designed prompts found to be useful in the litcrature are: 'a photo of a {class}', photo of person doing factivity class}' elc. Given C' class names, the lext encoder generates C' text. embeddings [ For test image I with embedding x,, the prediction probability p(y(I) is calculated as:
exp(sim(XI, w2)/T) p(ul1) = Ee exp(sim we)/T)
(2)
where, is temperature hyperparameter and sim(.) denotes cosine similarity function.
Text and Visual Prompt Learning. To overcome the shortcomings of hand-cngineered prompts, prompt learning aims to learn continuous vectors at each input token using a small amount of labeled data. Given a pre-trained model, a set of N learnable vectors are introduced in the input space. In order to learn the language prompts, set of prompt vectors T = [t( [e the input emboddings take the form [t),t²
are introduced in the text branch of the VLM. Now, ,2, ,..., N Jem1: where c stands for the word embodding of the ch introduced in the vision branch together with the input image tokens
class label. Similarly, V = {vi],N to learn the visual prompts. After introducing the prompts at the input layer of the vision encoder, the formulation for the / layers are modified as,
[x1 Z1, E1] = L1 ([x0, V, Eol)
(3)
[x Z, Ej] = L; Zi E--1)) Vi 2,,,....
where, Zi represents the features computed by the 7th transformer layer. During training, only these specff text prompt (T) and visual prompts (V) are updated, the VLM remains unchanged.
3.2 XPL
The proposed XPL framework leverages on the unlabeled data a very low labeled data regime to learn prompts that are more generalizable and enhance downstream classification performance. Though traditionally not used in prompt learning, semi-supervised approaches like pseudo-labeling and consistency regularization have demonstrated great performance in recognition (Arazo et al. 2020 Berthelot al. 2019a b Chen et al. 2020a Miyato et al. 2018 Singh et al. 2021). We propose to leverage on the huge pool of unlabeled images to shine light into the gaps between handful of labeled examples. One idea in using unlabeled data is to generate different views of the same input by augmenting it differently and force the deep network to predict the same information from the two views.
Typically, single model trained on handful of labeled data is used for such semi-supervised learning. In our cross-model approach introduce auxiliary network in addition to the primary VLM and ask them to produce the supervision for each other that in effect, encourages to learn complementary representations for the same unlabeled data. As seen in Figure 2 given an unlabeled image I, both the networks get two distinct views Ioo and Istr of the image using a 'weak' and a 'strong' augmentation respectively. 'Weak' augmentation is standard flip-an--shift operation while RandAngment (Cubuk et al. 2020) is used for 'strong' augmentation. In our multi-modal approach, to achieve mutual collaboration between the text and visual prompts, instead of using two distinct prompts in the text and visual branches, we derive the visual prompts V directly from the text prompts T using coupling function F(.), i.e,, v = =(('). We implement. F(.)
4

---

Published in Transactions on Machine Learning Research (06/2024)
1p"]
[c,
Labeled Samples
text feature (w*)
Ground Truth
Lyp Vp
VUM
Cross-entropy loss
angmemtation
visual feature (x₂
my [c,c
text feature (wP)
Pseudo-label p.ik
weak angmentattin
P.wk
VLM
p.str
visual feature (2, we
Unlabeled Samples
visual feature (x
Cross-entropy loss
Cross-entropy
loss
NZJ
[el cu
Pseudo-label (qawk)
text feature (w')
q
aww
gastr
visual feature (x, visual feature (x 3,
angmeniation
Primary network
parameters
Auxiliary
Frozen parameters
Figure Illustration of our XPL approach. Our approach consists of primary and auxiliary networks that share the same pretrained frozen VLM. The primary network accepts text and visual prompts (Tp and Vp respectively) with N tokens while the auxiliary network gets prompts (T, and V. respectively) with half the number of tokens. The visual prompts are generated from the textual prompts by learnable coupling function F(.). At first, the prompts are learned using limited labeled data (upper portion of the red dotted line). Subsequently for the unlabeled samples (lower portion of the red dotted line), in absence of labels, prompts are trained by encouraging representations to match in both networks. This done by minimizing the cross-entropy loss between pseudo-labels generated by the auxiliary network and the predictions made by the primary and vice versa. Given image at test. time, only the primary network is used for inference.
as simple linear layer. For the primary network, the two prompts are denoted as T, and Vp respectively. Similarly, the same for the auxiliary network are T, and Va respectively.
Given few labeled and large amount of unlabeled data, goal is to learn set of prompt vectors for both Tp and T, as well as the coupling function F(.). To better capitalize on the complementary information from the two networks, we propose to use prompts of different lengths (i.., different N) in them. Models with different nunber of prompt vectors exhibit markedly different behaviors in regards to category-wise performance. As the two models with different prompt lengths differ in what they learn, they can complement. in generating the supervision for each other. Our primary and auxiliary networks use N and N/2 prompt vectors respectively, i.e., Tр = (t)) and T = (+
Supervised Training. A labeled image I, with groundtruth class e; only weakly augmented and passed through the model with associated text and visual prompts. Similar Eq. g the prediction probabilities in the primary and auxiliary networks are given by,
exp(sim(xf,, w., I/T
p 1.) exp(sim(xf.. w2.)
(5) where, the superscripts p and a denote the primary and the auxiliary networks respectively. Given the number of labeled images b in batch, the supervised losses of the networks are given by, caup= -- Eb_ log pu? [I)) and Leup - log
p I. Ee exp(sim(xf; w²)/7)
(4)
Ec exp(sim(xf,, w
5

---

Published in Transactions on Machine Learning Research (06/2024)
Cross-model Unsupervised Training. For unlabeled image I,, the weak and strongly augmented versions I5t and 1 are passed through both the networks along with the learnable text and visual prompts. The final layer of the primary network's vision encoder generates two [CLS] embeddings xij and x respectively for 15th and 15t The language encoder generates C text embeddings {wele Probabilities of the weakly and strongly augmented images to belong to class are given by, exp(sim(x{] ww)/7)
exp(sim(x1,) w!)/r)
ply? [T,)
(6)
ply 11) Ei=1 "xp(ssisxsx(( w()/T)
(7)
Eiz exp(sim(x()) w()/r)
For all C classes, these are collected as the weak and strong probability distributions, qf qf
(p(p (ooP"/WIIL) .. -PUC [I]] and qf = [p(pp [Ty), (I),, plu8 strong probability distributions of the same image from the auxiliary network are obtained as q; and 9, afat respectively. The pseudo label from the weakly augmented image in the primary network is given by, af which is an one-hot vector with 1 in the position of arg m Likewise, 9, denotes the pseudo label from the weakly augmented image in the auxiliary network. The cross-model unsupervised losses are enforced as;
[,]]. In a similar manner, the weak and
jub
ab
cp
1 (max(q)
w.) 2 p)H a
p.str).
L
(max(q² ). > p)H ,9 .g.atr)
(8)
jub
jub
1=1
j=1
where, / is the ratio of the number of unlabled to labeled examples in minibatch, p is suitable threshold for gelling the pseudolabels and HG) denotes the cross-entropy function. Overall, the loss function for learning the prompt vectors involving the limited labeled data and the unlabeled data is,
L Lp + Loup + MC2 + C
(9)
where A denotes a hyperparameter for scaling the relative weights of the unlabeled losses.
Inference. After training, we only use the primary network for inference. At test time, an image is passed through the vision encoder and the prompts along with different class names are passed through the text encoder. The class giving the maximum cosine similarity with the extracted visual features is taken as the predicted class of the test image.
4 Experiments
In this section, we investigate XPL and aim to address three primary research questions. Q1: Do prompts learned using XPL effectively leverage unlabeled data for semi-supervised classification? Q2: How does XPL benefit from the novel cross-model design over other methods? Q3: Is XPL robust towards various distribution shifts in the training data and can it generalize to unseen classes?
4.1 Experimental Setup
Datasets. We evaluate XPL on 15 diverse classification datasets, namely, (a) Natural Images: CropDis-
eases
(Mohanty et al. 2016),
Deep Weeds (Olsen et al., 2019), Caltech101 Fei--Fei et al., 2004). Oxford-
Pets Parkhi et al. 2012)
1lowers102 (Nilsback & Zisserman 2008
UCF-101 (Soomro et al.) 2012) )
ImageNet. (Deng et al. 2009
StandfordCars (Krause et al. 2013); (b) Aerial Images: FuroSAT (Helber et al.
2019: (c) Medical Images: ISIC (Codella et al.) 2019)), Chest.X (Wang et al. (017)): (d) Ilussrattvve Images:
Kaokore Tian et al.
2020); (e) Texture Images: DTD (Cimpoi al. 2014); (I) Symmollcc Images: USPS (Hull
1994] (g) Structured Images: Clevr-Count (Johnson et al. 2017). For experiments under domain-shift, we
use the DomainNet (Peng et al.) 2019 dataset.
Baselines. Being one of the first works in multi-modal semi-supervised prompt learning, we carefully design the baselines for a comprehensive assessment, The baselines we are going to describe are different in terms of whether or not they use unlabeled data, 2. learn text or visual prompts or both and 3. use auxiliary network along with the primary network. Table lists these baselines in terms of the above three aspects for quick reference. We start with the simplest of baselines first Éee., we neither use unlabeled data nor the
6

---

Published in Transactions on Machine Learning Research (06/2024)
Baselines
Data
Prompts
Networks
Labeled
Unlabeled
Text
Visual
Primary
Auxiliary
TPL
VPL
MPL
TPL
VPL
V
V
MPL
XTPL
XVPL
XPL (Ours)
Table 1: Basclines and our approach. First three rows (TPL, VPL and MPL) use only the primary network and is trained using labeled data only. The baselines in the middle three rows (TPL", VPL" and MPL") are trained on both labeled and unlabeled data but still use only the primary network. TPL and TPL" learns only text prompts whereas VPL and VPL" learns only visual prompts. Baselines MPL and MPL" learns both text and visual prompts. The next two rows (XTPL and XVPL) use both labeled and unlabeled data as well as both primary and auxiliary networks, but learn either text or visual prompts. The last row (XPL) is our proposed approach which uses both labeled and unlabeled data, learns both text and visual prompts as well as uses both primary and auxiliary networks.
auxiliary network. Specifically, we aim to learn prompls by passing only labeled data through the primary network. The first such baseline is termed as Text Prompt Learning (TPL) which learns only textual prompts following CoOp (Zhou et al.) 2022b), while the second one is Visual Prompt Learning (VPL) which learns only visual prompts. After these two uni-modal baselines we go to the next baseline Multi-modal Prompt Learning (MPL) which jointly learns both textual and visual prompts. Note that TPL, VPI, and MPL (first three rows in Table 1 operate on labeled data only and at the same time, uses only the primary network. We now leverage unlabeled data in baselines TPL", VPL", and MPL" (middle three rows in Table I which uses unlabeled data along with the labeled data and employ two different augmentations to the unlabeled data. However, in absence of the auxiliary network the pscudo-label for the strongly augmented image comes from its weakly augmented view in same network i.e., the primary network. The next two baselines XTPL and XVPL (two ienultimate rows in Table [ use the cross-model architecture i.e., both primary and auxiliary networks but learn only text prompts or only visual prompts respectively. Note that XTPL and XVPL uses both labeled and unlabeled data in doing so. Finally, the proposod approach XPL explores the use of all three aspects i.e., unlabeled data, prompts from both modalities and the cross-model architecture. We show selected baselines in the main paper while compare with the rest in the appendix.
Implementation Details. We randomly sample 1%, 5%, and 10% of labeled data from cach class and consider the rest as unlabeled, following (Sohn et al. 2020). For few-shot evaluation, we follow CoOp (Zhou et al. 2022b) to obtain the splits. For the primary network, the number of learnable tokens for the text and visual prompts is set to 16, while in the auxiliary network, it set to 8. We set the hyperparameters X = 1, == 7, and p == 0.7. We choose the hyperparameter values after a small-scale hyperparameter sweep which are detailed in Section 4.3 We train using batch size of either 32 or 64 depending on the backbone. We run all experiments for 250 epochs over three random seeds and report the mean values. We use 4 NVIDIA Tesla V100 GPUs to conduct all our experiments.
4.2 Main Results and Comparisons
Figure and show the performance comparison of XPL with the basclines using VIT-B/16 backbone. In the subsequent paragraphs, we present a summary of the experimental results and key findings that motivated the development of the proposed framework.
Multimodal Prompt Learning. First, we discuss the superiority of multi-modal prompt lcarning in extracting rich information from both text and images, highlighting its advantages over unimodal approaches. As can be seen in Figure Ba] MPL outperforms TPL and VPL consistently on average e.g., with 10% of the data of a single label per class (1-shot), MPL outperforms TPL and VPL by 1.3% and 8. 8.2%, respectively (ref. Appendix A. This finding corroborates that adaptation of both the text and image encoder is more effective than adapting single encoder.
labeled the improvements are 3.1% and 6.1%. In the extreme case
7

---

Published in Transactions on Machine Learning Research (06/2024)
100
Accuunacy
labeled
Proportion
Proportion labeled
Proportion labeled data
(a) Avorage
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
Accure
Proportion of labeled data
Proportion of labeled data
Proportion labaled data
Proportion of labeled dats
(e) CropDiaeases
(f) Weeds
(g) Kaokore
(h) DTD
Accuretty
80
16
Propartion labeled data
Proportion
Proportion labeled data (k) UCF-101
Proportion data
(i) USPS
(i) Clevr-Count
(l) Caltech101
Accuracy
AcCursscy
Accurno
Proportion labeled
Proportion labeled data
Proportion labeled
Proportion laheled
(m) OxfordPets
StandfordCars
(a) Flowers102
(p) ImageNet
Figure 3: Performance of XPL on 15 datasets with VIT-B (TTBB/16 16 using only small percentage of labeled training data. XPL leverages on the unlabeled data the most and boasts the performance across all scenarios.
100
100
90
MPLA
90
MPLP
80
80
Accuracy
70
Accuracy
70
60
60
50
50
40
40
30
30
20
20
10
10
Average EunSAr
ISC
Chest Xray
Coppiseanes
DespHeeds
Kaokore
DTD
usps
Gour.count
JCr-101
101tech101
OxfordPets
CandfordCara
Rowerss02
imageNet
Average
EurSSAT
Chest-Xray
IiC
CopDiseaser
Weeds
Konkore
DTD
USps
Cleur-Count Count
UCF-101
Coltech101
OrfordPets
StandfordCCa
Flowers102 122
AmageMot
(a) 2-shot
(b) 1-shot
Figure Few-shot performance of XPL on datasets with ViT-B/16. XPL leverages on the
unlabeled data the most and boosts the performance across all scenarios.

---

Published in Transactions on Machine Learning Research (06/2024)
H
H
TFL
T4.43
40.53
49.17
37L
92. 12.19
64.74
65:09
TPL
74.30
19. 20
96.51
TEL
19:00
24.41
21.37
7PL
74.76
41.43
60.13
172
93. 90
64.88
60. 00
VEL
74.52
20.89
32.01
77L
22.40
24.48
23.30
HPG
77.28
44.07
64.47
30L
53. 49
65-12
69. 45
RPL
74.90
28 91
39.50
28.78
26.32
20. 94
80.79
53.32
06
1PC
80
58 90
73.52
XPL
78 40
80. 80
79.58
32. 70
33. 00
32.
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest- -tray
89-10
19:00
31:31
TPL
76,
13.80 80
23.33
TPL
40: 70
36.54
78.44
54:24
VPL.
87.60
17:43
07
1PL
80.
14.76
WLL
40: 76
36.28
79.00
10
00.2 2
19:92
32.80
1PL
61.8
18:41
HPL
40
36.90
78-70
42. 00-
6.3.80
20.
(e) CropDiseases
(I) DeepWeeds Weeds
(g) Kaokore
(h) DTD
88:00
36.
00:15
222
21:80
50
TPL
04.
90:00
98:00
47.30
68.25 28
22.50
67.70 77
70.72
(i) USPS
(i) Clevr-Count
(k) UCF-101
(1) Caltech101
H
98 00.60
70.15
0G
06.00
67.50
22.32
NPL
04.
80
70.90
00:30
69 21
07.54
63.20
76.
XPL.
87
58
73.44
74.59
71.82
78. 18
00.87
(m)) OxfordPets
(n) StandfordCars
(0) Flowers102
Table 2: Comparison of XPL with TPL, VPL and MPL in generalization from base to new classes. For a given datasct, the text and visual prompts are learned using a subset of classes (S) and evaluated on the rest of the classes (U). The results portray the strong generalizibility of our cross-model prompt learning approach. H refers to the Harmonic mean.
Leveraging Unlabeled Data. Here, we demonstrate the sub-optimalitty of disregarding unlabeled data in MPL, which can lead to loss of valuable knowledge. With unlabeled data, MPL" achieves a significant gain over MPL specifically in the low-labeled data regime. E.g., 3.5% average improvement in 1-shot scenario can be seen in Figure 4b] It also significantly helps in challenging datasets like EuroSAT and CropDiseases, e.g., performance improves by 11.9% and 17.9% respectively in 1-shot scenario, as can be seen in the same Figure.
Cross-Model Design. We now showcase the effectiveness of our cross-model design, which harnesses complementary knowledge from both models. As can be seen in Figures 3 and 4 XPL outperforms all the baselines in all the settings showing the effectiveness of the cross-model design. E.g., in Figure 4b] XPL provides 2.9% average improvement over the strongest baseline MPL" in 1-shot scenario. Moreover, XPL offers significant jump of 5% for the fine-grained WeepWeeds dataset in the same 1-shot sctup validating the importance of harnessing complementary knowledge (hrough our unique design.
Robustness to Domain Shift in Unlabeled Data. Adapting models to downstream data often overfits to that specific task and fails to generalize towards domain shifts. This behavior is specifically common in low-labeled data regime. For domain D with a given amount of labeled ((Di)) and unlabeled data (Pru)), we define mixture fraction 17 which signifies that 17 fraction of the unlabeled data (17 x (Pul) comes from a different domain D while (1 17) fraction of it ((1 - 1) x (P,,)) comes from the same domain D. We consider two scenarios: when all the unlabeled data belong to D (n 0), and when they belong to D (1 === 1)) Table 3 shows the classification accuracy on D with 10% labeled training data from the same domain. We compare with the strongest baseline MPL" on three pairs of domains from the Office-31 (Saenko et al., 2010)) and three pairs of domains from the DomainNet (Peng et al.) 2019) dataset. As can be observed, XPL consistently outperforms MPL" irrespective of the domain shift for both the datasets. F.g., for D=qdr and Đ=skt,, we compare the performance of the no domain shiftt scenario (n = 0) with that of the marimum domain shift (n - 1), "P accuracy drops by 2% (31.9% vs 29.9%) while XPL shows a drop of mere 0.7% (35.2% vs 34.5%) while outperforming "PL" by 4.6%. This corroborates the robustness of XPL towards ofitt--distti data.
Generalization from Seen to Unseen Classes. In Table 2 for a given datasct, we train on a subset of classes (seen) and show generalization performance to the rest of the classes (unseen). We compare XPL with TPL, VPL and MPL for accuracy on the seen classes (S) and the unseen classes (U) and their harmonic mcan (H) which highlights the generalization tradc-off (Xian et al., 2017). XPL consistently outperforms MPL on unseen classes across all the datasets, e.g. an improvement of 3.78% (55.12% vs 58.90%) and 7.68% (25.32% 33.00%) on EuroSAT and Chest-Xray datasets respectively. Even on the average performance
9

---

Published in Transactions on Machine Learning Research (06/2024)
Office311
DomainNet
Method
(D=A, D=W)
(D=W, D=D]
(D=D, D=A)
(D=rel, D=pnt)
(D=clp, D=inf)
(D=qdr, D=skt)
(10% labeled data)
n=0 0
n=
== 0
==
== 0
1)
==
0
n=
y= 0
8=
n=
1
MPL."
82.8
81.7
86.4
85.2
84,2
81.9
78.0
77.7
67.4
67.0
31,9
29.9
XPL (Ous)
84.7
$1.0
88.2
87.1
85.5
84.6
79.1
78.6
68.0
67.9
35,2
34.5
Table 3: Performance under domain shift in Office-31 and DomainNet. Numbers show the accuracy on test partition of domain D when the models are trained with 10% labeled data from D and two different. proportions of unlabeled data (17) between D and D. XPL achieves the best performance even this challenging scenario for both the datasets.
Accurncy
MDCS
Maco
Proportion labaled
Propertion labaled data
Proportion labeled data
Propartion labeted data
Proportion labeled data
EuroSAT
(b) ISIC
(0) Chest-Xray
(d) CropDiseases
Average
Figure 6: Comparison with self-supervised baselines. Plots show the performance comparison of XPL with PL which uses vanilla pseudo-label training and MoCo which uses momentum encoder for self-supervision. XPL consistently outperforms both PL and MoCo across all the 4 datasets.
across all datascts, XPL surpasses MPL by a significant margin of 7.59 (62.06% vs 54.47%). Superior harmonic mean across the datasets substantiates that learning multi-modal prompts with complementary knowledge harnessed from the cross-model architecture helps improve the generalization to unseen classes.
Different VLM Backbones. We show the gencralization
of XPL by replacing the VLM backbones and architect In Figure [ average accuracies all datasets excluding
Accuracy
Accuracy
ImageNet using CLIP VIT-B/32 (Radford et al. 2021 and DeCLIP ViT-B/32 (Li et al. 2021) backbones are reported. XPL consistently outperforms the baselines and obtains state- of-the-art performance for both models. E.g., XPL outper- forms MPL" by 1.9% (53.9% vs 52.0%) when 1% labeled data is used. This shows the effectiveness of XPL in harnessing from stronger backbones
NFL
Proportion labeled data
Proportion labeled
(a) CLIP VIT-B/32
(b) DeCLIP VIT-B/32
complementary information like DeCLIP which has already been trained with extensive self-supervision.
Figure 5: Performance with different VLM backbones. Plots show average accuracy using CLIP ViT-B/32 and DeCLIP ViT-B/32. XPL out- performs all baselines and obtains the best.
Comparison with self-supervised basclines. In order to assess the effectiveness of the cross-model strategy, in Figure we compare XPL with two self-supervised baselines namely, and MoCo. In PL we have single model and perform vanilla pseudo-label training (Lee 2013) on the unlabeled data in addition employ the self-supervision strategy of (He et al. 2020) using momentum encoder performance of both MoCo and PL fails to reach that of XPL across the 4 datasets, e.g., on average PL and MoCo shows 3.5% and 2.8% lower accuracy than XPL respectively for 1% labeled data (70.5% vs 71.2% vs 74.0%). This signifies the importance of the cross-model strategy in alleviating noisy and incorrect pseudo-labels by leveraging complementary information from both the networks.
the supervised loss on the labeled data. Similarly, in MoCo, we
single model. The
4.3 Ablation Studies
We perform ablation studies on 4 diverse datasets (unless otherwise specified) namely EuroSAT (Helber et al.) 2019), ISIC (Codella al.) 2019), Chest.X (Wang et al. 2017) and CropDiscases (Mohanty et al.) (2016) test the effectiveness of different components of XPL. For each ablation study we report the performance on the individual dataset well the average performance across these 4 datasets.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Different Prompt Lengths. In the main experiments we learn prompts of length 16 and 8 for the primary and the auxiliary network respectively. Figure 7 shows the performance using prompts of lengths 8 and 4 respectively (XPL (8, 4)), on 4 datasets. As expected, using shorter prompt lengths drops the performance since the number of learnable parameters decreases. E.g., on average, the accuracy drops by 3.4% (70.6% vs 74.0%) when we have 1% of labeled data. We also ran an experiment to see if using same number of prompts two paths can harness the complimentary information as well. In two different variations of this, we used 16 prompts (XPL (16, 16)) and prompts (XPL (8, 8)) in both primary and auxiliary paths. As seen, compared to the proposed approach XPL (16,8), the performance diminishes in both XPL (16, 16) and XPL (8, 8) showing the utility of using different prompt lengths in primary and auxiliary models. Lastly, we tried to see if increasing the ratio of the number of prompt vectors in the two paths helps more. As seen, if we use 32 and 8 prompls in the two paths (XPL (32.8) the performance diminishes which is possibly due to a large mismatch in the capacities of the two paths.
Accuracy
ACCurnTT
Proportion labeled data
Propertion labeled data
Propoction of labeled data
Propartion labeted data
Proportion of labeled data
(a) EuroSAT
(b) ISIC
(c) Chest-Xray
(d) CropDiscuses
(e) Average
Figure 7: Different Prompt Lengths. Plots show the accuracy curves for XPL using different prompt lengths. XPL (M, N) learns prompts of length M and N for the primary and auxiliary network.
Varying Prompt Positions. In this ablation study, we observe the effect changing the relative positions of the class token [CLS] and the prompt vectors, as an additional attribute instead of the length of the learnable prompts. In XPL, the [CLS] token was placed at the 'end' of the learnable prompt vectors for both primary and auxiliary branches. Here, consider two setups with the same prompt. lengths of 16 for both the branches: (1) [CLS] token is positioned at the end i.e., after the prompt vectors in the primary branch, while at the beginning in the auxiliary ('beg', 'end'); (2) [CLS] token is positioned at the middle of the prompt vectors in the primary branch, while at the end in the auxiliary ('mid', 'end'). As can be observed over all the datasets across 2 different proportions of labeled data (1% and 5%), changing the class token positions does not distinctively affect the performance of our approach. Rather we have seen that the lengths of the prompts play more significant role in the cross-model approach. The use of different. prompt lengths harnesses the most complementary information and provides the best performance.
Accuraty
Proportian of labeled data
Proportion labeled data
Proportion of labeled data
Propartion labeled data
Proportion of labeled data
(8) EuroSAT
(b) ISIC
(c) Cheat-Xray
(d) CropDieeases
(e) Average
Figure Varying the position prompts the two branches. Plots show the accuracies for XPL after appending the class tokens, [CLS), at different positions of of the learnable prompt vectors for both primary and auxiliary branches. XPL (M, N) (posl, pos2) learns prompts of length M and with the [CLS] token appended in the posl and pos2 positions for the primary and auxiliary network respectively. Here beg, mid and end refers to pulting the [CLS] loken in the beginning, mide
Effect of Hyperparameters. In this Figure shows the average performance over datasets, EuroSAT, ISIC, Chest- Xray and Cropdiseases. First, we vary X (ref. Eq. 9) to 0.5, 1.0 and 2.0 in XPL and obtain the best performance when all the losses are equally weighed (i.e. X = 1.0) Figure 10: Effect of Hyperparameters. Plots analyze the average and is used in our experiments. The ratio performance across 4 datasets by varying hyperparameters X, and
Accuracy
2.00
2.57
Propartion
Proportion
Propartion
(a)
(b) 1
(c) P
11

---

Published in Transactions on Machine Learning Research (06/2024)
Proportion
Propartion
Proportian
Proportion
Propartion
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
(e) CropDiseases
Proportion of labeled data
Proportion of labeled data
Proportion of labeled date
Proportion of labeled data
Proportion of labelled data
(f) DropWoods
(x) Kaokore
(h) DTD
(i) USPS
(i)
Clevr-Count
Accuray
Proportion labeled
Proportion labeled
labeled
labeled
labeled
(k) UCT-101
Calterh101
(m) OxfordPets
(n) StandfordCars
(o) Plowers102
Figure 9: Effect of coupling function F(-). Accuracy across 14 datasets shows that mutual collaboration between the text and visual prompts through F() is necessary for improved performance.
of unlabeled to labeled data j (ref. Eq. is important in deciding the performance. We vary μu to 5, 7, and 9. The performance increases with higher values of μ,, however, scaling up μ often requires high computational resources. We observe negligible improvement beyond μ = 7 and hence use that for XPL. We also vary the to 0.6, 0.7, and 0.95. We obtain the best performance at p 0.7 and
pseudo-label threshold p (ref. Eq. 6 7 use it for all the experiments.
Effect of coupling function F(.) As shown in Figure 2 we use a coupling function F(.) to ensure mutual collaboration between the text and visual prompts (hence the encoders). In order to study its effect, we remove F()) and independently learn the text and visual prompts for XPL, MPL", MPL, resulting XPL-I, MPL"-I, and MPL-I respectively. We show the individual performances of these baselines in all 14 the datasets along with the average performance for 1%, 5% and 10% proportions of labeled data in Figure g and 2-shot and 1-shot performances in Appendix (Figure [5). As can be observed in both the settings, removing F(.) decreases the average performance, e.g., 5.8% (72.4% vs 66.6%) for XPL with 1% labeled data and hence ensuring that mutual coherence between the text and visual prompts is crucial for better performance.
methods
5 Conclusion
We present XPL, a novel cross-model framework for multi-modal, semi-supervised prompt learning towards parameter-cfffiicnt adaptation of large pretrained VLMs to diffcrent downstream tasks. We identify that directly using the same adaplation model produce confident pseudo-labels for the unlabeled data may miss crucial category-wise information. novel cross-model semi-supervision is pionecred leverage the complimentary knowledge learned by models with different length prompts significantly improving the performance. We demonstrate the effectiveness of our proposed approach on fourteen benchmark datasets, outperforming several competing mothods. Our research can help reduce burden of collecting large-scale supervised data in many real-world vision applications by transferring knowledge from large pretrained VLMs. Linitations of our research are difficult to predict, however, using more data, albeit unlabeled may mean more computation, but this comes with a lot of savings in human annotation efforts for a similar performance gain in fully supervisod setup.
12

---

Published in Transactions on Machine Learning Research (06/2024)
References
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- Supervised Learning by Cross Audio-Video Clustering. Advances in Newral Information Processing Systems, (199758-977)) 2020.
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-Taiie Labeliig and Confirmation Bias in Deep Semi-Supervised Learning. In International Joint Conference on Neural Networks (IJCNN), pp. 1-8, IEEE, 2020.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with Pseudo-Ensembles. In Newrol Information Processing Systems, pp. 2365-3373, 2014.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring. In International Conference on Learning Represenlations, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A Holistic Approach to Semi-Supervised Learning. In Neural Information Processing Systems, pp. 5050 5060, 2019b.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Superrised Learning (chapelle, et al., eds.; 2006) Book Reviews]. Transaclions on Neural Nelworks, 20(3):542-5444 2009.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp.
104-120, 2020b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606 3613, 2014.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC).
arXiv preprinl vrxxi:1902. 03368, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical Automated Data Augmentation with Reduced Search Space. In Proceedings of the IEEE/CVF conference. on computer vision and pattern recognition workshops, pp. 702 703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pullern recognition, pp. 248 255,, 2009.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative Unsuper- vised Feature Learning with Convolutional Neural Networks. In Neural Information Processing Systems, pp. 766 774, 2014.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXie preprint arXiv:2010.1 11929, 2020.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and paltern recognition workshop, pp. 178-178, 2004.
13

---

Enrico Fini, Pictro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, and Michal Drozdzal. Improved baselines for vision-language pre-training. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http://ppnreeiieee.eeeerrrrraaaa Published in Transactions on Machine Learning Research (06/2024)
Featured Certification.
Andreas Fürst, Elisabeth Rumetshofer, Viet Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, David Kreil, Michael Kopp, Günter Klambauer, Augela Bitto-Nemling, et al. Cloob: Modern hopfield networks with infoloob outperform clip. arXiv preprint arXiv:2110.110 111161,101 2021.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-Adapter: Better Vasion-Language Models with Feature Adapters. arXiv preprint or 10. 00.0444.. 2021.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, and Chunhua Shen. PyramidCLIP; Hierarchical Feature Alignment for Vision-language Model Pretraining. arXio preprint or arxin:224.. 14095, 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. arXiv preprint arXiv:1803.00 077.7778 2018.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyelic Contrastive Language-Image Pretraining. Xixi preprinl 3rXiv:2205.1442 14459, 2022.
Yves Grandvalet and Yoshua Bengio. Semi-Supervised Learning by Entropy Minimization. Neural Information Processing Systems, pp. 529-536, 2005.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirk-Kikkatrick, and Graham Neubig. Towards Unified View of Parameter-Efffcient Transfer Learning. In International Conference Learning Representations, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised Prompt Learning for Vision- Language Models. arXiv preprint arXiv:2204.000. 03649, 2022.
Jonathan J. Hull. A Database for Handwritten Text Recognition Research. IEEE Transactions on pottern analysis and machine intelligence, 16(5):550 554, 1994.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text. supervision. In International Conference on Machine Learning, pp. 4904 4916, 2021.
Chaoya Jiang, Wei Ye, Haiyang Xu, Songfang Huang, Fei Huang, and Shikun Zhang. Vision Language Pre-training by Contrastive Learning with Cross- Similarity Regulation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 14660 14679, Toronto, Canada, July 2023. Association for Computational Linguistics.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Dong-Hyun Lec. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural
networks. In International Conference on Machine Learning Workshop, volume 3, pp. 2, 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effcient prompt tuning. arXiv preprint orXic:2104.0800 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. orXiv preprint ar Xrv: 03557, 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137, 2020.
Vangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Iman Image Pre-training Paradigm. arxiv preprint Xiv:22110. 10.05208, 2021.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt Distribution Learning. In Proceedings of the IEER/CVF Conference on Computer Vision and Pottern Recognition, pp. 5206-522 5215,, 2022.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE trunsactions on pallern analysis and machine intelligence, 41(8):1979 19999,9999--11 2018.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using Deep Learning for Imago-based Plant Disease Detection. Frontiers in plant science, 7:1419, 2016.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprinl Xix:v:21 12. 12750, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number classes. In 2008 Sizth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722 729, 2008.
Alex Olsen, Dmitry A Konovalov, Bronson Philippa, Peter Ridd, Jake C Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, et al. WeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning. Scientific reports, 9(1):2058, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 3505, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Momont Matching for Multi-source Domain Adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406 1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748 80688873, 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting Visual Category Models to New Domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Manli Shu, Weili Nic, Do-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. arXiv preprint ar 07X: 2022.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Semi-supervised action recognition with temporal contrastive learning. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10389- 10399, 2021.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-LLang Li. Fixmatch: Simplifying Semi-supervised Learning with Consistency and Confidence. Advances in neural information processing systems, 33:596-660 608, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:
2012.
Alessandro Sordoni, Eric Yuan, Marc- Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint Prompt Optimization Stacked Lims Using Variational Inference. Advances in Neural Information Processing Systems, 36, 2024.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXic:1908.0885 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers.
arXiv preprinl xiv:: 199::
908. 008.07490, 2019.
Yingtao Tian, Chikahiko Suzuki, Tarin Clanuwat, Mikel Bober-Irizar, Alex Lamb, and Asanobu Kitamoto. Kaokore: A Pre-modern Japanese Art Facial Expression Datasot. ar preprint N5a:2002208595 2020.
Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-Guided Text Prompt for Vision- Language Pro Pra-ingg In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
11942-23255) June 2023.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-superrised Classification and Localization Common Thorax Diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data Efficient Language- supervised Zero-shot Recognition with Optimal Transport Distillation. arXie preprint ar XrXiv:2112 02.09445. 2021.
Yongqin Xian, Bernt Schicle, and Zeynep Akata. Zero-shot Learning-the Good, the Bad and the Ugly. In Proceedings of the IEEE conference on compuler vision and pollern recognilion, pp. 4582 -5511 2017.
Yinghao Xu, Fnngyun Wei, Xiao Sun, Ceyuan Yang, Yujun Shen, Bo Dai, Bolei Zhou, and Stephen Lin. Cross-Model Pseudo-labeling for Semi-Supervised Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2959 2968, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. orXiv preprint a1Xiv:2111.07783, 2021.
Renrui Zhang, Rongvao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-Adapter: Training-free free Alip-Adapter for Better Vision-Language Modeling. arXiv preprint arXiv:2111.032 0333300000 2021.
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual Probing is [MASK]; Learning vs. Learning to Recall. arXiv preprint 25f::2104.052.4 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022a.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to Prompt for Vision-language Models. International Journal of Computer Vision, (000))))337-245 2022b.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large Language Models are Human-level Prompt Engineers. arXiv preprint arXiv:221 1. 1.01910, 2022c.
17

---

Published in Transactions on Machine Learning Research (06/2024)
A
Appendix
Here we provide additional experiments and visualizations on the dalasets to further explore the XPL approach. These are summarized in the following Table 4
Section
Content
A.1
Leveraging Unlabeled Data for Uni-modal basclines
A.
XPL in Uni-modal setting
A.3
Effect of Coupling Function in Fewshot Setting F(.) Different VLM Backbones
A.4
A.5
Qualitalive Results
A.6
t-SNE Visualizations
Table 4: Overview of Appendix.
Code. Please refer to XPL code.zip in the appendix material for our code submission. We will make the code public.
A.1 Leveraging Unlabeled Data for Uni-modal baselines
In this section of the appendix, demonstrate the sub-optimality of disregarding unlabeled data for the uni-modal basclines TPL and VPL in similar manner shown for the multi-modal bascline in scction 4.2 of the main paper. Both TPL" and VPL" obtains a significant gain in performance as can be seen in Figure [1 On average, TPL" helps to perform better by 3% than TPL, whereas, VPL" shows 2% gain in accuracy over VPL, when using only 1% labeled data. Similar trend also observed for 2-shot and 1-shot scenarios as shown in Figure XPL retains the supremacy across all the baselines in both few-shot setting and well as in low percentages of labeled data.
Proportion lsbeled data
Proportion laboled data
Proportion Inbeled data
Proportion Inbeled data
Propartion labeled data
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
(e) CropDiseases
Accuracyy
Proportion labeled data
Propartion febeled data
Propartian labeled
Proportion labeled
Propartion labeted data
(f) DeepWoeds
Kaokore
(b) DTD
USPS
Clevr-Count
Accuraty
Proportion
Proportion data
Propertion labeled data
Proportion of Inbeled data
Propartion of Inbsted data
(k) UCF-101
()) Caltorh101
(m) OxfordPets
(n) StandfordCars
(a) Plowers102
Figure 11: Performance of TPL, TPL", VPL, VPL and XPL on 14 datasets with ViT-B/16 using only small percentage of labeled training data. The uni-modal baselines TPL and VPL" leverage the uulabeled data to obtain performance gain over TPL and VPL respectively across scenarios. XPL leverages on unlabeled data the most and obtains maximum boost in the performance.
18

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
90
90
80
80
Accuracy
70
Accuracy
70
60
60
50
50
40
40
30
30
20
20
10
10
Average
EuroSAT
ISIC
Chestoray
CropDiseanes
Despheeds
Kaokare
DTD
uSPs
Beur-Count
UCF-101
Caltech10
OxforePets
SandfordCars
Flowers10
imageNes
Average
EurosAr
Chest-krap
GropDiseaees
DeepWeedds
Kaolore
DTD
USPS
Clev--oont
UCT-101
Caltec1
OefordPats
CandfordCaars
Rowas102
imageNee
(a) 2-shot
(b) 1-shot
Figure 12: Few-shot performance of TPL, TPL", VPL, VPL" and XPL on 15 datasets with VIT-B/16. Even
in 2-shot and 1-shot scenarios the uni-modal baselines TPL" and VPL leverage on the unlabeled data to obtain performance gain over TPL and VPL respectively across scenarios. XPL leverages on the unlabeled data the most the few-shot setting to give the highest performance.
A.2 XPL in Uni-modal Setting
Here, we showcase the importance of multi- prompt learning to extract richer information from both text and images compared to the unimodal approaches. As can be seen in Figure [3] consider two baselines XTPL and XVPL, having only text prompts and only visual prompts respectively the two uni-modal variants of XPL. In both low proportions labeled data (Figure 13) and few-shot settings (Figure [4], XPL obtains the most hike in accuracy over both XTPL and XVPL. Even for challenging datasets like DeepWeeds (Olsen et al.) 2019) (refer Figure 13fg and Clevr-Count (Johnson et al. 2017) (refer Figure 13j), XPL shows the supremacy in performance by almost 10% and 35% gains respectively when using only 5% labeled data.
A.3 Effect of coupling function F(.) in fewshot setting
Here we explore the effects of using the coupling function F(-) on the individual performances of all 14 the datasets the fewshot scenarios. We compare the performances of XPL, MPL", MPL with XPL-I, MPL"-I, and MPL-I 2-shot and 1-shot scttings respectivcly in Figure 15 As observed, following similar trend in Figure of the main paper, using F()) boosts the average performance across all the datasets.
A.4 Different VLM Backbones
We have shown the supremacy of XPL with other VLM architectures, CLIP ViT-B/32 (Radfrdd et. al., 2021 and DeCLIP ViT-B/32 (Li et al. 2021) in Figure 5 of the main paper. Here, we illustrate those plots providing the variation in performance across the individual 14 datascts with low proportions of training data for CLIP VIT-B/32 in Figure 16 and DeCLIP ViT-B/32 in Figure 18 The average plots from the main paper (refer Figure 5) have also been included in Figures 16a and 18a for CLIP ViT-B/32 and DeCLIP ViT-B/32 respectively, for reference. We explore the performances with the two VLM backbones under few-show setting
as well and plot the accuracies in Figures 17 for CLIP ViT-B/32 and 19 DeCLIP ViT-B/32 respectively.
A.5 Qualitative Results
Figure 20 shows the qualitative examples for comparing the performance of XPL with the baselines of TPL, VPL, MPL and also the next-best MPL". As can be seen, XPL proves its supremacy in identifying diverse image samples such as different landscapes EuroSAT (Helber al. (019) (Figure (0a)). flower types Flowers102 (Nilsback & Zisserman 2008) (Figure 20b) and also animals in OxfordPets (Parkhi et al. 2012
(Figure 20c
19

---

Published in Transactions on Machine Learning Research (06/2024)
AccuPmE
Proportion labeled data
Proportion labeled data
Propartian
Propartion
Propartion data
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
(e) CropDiseases
Accuracy
Proportion labeted data
Proportion lebeled data
Proportion Tebeled
Proportion lebeled
Proportion Imbeted data
(f) WeepWords
Kaokore
(h) DTD
USPS
Clevr-Count
Accuracy
Un
Un
Proportion
Proportion
Proportion laboled
Proportion of data
Proportion
(k) UCF-101
()) Caltech101
(m) OxfordPets
(n) StandfordCars
(o) Flowers102
Figure 13: Performance of XTPL, XVPL and on 14 datasets with ViT-B/16 using only small percentage of labeled training data. XPL obtains higher performance gain over XTPL and XVPL respectively across all scenarios. The adaptation of both the text and image encoder in XPL is more effective than adapting single encoder as XTPL and XVPL.
100
100
90
90
XVPL
XVPL
80
80
Accuracy
70
Accuracy
70
60
60
50
50
40
40
30
30
20
20
10
10
Average
EUrSAT
Chest Xhayt-raay
BNC
CopDiseaees
DeepWeads
raokore
CTD
usPs
Coant
UCT-101
Caltech101
OrforePPets
CandfordCars
fooersl02
Average
EuroSAT
Fic
Chest Xray
CopDiseates
Weeds
Kaolare
DTD
USPS
Cluur.Count
UCr-101
1
OrfordPets
ScandforeCars
Flowers22
(a) 2-shot
(b) 1-shot
Figure 14: Few-shot performance of XTPL, XVPL and XPL on 14 datasets with VIT-B/16. XPL obtains higher performance gain over XTPL and XVPL respectively both 2-shot and 1-shot setting across datasets. The adaptation of both the text and image encoder in XPL is more effective than adapting a single encoder as in XTPL and XVPL.
20

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
90
MCN
90
80
80
Accuracy
70
Accuracy
70
50
50
40
40
30
30
20
20
10
10
Average
BuroSAT
EBI
Chest KrestXray
CropDiseanes
DeepWieeds
Faotore
DTD
USPS
Count
UCF-101
Caltech101
OrfordFate
SandfordCars
Roowrss02
Average
EuroSAT
ISC
Chest-ray
GropDiseattes
DespWeeds
Kanfore
LTD
USps
Cleve-ounnt
UCT-301
Caltechio:
OrforsPett
SaandforiCann
Flowers 102
(a) 2-shot
(b) I-shot
Figure 15: Effect of coupling function F(.). Fewshot performance across 14 datasets also shows mutual
collaboration through F(.) is necessary for performance gain.
Proportion of inbeled data
Proportion of lnbeled data
Proportion of inbeled data
Proportion Inbeled data
Proportion inbeled data
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
(c) CropDiscases
Accurary
HFL
MPL
lnboled data
Inbeled
Proportion laboled
Proportion labeled
Proportion labeted
(f) Doep Woeds
(x)
(h)
USPS
Clevr-Count
Accuraty
M01
Proportion labeled date
Proportion labeled data
Proportion of labeled data
Proportion of inbeled data
Proportion of labeled data
(k) UCF-101
(I) 1altoch101
(m) OxfordPets
(n) StandfordCars
(a) Flowers102
Figure 16: Performance using CLIP VIT-B/32, Plots show accuracy across datasets using CL.P ViT-B/32.
XPL outperforms all the baselines for each dataset and obtains the best performance.
21

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
XPL
XP.
90
90
MPL
90
80
TPL
Accuracy
70
Accuracy
70
50
60
50
50
40
40
30
30
20
20
10
10
Avarage
BuroSAr
Chest-trap
CopDlsnates
Heepss
Kaakora
DTD
USPS
Ceun-
UCF-101
caltech10s
OvfordPets
SandfordCans
Flowersi02
Average
EuroSAr
IBC
Chest Chestxtray
CooDlseares
Deepleeds
Kaokore
DTD
USPS
Cewr-Count
UCF UCF-101 -01
Caltechi 011
OrfordPets
StantfordCCan
Flowerstdg 02
(a) 2-shot
(b) 1-shot
Figure 17: Few-shot performance using CLIP ViT-B/32. Plots show accuracy across 14 datasets. XPL
outperforms all the baselines for each dataset and obtains the best performance.
Rccurnry
Proportion of inbeled data
Proportion of labeled data
Proportion of lnbeled data
Proportion Inbelee data
Proportion lnbeled date
(a) Average
(b) EuroSAT
(c) ISIC
(d) Chest-Xray
(c) CropDiscases
ACQURERY
30
0%
10.
Proportion lnboled data
Inbeled
Proportion labeled
Proportion labeted
(f) Deeps
(g) Kaokore
(l) DTD
USPS
Clevr-Count
Accuraty
VEL
Proportion labeled data
Proportion laboled data
Proportion laboled data
Proportion of Inboled data
Proportion of Iabeled data
(k) UCF-101
()) Caltoch101
(m) OxfordPets
(n) StandfordCars
(a) Flowers102
Figure 18: Performance using DeCLIP VIT-B/32. Plots show accuracy across datasets using CLIP VTT-BB32.
XPL outperforms all the baselines for each dataset and obtains the best performance.
22

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
XPL
XPL
90
90
80
80
TPL
Accuracy
70
Accuracy
70
60
60
50
50
40
40
30
30
20
20
10
10
Average
BuroSAT
Chest-tray
Copplseaees
WeepWesds
Kaskore
DTD
USPs
Count
UCF-101
caltech10:
Oviordnets
StandforCCars
Flawerst02
Average
EuroAAr
Chest Chest-tray
BSC
CroDlseeate
DeepWesdd
Kookare
DTD
USPS
CewrCount
MCF-101
caltech 111
OrfoniPets
StanifordCCas
Finwerst02
(a) 2-shot
(b) 1-shot
Figure 19: Few-shot performance using DeCLIP ViT-B/32. Plots show accuracy across 14 datasets. XPL
outperforms all the baselines for each dataset and obtains the best performance.
A.6 t-SNE Visualizations
Figure shows the t-SNE visualizations of XPL along with the next-best baseline MPL" and also uni-modal VPL, TPL across 3 datasets of EuroSAT (Helber et al. 2019: (Figure 21a Flowers (Nilsback & Zisserman, 2008 (Figure 21b) and OxfordPets (Parkhi al.) 2012) (Figure 21ch. Inspite of diverse datasets, XPL portrays the most consistent clustering and class- wise discriminative live acoss all the 3 datasets, showing the efficacy of our cross-model approach in learning discriminative features in multi-moda setting.
23

---

Published in Transactions on Machine Learning Research (06/2024)
TPL: AnnualCrop
TPL: Highway
TPL: Residential
TPL: PermanentCrop
VPL: Highway
VPL: Industrial
VPL: Industrial
VPL: Pasture
MPL: Pasture
MPL: River
MPL: Pasture
MPL: HerbaceousVeggetttin
MPL River
MPL Pasture
MPL : Highway
MPL": Forest
XPL: SeaLake
XPL: PermanentCrop
XPL: Highway
XPL River
(a) EuroSAT
TPL: cyclamen
TPL: tree mallow
TPL: balloon flower
TPL: blackberry lily
VPL: siam tulip
VPL: passion flower
VPL: sweet william
VPL:
balloon flower
MPL: toad lily
MPL: magnolia
MPL: corn poppy
MPL: wallflower
MPLL: ": bougainvillea
MPL magnolia
MPL daffodil
MPL Ue foxglove
XPL: bougainvillea
XPL: magnolia
XPL: prince of wales feathers
XPL; trumpet creeper
(b) Flowers
TPL: beagle
TPL: Maine_ Coon
TPL: Bengal
TPL: chihuahua
VPL: miniature_ miniaatre_ppinshherre
VPL: shiba_ine
VPL: havanese
VPL: Ragdoll
MPL: chihuahua
MPL: pug
MPL: pomeranian
MPL: boxer
MPL Bombay
MPL": boxer
MPLL" Persian
MPL" Pug
XPL: Abyssinian
XPL: american bulldog XPL: Persian
XPL: Pug
(c) OxfordPets
Figure Qualitative examples comparing XPL with TPL, VPL MPL baselines. We compare performances datascts, EuroSAT (Helber et al. 2019), Flowers102 (Nilsback & Zisserman 2008) and OxfordPets (Parkhi et al. 2012) trained using 1% labeled data with CLIP VIT-B/16. The correct predictions are marked green while the incorrect predictions have been marked red.
24

---

Published in Transactions on Machine Learning Research (06/2024)
(a) XPL
b MPL
(c) VPL
(d) TPL
(a) EuroSAT
XPL XPL
(b) MPL
(c) VPL
(d) TPL
(b) Flowers102
(a) XPL
(b) MPL
(c) VPL
(d) TPL
(c) OxfordPets
Figure 21: Feature Visualization using t-SNE. Figure shows the t-SNE visualizations XPL along with different baselines of MPL", VPL and TPL diverse datasets, EuroSAT (Helber et al. 2019), Flowers102 (Nilsback & Zisserman 2008 and OxfordPets (Parkh et al. 2012) trained using 1% labeled data with CLIP ViT-B/16. XPL forms most consistent clustering and performs better classwise discrimination viewed in color.
the diverse datasets. the Best.
25


Reviews:
[{"review_id": "ec032xZz6H", "reviewer": "Reviewer_LUr8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Overall, the paper is well-written and easy to read.\n- The paper addresses and validates the concept of training on unlabeled data for prompt learning in low-labeled data regime. With the cross-model and multi-modal prompt learning, the proposed method achieves superior results in comprehensive experiments. The ablation studies also validate the design choices the authors made, which strengthens quality of the paper.\n\n\nWeaknesses\n- What are the main contributions the authors claim? Unsupervised training for prompt learning (MPL vs MPL^u), cross-model prompt learning with different number of prompt vectors (MPL^u vs XPL), multi-modal prompt learning (TPL/VPL vs MPL), or all of them? Which are the ones that this paper first try or there are previous works exploring similar approaches to? Please make this clear.\n- All the models used in the experiments are based on the same architecture and training objectives with the addition/subtraction of some components. Regarding the above point, if there are previous works that explored similar approaches, the authors should compare with them.\n- While the authors provide extensive experiment results to validate the proposed method, there are still remaining design choices not addressed. For example, does XPL use any constraints to stabilize cross-model representation learning, such as stop gradients as in BYOL [1]? Why did the authors use hard (one-hot) pseudo-labels for unsupervised training? Why did the authors use only weak augmentation for supervised training? And does scaling up model capacity (size) affect the generalization performance of XPL? (Figure 5 compared the same size models with different training recipes). The authors also did not explicitly show the effectiveness of learning prompts invariant to different views of the same image, which needs comparison with the cross-model training, but using weak augmentation only.\n- Some experimental details are missing, e.g., how the authors split the classes of each dataset for evaluating generation from seen to unseen classes (Table 2), and how they sampled 1%/5%/10% of labeled data.\n- MoCo is a self-supervised baseline for learning visual (that is, image-only) representation while XPL uses cross-modal contrastive losses, so the comparison in figure 6 seems to be not fair to me. Please elaborate on how the authors applied MoCo for the experiment.\n\nMinor questions\n- Appendix A.1 does not include comparison between MPL and TPL/VPL.\n\n[1] Jean-Bastien Grill et al., Bootstrap your own latent: A new approach to self-supervised Learning. NeurIPS 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the effect of prompt learning from unlabeled data for pretrained vision-language models (VLMs), especially in low-labeled data regime (few-shot learning or semi-supervised learning). With both supervised multi-modal training on labeled data and unsupervised multi-modal, cross-model training on unlabeled data, the proposed method, XPL, achieves superior results on 15 diverse image classification datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns raised above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W8jkQCgctD", "reviewer": "Reviewer_tbUN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* This work makes a bridge between prompt learning and semi-supervised learning. \n* The paper has multiple experimental results to assess various aspects of the improvements. \n* Various settings for prompt learning are compared with, c.f. Table 1.\n\n\nWeaknesses:\n\n* Although the paper shows promising experimental results, a more theoretical or grounded explanation for the method is missing. One could interpret the method of ‘augmenting multiple views and regressing a common representation’ as maximizing/minimizing specific forms of information [1][2]. Such an explanation could help to interpret Figure 2.\n* Most datasets in Section 4.1 are relatively small, other than ImageNet. Although the paper argues for few-show learning, that performance could still be measured on larger datasets. \n\n[1] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. \"Representation learning with contrastive predictive coding.\" arXiv preprint arXiv:1807.03748 (2018).\n\n[2] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, Mario Lucic:\nOn Mutual Information Maximization for Representation Learning. ICLR 2020\n\n[3] Xie, Qizhe, et al. \"Self-training with noisy student improves imagenet classification.\" CVPR. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of adjusting pre-trained vision-language models to multiple downstream tasks. Instead of fine-tuning the VLM's weights, the paper focuses on the prompts being used as input to the VLM. Adjusting prompts for each new downstream task requires domain-specific heuristics, and this paper aims to alleviate that. The approach works by generating multiple views of unlabeled data and incurring a loss in learning similar representations between the views. The experimental results show improvements on a variety of 16 datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Question:\n\n* Figures 1c and 1d show that the proposed XPL approach performs similarly to the CoOp approach at only a 1% labeling rate. However, what are the absolute numbers of data? It is not clear if either a) the amount of labeled data is fixed, and this method manages to use ‘additional’ unlabelled data, or b) the amount of total data is fixed, and this method manages to make do with ‘less’ labeled data. (Same for Figure 3)\n* The semi-supervised approach seems similar to ‘Noisy-student’ from [3]. What are the similarities/differences w.r.t. that paper?", "broader_impact_concerns": "Semi-supervised learning has a positive impact, both on being able to use more data, and possibly to save compute.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vSSrxHDyMP", "reviewer": "Reviewer_W2o1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n1. Interesting General Idea:\nThe concept of utilizing semi-supervised training to enhance the learning of prompts is intriguing and represents a novel approach in the field. This methodology could pave the way for more efficient learning paradigms, especially in scenarios where labeled data is scarce but unlabeled data is abundant. The idea is not only innovative but also very timely, given the increasing interest in making machine learning models more data-efficient and capable of learning from limited supervision.\n\n2. Comprehensive Experiments and Ablation Studies:\nThe thoroughness of the experiments and the inclusion of ablation studies significantly strengthen the paper. These studies provide valuable insights into the impact of various components of the proposed method, offering a clear understanding of what contributes most to its effectiveness. Such comprehensive experimentation is crucial for validating the robustness and generalizability of the proposed approach. It also helps in establishing a benchmark for future work, encouraging more transparent and replicable research in the community.\n\n\n# Weaknesses:\n\n1. Inaccuracy in Abstract regarding Prompt Learning:\nThe statement in the abstract that prompt learning focuses solely on learning continuous soft prompts overlooks significant existing work on learning discrete prompts. This oversight might mislead readers about the scope and diversity of prompt learning methodologies. Acknowledging the existence and relevance of discrete prompt learning methods [1, 2], although in the NLP domain, is important for providing a holistic view of the field and ensuring that the contributions of the paper are accurately positioned within the broader research landscape.\n\n2. Lack of Distinction between Equations 4 and 5:\nThe similarity between Equations 4 and 5, with the only difference being the subscript of primary/auxiliary networks, raises questions about the novelty or necessity of presenting both as separate equations. This could potentially confuse readers or dilute the perceived contribution of the mathematical formulation. Clarifying the specific roles or implications of these equations in the context of the proposed method could help in underscoring their relevance.\n\n3. Questioning the Advantage of the “Networks” Column in Table 1:\nThe presentation of the “Networks” column as an advantage in Table 1 might be misleading, as employing an auxiliary network could introduce additional computational overhead and complexity, which might not be preferable in all contexts. Efficiency is a critical consideration in the deployment of machine learning models, especially in resource-constrained environments. Highlighting this aspect could offer a more balanced view of the trade-offs involved in the proposed method.\n\n4. Exclusion of Baseline Comparisons in Sections 4.2 and 4.3:\nThe absence of a baseline comparison that lacks any visual/text prompt in Sections 4.2 and 4.3 is a missed opportunity to demonstrate the significance and added value of the prompt learning methods being proposed. Including such a baseline would provide a clearer picture of the performance improvements attributable to the introduction of prompts, thereby strengthening the argument for the effectiveness of the proposed approach.\n\n5. Selection of Few Labeled Data and Sensitivity to Distribution Gaps:\nThe method's sensitivity to the selection of the few labeled data and potential distribution gaps between the labeled and unlabeled datasets is a critical concern that warrants further exploration. Understanding how the selection process and any distribution disparities affect the performance of the method is essential for assessing its robustness and applicability to real-world scenarios. Addressing these concerns could greatly enhance the paper's contributions by providing guidelines for effectively implementing the proposed method in diverse settings.\n\n[1] Large language models are human-level prompt engineers\n\n[2] Joint Prompt Optimization of Stacked LLMs using Variational Inference", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces XPL, a semi-supervised framework for learning prompts in vision-language models (VLMs), leveraging both labeled and unlabeled data. It identifies the limitations of prior supervised prompt learning approaches, which depend heavily on labeled data, and proposes utilizing the vast amounts of available unlabeled data to improve learning efficiency and model performance. XPL employs a novel cross-model design where two models with different prompt lengths learn complementary knowledge from unlabeled data, using a coupling function to ensure consistency between text and visual prompts. This approach not only improves the adaptability of VLMs to various downstream tasks but also enhances their generalization ability, as demonstrated by extensive experiments across multiple datasets. The results show that XPL significantly outperforms existing methods, especially in low-data regimes, showcasing its effectiveness in utilizing unlabeled data and its potential to reduce the reliance on large labeled datasets for training VLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I hope the authors could address my concerns above.", "broader_impact_concerns": "This paper will contribute a broad impact to the vision-language community for this semi-supervised learning metho.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NylgvYIWiz", "reviewer": "Reviewer_G6Rs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Innovative Approach: The framework introduces a novel method for leveraging unlabeled data in vision-language models, potentially improving model performance without additional labeled data.\n- Comprehensive Analysis and Study: The framework includes a thorough examination of various aspects such as the impact of different augmentation techniques, prompt lengths, and semi-supervised learning strategies. This comprehensive analysis provides valuable insights into the effectiveness of the proposed approach and contributes to a deeper understanding of the underlying mechanisms.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Introduce XPL, a cross-model framework for semi-supervised prompt learning in VLMs.\n- Propose a semi-supervised prompt learning approach, making prompts invariant to diverse views of unlabeled samples.\n- Utilize various image augmentations and prompt length variations to generate multiple views of unlabeled samples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can author visualize the learned text and visual prompts? Visualizations can enhance the interpretability of the model's behavior, aiding in understanding its inner workings and potential biases. Providing clear and insightful visualizations could be crucial for readers to assess the validity and effectiveness of the proposed approach.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel framework, XPL, for semi-supervised prompt learning in vision-language models (VLMs). This approach leverages unlabeled data to enhance the adaptability of pretrained VLMs to multiple downstream tasks without extensive retraining. XPL employs a dual model architecture where different prompt lengths learn from unlabeled samples through a cross-model design, improving learning efficiency and generalization.

There is a unanimous decision to accept the paper. Using semi-supervised learning on prompt learning for multimodal models is a novel contribution. The methodology is rigorously tested on diverse datasets with significant improvements. The framework also shows strong performance with out-of-domain unlabeled images. On the negative side, there are concerns regarding: 1) the lack of theoretical grounding for why the method works, 2) connections to existing literature in both prompt tuning and semi-supervised learning, and 3) other experimental improvements and clarifications. The authors did a good job addressing the concerns with their rebuttal. Overall, this is a solid work with an interesting problem, comprehensive results, and clear improvements. The AE thus recommends acceptance to TMLR. The authors should carefully revise their paper and address all reviewer concerns in their final version.
Decision: Accept (poster)