Learning from True-False Labels via Multi-modal Prompt Retrieving
Authors: Zhongnian Li
Jinghao Xu
Peng Ying
Meng Wei
Xinzheng Xu
Venue: ICML 2025 poster (2025)
Keywords: Weakly supervised learning; True-False labels; Multi-modal prompt retrieving; Multi-class classification;
Primary area: general_machine_learning->everything_else
TL;DR: We propose a novel weakly supervised setting, True-False Label (TFL), which generates reliable labels using vision-language models. Besides, we introduce a risk-consistent method to learn from TFLs via multi-modal prompt retrieving.

Abstract: Pre-trained **V**ision-**L**anguage **M**odels (VLMs) exhibit strong zero-shot classification abilities, demonstrating great potential for generating weakly supervised labels. Unfortunately, existing weakly supervised learning methods are short of ability in generating accurate labels via VLMs. In this paper, we propose a novel weakly supervised labeling setting, namely **T**rue-**F**alse **L**abels (TFLs) which can achieve high accuracy when generated by VLMs. The TFL indicates whether an instance belongs to the label, which is randomly and uniformly sampled from the candidate label set. Specifically, we theoretically derive a risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs. Besides, we propose a convolutional-based **M**ulti-modal **P**rompt **R**etrieving (MRP) method to bridge the gap between the knowledge of VLMs and target learning tasks. Experimental results demonstrate the effectiveness of the proposed TFL setting and MRP learning method. The code to reproduce the experiments is at https://github.com/Tranquilxu/TMP.

Learning from True-False Labels via Multi-modal Prompt Retrieving
Zhongnian Li
Jinghao Xu'
Peng Ying'
Meng Wei Xinzheng Xu 123
Abstract
mains in the time-consuming process of collecting such annotated datasets. To address this challenge, various forms of weakly supervised learning have been proposed and ex- plored in a range of settings, including semi-supervised learning( Van Engelen & Hoos, 2020; Cao et al., 2022; Guo et al., 2022; Wei et al., 2024), positive-unlabelet learn- ing(Sakai al., 2018; Chapel al., 2020; Hu al., 2021; Li et al., 2024b; 2025), noisy-label learnin((Menon et al., 2015; Ghosh et al., 2017; Han et al., 2020), partial-label learning(Cour et al., 2011a; Feng et al., 2020b; Xia et al., 2023; Li et al., 2024a). and complementary- learn- ing(Ishida et al., 2017; Yu et al., 2018; Gao & Zhang, 2021; Wei et al., 2023).
Pre-trained Vision-Language Models (VLMs) exhibit strong zero-shot classification abilities, demonstrating great potential for generating weakly supervised labels. Unfortunately, exist- ing weakly supervised learning methods are short of ability in generating accurate labels via Vs. In this paper, we propose a novel weakly super- vised labeling setting, namely True-False Labels (TFLs) which can achieve high accuracy when generated by VLMs. The TFL indicates whether instance belongs to the label, which is ran- domly and uniformly sampled from the candidate label set. Specifically, we theoretically derive a risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs. Besides, we propose convolutional- based Multi-modal Prompt Retrieving (MRP) method to bridge the gap between the knowl- edge of VLMs and target learning tasks. Ex- perimental results demonstrate the effectiveness of the proposed TFL setting and MRP learning method. The code to reproduce the experiments is at github. htt.tuu.comm//eoooeeoeeeeeeeeeee Tramquilxu/TMPP
Recently, pre-trained Vision- Language Models (VLMs) (Radford et al., 2021; Li al., 2022; Liu et al., 2024) trained on large-scale labeled data have achieved remark- able results and exhibit significant potential in generating high-quality weakly supervised labels, thereby reducing an- notation costs(Menghini et al., 2023). Unfortunately, as shown in Figure 1 (a), the VLMs supervised labels are often of low quality because the zero-shot results from VL.Ms may be incorrect(Wang et al., 2022: Menghini et al., 2023). Specifically, the VLMs supervised label will be annotate as "Sea horse" (i.e., label with the highest confidence), while the ground-truth label is "Bass". These VLMs supervised labels with noise semantics may degrade the performance of models on target learning tasks. This fact further inspires us to explore and leverage novel weakly supervised label- ing settings to harness VLMs for generating higher-quality labels.
1. Introduction
In recent years, supervised learning has exhibited remark- able performance across a diverse range of visual tasks, including image cnassffiaatio((ooovittitt et al., 2021). ob- ject detection(Zong et al., 2023), and semantic segmenta- tCon et al., 2018). This success can be largely at- tributed to the abundance of fully annotated training web data the internet. However, a significant challenge re-
In this paper, we propose a novel weakly supervised classi- fication setting: learning from True-False Labels (TFLs), which can achieve high accuracy and avoid the cost of man- ual annotation when generated by VLMs. Besides, the utilization of TFLs can markedly enhance the efficiency of human annotation. The TFL indicates whether an instance belongs to the label, which is randomly and uniformly sam- pled from the candidate label set. Specifically, instance will be annotated with "True" label when it belongs to the sampled label, and with a "False" label when it does not. For example, as illustrated in Figure I (b), for an image with the ground-truth label "Bass", annotators will easily annotate the instance with "Bass" and "True" label when the randomly sampled label is also "Bass". Conversely, the
"School of Computer Science and Technology, China Uni- versity of Mining and Technology, Xuzhou, China "Mine Dig- itization Engineering Research Center of the Ministry of Edu- cation, China University of Mining and Technology. Xuzhou, China State Key Lab. for Novel Software Technology, Nanjing University, Nanjing, China. Correspondence to: Xinzheng Xu <xxzheng@ www.t.cu.mmt......
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Ground Truth: Bass
Candidate Label Set
1. Ant
Wild cat
Butterfly
Panda
Elephant
6. horse
7. Nautilus
8. Crab
Rooster 10.Lobster
99, Bass
1. Ant
7. Nautilus
Human
hurse
57.47%
VLMs
Bass
13.02%
Buttertly
9. Rooster
Labster
10.31%
Panda
Elephant
10.Lobster
Nautilus
3.19%
6. Sen horse
99. Bass
Crab
3.19%
Sampling
Sampling
Bass
Bass
True
Not the highest confidence
Crab
False
Crab
Elephant
Elephant &
False
True-False Labeling
(a) bases- Labeling Process
(b) Human-based Labeling Process
VLMs Supervised Label
Human Supervised Label
True-False Label
Randomly Sampled Label
True
False
Hinary True-False Label
Figure comparison between traditional labeling and TF labeling (both VLMs-based and human-based labeling process). The VLMs zero-shot result depicted in the figure derived from CLIP with ViT-L/14(Radford et al., 2021). The example images and categories are derived from Caltech-1011 101 Fei et al., 2004). Truc-False labels can enhance both the accuracy of VLMs supervised labeling and the efficiency of Human-based labeling.
annotators can annotate the instance with "Elephant" and "False" label when the randomly sampled label is "Elephant". Moreover, as illustrated in Figure 1 (a), when the zero-shot results from VLMs are incorrect, the TFLs generated by VLMs are almost always accurate. Overall, this novel label- ing setting can effectively leverage the knowledge of VLMs for generating high-quality labels. Additionally, the TFL.s can enhance the efficiency of the human labeling process by reducing the time cost for browsing the candidate label set.
process.
A risk-consistent method introduced to explore and utilize the conditional probability distribution informa- tion of TFLs instead relying solely on labels. The conditional probability distribution information can be easily obtained by VI.Ms.
A convolutional multi-modal prompt retrieving method proposed to bridge the gap between the knowledge of VLMs and target learning tasks. To the best of our knowledge, this is the first convolutional- based prompt learning approach for weakly supervised learning.
In this paper, we propose risk-consistent method to learn from True-False labels via Multi-modal Prompt retrieving (TMP). Specificalyy. we theoretically derive risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs instead of relying solely on labels. Besides, introduce novel prompt learning method called MRP learning, which can bridge the gap between pre-training and target learning tasks. Extensive experiments on various datasets clearly demonstrate the effectiveness of the proposed TFL setting and MPR learning method.
2. Related Work
2.1. Weakly Supervised Learning
Weakly supervised learning aims to construct predictive models by learning from large number of training samples that contain incomplete, inexact, or inaccurate supervision information(Zhou. 2018). These weakly supervised learning approaches include but not limited to semi-supervised learn- ing( Van Engelen & Hoos, 2020; Cao et al., 2022; Guo et al., 2022; Wei et al., 2024), partial-label learning(Sakai et al.,
Our main contributions are summarized as follows:
We propose novel labeling setting for weakly super- vised classification, which can effectively leverage the knowledge of VLMs for generating high-quality la- bels and enhance the efficiency of the human labeling
2018; Chapel et al., 2020; Hu et al., 2021; Li et al., 2024b:
2025) and complementary-labe learning(Ishida et al., 2017;
Yu al., 2018: Gao Zhang, 2021: Wei al., 2023).
2

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Semi-supervised learning assumes the presence of both la- beled and unlabeled data in the training set. It mainly in- & Ben-
language representations.
Previous prompt learning approaches have typically focused on directly learning the prompt itself. In contrast, our method involves training a convolutional neural network to retrieve the prompt embeddings.
cludes entropy minimization methods gio, 2004; Lee et al., 2013), consistency regularization methods(Sajjadi et al., 2016; Tarvainen & Valpola, 2017; Miyato et al., 2018), and holistic methods(Berthelot et al., 2019; Sohn et al., 2020; Sosea & Caragea, 2023). Partial- label learning involves training instances with set of po- tential candidate labels, where only one is assumed to be correct but is unknown. This approach can be cat- egorized into identification-base strategies(Zhang et al., 2016; Tang & Zhang, 2017; Xu et al., 2019) and average- based strategies(Cour et al., 1b; Hüllermeier & Beringer, 2005), depending on how they handle candidate labels. Complementary-label learning (CLL) assigns a label which specifies the class that an instance does not belong to. Ishida et (Ishida et al., 2017) design unbiased risk estima- tor (URE) with a solid theoretical analysis, which enables multi-class classification with only complementary labels. Subsequently, various models and loss functions are incor- porated into the CLL framework(X) Yu et al., 2018; Gao & Zhang, 2021; Ishida et al., 2019).
3. Method
In this section, we provide a detailed description of risk- consistent method to learn from True-False labels via Multi- modal Prompt retrieving (TMP). Firstly, we introduce the problem definition and the labeling process of TFL. Be- sides, we theoretically derive a risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs instead of relying solely on labels. Sub- sequently, we propose convolutional-basel multi-modal prompt retrieving method to bridge the gap between the knowledge of VLMs and target learning tasks. Finally, we illustrate the architecture of TMP.
3.1. True-False Labels
In contrast to the previous approach, we now consider an- other scenario, namely True-False Labels (TFLs) learning, In this setting, the TFL indicates whether an instance be- longs to the label, which is randomly and uniformly sampled from the candidate label set. Specifically, annotator only needs provide the binary TFL (i.e., "True" "False") according to the randomly sampled label. To illustrate, as shown in Figure I (b), when considering the candidate la- bel set, ("ant", "wild cat", "butterfly", "panda", "elephant", "sea horse", ... "bass", ...), for "bass" image, the anno- tator can readily assign a "Truc" label when the randomly sampled label is the "bass". contrast, the annotator can readily assign "False" label when the randomly sampled label is the "clephant". Moreover, as illustrated in Figure I (a), the zero-shot results with the highest confidence from the VLMs annotate the instance with "sea horse". However, when the randomly sampled label is "crab", the TFL of this instance will be "crab" and "false", which is accurate. Com- pared to the traditional labeling, TFL effectively leverage the knowledge of VLMs for generating high-quality labels. Additionally, the TFLs enhance the efficiency of the human labeling process by reducing the time cost for browsing the candidate label set.
Recent studies have explored the potential of reducing anno- tation costs with weakly supervised learning. Unfortunately, these methods struggle to leverage the knowledge of VLMs to generate usable labels. Consequently, we propose the True-False label setting, which can achieve high-accuracy when generated by VLMs.
2.2. Prompt Learning in VLMs
The role of the prompt is primarily to provide the model with context and paramcter information about the input. Prompts can help the model understand the input's intention and generate an appropriate response(Liu et al., 2023; Lei et al., 2024).
CLIP(Radford et al., 2021) introduces prompt to the CV and multi-modal domains by converting image category la- bels into text sequences as a hand-crafted language template prompt, such as "a photo of a (CLASS)". CoOp(Zhou et al., 2022b) transforms CLIP's hand-crafted template prompts into set of learnable continuous vectors, which opti- mized from few-shot transfer. CoCoOp(ZZhu et al., 2022a) enhances CoOp by training a lightweight neural network to generate input conditional vectors for each image, result- ing in better performance on new classes. VPT(Jia et al., 2022) introduces a small number of trainable parameters into the input space while keeping the pre-trained Trans- former backbone frozen. These additional parameters are simply prepended into the input sequence of each Trans- former layer and learned together with a linear head during fine-tuning. (Khhttak et al., 2023) develops a multi- modal prompt to improve consistency between visual and
Accuracy superiority. As shown in Table we demon- strate the accuracy superiority of TFL over traditional la- beling. When utilizing CLIP for annotation, CLIP provides the binary TFL. by determining whether the CLIP zero-shot result are consistent with the randomly sampled label. TFLs generally demonstrate an impressive accuracy rate exceed- ing 99.5%, which substantiates the effectiveness of TFL. Moreover, it is worth noting that, according to previous
3

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Table 1. The TFL. annotation details of five benchmark datasets. We demonstrate the accuracy of THL. by utilizing VL. (i.e., CLIIP
ViT-L/14(Radford et al., 2021)) for annotation. We show the number of true labels and the efficiency of TFL by human annotation. #
denotes the number of, and x denotes times.
Basic Information
Accuracy
Efficiency
# Classes
Training Set
# True Labels
Mislabeled
Error Rate (%)
Labeling Speed Multiplier Ven (x)
CIFAR-100 100
100
50000
498
256
0.512
50
Tiny ImageNet
200
100000
507
267
0.267
100
Caltech-101
102
6400
0,281
51
Food-101
101
75750
795
153
0.202
50.5
Stanford Cars
196
8144
42
20
0.246
98
Average
142.8
0.302
69.9
studies(Northeur al., 2021), TFLs generated by VLMM have, to some extent, achieved superior accuracy compared to traditional manual labeling.
to mitigate bias amplification through stochastic sampling, since such error propagation fundamentally stems from de- terministic labeling mechanisms. Through its implementa- tion, TFL achieves over 99% annotation accuracy, result that significantly improves labeling precision via confidence- label generation. This advancement substantially reduces noise propagation by effectively min- imizing error accumulation in pseudo-labeling iterations. OHybrid supervision mechanism: TFL integrates strong supervision from retained true labels to provide semantic correction anchors. This hybrid labels provide explicit opti- mization direction, enhances noise robustness and improves stability in complex scenarios.
Efficiency superiority. As shown in Table 1, we demon- strate the efficiency superiority of TFL over traditional label- ing. Specifically, we introduce a labeling speed multiplier Um to quantify the efficiency of TFL. We define the time it takes for an annotator to determine whether an instance be- longs to a label as the unit time. Thus, the time required for TFL to label an instance 1 unit (i.e., Irr = 1). contrast, traditional numan-basee labeling process require annotators to browse through half of the candidate label set average, so the mathematic expectation of traditional labeling time to is K where K is the number of candidate labels. Then fo_. As shown in Table 1, TFL demonstrates an average labeling efficiency that is 69.9 times higher than traditional labeling across five datasets, which substantiates the efficiency of TFL. The efficiency of TFLs can be attributed to the fact that this labeling approach significantly reduces the time spent browsing the candidate label set.
2
3.2. Problem Setup
Um can be formulated as Um byp"
In multi-class classification, let X' E TR:d be the feature space and y = [K] be the label space, where d is the feature space dimension; [K] = {1..... K]: and K > is the num- ber of classes. Suppose D = {(x1,, 31) is the dateset where xι E X, y1 E y and N denotes the number of train- ing instances. We assume that ((1). y/ ))) are sampled independently from an unknown probability distribution with density p(a,, y). The goal of ordinary multi-class clas- sification is to learn classifier f(x) : {l...., K} that minimizes the classification risk with multi-class loss (()()),))
Key advantages. The TFL. framework introduces three key advantages that fundamentally address limitations in standard self-training methods: Random label sampling as implicit regularization: TFL employs uniform random label sampling as implicit regularization prevent model overfitting. This stochastic mechanism in TFL, analogous to established techniques like dropout, SGD and random data augmentation, promotes better generalization solutions(A)) Ali et al., 2020; Wang et al., 2024). Unlike confidence- based pseudo-labeling in self-training methods, which amplifies CLIP's inherent biases through error propagation( Wang et al., 2022: Menghini et al., 2023). @ High-accuracy TFL for bias amplification mitigation: Prior studies(Wang et al., 2022; Menghini et al., 2023) have demonstrated that pscudo-labeling noise can create cumulative effect on classes with inaccurate pseudo-labels, which progressively amplifies the model's 's inherent bias toward certain classes. This observation directly motivates TFL's design objective
R() Epp(..) ((((s)
(1)
Egzz E,_ plu plu= (()o))
where E denotes the expectation.
In this paper, we consider the scenario where each in- stance is annotated with TFL Y instead of ordinary class label y. Suppose the TF labeled training dataset DIF { Yi) is sampled randomly and uniformly from an unknown probability distribution with density p(r, YY. Y = (0t,s) is a TFL where `t E y is the ran- domly sampled label and 8; E {0, 1) represents whether instance xt belongs to category ÿt. Specifically, st = O signifies that the instance Iú does not belong to the cate-

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
gory Wi and it = 1 denotes that the instance iy belongs to the category yt- Similarly, the objective is to learn classi- fier f(x) x + (.,...,, K from the TF labeled training dataset, which can accurately categorize images that have not been previously observed.
3.4. Multi-modal Prompt Retrieving
In this section, we introduce convolutional- Multi- modal Prompt Retrieving (MPR) method to bridge the gap between the knowledge of VLMs and target learning tasks. MPR supplements discriminative features from weakly su- pervised data through cross-modal retrieval, reducing learn- ing complexity while improving model robustness. Specifi- cally, we retrieve visual and textual embeddings by learning a convolutional based prompt network on top of CLIP. The learnable convolutional network retrieves domain- aware em- beddings (e.g., culinary textures in Food-101 addressing CLIP's generic prompt limitations.
3.3.
Estimator
In this section, based on proposed problem setup, we present a risk-consistent method(Feng et al., 2020c;a; Xu et al., 2022). To rigorously depict the connection between ground- truth label and TFL, we introduce the following assumption.
Assumption 3.1. (TFLs Assumption). Since (zr,g) y) is sam- pled randomly and uniformly from an unknown probability distribution with density p(z, y). the conditional probability K 1,z)) === is under
The overall architecture of the MPR is shown in Figure 2. Note that the base models of CLIP(Radford et al., 2021) is frozen in the entire training process. MPR is comprised of two distinct components, Textual Prompt Retrieving (TPR) and Visual Prompt Retrieving (VPR). They share one convolutional based prompt network for prompt retrieving.
distribution of TFLs {p(y = =ly i, 8
the TFLs assumption as follows:
ply= = 1yy L, 1.2) =pl = 2/y 2, 1,2)
First of all, we select a matrix M E R#XW*BB whose elements are all initialized to 1. This matrix will be fed into a convolutional based prompt network Jenn(-) and the image encoder gr(-) to obtain prompt embedding 4p gi (genn (M)).
= = Klū = K2 1,
It is worth noting that we cannot employ the conditional probability p(y = i|zz) in (I) directly since we do not have access to ordinary supervised data. Fortunately, we can use TFLs data to represent it by introducing the TFLs conditional probability p(y = i = j.8 = 0fzr).
For the TPR, we the text prompt template "This is photo of (Radfod et al., 2021), where "[CI.S]" rep- resents category labels. By putting text prompts for all cate- gories {P:T) Jiz1 into the text encoder gr(-), we obtain the text embeddings Qr = {17") K ) for all categories, where = (P!(
K
Lemma 3.2. Under the TFLs Assumption 3.1, the condi- tional probabilities ply ilı)) can be expressed as:
ply- i|(z) = p(ū = i, = +
For the VPR, we randomly sample images { C the dataset to create the retrieval image set, where C' denotes the image number of the retrieval image set. These images are then fed to the image encoder 91 (-) to obtain the image embeddings Q,- {o
En. ply = 1 = J. = 0, - j. -, 0|z)
n=1 from
Theorem 3.3. To deal with TFL learning problem, accord-
ing the Assumption 3.1 and Lemma 3h2 classification risk R()) in Equation (1) could be rewritten as R F (f) = E, ((f(x), B:ptt...=11
- -. where q = (P!)).
(4)
Besides, we retrieve Kr text embeddings and Kr image embeddings that are most similar to the prompt embedding, respectively.
where [(f(x).)) = Eim.ing ply i|y j,8
0, O.x)((((x)) 1). The proof is provided in the Appendix A.2.
Remark 3.4. To fully explore and leverage the prior knowl- edge of VLMs, we employ VLMs to precisely estimate condi- tional probability distributions ply ily j. 0, 2)) in Theorem 3.3. And then the empirical risk estimator can be expressed as:
(1) Kr
9" = - 5 = Top- (r (p))
e0
(6)
d' = {ot)
{ r=1
K1
= -Top-K gp))
of
1 Nr ((f()))))++ ( = NE ETT
NT E NT C(f(a)), 3
where cos(:..) denotes cosine similarity, and the (cos( Top-K(cos(QU. qi
(5) where Nr and Ny denote the number of instances with binary TFL s = 0 and = 1. Then, we can learn a multi- class classifier f(x): : x - (1....,
9p)) will retrieve top K vectors with the highest similarity to vector Sp from Q. Ky and K2 are hyperparameters to balance TPR and VPR.
".....,KK K) by minimizing the proposed empirical approximation of the rosk-consistent estimator in Eq (5).
These embeddings are then flattened for further process- ing. Then we obtain the TPR embeddings gr and VPR
5

---

Learning from True-False Labels via Multi-modal Prompt Retrieving
MPR
Text Embeddings
Frozen
Trainable
Text Prompts This is photo of (CLS)
Concat
+ Weighted Sun
Text
Imoge Encoder
Top-Kr
Flatten
Encoder
Prompt Embedding)
TPR
VPR
Initial Matrix
Feliat
Prompt Network
Pone
Convolutional based
Chesdfixx
Prompt Network
Cosine Simularity
Retrieval Image Set
Image Embeddings
Top-
Flatten
Softmax
Kisk-Consistent
Randomly Sample
Instances
Text Promits This a photo of [CLS]
Sample
Softmax
+
Dataset
Only for instances with False label
Figure architecture of TMP, including MPR and risk-consistent estimator, MPR retrieve visual and textual embeddings by learning convolutional-basee prompt network top of CLIP. The goal of risk-consistent estimator is to explore and utilize the conditional probability distribution of TFLs.
emheddings @r, which make up the MPR embeddings.
edge of VLMs, we employ VLMs to precisely estimate p'y = i = j. s = 2). Specifically, we could get the conditional probability distributions of linear classifier Pr.c which formulated follows:
rr 9 gr 9
9KT)
T
.9K,) .g9,),
The (-) function flattens input vectors by reshaping them into a one-dimensional vector.
PLC Softmmat(alCommaaggiia 2 (r))))))
(8)
where gi(-) is linear classifier and Concat(-) concatenates the given sequence of vectors. Besides, we obtain the con- ditional probability distributions POLIP from CLIP, which can be formalized as follows:
To the best of our knowledge, MPR is the first convolutional- based prompt learning approach for fine-tuning VLMs. Un- like directly optimizing the text or image itself (Zhou et al., 2022b; Bahng el al., 2022), Convolutional Neural Networks (CNNs) offer significant advantages in capturing local fea- tures. Direct image optimization, on the other hand, requires independent updates for each pixel, resulting in lack of correlation between them. MPR enhances both textual and visual modalities by providing supplementary information without requiring additional data. Additionally, MPR is straightforward and relatively inexpensive in terms of com- putational resources compared to other multi-modal prompt learning methods.
Pelir Soffmarrcos(gr (x), Qr))
(9)
Finally, we obtain the empirical conditional probability dis- tribution through linear weighted sum method, which can be formalized as follows:
p(v i1y = XPIC (1 A)POLIP (10)
where À E [0, 1] conditional probability hyperparam- eter that allows our model simultaneously leverage the knowledge from VLMs and the learned model to enhance the performance of classification.
3.5. Practical Implementation
In this section, we introduce the practical implementation of the proposed method.
The conditional probability distribution p(u = -ly - j.s-
0, x)) can be estimated as p(y = ğ = j. = 0, Then we can calculate the empirical risk-consistent cross-entropy loss based on the p(y ilğ j. 0, 0,2;, optimize
Conditional probability distribution. Notice that min- imizing R requires estimating the conditional proba- bility distributions ply = ilğ = j.s = 0,x) in Theo- rem 3.3. To fully explore and leverage the prior knowl-
both the linear classifier and the convolutional-be based prompt network.
6

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Model & Algorithm. The convolutional-basee prompt network comprises four convolutional layers (model archi- tecture in Appendix A.3). For a comprehensive understand- ing, Algorithm in Appendix A.4 outlines the overall pro-
ORCA(Cao al., 2022) and NACH(Guo et al., 2022); The SSL methods aiming to classify both seen and unseen classes effectively. In our experiments, we treat instances with = 1 as supervised data and instances with = 0 as unlabeled data.
cedure.
PaPi(Xia al., 2023): An PLL method eliminating noisy positives and adopting a different disambigua- tion guidance direction. In our experiments, we treat instances with = 1 as supervised data and instances with 8 = 0 as partial-labeled data. Then we consider all categories other than the randomly sampled label as the candidate label set, for purtial-labeled data.
Loss f'unctions. Many loss functtions satisfy our method, such logistic loss (() log(1 e-wf(=)), MSE loss [()()),)) (y - f()))) etc. In our experiments, we utilize the widely used cross-entropy loss function in multi-class classification ((f(),,)) v) = -y (x)(
4. Experiments
CLL with WL(Gao & Zhang, 2021): A CLL method with a weighted loss. In our experiments, we treat instances with s = as supervised data and instance with 8 = as complementary-laber consider the randomly sampled label as the class label that the instance does not belong to for complementary- labeled data.
4.1. Experimental Setup
Dataset. The efficacy of our method was evaluated on five distinct multi-class image classification datasets that feature both poarse-grained (CIFAR-100(K)) (00((Krrheesky et al., 2009), Tiny ImageNet(Le& & Yang. 2015) and (FFeici- et al., 2004)) and fine-gras tne-grained 2014) and Stanford Cars(Krause et al., 2013)) classification in different domains. For each dataset, the label of each image in the training set is replaced with the True-False Label (TFL), and the labels in the test set remain unchanged from the ground-truth labels. More information related to the datasets is shown in the Appendix A.5.
data. Then we
et al.,
CLIP Linear ProbeoR et al., 2021) (CLIP LP): A VLMs-based approach which trains an additional linear classifier on top of CLIP's visual encoder. In our experiments, we employ three different settings: CLIP LPS, CLIP LPT, and CLIP LPP. CLIP LPS denotes the application of all labeled training data in a fully- supervised learning framework. CLIP LPT denotes learning using TFLs (i.e., only use instances with s = 1 as supervised data). CLIP L.PP denotes the application of CLIP supervised labeled data in a fully-supervised learning framework.
Implementation details. To ensure fair comparisons, for all experiments, we use CLIP with ViT-L/14 as the vision backbone, and employ the AdamW pprimizer(Loshchiine Hutter, 2019) for the linear classifier with an initial learning rate of Le 33 a weight decay parameter set to 0.9, and the minimum learning rate of 5e 6 Unless otherwise noted, all models are trained for 50 epochs with a batch-size of 256 on single NVIDIA RTX 4090 GPU. In our experiments, employ the AdamW optimizer for the convolutional- based prompt network with an initial learning rate of 8e-2 a weight decay parameter set to 0.01, and the minimum learning rate of 5e-1. The hyperparameters Ky and K, are set to 15 and 5, respectively. The size of matrix M is set to 224 x 224x x 1.
&
To ensure that the only variable is the algorithm, we re- place their original visual encoders with the same CLIP's visual encoder, and used the same linear classifier across all experiments.
4.2. Results of VLMs-based TFLs
In this section, we utilize the TFLs generated by CLIP with ViT-L/14. From the results presented in Table 2, it can be observed that the proposed method consistently outperforms all weakly supervised baselines by large margin (over 10%), especially on fine-grained datasets such as Stanford Cars (over 60%). These results demonstrate that traditional weakly supervised learning methods struggle to effectively leverage the prior knowledge of VLMs. In contrast, our approach can more fully exploit the capabilities of VLMs.
Compared methods. To assess the efficacy of the pro- posed approach, a thorough evaluation is conducted through comparisons with weakly supervised learning methods, in- cluding semi-supervised learing (SSL) methods, partial- label learning (PLL) methods and complementary-labee learning (CLL) methods. VLMs-based approaches are also considered. Specifically, the image encoders in the weakly supervised methods mentioned above were built using the same CLIP image encoder, transforming them into more competitive weakly supervised linear probe models. The key summary statistics for the compared methods are as follows:
Furthermore, our approach exhibits performance enhance- ments over other methods based on VLMs. Specifically, our method outperforms zero-shot CLIP on all datasets, with an average improvement of nearly 2%. Additionally, our method converges more rapidly than the CLIP linear probe.
7

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Table Comparison results VI.Ms-based TFLs in of classification accuracy (the higher, the better).
The best accuracy
highlighted in bold. TMP (VLMs) denotes the results on VLMs-based TFLs.
CIFAR-100 Tiny ImageNet Caltech- 101
Food- 101
Stanford Cars
Average
VLMs Supervised Labels Learning
CLIP LPP(Radford et al., 2021)
77.4
74.71
88.74
93.30
71.56
81.14
Weakly Supervised Learning Methods
ORCA(Cao et al., 2022)
53.26
19.21
14.40
7.96
7.25
20.42
NACH(Guo et al., 2022)
64,42
35.09
21.39
12.62
4.53
27.61
PaPi(Xia al., 2023)
63.73
41.50
43.27
81.94
10.19
48.13
CLL with WL(Gao & Zhang, 2021)
59.05
44.21
44.79
85.46
10.40
48.78
VI.Ms-based Methods
Zemo-shot CIIPPRadoord al., 2021)
75.58
72.66
87.24
92.76
70.76
79.80
CLIP LPT(Radford et al., 2021)
25.25
18.24
22.63
66.69
4.96
27.55
CLIP (200 (anchs)(Radford et al., 2021)
27.86
20.26
25.07
69.35
5.53
29.61
TMP (VLMs)
78.22
75.14
89.14
93.52
72.52
81.71
Table Comparison results human-based TFLs in terms of classification accuracy (the higher, the better). The best accuracy highlighted in bold. TMP (human) denotes the results on human-based TFLs data and TMP (VLMs) denotes the results on VLMs-based TFLs.
CIFAR-100
Tiny ImageNet Caltech- 101
Food- 101
Stanford Cars
Average
Human Supervised Labels Learning
CLIP LPS(Radford et al., 2021)
85.81
85.31
96.76
94.94
87.71
90.11
Weakly Supervised Learning Methods
ORCA(Cao el al., 2022)
50.54
17.65
14.14
7.82
NACH(Guo et al., 2022)
63.46
31.78
15.09
8.51
5.57
24.89
PaPi(Xia al., 2023)
60.69
40.80
47.06
80.31
6.00
46.97
CLL with WL(Gao & Zhang, 2021)
63.25
51.54
50.73
87.85
9.64
52.60
VLMs-based Methods
Zero-shot CLIP(Radford al., 2021)
75.58
72.66
87.24
92.76
70.76
79.80
CLIP LPT(Radford et al., 2021)
25.60
21.37
21.36
68.23
3.69
28.05
CLIP LPT (200 epochs) et al., 2021)
28.06
23.39
24.31
70.65
4.24
30.12
TMP (VLMs)
78.22
75.14
89.14
93.52
72.52
81.71
TMP (human)
78.72
75.84
90.60
93.55
72.60
82.07
Surpassing the performance the CLIP linear probe trained for 200 epochs, achieve better results after just 50 epochs. This is due to the limited number of instances with that the CLIP linear probe can utilize. Besides, the results of TMP on all five datasets are better than VLMs supervised learning results. TMP uses less supervisory information and achieves better results, which demonstrates the potential of TFL as a labeling setting.
rable fully supervised approaches on Food-101 These experimental results demonstrate that method effectively bridge the gap between knowledge of CLIP and target learn- ing tasks. It is worth noting that in some weakly supervised methods, training with TFLs generated by CLIP can achieve even better results compared to those presented Table 3. Specifically, the performance 100 has improved by over 4%. heuristic reason for this that TFL. generated by CLIP may correct inherent noise in the original dataset.
CLL method on CIFAR-
4.3. Results of (luaan-based TFLs
We use the ground-trutt labels to generate TFLs for the training set. All remaining settings are identical to those in section 4.2. Table exhibits similar trend to Table 2. Compared to other methods, method achieves the best results on all datasets, which substantiates the effectiveness of TMP. Besides, our method achieves performance compa-
TLMs-based Human-based TFLs. As shown in the last two rows of Table 3, there is no significant performance difference between TMP (human) and TMP (VLMs) (with an average difference of no more than 0.3%), indicating that VLMs are capable of generating sufficiently high-quality TFL..
8

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Cahtcch 101 CIFAR-100
565.3
Table Experimental results the influence of MPR. w/o de-
Average Traninn Epesh beconds
notes without the component. Experiments are performed on human- based TFLs data.
400
Caltech-101
Food-101
Stanford Cars
Average
358.0
TMP(human) 90.60
93.55
72.60
85.58
3000
w/o MPR
88.81
93.47
71.25
84.51
w/o TPR
89.80
93.53
72.25
85.19
w/o VPR
90.23
93.52
72.53
85,43
136.8
200-
60.7
74.7
26.8
TMP
NACH
WL
90
Figure 4. Comparison of training cost between TMP and weakly supervised learning methods. The numbers in the figure represent the average time (in seconds) required to train each method for single epoch. The experiments utilize the CIFAR-10 and Caltech- 101 datasets, conducted on a single NVIDIA RTX 4090 GPU.
09
6.9
0.4
Hypepammeerr
Figure 3. Experimental results on the influence of the conditional probability hyperparameter A. Experiments are performed human-based TFLs data.
ing methods. The experiments utilize the CIFAR-100 and Caltech-101 datasets, conducted on single NVIDIA RTX 4090 GPU, with all other settings consistent with those of the previous experiments. As shown in Figure 5, we mea- sure the time (in seconds) required to train each method for one epoch. TMP significantly reduces training time com- pared to weakly supervised learning methods. Specifically, the training time for ORCA, NACH, and PaPi exceeds twice that of TMP. Additionally, the training times for TMP and CLL with WL are comparable. This advantage is attributed to the fact that weakly supervised methods often necessi- tate multiple iterations over unlabeled data within a single epoch. Our experiments demonstrate that TMP achieves higher accuracy while requiring less training time.
4.4. Influence of MPR
In Table 4, we explore the effectivvness of our proposed MPR method, which consists of TPR and VPR compo- nents, on a coarse-grained dataset (Caltech-101 101) and two fine-grained datasets (Food- 101 and Stanford Cars). We conducted experiments by individually removing the TPR and VPR, as well as removing the MPR as a whole. The results show that the removal of each component leads to some degree of performance degradation. Specifically, each component (TPR and VPR) leads to an average performance improvement of more than 0.5%. This improvement con- firms the discussion in earlier section that MPR provides more related information bridge the gap between the knowledge of VLMS and target learning tasks.
5. Conclusion
4.5. Influence of the Hyperparameter À
In this paper, we investigate a novel weakly supervised learn- ing problem called learning from True-False Labels (TFLs), which can significantly enhance the quality and efficiency of annotation. this novel labeling setting, the TFL indicates whether an instance belongs the label that is randomly and uniformly sampled from the candidate label set. We the- oretically derive risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs. Besides, we introduce novel prompt learning method called MRP learning, which can bridge the gap be- tween the knowledge of VLMs and target learning tasks. The experimental results demonstrate the effectiveness of the proposed TFL setting and MRP learning method.
We check how performance varies w.r.t À on coarse- grained dataset (Caltech-101) 101) and a fine-grained dataset (Stanford Cars), Figure 3 shows that as À increases from 0 to 0.9, there a gradual decline in performance. Specifically. these two datasets achieved the best accuracy at = 0.1 (i.e., 90.60 on Caltech-101 and 72.60 on Stanford Cars), which demonstrates the necessity of simultaneously leverag- ing the knowledge from VLMs and the learned model. Note that our method demonstrates stable performance when the conditional probability hyperparameter À is within the range [0, 0.9). The reason may be that TMP could adaptively ex- plore the relationship between VLMs and the learned model to effectively estimate the probability distribution.
Limitations and future directions. A limitation of this method is that it does not use test data to improve model performance. In the future, it is interesting to explore a test- time prompt retrieving method which can learn adaptive prompts with single test instance.
4.6. Comparison of Training Cost
In this section, we compare the training cost of our proposed TMP method with those of various weakly supervised learn-

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Acknowledgements
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, 2021.
This work was supported by the National Natural Science Foundation of China (No.62306320, 61976217), the Open Project Program of State Key Lab. for Novel Software Technology (No. KFKT2024B32), and the Natural Science Foundation of Jiangsu Province (No. (KK20210633)
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested 101 object categories. In 2004 Conference on Computer Vision and Pattern Recog- nition Workshop, pp. 178-178, 2004.
Impact Statement
The primary objective of this paper to advance the field of machine learning. While our work carries numerous potential societal implications, deem it unnecessary to explicitly highlight them within this context.
Feng, L., Kaneko, T., Han, B., Niu, G., An, B., and Sugiyama, M. Learning with multiple complementary labels. In Proceedings of the International Conference on Machine Learning, pp. 3072-3081, 2020a.
References
Ali, A., Dobriban, E.,, and Tibshirani, R. J. The implicit reg- ularization of stochastic gradient flow for least squares, In Proceedings of the International Conference on Machine Learning, pp. 233-244, 2020.
Feng, L., Lv, J., Han, B., Xu, M., Niu, G., Geng, X., An, B., and Sugiyama, M. Provably consistent partial-label learning. Advances in Neural Information Processing Systems, -19940,,0960,, 2020b.
Bahng, H., Jahanian, A., Sankaranarayanan S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274. 2022.
Feng, L., Lv, J., Han, B., Xu, M., Niu, G., Geng, X., An, B., and Sugiyama, M. Provably consistent partial-label learning. Advances in Neural Information Processing Systems, 33:10948-10960, 2020c.
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. A. Mixmatch: A holistic ap- proach to semi-supervised learning. Advances in Neural Tnformation Processing Systems, 32, 2019.
Gao, Y. and Zhang, M.-L. Discriminative complementary- label learning with weighted loss. In Proceedings of the International Conference Machine Learning, pp. 3587-3597, 2021.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 101- mining discriminative components with random forests. In Proceedings of the European Conference on Computer Vision, pp. 446 461, 2014.
Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss func- tions under label noise for deep neural networks. In Pro- ceedings of the AAAI conference on Artificial Intelligence, volume 31, 2017.
Cao, K., Brbic, M., and Leskovec, J. Open-world semi- supervised learning. In Proceedings of the International Conference Learning Representations, 2022.
Grandvalet, Y. and Bengio, Y. Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, 17, 2004.
Chapel, L., Alaya, M. Z., and Gasso, G. Partial optimal tranport with applications on positive-unlabeled learning- Advances in Neural Information Processing Systems, 33: 2903-2913, 2020.
Guo, L.-Z., Zhang, Y.-G.. Wu, Z.FF, Shao, J.-J., and Li, Y.- F. Robust semi-supervised learning when not all classes have labels. Advances in Neural Information Processing Systems, 35:3305-3317 2022.
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision, pp. 801-818, 2018.
Han, B., Niu, G., Yu, x., Yao, Q., Xu, M., Tsang, I., and Sugiyama, M. Sigua: Forgetting may make learning with noisy labels more robust. In Proceedings of the International Conference on Machine Learning, pp. 4006- 4016, 2020.
Cour, T., Sapp. B., and Taskar, B. Learning from partial labels. The Journal of Machine Learning Research, 12: 1501-1536, 2011a
Hu, W., Le, R., Liu, BI, Ma,
Ma, J., Zhao, D., and Yan, R. Predictive adversarial learning from positive and un- labeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7806-7866 00-784444 2021.
Cour, T., Sapp. B., and Taskar, B. Learning from partial labels. The Journal of Machine Learning Research, 12: 1501-1536, 2011b. 1b.
10

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Hüllermeier, E. and Beringer, J. Learning from ambiguously labeled examples. In Proceedings of the International Symposium on Intelligent Data Analysis, pp. 168-179, 2005.
global unannotated categories. In Proceedings of the International Conference on Multimedia Retrieval, pp- 561-569, 2024b.
Li, Z., Wei, M., Ying, P., and Xu, X. ESA: example sieve approach for multi-positive and unlabeled learning. In Proceedings of ACM International Conference on Web Search and Data Mining, pp. 439-447, 2025.
Ishida, T., Niu, G., Hu, W., and Sugiyama, M. Learning from complementary labels. Advances in Neural Infor- mation Processing Systems, 30, 2017.
Ishida, T., Niu, G., Menon, A., and Sugiyama, M. Complementary-label learning for arbitrary losses and models. In Proceedings of the International Conference on Machine Learning, pp. 2971 2971-2980, 2019.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in Neural Information Processing Systems, 36, 2024.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision, pp. 709-727, 2022.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. In Proceedings of the International Conference on Learning Representations, 2019.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19112-19122, 2023.
Menghini, C., Delworth, A., and Bach, S. Enhancing clip Exploring pseudolabeling for limited-label prompt tuning. Advances in Neural Information Pro- cessing Systems, 36:60984-61007, 2023.
with clip:
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d ob-
ject representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 554-561, 2013.
Menon, A.,
Van Rooyen, B., Ong, C. S., and Williamson, Learning from corrupted binary labels via class- probability estimation. In Proceedings of the Interna- tional Conference Machine Learning, pp. 125-134, 2015.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3, 2015.
Miyato, T., Maeda, S.-i., Koyama, M., and Ishii, Vir- tual adversarial training: regularization method for su- pervised and supervipd learning. IEEE Transac- lions on Pallern Analysis and Machine Intelligence, 41 19):197919993, 2018.
Lee, D.-H. et al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural net- works. In Workshop on Challenges in Representation Learning, ICML, volume 3, pp. 896, 2013.
Northcutt, C. G., Athalye, A., and Mueller, J. Pervasive la- bel errors in test sets destabilize machine learning bench- marks. Vanschoren, and Yeung, S. (eds.), Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Lei, Y., Li, J., Li, Z.,, Cao, Y., and Shan, H. Prompt learn- ing in computer vision: survey. Frontiers of Informa- tion Technology & Electronic Engineering, 25(1):42-63, 2024.
Li, J.. Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping tunguage-image pre-training for unified linguael understanding and generation. In Proceedings of the In- ternational Conference on Machine Learning, pp. 12888- 12900, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp. 8748-8788 8763, 2021.
Li, Z.., Wei, M., Ying, P., Sun, T., and Xu, Learning from concealed labels. In Proceedings of the ACM In- ternational Conference Multimedia, pp. 7220-7228, 2024a.
Sajjadi, M., Javanmardi, M., and Tasdizen, T. Regulariza- tion with stochastic transformations and perturbations for deep semi- supervueeeeieed learning. Advances in Neural Information processing Systems, 29, 2016.
Li, Z., Ying, P., Wei, M., Sun, T., and Xu, X. Prompt expending for single positive multi-label learning with

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Sakai, T., Niu, G., and Sugiyama, M. Semi-supervised auc optimization based on positive-unlaheled learning. Machine Learning, 107:767-794, 2018.
Xu, N., Qiao, C., Lv, J., Geng, X., and Zhang, M.-L. One positive label is sufficient: Single-positive multi-label learning with label enhancement. Advances in Neural Information Processing Systems, 01221765-21777 2022.
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Tnfor-
Yu, X., Liu, T., Gong, M., and Tao, D. Learning with biased complementary labels. In Proceedings of the European Conference on Computer Vision, pp. 68 68-83, 2018.
mation Processing Systems, 33:596-608, 2020.
Zhang, M.-L.., Zhou, B.-B., and Liu, X.-Y. Partial label learning via feature-aware disambiguation, In Proceed- ings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1335-1344, 2016.
Sosea, T. and Caragea, C. Marginmatch: Improving semi- supervised learning with pseudo-margins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15773-15782, 2023.
Tang, C.-Z. and Zhang, M.-L. Confidence-ratee discrimi- native partial label learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16816-16825, 2022a.
Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neu- ral Tnformation Processing Systems, 30, 2017.
Zhou, K.. Yang. J., Loy, C. C., and Liu, Learning to
prompt for vision-language models. International Jour- nal of Computer Vision, 130(9))2337-2344 2022b.
Zhou, Z.-H. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-555 53, 2018.
Van Engelen, E. and Hoos, H. H. survey on semi- supervised learning. Machine Learning, 109(2):373-44, 2020.
Zong, Z., Song, G., and Liu, Y. Detrs with collabora- tive hybrid assignments training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6748-6758, 2023.
Wang, W., Ishida, T., Zhang, Y., Niu, G., and Sugiyama, M. Learning with complementary labels revisited: The selected-compe didected-comppetelyy-ee at random setting is more practical. In Proceedings of the International Conference on Ma- chine Learning, 2024.
Wang, X., Wu, Z., Lian, L., and Yu, S. X. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657, 2022.
Wei, M., Zhou, Y., Li, Z. and Xu, X. Class-imbalanced complementary-Laber learning via weighted loss. Neural Networks, 166:555-565. 2023. doi: 10. 1.1016/J.NUNEE7 2023.07.030.
Wei, M., Li, Z., Zhou, Y., and Xu, X. Learning from re- duced labels for long-tailed data. In Proceedings of the International Conference on Multimedia Retrieval, pp- 111-119, 2024.
Xia, S., Lv, J., Xu, N., Niu, G., and Geng, X. Towards effec- tive visual representations for partial-label learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15589-15598, 2023.
Xu, N., Lv, J., and Geng, X. Partial label learning via label enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5557-5564, 2019.
12

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
A. Appendix/ / supplemental material
A.I. Proof of Lamme 3.2
Lemma A.1. Under the TFLs Assumption 3.1, the conditional probabilities pyy = (|rr) can be expressed as:
K
ply = 1|r) =p(ÿ i, = 1/2) +
Σ
= -, 0/z)
(1))
Proof. According to Assumption 3.1, Bayes Rule and Total Probability Theorem,
p(y = d)x)) =p = i,s 1|x) + ply i, = 0/2²)
Σ ply p. 1|x) Σ
0
j=1
K
Σ
ply = -, 1,
1,2)+ +
Σ
ply = i j|s 0) = 0/2²)
j=1
K
=
Σ
plu
- - 0) = j|s = 0, x)p(( = 0/z²
K
=p(ō = i,8 1|))
Σ j=1,
- = j. = 0, j - 0/z)-
A.2. Proof of Theorem 3.3
Theorem A.2. To deal with TF label learning problem, according to the Assumption 3.1 and Lemma 3.2,the classification risk R(f) in Equation (1) could be rewritten as
+ ( ((f RfF(f) = [()
(12)
where E(f(x),D) = E ply il j (f (,,)=
Proof. According to the Assumption 3.1 and Lemma 3.2
( = ppp(,,)) [C(f(x).v))
K
= En
Σ
K
K
= E pm
ply = j.8 = 0: + Ennn Σ -
i= =1,
K
K
K
E -E
plū = j, = 0|z)
Σ im 1
-, + Evana
f=1
K
-Bplr.g.e00) 0)
Σ
ply jjs + 1 ()
im1,5//
= =E -E((,,0.=0) [(f(x), (7) + E FPt.,
13

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
A.3. The specific architecture of the convolutional-based prompt network
As shown in Figure 5, the convolutional-bact based prompt network consists of four CNN blocks, each with varying input and output channel configurations. Within each CNN block, there is convolutional layer with kernel size of and stride of 1, followed by batch normalization layer, a Leaky ReLU activation layer, and dropout layer.
Output
HXW x3
Output
CNN Block
Dropout
HxW xi44
CNN Block
Leaky ReL
HxW x8
Batch Norm
CNN Block
Cony 3x3
HxW x64
CNN Block
Input
HxW XC
Input
(8) Overall framework
(b) The CNN Block
the CNN besed prompt network
Figure 5, Overall framework of the convolutional-basee prompt network
The CNN-based network used in our study contains 150,531 parameters, which is comparable the 150,528 pixels in an image of size 224 x 224 3. Despite similar numbers of parameters, CNN offer significant advantages in capturing local features, unlike direct image optimization, which would require independent updates for each pixel, leading to lack of correlation between pixels and computational inefficiency. Additionally, CNN offer key advantages like reduced computational complexity through parameter sharing and parallel execution on hardware accelerators.
A.4. Overall algorithm procedure
Algorithm illlustrates the overall algorithm procedure. Through this process, we can learn a high-quality linear classifier and a convolutional-based prompt network. This convolutional-based prompt network retrieve multi-modal prompts to bridge the gap between knowledge of VLMs and target learning tasks. ensure stable optimization of the parameters the convolutional-basee prompt network, we introduce hyperparameter to control the process.
A.5. The details of datasets
In this section, we provide a detailed description of datasets used in our experiments.
CIFAR-100(Krrihevskky et al., 2009): A coarse coarse-grained dataset comprising 60,000 color images divided into 100 classes. Each image given 32x3 format, and each class contains 500 training images and 100 test images.
ImagemmaeeNee(Le & Yang. 2015): A coarse-grained dataset consists of 100,000 color images divided into 200 classes. Each image is given in a 64x3 format, and each class contains 500 training images, 50 validation images and 50 test images.
et al., 2004): A coarse-grained dataset comprises images from 101 object categories and a background category that contains the images not from the 101 object categories. Each object category contains approximately 40 to 800 images, with most classes having about 50 images. The image resolution is approximately 300x200 pixels.
14

---

Learning from Truc-False Labels via Multi-modal Prompt Retrieving
Algorithm 1 TFL learning via MPR
Input: The TF labeled training set Drp = ((r,, (Wt, +=) The convoluiional-basel prompt network gean (-); matrix M. whose elements all The CLIP's image encoder gr (-); The number of epochs T The Stability Optimization hyperparameter m;
Output: Model parameter 01 for the linear classifier; Model parameter 0g for Icnn
1: for i to 7' do
2: Shuffle DrF = {(r; (Ti, st)) into B mini-batches;
3: Up 91 (M));
4: Calculate eT and by by Eq.(7);
5: for b 0 B do
6: Fetch mini-batch DA from DrF;
7: Calculate ply y = j. 0, by Eq.(10);
8: Update the linear classifier's parameters 01 by Arr in Eq,(5);
9:
if (%mm = 0 then
10: Update the convolutional-base prompt network's parameters 0g by RTF in Eq.(5);
11: end
end for end for
13:
(rod-101(Bosse (Bossad et al., 2014): fine-grained dataset in the food domain, comprising 101,000 images divided into 101 food categories. Each class contains 750 training images and 750 test images. The labels for the test images have been manually cleaned, while the training set contains some noise.
Stanford Cars(Krause et al., 2013): fine-grained dataset in the car domain, comprising 16,185 images categorized into 196 car classes. The data is divided into almost 50-50 train/test split with 8, 144 training images and 8,041 testing images. Categories typically the level of Make, Model, Year. The images 360x240 pixels.
15


Reviews:
[{"review_id": "uxWx4LYh5M", "reviewer": "Reviewer_hKGd", "summary": "This paper introduces a weakly supervised learning framework that leverages True-False Labels (TFLs) to enhance annotation quality and efficiency. In this setting, each instance receives a binary label indicating whether it belongs to a randomly sampled candidate class, thereby mitigating errors common in conventional vision-language model outputs. The authors derive a risk-consistent estimator to fully exploit the conditional probability distribution of TFLs. Furthermore, a convolutional-based Multi-modal Prompt Retrieving (MPR) method is proposed to effectively align pretrained vision-language model knowledge with target learning tasks, addressing the inherent label noise issue.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive comments. We will address each concern point by point:  \n**Experimental Designs (Ablation study on the hyperparameters $K_T$ and $K_I$)**  \nR1: We conducted comprehensive ablation studies on the hyperparameters $K_T$ and $K_I$. Experimental results demonstrate that performance variation remains within 0.5% across all datasets when adjusting $K_T$ (text prompts) and $K_I$ (image prompts) between 5-20. This stability stems from the inherent design of our MPR ($Top-K(\\cos(\\cdot, \\cdot))$), where retrieved embeddings exhibit high cosine similarity and contain domain-specific information relevant to downstream tasks.  \n\n**W1-1 (Underrepresentation of rare classes)**  \nR2-1: In real-world annotation scenarios, rare classes are generally unknown a priori. Deliberate selection of rare classes risks inducing additional imbalance, which could exacerbate annotation bias. Under these practical constraints, the uniform random sampling in TFL represents the most feasible solution, as it aligns with practical annotation workflows while reducing labeling complexity and noise.  \n\n**W1-2 (Model's performance on less frequent categories)**  \nR2-2: TMP addresses class imbalance through the risk-consistent estimator, which reweights less frequent categories using CLIP's prior knowledge preserves recognition capabilities for rare classes. Empirical validation across all five datasets—including naturally imbalanced ones like Caltech-101—confirms competitive performance (**90.60% accuracy**).  \n\n**W2 (Architecture Choice - Convolutional Limitations)**  \nR3: Our convolutional MPR design is motivated by two key considerations:  \n1) **Characteristics of Retrieval**: As MPR focuses on retrieval rather than capture complex semantic relationships, the extraction of task-relevant information benefits more from local feature matching than complex semantic relationships. For prompt retrieval, the experimental results show that demonstrates that local texture patterns (captured by CNNs) are more effective than global attention mechanisms (by 3%).  \n2) **Computational Efficiency**: CNNs achieve faster training speed compared to Transformers, which is crucial for weakly supervised scenarios with limited computational budgets.  \n\n**W3 (The binary labeling scheme might lack the granularity required to distinguish subtle differences)**  \nR4：In real-world annotation scenarios involving subtle differences, the binary labeling scheme proves simpler to implement than multi-class annotation. Annotators only need to determine whether a randomly provided candidate label is correct, rather than selecting one from many visually or semantically similar options. Compared to conventional labeling methods that require annotators to distinguish between nuanced categories, TFL significantly reduces the skill requirements for annotators while maintaining theoretical rigor. This approach is particularly advantageous for datasets with high inter-class similarity (e.g., Stanford Cars), where traditional labeling often struggles with ambiguity.  \n\n**Question (Hyperparameter Stability)**  \nR5: In our experiments, the observed sensitivity is not a universal phenomenon. We attribute λ's instability to CLIP's zero-shot capability: weaker zero-shot performance (e.g., Stanford Cars) introduces noisier conditional probability estimates $p(y|\\bar{y}, s=0, x)$ forcing greater reliance on CLIP’s prior $P_{\\text{CLIP}}$ and amplifying sensitivity.", "claims_and_evidence": "Overall, the submission's claims are clear and convinced.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria seem to be suitable for the problem at hand.", "theoretical_claims": "I’ve glanced through the proofs for the theoretical claims, but I didn’t rigorously verify their accuracy.", "experimental_designs_or_analyses": "Missing an ablation study on the hyperparameters K_T and K_I. Without this analysis, it remains unclear how sensitive the model’s performance is to the number of retrieved text and image embeddings.", "relation_to_broader_scientific_literature": "The paper advances weakly supervised learning by introducing True-False Labels with multi-modal prompt retrieving, extending unbiased risk estimation and leveraging vision-language prompt techniques.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper introduces True-False Labels, reducing annotation cost and label noise by using binary decisions on randomly sampled candidate labels, thereby simplifying the labeling process.\n2. Extensive experiments across diverse datasets demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. The random sampling of candidate labels assumes a uniform distribution, which may fail to reflect the natural class imbalance often present in real-world datasets. This could lead to underrepresentation of rare classes, potentially impacting the model’s performance on these less frequent categories.\n\n2. The convolutional-based prompt retrieval approach, while efficient, might not capture complex semantic relationships between visual and textual modalities as effectively as transformer-based architectures, potentially limiting its expressive power.\n\n3. In datasets with high inter-class similarity, the binary labeling scheme might lack the granularity required to distinguish subtle differences. This could lead to confusion between similar classes, resulting in an increased rate of misclassification.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "The hyperparameter λ balances the contributions from the pre-trained model and the learned model. As illustrated in Fig. 3, this parameter is highly sensitive on Stanford Cars. I'm curious whether this instability and inconsistent performance across different datasets is a common phenomenon.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kj3AOPdCsg", "reviewer": "Reviewer_ry5t", "summary": "The paper proposes TFLs, a weakly supervised framework leveraging VLMs to generate high-accuracy labels efficiently. A risk-consistent estimator exploits TFLs’ conditional probabilities, and MPR aligns VLMs with target tasks. Experiments show significant gains over baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive comments. We will address each concern point by point:  \n**Q1: Class Imbalance in TFLs**  \nR1: We appreciate the reviewer's insightful observation regarding class imbalance. The imbalance between True and False labels is an inherent characteristic of the TFL framework. However, we have taken this into account in our design. The proposed risk-consistent estimator in the TMP is specifically designed to mitigate the effects of such imbalance. By estimating the probability distribution over categories, the estimator effectively reduces the bias introduced by the overrepresentation of False labels, ensuring balanced learning outcomes. The experimental results show that this approach enables the model to handle label imbalance without significant degradation in performance.  \n\n**Q2: Multi-Label/Open-Vocabulary Extensions**  \nR2: We thank the reviewer for identifying this impactful research direction.\n1. For **multi-label classification**, we can extend the single positive label framework [1] by assigning one True label (indicating presence) or one False label (indicating absence) per candidate label for each instance.  \n2. For **open-vocabulary scenarios**, the samples that do not have an overwhelming advantage in the highest output confidence of the model can be selected as the new class samples, i.e., the samples have relatively high confidence in multiple categories. We conducted preliminary experiments in which the samples belonging to the new class were input into the already trained model. The experimental results indicated that the recognition accuracy for the new class was approximately 54%.  \n\n**W1: Single Judgment Limitation**  \nR3: We appreciate this thoughtful suggestion. We respectfully clarify that while multiple judgments could theoretically help, our experiments show:  \n1. **Diminishing Returns**: Dual judgments only improve accuracy by 0.3% on CIFAR-100/Stanford Cars.  \n2. **Efficiency Trade-off**: Each additional judgment linearly increases annotation costs, contradicting TFL's efficiency goals.  \n\n**W2: Fine-Grained Performance Gap**  \nR4: We gratefully acknowledge this valuable critique. We agree this warrants deeper investigation. Two key factors explain the gap:  \n1. **Semantic Overlap**: Fine-grained classes (e.g., car) share more visual features than coarse ones (e.g., animals), reducing CLIP's discriminative power.  \n2. **Prompt Sensitivity**: As shown in [2], class descriptions prompt from LLM may improve this gap.  \n\nReference:  \n[1]\tZhou D, et al. Acknowledging the unknown for multi-label learning with single positive labels, ECCV, 2022.  \n[2]\tPratt S, et al. What does a platypus look like? generating customized prompts for zero-shot image classification, ICCV, 2023.", "claims_and_evidence": "The claims are supported clearly in the current form.", "methods_and_evaluation_criteria": "They make sense to some extent.", "theoretical_claims": "I check them.", "experimental_designs_or_analyses": "I check them", "relation_to_broader_scientific_literature": "Refer to the detailed comments.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**  \n1. The integration of VLMs with weakly supervised learning is a compelling contribution. The method achieves a high annotation accuracy of over 99.5% in experiments while significantly reducing human effort (69.9× speedup compared to traditional labeling).  \n2. The paper is well-structured, with a persuasive narrative that clearly motivates the problem and technical contributions. A detailed theoretical proof of the risk-consistent estimator is provided, solidifying the method’s foundation. Experimental results demonstrate the effectiveness of the proposed TFL setting and MRP learning method.  \n3. The MPR method represents the first attempt to fine-tune VLMs through prompt retrieval (as opposed to directly learning textual embeddings), offering a fresh perspective on prompt engineering.  \n**Weaknesses:**  \n1. Each TFL provides only a single binary judgment (True/False) per candidate label. While efficient, this may restrict the richness of supervision, particularly for ambiguous or fine-grained classes. Extending the framework to allow multiple judgments (e.g., sampling multiple candidates per instance) could enhance its robustness.  \n2. Although experiments include fine-grained datasets (e.g., Stanford Cars), the gains here are slightly smaller than those for coarse-grained tasks. Further analysis on how class granularity impacts performance would enhance understanding.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. By design, the number of \"False\" labels in TFLs will vastly exceed \"True\" labels, especially as the candidate label set grows. How does the proposed method address potential class imbalance during training?  \n2. Extending TFLs to multi-label classification or open-vocabulary scenarios appears promising. Could the authors elaborate on potential extensions in future work?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKTqVBoxZa", "reviewer": "Reviewer_X5ci", "summary": "This paper proposes a novel weakly supervised setting called True-False Labels (TFLs), leveraging VLM to reduce the difficulty of manual annotation. TFLs indicates whether a sample belongs to a label randomly and uniformly sampled from a candidate label set. In addition, this paper derives a risk-consistent estimator to explore and utilize the conditional probability distribution information of TFLs and introduces a Multimodal Prompt Retrieval to bridge the gap between VLM knowledge and the target learning task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive comments. We will address each concern point by point:  \n**Q1: Assumption 1 appears problematic**  \nR1: In fact, Assumption 1 emphasizes that consistent classifiers can learn from pre-existing VLM-generated or human-annotated TFLs data, i.e., TFLs consistent learn assumption, rather than focus on the results of labeling the data must be perfect.  \nFurthermore, high-performance VLMs exhibit minimal annotation errors, with accuracy levels comparable to or surpassing human performance [1,2]. This suggests that they can satisfy the TFLs consistent learn assumption. Besides, in real-world scenarios, the goal of TFLs labeling is to mitigate noise introduced by VLMs-based annotations, ensuring that the labeled data adheres to the TFLs consistent learn assumption as closely as possible.  \n\n**Q2&W1: Motivation for MPR requires clarification**  \nR2: We have described the motivation for MPR in lines 104-109 and 290-294, among others, in the paper. More details are as follows:  \n1) **Weak Supervision Enhancement**: MPR supplements discriminative features from weakly supervised data through cross-modal retrieval, reducing learning complexity while improving model robustness.  \n2) **Task-Specific Adaptation** (Section 3.4): Our learnable convolutional network retrieves domain-aware embeddings (e.g., culinary textures in Food-101), addressing CLIP's generic prompt limitations.  \n3) **Modality Alignment** (Eq.7-9): By dynamically aggregating Top-K visual and textual prompts, MPR enhances modality consistency while preserving VLM knowledge.  \nAblation studies (Table 4) confirm MPR contributes 1.07% average accuracy improvement across datasets. We will strengthen the methodological motivation in Section 1.  \n\n**Q3: Equation (9) image encoder clarification**  \nR3: The revised formulation is:  \n$P_{CLIP} = \\text{Softmax}(\\cos(g_I(x), \\mathbb{Q}_T))$  \nwhere $g_I(\\cdot)$ denotes CLIP's image encoder. We will update Eq.9 and ensure consistency in all algorithm descriptions.\n\n**Q4: Effectiveness of TMP**  \nR4: As shown in Tables 2&3, TMP achieves optimal performance across all datasets. Notably, optimal performance never occurs at λ=0. This demonstrates that our balanced integration strategy (λ=0.1) optimally combines VLM knowledge with task-specific adaptation, thereby validating TMP's effectiveness.  \n\n**Q5: Line 372's \"approaching fully supervised\" claim may be overstated**  \nR5: Compared to baseline methods, our approach achieves performance approaching fully supervised levels on specific datasets. For example, on the Food-101 dataset (93.55% vs. 94.94% fully supervised). However, a 15% performance gap persists on the Stanford Cars dataset. We have revised the statement to: \"Our method achieves performance comparable to fully supervised approaches on Food-101.\"\n\n**Q6: Hyperparameter m determination**  \nR6: These hyperparameter $m$ were selected through experimentation to strike a tradeoff between precision and efficiency (when m=1, the accuracy is the highest, but the training cost is relatively high).  \nReference:  \n[1] Street W, et al. LLMs achieve adult human performance on higher-order theory of mind tasks, arXiv, 2024.  \n[2] Kapania S, et al. \"Because AI is 100% right and safe\": User attitudes and sources of AI authority in India, CHI, 2022.", "claims_and_evidence": "The paper presents a relevant and important problem, and the claims made are generally supported by evidence. However, the motivation behind the proposed framework is not sufficiently clear, making it difficult to fully understand how the proposed approach can effectively address the problem. The lack of clarity regarding the framework's motivation undermines the convincingness of the presented evidence and solution.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are suitable for the problem and application at hand.", "theoretical_claims": "Yes. Assumption 1 does not account for the case where the TFL is incorrect. Other parts of the proofs seem to be correct with no issues.", "experimental_designs_or_analyses": "The analysis of the hyperparameter $\\lambda$ lacks convincing evidence, as the performance improves when its value is small. This raises concerns about the lack of support for the effectiveness of TMP.", "relation_to_broader_scientific_literature": "The paper builds upon two key areas of research: weakly supervised learning and vision-language models (VLMs). It primarily leverages the strong generalization ability of VLMs to address weak supervision and proposes a novel annotation strategy. In terms of problem formulation, the work is more closely related to prior research on pseudo-label learning.", "essential_references_not_discussed": "I have not identified any essential related works that are missing from the paper at this time.", "other_strengths_and_weaknesses": "**Strengths:**\n1. This paper proposes a novel weakly supervised learning annotation method, expanding the scope of weak supervision approaches.\n2. The integration of VLMs to address weak supervision is an effective combination and is a promising research direction.\n\n**Weaknesses:**\n1. The introduction spends a significant portion explaining label annotation, which makes the latter part of the introduction insufficient and difficult to understand the motivation. Additionally, there is a lack of smooth transition to the methodology, making the proposed method less understandable.\n2. The overall organization of the paper could be improved in terms of readability. In addition, Table 1 lacks necessary explanations, making it difficult to understand the meaning of certain columns. Improving readability would enhance clarity.", "other_comments_or_suggestions": "Please refer to the Questions For Authors.", "questions_for_authors": "1. Assumption 1 appears problematic, as it seems to disregard cases where TFL are incorrect.\n2. Why was prompt retrieving designed? The method lacks sufficient explanation, and I did not fully understand the motivation behind introducing the TMP framework. Could this be clarified by providing further details in the introduction and starting from Section 3.4?\n3. In Equation (9), does $x$ not need to go through the image encoder? \n4. When the hyperparameter $\\lambda$ is small, the performance improves, but this seems to lack evidence supporting the effectiveness of TMP. \n5. In right-hand side of line 372, the paper states that \"our approach has achieved results approaching those of the fully supervised method\". Does this refer to the first row of the table 3? If yes, there is still a noticeable gap, so it may be worth reconsidering whether this statement is appropriate.\n6. In line 718 of the appendix, how is the hyperparameter $m$ determined? This is not explained in the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fZPZCKMZVA", "reviewer": "Reviewer_peko", "summary": "This paper propose a novel weakly supervised labeling setting, namely True-False Labels (TFLs) which can achieve high accuracy when generated by pre-trained Vision-Language Models (VLM). Moreover, the paper derived a risk-consistent loss for this setting and propose a convolutional-based Multi-modal Prompt Retrieving (MRP) method to bridge the gap between the knowledge of VLMs and target learning tasks.  Experimental results demonstrate the effectiveness of the proposed TFL setting and MRP learning method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive comments. We will address each concern point by point:  \n**Q1&Method (Novelty & Comparison with Self-Training)**  \nR1: Our TFL framework introduces three key innovations that fundamentally address limitations in standard self-training and prior work like Candidate Pseudo-Label Learning[1]:  \n1. **Random Label Sampling as Implicit Regularization**:  \nTFL employs uniform random label sampling as implicit regularization to prevent model overfitting. This stochastic mechanism in TFL, analogous to established techniques like dropout, SGD and random data augmentation, promotes better generalization solutions [3]. Unlike confidence-based pseudo-labeling in self-training and [1], which amplifies CLIP's inherent biases through error propagation [1,2].  \n2. **High-Accuracy TFL for Bias Amplification Mitigation**  \nPrior studies [2,3] have demonstrated that pseudo-labeling noise can create a cumulative effect on classes with inaccurate pseudo-labels, amplifying the model's inherent bias toward certain classes. The experimental results in Table 6 of [1] further support this conclusion, showing that accuracy improvements tend to be more pronounced on datasets where Zero-shot CLIP initially performs better. This observation aligns with TFL’s design motivation to mitigate bias amplification through stochastic sampling. TFL achieves over 99% annotation accuracy compared to the 85% accuracy shown in [1] (Fig.2). This significant improvement in labeling precision (via confidence-ranking-independent label generation) substantially reduces noise propagation.  \n3. **Hybrid Supervision Mechanism**  \nTFL integrates strong supervision from retained true labels to provide semantic correction anchors, whereas [1] relies solely on candidate pseudo-label. The utilization of hybrid supervision is mathematically formalized through our risk-consistent estimator (Eq.5). This hybrid labels provide explicit optimization direction, enhances noise robustness and improves stability in complex scenarios.  \n\n**Q2&E1 (Poor Performance of Weak Supervision Baselines)**  \nR2: We rigorously verified our implementation (code available at [Anonymous GitHub](https://anonymous.4open.science/r/TMP-2D10)) under identical configurations (epochs, architecture, hyperparameters). The observed performance gap primarily arises from **Large Label Space Challenges**:  \n- **TFL data contains numerous unseen class for semi-supervised learning** (e.g., 196 classes with only 42 supervised samples in Stanford Cars). This results in most classes lacking supervised samples. Existing semi-supervised methods for unknown classes fail to handle such a high proportion of unseen categories, leading to catastrophic performance collapse.  \n- **The candidate sets generated by TFL are excessively large for partial/complementary-label learning**. Oversized candidate sets (>100 classes) severely degrade disambiguation capabilities. In the weak supervision community, Complementary-label learning[5] and partial-label learning[6] typically uses candidate sets containing ≤30 classes.  \nOur method overcomes this through:  \n   - MPR(Sec. 3.4)  \n   - CLIP prior integration(Eq. 9-10)  \n\n**E2&Method (Baseline Comparisons)**  \nR3: In our experiments, we have conducted comprehensive comparisons with relevant works (Tables 2&3), consistently demonstrating superior performance. We further expanded comparisons to include: Few-shot methods(Tip-Adapter), Semi-supervised VLM fine-tuning approaches(FineSSL), More sophisticated methods utilizing complementary labels(DebiasPL), Advanced complementary-label methods(SCARCE). As shown below:  \n| Method | CIFAR-100 | Caltech-101 |\n|:---:|:---:|:---:|\n| Tip-Adapter  | 76.47 | 86.77 |\n| Tip-Adapter-F | 77.40 | 86.81 |\n| FineSSL | 67.28 | 28.72 |\n| DebiasPL | - | 63.40 |\n| SCARCE | 44.06 | 39.39 |\n| CPL+LaFTer | 77.30 | 93.40 |\n| TMP | 78.72 | 90.60 |\n\nNotably, TMP achieves comparable performance to CPL+LaFTer on CIFAR-100 and Caltech-101, despite the fact that CPL+LaFTer requires more resources, such as leveraging additional LLM knowledge, and utilizes iterative pseudo-label refinement (T=10). These results conclusively validate TMP's effectiveness through its noise-robust probability estimation framework.  \nReference:  \n[1]\tZhang J, et al. Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data, ICML, 2024.  \n[2]\tMenghini C, et al.. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning, NeurIPS, 2023.  \n[3]\tWang X., et al. Debiased Learning from Naturally Imbalanced Pseudo-Labels, CVPR, 2022.  \n[4]\tAli A, et al. The implicit regularization of stochastic gradient flow for least squares, ICML, 2020.  \n[5]\tWang, Wei et al. Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical, ICML 2024.  \n[6]\tXia S, et al. Towards effective visual representations for partial-label learning, CVPR 2023.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods have some limitations that reduce the significance of the paper's contributions:\n\nFirstly, the proposed weak supervision setting appears to be a straightforward combination of full supervision and complementary labeling. The authors do not sufficiently justify why and how this particular weak supervision setting is advantageous for VLM fine-tuning compared to existing weak supervision setting. For instance, prior work such as [1] using partial label setting for fine-tune VLM. The paper would benefit from a more thorough comparison and analysis of how the proposed approach improves upon or differs from existing weak supervision setting in the context of VLM fine-tuning.\n\nThe proposed  methods lacks novelty, as it is essentially equivalent to the widely-used self-training approach in weak supervision learning. Specifically, the proposed method is essentially equivalent to a two-stage self-training approach. In the first stage, complementary labels or ground-truth labels are assigned to the samples. In the second stage, the model learns from the samples with complementary labels by utilizing the predicted posterior probabilities. This formulation aligns closely with conventional self-training frameworks, where pseudo-labels are iteratively generated and refined to improve model performance. The authors should clarify how their approach differs from or advances beyond this well-established paradigm, as the current presentation does not sufficiently highlight novel methodological contributions.\n\nIn summary, while the paper explores an interesting direction, the lack of significant novelty in the proposed methods and insufficient comparison to existing approaches reduce the overall impact of the work. The authors should address these limitations by providing a more thorough analysis of how their approach advances the state-of-the-art real-world VLMs fine tune.\n\n[1] Zhang, Jiahan et al. Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data. ICML (2024).", "theoretical_claims": "Yes.  All proofs for theoretical claims in this  paper are correct.", "experimental_designs_or_analyses": "Yes, I have carefully reviewed the experimental designs and analyses, and several issues need to be addressed to ensure the soundness and validity of the results:  \n1. **Unusual Weak Supervision Results**: The results of the weakly supervised learning methods in Table 2 and Table 3 appear to be quite unusual. Specifically, the fine-tuned results are significantly worse than the zero-shot results, which is counterintuitive and raises concerns about the validity of the experimental setup or implementation. The authors should provide more detailed training configurations and hyper parameters for these methods to ensure reproducibility. Additionally, they should carefully examine whether the poor performance is caused by suboptimal training details, such as learning rates, optimization strategies, or data pre-processing. Without a thorough investigation and clarification, the credibility of the reported results remains questionable.  \n2. **Insufficient Baseline Comparisons**: The current experimental design lacks comprehensive comparisons with relevant baselines, which limits the ability to assess the effectiveness of the proposed method. Specifically:  \n   - For only using supervised samples, some state-of-the-art few-shot fine-tuning methods such as [1] should be included as baselines.  \n   - For the semi-supervised setting, advanced semi-supervised methods specifically designed for fine-tuning models [2, 3] should be compared.  \n   - For complementary labeling, more sophisticated methods utilizing complementary labels such as [4] should be incorporated   \n   - Finally, self-training methods under other weak supervision settings [5] should also be added to the comparison.  \n\nBy expanding the comparison to these relevant baselines, the authors can provide a more comprehensive evaluation of their method's performance and better highlight its potential advantages over existing approaches. Without such comparisons, it is difficult to assess whether the proposed method truly advances the state-of-the-art for VLMs fine-tuning.  \n\nIn summary, while the paper explores an interesting direction, the experimental design and analysis need significant improvements to ensure the validity and soundness of the results. Addressing these issues would strengthen the paper's contributions and provide a more convincing evaluation of the proposed method.  \n\nReference:\n\n[1] Zhang, RenRui et al. Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling\n\n[2] Gan, Kai ea al. Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning\n\n[3] Wang, XuDong et al. Debiased learning from naturally imbalanced pseudo-labels\n\n[4] Wang, Wei et al. Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical\n\n[5] Zhang, Jiahan et al. Candidate Pseudo label Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data", "relation_to_broader_scientific_literature": "The key contributions of this paper are related to the broader scientific literature in the following ways:\n\n1.**Novel Weak Supervision Annotation (True-False Labels)**: Compared to prior work, such as [1], this paper proposes a new weak  supervision annotation scheme—True-False labels—specifically designed for fine-tuning foundation models. This annotation can be automatically generated by VLMs, which distinguishes it from traditional weak supervision approaches that often rely on manual or heuristic labeling. This contribution addresses a gap in the literature by providing a more scalable and efficient way to generate weak supervision signals for fine-tuning.  \n2. **Fine-Tuning Method for True-False Labels**: Building on the proposed annotation scheme, the authors design a fine-tuning method tailored to True-False labels, which demonstrates competitive performance. This methodological advancement extends the existing literature on weak supervision by showing how such annotations can be effectively utilized to improve model performance. The results suggest that True-False labels, despite their simplicity, can serve as a viable alternative to more complex weak supervision strategies.  \n\nReference:\n\n[1]  Zhang, Jiahan et al. Candidate Pseudo label Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data. ICML (2024).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Other Strengths:\n1. **Novel Weak Supervision Annotation Scheme**: The introduction of True-False labels as a new weak supervision annotation method is a creative and potentially impactful contribution. This approach leverages the capabilities of Vision-Language Models (VLMs) to automate the annotation process, offering a scalable and efficient alternative to traditional weak supervision techniques.  \n2. **Competitive Performance**: The proposed fine-tuning method, tailored to the True-False labels, demonstrates competitive performance on the evaluated benchmarks. This suggests that the method is effective in leveraging weak supervision signals to improve model performance.  \n3. **Relevance to Broader Research Trends**: The work aligns with the growing interest in reducing reliance on expensive manual annotations while maintaining or improving model performance. The proposed method could inspire further research into automated weak supervision strategies for fine-tuning foundation models.", "other_comments_or_suggestions": "I don't have any further comments of suggestions.", "questions_for_authors": "**Question 1: Comparison with Standard Self-Training and Prior Work**  \n\nThe proposed method appears to be essentially equivalent to a two-stage self-training approach:  \n1. In the first stage, complementary labels or ground-truth labels are assigned to the samples.  \n2. In the second stage, the model learns from the samples with complementary labels by utilizing the predicted posterior probabilities.  \n\nWhile this formulation shares similarities with standard self-training and prior work such as [1], it is unclear why the proposed method is superior. Specifically:  \n- **Compared to standard self-training**: What are the advantages of using complementary labels (True-False labels) over traditional pseudo-labels? Is there a theoretical or empirical justification for why this approach leads to better performance or faster convergence?  \n- **Compared to [1]**: How does the proposed method improve upon or differ from the weak supervision strategies explored in [1]? For instance, does the use of True-False labels provide better robustnessor more efficient utilization of unlabeled data?  \n\nI hope authors provide a more in-depth discussion, supported by either theoretical analysis, empirical results, or intuitive insights, to clarify the unique benefits of their approach over existing methods. \n\n**Question 2: Poor Performance of Previous Weak Supervision Methods**  \n\nThe results of previous weak supervision methods, as shown in Table 2 and Table 3, are notably poor—even worse than the zero-shot performance. This raises significant concerns and requires further explanation:  \n1. **Potential Causes**: Could the poor performance be attributed to suboptimal training configurations, such as inappropriate learning rates, insufficient training epochs, or inadequate hyper parameter tuning? Alternatively, is there an inherent limitation in the design of these weak supervision methods that makes them unsuitable for the task or dataset at hand?  \n2. **Proposed Method's Advantages**: The authors should clearly explain how their training method differs from previous weak supervision approaches and why it avoids the pitfalls that led to the poor performance of those methods.\n\nReference:\n\n[1] Zhang, Jiahan et al. Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data. ICML (2024).\n\nI appreciate the authors' efforts in this work. I would like to raise my score if my concerns were addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After review, the paper received three positive and one negative ratings. Following the authors' rebuttal, one reviewer increased the rating, while the other three maintained their original scores. All reviewers acknowledged the paper's contributions and expressed satisfaction with its technical merits.

The AC concurs with the reviewers' assessments and recommends acceptance.
Decision: Accept (poster)