Unlocking the Power of Co-Occurrence in CLIP: A DualPrompt-Driven Method for Training-Free Zero-Shot Multi-Label Classification
Authors: Ming-Kun Xie
Zhiqiang Kou
Zhongnian Li
Gang Niu
Masashi Sugiyama
Venue: ICLR 2026 Poster (2026)
Keywords: Zero-shot multi-label classification
co-occurrence
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Contrastive Language-Image Pretraining (CLIP) has exhibited powerful zero-shot capacity in various single-label image classification tasks. However, when applying to the multi-label scenarios, CLIP suffers from significant performance declines due to the lack of explicit exploitation of co-occurrence information. In pretraining, due to the contrastive property of its used objective, the model focuses on the prominent object in an image, while overlooking other objects and their co-occurrence relationships; in inference, it uses a discriminative prompt containing only a target label name to make predictions, which does not introduce any co-occurrence information. Then, an important question is as follows: \textit{Do we need label co-occurrence in CLIP for achieving effective zero-shot multi-label learning?} In this paper, we propose to rewrite the original prompt into a correlative form consisting of both the target label and its co-occurring labels. An interesting finding is that such a simple modification can effectively introduce co-occurrence information into CLIP and it exhibits both good and bad effects. On the one hand, it can enhance the recognition capacity of CLIP by exploiting the correlative pattern activated by the correlative prompt; on the other hand, it leads to object hallucination in CLIP, where the model predicts objects that do not actually exist in the image, due to overfitting to co-occurrence. To address this problem, we proposed to calibrate CLIP predictions by keeping the positive effect while removing the negative effect caused by suspicious co-occurrence. This can be achieved by using dual prompts consisting of the discriminative and correlative prompts, which introduce label co-occurrence while emphasizing the discriminative pattern of the target object. Experimental results verify that our method can achieve performance than the state-of-the-art methods.

Published as a conference paper at ICLR 2026
UNLOCKING THE POWER OF CO-OCCURRENCE IN CLIP: A DUALPROMPT-DRIVEN METHOD FOR TRAINING-FREE ZERO-SHOT MULTI-LABEL CLAS- SIFICATION
Ming-Kun
Kou", Zhongnian Li', Gang Niu', Masashi Sugiyama 1.3 RIKEN Center for Advanced Intelligence Project 2 Hong Kong Polytechnic University
The University of Tokyo China University of Mining and Technology
mi nun. xieê zieeriie ri mer.kken.33
(zgiang, kou, gang miu. ) gma i..iil... com shongnianle feuutt. edu.cn sugiek.u-tokyo.as. u-tokyo. ac.
ABSTRACT
Contrastive I. Language-Image Pretraining (CLIP) has exhibited powerful zero-shot capacity in various single-label image classification tasks. However, when ap- plying to the multi-label scenarios, CLIP suffers from significant performance declines due to the lack of explicit exploitation co-occurrence information. pretraining, due to the contrastive property of its used objective, the model fo- cuses on the prominent object in an image, while overlooking other objects and their co-occurrence relationships; in inference, containing only target label name to make predictions, which does not intro- duce any co-occurrence information. Then, an important question as follows: Do we need label co-occurrence in CLIP for achieving effective zero-shot multi- label learning? In this paper, we propose to rewrite the original prompt into a correlative form consisting of both the target label and its co-occurring labels. An interesting finding is that such a simple modification can effectively introduce co-occurrence information into CLIP and it exhibits both good and bad effects. On the one hand, can enhance the recognition capacity of CLIP by exploiting the correlative pattern activated by the correlative prompt; the other hand, it leads to object hallucination in CLIP, where the model predicts objects that do not actually exist in the image, due overfitting co-occurrence. To address this problem, we proposed to calibrate CLIP predictions by keeping the positive effect while removing the negative effect caused by suspicious co-occurrence. This can be achieved by using dual prompts consisting of the discriminative and correlative prompts, which introduce label co-oceurrence while emphasizing the discrimina- tive pattern of the target object. Experimental results verify that our method can achieve better performance than the state-of-the-art methods.
uses a discriminative prompt
INTRODUCTION
Vision-Language Models (VLMs) trained on massive cheaply collected data have demonstrated im- mense potential in many realistic tasks, e.g., object detection (Gu et al. 2022), semantic segmen- tation Lin et al. (2023), and anomaly detection (Zhou et al., 2024). As representative model, CLIP performs pretraining by aligning large-scale image-text using contrastive objective, showing strong zero-shot generalization ability in various downstream tasks. To perform zero-shot image classification, vanilla strategy is to expand each label name with of prompt templates, e.g., "A photo of a {label] }", showing impressive zero-shot classification performance in many benchmark datasets Radford al. (2021). Although CLIP has achieved successes in the single-label zero-shot classification tasks, it often suffers from unfavorable performance when applied to more realistic multi-label scenarios, where an image is often associated with multiple labels.
"Corresponding author

---

Published as a conference paper at ICLR 2026
microwave
buste
bestl
0::0
0.22
0.03
0.35
0.13
1.00
0.59
0.06
0:40
0.02
0.2
0:02 0.04
0.31
0.16
0:34
0.25
oreet
0.65
0.03
0.29
0.20
ores
col)
0.33
0.03
0.0
0:05
0.06
7,00
0.21
6.21
0.31
toanter
0.00
0.05
0.00
0.00
0.42
lost
0:15
0.00 0.11
0.35
0.90
rofnigreier
sink
0.05
6.11
0.01
0.54
0.06
slok
0.14
1.00
koifs
0.04
0.54
0.03
0.13
0:52
0.94
kaita
0.52
0.46
9.41
1.00
0:27
0.34
0.15
0.25
0.01
9.20
0.56
reffigurate
0.26
0.08
spenn
0.00
a.cs
0.04 0.8 9.20 0.13
speat
0.54
0.55
0.32
0.32
2.00
9:00
miceow
orea
teaster
slnk
refrigorator
microaate
Dr60
roastor
slok
reffigerator
bowl
now
0. wos
0.10 0:21
bow!
0.51
0.19
0.2
2:30
bottl
tack knfe
specns
bool
bottle
Lack
knffe
bost
(a) CLIP Predictions
(b) True Labels
(c) CLIP Predictions
(d) True Labels
Figure I: The co-occurrence probabilities of CLIP predictions and true labels for groups of classes.
Without explicitly exploiting label correlations, CLIP is hard to preciscly model the co-occurrence among labels.
There are two major challenges in achieving effective zero-shot multi-label classification. As pointed by (Lin et al., 2024), the first challenge is that CLIP tends to capture the global features of entire image dominated by the most prominent object, while neglecting the local features of others. This limits its ability to recognize multiple objects simultaneously, often leading to the precise recognition of only the most prominent object while overlooking others. To address this, TagCLIP (Lin et al.) 2024) adopted the Vision Transformer (ViT) (Dosovitskiy et al.] (202I) as the backbone of CLIP to explore the local features of patches, which are usually useful for identifying inconspicuous objects. However, one limitation of this method lies in its heavy reliance on ViT to obtain local features of patches for performing classification, which reduces its universality and prevents it from being applicd to other backbones, e.g., ResNet (He et al. 2016).
The other challenge is that CLIP does not explicitly leverage label correlations, i.e., co-occurrence relationships, during both pretraining and inference, which has been proven to be indispensable for multi-label classification Chen et al. (2019); Lanchantin et al. (2021). In pretraining, the contrastive objective used in CLIP leads its image encoder to focus on the prominent object in an image (Lin) al. 2024), while neglecting others and their co-oceurrence patterns; in inference, using dis- criminative prompt such as "A photo of a {label}" does not introduce any co-oceurrence information. To verify this, we perform zero-shot multi-label classification on benchmark dataset, MS-COCO et al. (2014). and show the co-occurrence probabilities of vanilla CLIP predictions and true labels in Figure [ Considering that is too large to display the entire matrix in the main text, we select two subsets of frequently co-occurring objects and show their co-occurrence proba- bilities. From the figure, there is significant gap between the estimated co-occurrence probabilities and the true ones, which means that vanilla CLIP cannot precisely model co-occurrence relation- ships. This indicates that cannot effectively reduce the complexity of the output space by captur- ing prior information of label relationships, which often leads to missing labels and consequently degrades model performance. An important question has emerged: Do we need label co-occurrence in CLIP to achieve effective zero-shot multi-label classification?
In this paper, unlike the previous work that focuses on enhancing image feature representation, we propose to rewrite the original discriminative prompt and obtain a correlative prompt for each target label by including its co-occurring labels. An interesting finding is that such a simple modification effectively introduces co-occurrence information, prompting the model to capture co-occurrence patterns, which enhances its ability to recognize multiple objects. Unfortunately, we find that CLIP tends overfit to the co-occurrence relationships introduced by the correlative prompts. This leads the model to suffer from an object hallucination issue, where it makes predictions even when the target object does not exist in the image while its co-occurring objects do. To address this problem, inspired by the recent study (Xie et al. (024), we build a causal inference framework by modeling the co-occurrence as a mediator. This enables us to calibrate the CLIP predictions by removing the biased prediction caused by mediated effect from the prompt of co-occurring labels to target prediction. In practical implementation, our derivation shows that this can be achieved simply by combining the outputs of the discriminative prompt and the correlative prompt. Comprehensive experimental results verify that our method can achieve better performance than state-of otate-of-the-an methods.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
Multi-label image classification aims to train a multi-label classifier that can predict all relevant la- bels for unseen images. The pioneering work (Wang al. 2016) combined Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) and developed CNN-RNN framework characterize the label correlations well the image-label relevance. To address the issue that dataset-level statistical correlations may not hold for every individual image, Structured Seman- tie Transfer (SST) (Chen et al.) 2022) introduced the Intra-image Semantic Transfer (IST) mod- ule, which learns an image-specific co-occurrence matrix and leverages this information to recover unknown labels. To overcome the challenge that incomplete annotations make it difficult esti- mate label co-occurrence, SCPNet (Ding et al. 2023) computed text features by feeding category names into the CLIP text encoder, and uses their pairwise similarities as a surrogate for the label co-occurrence. Zhang et al. 2024) developed large model to achieve the goal of recognizing any object. Yue et al. (2024) treated the task of object recognition as the task of next token prediction. Some studies train multi-label classifiers using only small number of labeled examples (Xie al., 2023
2024 Kou et al. (0025). Other studies transform multi-label information into label distributions effectively learn classification models (Kou al. 2024).
Xiao et al.
Prompt tuning (Zhou et al. 2022) learns continuous vectors as task-specific prompts based on small number of training examples. The idea has been applied multi-label image classification and achieved impressive performance (Sun et al.] [2022] [Hu et al., (003)) Given that images may be costly collect while texts are cheap to generate, Guo et al. (2023) proposed Texts as Images in prompt tuning (Tal) to adapt CLIP to multi-label image classification based on only the textual modality.
PRELIMINARIES
In our setting, the task is perform multi-label image classification, also known as object recogni- tion Yue et (2024) or image BoggingZhhagg et al. (2024), using CL.IP without training. Specifically, we are given a dataset [x; in consisting n test instances. Each instance is associated with an unknown label vector y: C V. where y {0, 1)s is the label space with possible class labels. Here, yk = indicates that the k-th label is relevant while yk = 0 indicates that it is not. Our task is develop an inference strategy that enables CLIP to accurately predict all relevant labels for each test instance. We begin with a brief introduction of CLIP. We use [q] to denote the integer set ().....q).
CLIP consists an image encoder Enci(.) and a text encoder Encr(-). To perform zero-shot classi- fication, conventional method construct a prompt Ps using template like "A photo of [labelg]] }", where labelı is the name of the k-th label. By using these two encoders, the image and text features can be obtain
ft Ency((;)) t Ener (P:), [g].
Then, by computing the cosine similarity between the image features f: and the text features t, and applying the softmax activation function, we obtain the predicted probabilities as
exp((fi, (x))
p(u 1|,,, Pa) Ei cxp((px,pp))cccco tx))
Vk [g].
where ( denotes the cosine similarity.
The original paper of CLIP has shown that prompt engineering and ensembling enhance the performance of its zero-shot classification (Radford et al., 202T). For example, the original sin- gle prompt can be expanded into a of prompts by including prompts such as "A satellite photo of a {label}" and "A photo of a big {label}".
4 Do WE NEED CO-OCCURRENCE IN CLIP?
As discussed before, a potential limitation of CLIP is that it does not explicitly exploit co-occurrence information during either the pretraining phase the inference phase. Co-occurrence information
3

---

Published as a conference paper at ICLR 2026
AP Changes
True Posilives
2038
CaP
False Positive
1938
Col
20.
750
*98
508
-10
nadyy
Nowe
Koste
cogs
carrol
trocout lanalige appl
ddgp mrack
hair di
reddy bos
Sorted Classes
Classes
Classes
(a) AP Changes
(b) True Positives
(c) False Positives
Figure 2: Empirical Validations on the advantages and disadvantages of CoP. (a) shows top 10
performance improvements and top 10 performance declines after using CoP. (b) shows the co- occurrence probabilities of CLIP predictions using CoP. (c) reports the number of false positives that contains co-oceurring objects predicted by DiP and CoP.
L
Fc
pd
p
L
Fd
Y
pd
Y
(a) CoP
(b) DiP
Figure 3: The causal graphs of DiP and CoP. pd and pe are the discriminative and correlative prompts. Fd and F are the discriminative and correlative features. Y is the prediction of the target label.
has long been proven to significantly enhance multi-labe classification performance (Chen et al., 2019). Consequently, an important question arises: Do we need co-occurrence in CLIP for zero- shol multi-label classification? To answer this question, we perform experiments on MS-COCO by incorporating co-occurrence information during the inference phase.
Specifically, for each label yi with the name labelş, we assume that there is set of co-occurring labels C. {label;]]] m where my the number of co-occurring labels for the label yı- We will discuss how to obtain the co-occurring label set in the latter section. Then we rewrite the prompt as "A photo of {labelx) often includee cucludes
{label1].... a (m }". This allows us to introduce co-occurrence information into CLIP inference through prompts. We refer to this prompt as the Correlative Prompt (CoP), while referring to the original prompt as the Discriminative Prompt (DiP). Figure 2(a) illustrates the performance improvements/decliine of CoP over DiP, sorted by the performance changes in terms of commonly used Average Precision (AP). As shown in the figure, some labels experienced significant performance improvements, while others saw substantial declines. Due to the page limit, we only show the top-10 labels with the highest performance improvements and the top-10 with the greatest performance declines. The complete results can be found in Appendix B To disclose the reason why the improvements are achieved on these labels, we report the number of true positive instances that contain the target label and its co- occurring labels, measuring how large co-occurrence helps the model to identify the target object. The metric measures the positive impact of co-occurrence on the CLIP predictions. Obviously, CoP makes far more correct predictions than DiP on most of these labels. This indicates that co- occurrence is useful for enhancing CLIP's s ability to recognize multiple objects.
In Figure 2(a) we also present top- 10 labels with the greatest performance declines achieved by CoP. To provide an explanation for this phenomenon, for each target label, Figure 2(c) illustrates the number of false positive instances that contain its co-occurring labels, measuring how many incorrect predictions are caused by the model's overfitting to co-occurrence. It is obvious to see that CoP makes far more incorrect predictions caused by the overfitting issue than DiP in almost all
4

---

Published as a conference paper at ICLR 2026
labels. While CoP benefits from co-oceurrence information, it still carries the risk of overfitting to it.
To provide a theoretical explanation of these phenomena, as shown in Fig- ure 3(a] we construct a causal inference framework to show how CoP leverages co-occurrence information. The framework includes three kinds of variables: the correlative prompt P consisting of the target label L' and its co-occurring labels L", the discriminative activated features F and the correlative activated features F and the model prediction Y for the target label. We use + to represent the causal link between any two variables, which indicates their casual effects. For the sake of comparison, Figure ((b) shows the causal graph of DiP, where pd represents the discrimina- tive prompt only containing the target label L', On the one hand, compared to DiP, which can only make predictions by activating the discriminative features through the single path L' + Fd - Y. CoP can still make predictions by activating the correlative features through an additional path (L',L") + F + Y. especially when the former does not work. This improves CLIP's ability to recognize inconspicuous objects by leveraging co-occurrence information, which is verified by the results in Figure 2(b) On the other hand, we consider a case where the target object does not exist an image but its co-occurring objects do. The discriminative features would not be activated by the target label L: in the correlative prompt, i.e., the path L' + Fdd - Y is disconnected, due to the fact that there is no target object; while the correlative features would be activated by the co-occurring labels L in the correlative prompt, i.e., the path L + F - Y is connected, due to the fact that there exist co-occurring objects. This enables the model to give a high probability of the target object that does not even exist, which can be supported by the results in Figure 2 The above discussions tell us that co-occurrence has not only a positive side: it enhances the CLIP's ability to identify the target objects, but also a negative side: it makes CLIP suffer from the overfit- ting issue and cause object hallucination (Biten et al.) 2022). The important question now becomes: Can we keep the positive side while mitigating the negative side of co-occurrence in CLIP?
A Causal Perspective
5 UNLOCKING THE POWER OF CO-OCCURRENCE
For CoP, the predicted probability can be written as p(yk = 1 i. Pk) or p(yk = 1 1,,;, Li L.). where PK is the correlative prompt consisting of the target label L and its co-occurring labels Lg for the k-th class. However, when there are only co-occurring objects in an image and no target object, CLIP still activates the correlative features by the co-occurring labels in the prompt through the path L + F + Y and gives a high probability of the target object. This indicates that CLIP suffers from overfitting to the co-occurrence, leading to object hallucination. To address this problem, we propose to calibrate the CoP prediction via the total direct effect (TDE) (Pearl (00TT. with the goal of removing the biased part of the prediction. For a given image æ, we define the TDE prediction with respect to the k-th class as
(ax = p(u: 1/x, T,k, I.). p(ı = 1 1/x,, T...)
The first term represents the positive causal effect, where CLIP makes prediction based on the target label prompt L' and its co-occurring label prompt L"; the second term represents the negative effect, where CLIP makes a prediction based on only the co-occurring label prompt. The minus sign indicates that we keep the positive effect while discarding the negative effect.
However, in our experiments, we find Eq. ①1 hardly works. This may be because CLIP often over- estimates the probability p(yk- = 1 xx L especially when some labels preferred by CLIP appear in L This leads the final predictions to underestimation, resulting in performance declines. To ad- dress this problem, we derive an equivalent form of Eq. ( which transforms it from a subtraction form into an addition form as
(x) p(y 1|x, PF) + p(u-- = (/x.P))
(2)
The second term represents the direct causal effect, where CLIP makes a prediction based on the discriminative prompt pd containing only the target label L'. The plus sign indicates that we en- hance the direct effect. An intuitive explanation of this transformation is that mitigating the indirect effect is equivalent to strengthening the direct effect. We refer to this method as DualPrompt.
'The detailed derivation can be found in Appendix
5

---

Published as a conference paper at ICLR 2026
Correlative Prompt
vk [4].A photo (label, J often contains (label,) a H
Text Encoder
Image Encoder
Discriminative Prompt
Text Encoder
vk (4). a (label,}
Figure 4: The framework of DualPrompt. It consists of two types of prompts. The correlative prompt used to maintain the good causal effect of co-occurrence by exploring correlative patterns, while the discriminative prompt is used to remove the bad causal effect by enhancing the discriminative responses.
4 an illustration of our proposed DualPrompt method. Unlike the vanilla method, the bottom branch, which uses the discriminative prompt, our DualPrompt method uses dual prompts, discriminative prompt and correlative prompt, achieve effective zero-shot multi-label classifi- cation.
How to Obtain Co-Oceurrence? Considering that co-occurrence has both universality, meaning that it is natural law used to describe the physical world, and specificity, meaning that it may vary across different datasets, we employ two methods to obtain co-occurrence information in ex- periments. Firstly, we ask ChatGPT--4 to generate up to labels that frequently appear with target objects in the same images. We find that the co-occurring labels generated by ChatGPT are not completely correct. A large ( often means that more labels without co-occurrence relationships introduced, which can mislead the model and result in performance declines. To avoid this problem, set to be small as 2 in experiments. The other method is to annotate tiny amount of train- ing data and only use it for estimating the co-occurrence probabilities. The most co-occurring labels for each target label can be found according the co-occurrence probabilities. Our empirical stud- ies in Section 6 show that very small number of examples, e.g. 1% training data in the MS-COCO dataset, are needed obtain an acceptable co-occurrence probability, which are often insufficient to fine-tune CLIP by existing methods (Sun et al. 2022) to achieve favorable performance.
EXPERIMENTS
We first compare our proposed method with the studies examine the contribution of each component.
methods; then, we perform ablation
6.1 EXPERIMENTAL. SETTINGS
Dataset To evaluate the performance of the proposed method, we perform experiments two benchmark datasets, including MS-COCO 2014 (MS-COCO for short) {Lin et al. 2014) and Vi- sual Genome $] (Krishna et al. 2017). MS-COCO contains 82,081 training images and 137 val- idation images for 80 classes, with an average of 2.9 labels per image. Visual Genome is a dataset that contains 108.249 images from 838 categories. By performing preprocessing done in the previous work (Xie et al. 2024), we obtain a dataset named VG-256 that contains 106,702 images and 256 classes. We randomly split the entire dataset into 70% training set with 74,691 images and a 30% validation set with 32,01 images. We directly evaluate our method and other methods on the validation set. The commonly used mean Average Precision (mAP) and per-class FI (F1) are used as the evaluation metrics. To ensure a fair comparison, we set the random seed to for all experiments.
https::/cocodaaasssttcco
https://hcmee washi
edu www.:eonnddddraaaaaaaaaaa sua www.saawwwggeeeneeeeeeeeeeeeeeee /rana///aaiiiaaaaaagnnn//e////ee htm1

---

Published as a conference paper at ICLR 2026
Table 1: The comparative results between the proposed method and the state-of-the-ar methods MS-COCO dataset. The best performance is highlighted in bold. Coo. Est. represents co-oceurrence probabilities estimation.
Backhone
Resolution
Training Data Usage
mAP
F1
DualCoOp
ResNet- 101
224 224
1% Data for Training
56.3
55.1
TaICLIP
ResNet-
224 x 224
1% Data for Training
56.6
55.7
TaICLIP
ResNet- 101
224 224
COCO Captions
65.1
CLIP
ResNet-101
224 x 224
None
62.9
59.8
DualPrompt
ResNet-101
224 x 224
None
65.5
61.7
DualPrompt
ResNet-101
224 x 224
1% Data Coo. Est.
67.1
63.0
DualCoOp
ViT-B/16
224 x 224
1% Data for training
55.1
54.4
TaICLIP
VIT-B/16
224 224
1% Data for training
63.6
55.9
CLIP
ViT-B/16
224 x 224
None
64.9
61.5
DualPrompt
ViT-B/16
224 x 224
None
67.7
63.6
DualPrompt
ViT-B/16
224 x 224
1% Data for Coo. Est.
69.4
65.0
TagCLIP
ViT-B/16
Original
None
68.7
65.2
DualPrompt + TagCLIP
ViT-B/16
Original Original
None
69.2
65.4
DualPrompt TagCLIP
ViT-B/16
1% Data for Coo. Est.
70.0
66.1
Table 2: The comparative results between the proposed method and the state-of-the-ar methods on VG-256 dataset. The best performance is highlighted in bold. Coo. Est. represents co-occurrence probabilities estimation.
Backbone
Resolution
Training Data Usage
mAP
F1
DualCoOp
ResNet- 101
224 x 224
2% Data for Training
30.4
33.5
TaICLIP
ResNet- 101
224 224
2% Data for Training
32.0
34.9
CLIP
ResNet-10 101
224 x 224
None
29.2
32.2
DualPrompt
ResNet- 101 224 x 224
None
33.5
36.1
DualPrompt
ResNet-101
224 224
2% Data for Coo. Est.
35.2
37.6
TagCLIP
ViT-B/16
Original
None
39.6
41.9
DualPrompt + TagCLIP
ViT-B/16
Original Original
None
40.2
42.4
DualPrompt TLgCLIP
VIT-B/16
2% Data for Coo. Estt.
40.7
42.7
For CLIP, we use ResNet-101 101 (He et al. 2016) and ViT-B/16 (Dosovitskiy al. 202T) as the backbones. For TagCLIP, we use the original resolution images as done in its original resolution of 224 x 224. Following the previous work (Lin al., 2024]. we use the 80 prompts used in CLIP (Radford et al.) (.2I). For prompt fine-tuning methods, when only 1% of the training data is available, the batch size set to 16; when 2%-5% of the training data is available, the batch size is set to 32. The other parameters use the reference values provided the original papers, All experiments were run on single NVIDIA A100 GPUs.
Implementation
paper; for other methods,
6.2 COMPARISON WITH STATE-OF-THE--RT METHODS
To validate the effectiveness of the proposed method, we compare it with two types of CLIP-based methods: i) two training-basee methods, DualCoOp (Sun et al.) (022), which performs prompt tuning based on subset of downstream training data, and Tal (Guo et al. (203),, which trains on subset of data curated caption data in the downstream task; two training-free methods, CLIP (Radford et al. 2021) and TagCLIP (Lin et al. 2024).
Table and Table report the results of the proposed method and other methods MS-COCO and VG-256, respectively. From the table, we can see that: i) DualPrompt outperforms both training- based and training-free methods with significant margin. ii) By combining DualPrompt and Tag-

---

Published as a conference paper at ICLR 2026
18
AP Increases
AP Increases
10.
20
5
10
-10-
-15
10
20
30
40
50
60
70
80
10
20
30
40
50
60
70
80
Class Indices
Class Indices
(a) CoP vs. DiP
(b) DualPrompt vs. CoP
Figure 5: Comparisons of per-class average precision among DiP. CoP, and DralPrompt. Compared with DiP, CoP shows improvements on some categories but decreases on others, whereas compared with CoP, DualPrompt achieves improvements on the vast majority of categories.
40
36
64-
36.
mAP
52
Bathspp
Daihheg!
34
50-
32
Peraxitgg of Triiiig (hi)
Peroningg of Tinizig Dax (h) (b) MS-COCO, CFI
Percentage Training Data (%)
Percentage of Trrininn Data
(a) MS-COCO, mAP
(c) VG-256, mAP
(d) VG-256, CFI
Figure 6: The performance curve with the increase of training data usage. For DualPrompt, training data is only used for estimating co-occurrence probabilities; while for DualCoOp, training data is used for model training.
CLIP, we achieve the state-of-the-ar performance on both two datasets. iii) For the two methods of obtaining co-occurrence labels, it is evident that co-occurrence estimation with tiny number of data is better than ChatGPT. The co-occurrence provided by ChatGPT universally existing regu- larity, while co-occurrence estimation is based on the prior of the downstream dataset. These results convincingly verify that the proposed method can achieve the state-of-the performance.
6.3 ABLATION STUDIES
We provide empirical validations how DualPrompt keeps the positive impact while removing the negative impact of co-occurrence. Figure iillustrates a stepwise evaluation on MS MS-COCO in terms of AP. From the figure, we see that CoP achieves better performance than DiP some classes, while performing worse on others. Overall, the performance of CoP is inferior to that of DiP. This because CLIP overfits to the co-oceuring labels given in the prompt, making many false positive predictions, which leads to a decline in model performance. To address this problem, our proposed method calibrates the predicted probabilities via TED measurement by removing the biased parts. DralPrompt achieves performance improvements in almost all classes, providing convincing evi- dence that can maintain the positive effect while removing the negative effect.
6.4 STUDY ON TRAINING DATA USAGE
In our method, we need to obtain the co-occurring labels for each target label. A solution is to estimate the co-occurrence probabilities based on very small subset of training data and obtain co-occurrence according to probabilities. In this subsection, we show that such small number of training data is insufficient for model training. Figure 6 illustrates the performance curves of our method and prompt tuning methods, DualCoOp and Tal. It is noteworthy that our method only uses the training data for estimating co-occurrence probabilities, while DualCoOp and Tal use them for fine-tuning the model. From the figure, it can be observed that with very small number of train- ing (1%). there is a significant performance gap between our method and prompt tuning methods. With the increase of training data, the gap gradually narrows, and eventually, Tal outperforms
8

---

Published as a conference paper at ICLR 2026
bartle
bottle
bade
boitle
0.20
0.04
10.00
0.12
9:53
0.06
9.06
10.00
4:03
0.34
0:31
0.16
0.14
9.26
sup
0.00
0.37
0.11
0.13
0.16
1.00
0.29
0.21
look
0.12
0.22
0.36
0.21
0.50
kade
laite
bnic
spoon
spoon
Spacn
0.13
0.57
bened
0.30
6.27
bone
0.12
bows
bosel
bonte
cup
tork
knife
opeon
bowl
bords
cap
fork
knife
greea
bowl
bottle
fork
knife
spocn
bowi
bottle
oop
fonk
Lnffee
bowl
(a) DiP
(b) CoP
(c) DramPtompp
(d) True
Figure 7: The co-occurrence probabilities of different predictions and true labels.
1st
5th
10th
15th
20th
1st
5th
10th
15th
20th
The text-image retrieval results of DiP (the first row), CoP (the second row) and Dual- Prompt (the last row) with respect cup (on the left side) and sink (on the right side). To show the representativeness of different methods, we present the images ranked st,, 5th, 10th, 15th, and 20th based on text-image similarities.
Figure 8:
method. This indicates that method requires such small number of training data estimate co-occurrence probabilities that they are not even sufficient for fine-tuning the model.
6.5 STUDY ON CO-OCCURRENCE ESTIMATION
In this subsection, we demonstrate the ability of different methods capture co-occurrence rela- tionships. Figure 7 iilustrates the co-occurrence probabilities estimated by different methods in the kitchenware scenario consisting of six classes: bottle, fork, knife, spoon, bowl. We can see that CoP performs better than DiP some classes, while performing worse others. This means that although CoP benefits from the advantage of correlative prompts, still suffers from the overfitting to co-occurrence. Suspicious co-occurrence hinders further performance improvement. To address this problem, DualPrompt aims to remove the negative impact caused by suspicious co-occurrence and achieve better co-occurrence estimation than both DiP and CoP.
6.6 TEXT-IMAGE RETRIEVAL RESULTS
To explore the mechanism behind DualPrompt, as shown in Figure g we visualize some images according text-image similarity scores. Three rows represent the text-image retrieval results of DiP, CoP and DualPrompt. For each method, we illustrate the images ranked Ist, 5th, 10th, 15th, and 20th according to similarity scores. From the figure, we can see that the retrieved images from different methods exhibit significant differences. DiP tends to retrieve the images dominated by the target object. sometimes make mistake due to the ambiguous object, e,g., incorrectly identify a vase as cup in the 15-th image. While CoP tends to retrieve images that depict scenes characterized by the target object and its co-occurring objects, such as a dining table full of plates or a fully equipped bathroom. This indicates that CoP identifies the target object by exploiting co-occurrence information. But it often suffers from the overfitting issue and make incorrect predictions based on only the co-occurring objects, e.g., there is no cup but only its co-occurring objects like bowl and

---

Published as a conference paper at ICLR 2026
bottle in the 15-th image. The images retrieved by DualPrompt seem be a combination of the first two methods. DualPrompt enhances the text-image matching ability by leveraging the advantages of the first two methods, which aims to identify the target object based discriminative features and correlative patterns.
7 CONCLUSION
The paper studies on zero-shot multi-label classification using CLIP without training. We find that compared the single-label scenarios, CLIP often obtain unfavorable performance in the multi- label scenarios due to the fact that it does not explicitly exploit co-occurrence both pretraining and inference. To address this problem, we propose the DualPrroppt method to use the discriminative prompt and correlative prompt simultaneously. The former contains the only target label, which aims capture discriminative features, while the latter contains the target label and its co-occurring labels, which introduces co-occurrence information. Moreover, we construct a causal inference framework provide theoretical explanation for our method. Extensive experimental results verify the effectiveness of the proposed method.
ACKNOWLEDGMENTS
MS was supported by JST ASPIRE Grant Number JPMIAP25BI.
REFERENCES
Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. Let there be clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1381-1390, 2022.
Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, and Liang Lin. Structured semantic transfer for multi-label recognition with partial labels. In Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 339-346, 2022.
Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5177-5186, 2019.
Zixuan Ding, Ao Wang, Hui Chen, Qiang Zhang, Pengzhang Liu, Yongjun Bao, Weipeng Yan, and Jungong Han. Exploring structured semantic prior for multi label recognition with incomplete labels. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3998-3407, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openrevie ew www.:nee/ffoooommmmmm wttp::/idiiiccdddocccccc cbddTTy
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Opcabbocyy object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022. URL /oppptttp/s///eeeeveieew net www.mooorrullia... 1nMMbb311111
Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2808 28882287,77 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ping Hu, Ximeng Sun, Stan Sclaroff, and Kate Saenko. Dualcoop++: Fast and effective adaptation to multi-label recognition with limited annotations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
10

---

Published as a conference paper at ICLR 2026
Zhiqiang Kou, Jing Wang, Jiawei Tang, Yuheng Jia, Boyu Shi, and Xin Geng. Exploiting multi-label correlation in label distribution learning. In Proceedings of the Thirty- Third International Joins Conference Artificial Intelligence, pp. 4326-4334 2024.
Zhiqiang Kou, Yucheng Xie, Hailin Wang, Junyang Chen, Jing Wang, Ming- Kun Xie, Shuo Chen, Yuheng Jia, Tongliang Liu, and Xin Geng. Rankmatch: A novel approach to semi-supervised label distribution learning leveraging rank correlation between labels. In The Thirty Anmual Conference on Neural Information Processing Systems, 2025. URL 2 1d0pmmmm21d08000.55555330
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Jack Lanchantin, Tianlu Wang, Vicente Ordonez, and Yanjun Qi. General multi-label image classi- fication with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16478-16488, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Yuqi Lin. Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15305- 15314, 2023.
Yuqi Lin, Minghao Chen, Kaipeng Zhang, Hengjia Li, Mingming Li, Zheng Yang, Dongqin Lv, Binbin Lin, Haifeng Liu, and Deng Cai. Tagelip: A local-to-global framework to enhance open- vocabulary multi-label classification of clip without training. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 3513-3521, 2024.
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncer- tainty and Artificial Intelligence, 2001, pp. 411-420. Morgan Kaufman, 2001.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8430-8439, 2019.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35:30569-30533 2022.
Jiang Wang, Yi Yang. Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285-2294, 2016
Jia-Hao Hao Xiao, Ming-Kan Kun Xie, Heng-Bo Fan, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Dual-decoupling learning and metric-adaptive thresholding for semi-supervised multi-label learn- ing. In European Conference on Computer Vision, pp. 437-454, 2024.
Ming-Kun Kun Xie, Jia-Hao Xiao, Hao-Zhe Liu, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Class-distrbbution-aww aware pseudo-labeling for semi-supervised multi-label learning. In Thirty-seventl Conference on Neural Information Processing Systems, 2023. URL https:
//openrevien http://forumm2.dd0000110ggg1g11p

---

Published as a conference paper at ICLR 2026
Ming-Kun Xie, Jia-Hao Xiao, Pei Peng. Gang Niu. Masashi Sugiyama, and Sheng-Jun Huang. Counterfactual reasoning for multi-label image classification via patching- based train-
ing. In Forty-first International Conference Machine Learning, 2024. URL https:
Kaiyu Yue, Bor-Chun Chen, Jonas Geiping, Hengduo Li, Tom Goldstein, and Ser-Nam Lim. Object recognition next token prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16645-16656, 2024.
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et Recognize anything: strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1724-1732, 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. AnomalyCLIP. Object- agnostic prompt learning for zero-shot anomaly detection. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httpss://operreviee net/ forum?id
buC4E9 XZE
12

---

Published as a conference paper at ICLR 2026
'able 3: The comparative results between the proposed method and the state-of-the-art methods Objects365 dataset. The best performance is highlighted in bold. Coo. Est. represents co-oceurrence probabilities estimation.
Backhone
Resolution
Training Data Usage
mAP
F1
CLIP
ResNet- 101
224 224
None
30.0
26.8
DualPrompt
ResNet-101
224 x 224
1% Data for Coo. Est.
34.5
30.0
TagCLIP
ViT-B/16
224 x 224
None
36.7
30.4
DualPrompt TagCLIP
ViT-B/16
224 224
1% Data for Coo. Est.
37.9
31.9
A DERIVATION OF EQ. (2)
Considering that a correlative prompt P consisting of the target label L' and its co-occurring labels LS, let {L', Li} and { [ represent mutually exclusive events, where Lğ (L)) represents we remove the co-occurring labels (target label) from the correlative prompt. By omitting the class index & for notational simplicity, we have
T(x)
=V 1|a L' L9) - plu = 1/2, L", L.)
=ply |0, L,, 1.9 pvy = 1, xc L Lo)
p(x, Le, L6)
ply = 1, œ, L", L') , p(ar, L- LT)
ply = 1, x Li: L9)
ply 1 L' 19)
ply = 1/,, L L' p(ar IL,L') ply 1|,, Lg, L (')p((, Lū, L')
ply 1/x, L', L9)
plæ, L 1g)
(1 p(x, L" L')
ppy 1|x L I9) + pax, Lg L') p(y 1 1 L.. L9)
x L", Lg)
p(x, L L
x ply 1, po) + Xply = 1|x, pd),
P(x.Li,L')
where À p L ((' can be regarded as a trade-off parameter. In our experiments, we find that our method achieves favorable performance by simply setting À to 1.
B COMPLETE RESULTS OF FIGURE 2(A)
AP DetcoeeeeDrrrr
Figure 9: The performance changes of CoP against DiP in terms of AP over all cases. The perfor- mance of nearly half of the labels improved, while the performance
other half decreased.
Figure illustrates the performance changes CoP against DiP in terms of AP over all classes. From the figure, can that nearly half of the labels experience performance improvements, while the other half suffer from performance declines. These results verify that co-occurrence has both good and bad effects.
13

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL RESULTS ON OBJECTS365
To verify that the proposed method has a broad range of applicability, we have added experimental results on more challenging dataset, Objects365 Shao al. 2019), which contains 365 classes. This dataset has an average of 6.17 labels per image, which is more than twice that of the COCO dataset (2.93). Table
the experimental results of our method and the comparison baselines on Objects365. For our method DualPrompt, we use 1% of the training data to estimate label co-occurrence. As shown in the table, DualPrompt significantly outperforms the baseline CLIP, achieving 4.5% improvement in mAP. After incorporating TagCLIP, the proposed method still yields 1.2% gain in mAP. These experimental results demonstrate the effectiveness of DualPrompt in challenging scenarios.
14


Reviews:
[{"review_id": "8AVb6omKHd", "reviewer": "Reviewer_QsUa", "summary": "This paper addresses the limitation of CLIP in zero-shot multi-label classification, where the model fails to explicitly leverage label co-occurrence relationships, leading to suboptimal performance. This paper proposes a novel method called DualPrompt, which introduces co-occurrence information by using two types of prompts: a discriminative prompt containing only the target label and a correlative prompt that includes both the target and its co-occurring labels. While the correlative prompt enhances recognition by capturing label dependencies, it also causes object hallucination due to overfitting to co-occurrence patterns. To mitigate this, this paper develops a causal inference-based calibration mechanism that retains the benefits of co-occurrence while suppressing its adverse effects. Extensive experiments on MS-COCO and VG-256 datasets demonstrate that DualPrompt outperforms both training-based and training-free state-of-the-art methods, achieving superior performance even without any training data.", "questions": "In Figure 6, why does DualCoOp++ (which presumably augments DualCoOp with additional parameters) yield lower mAP/F1 than DualCoOp?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tProposes a novel, training-free method (DualPrompt) that explicitly injects label-co-occurrence information into CLIP for zero-shot multi-label classification.\n2.\tIntroduces a lightweight dual-prompt strategy—discriminative vs. correlative—that can be applied to any CLIP backbone without training.\n3.\tProvides a principled causal-inference formulation that disentangles the helpful mediated effect of co-occurrence from the harmful “object-hallucination” effect, yielding calibrated predictions.\n4.\tDemonstrates consistent gains over both training-free (CLIP, TagCLIP) and training-based (DualCoOp, TaI) competitors on MS-COCO and VG-256.\n5.\tShows that only ~1 % of training data is sufficient to estimate reliable co-occurrence statistics, keeping data requirements minimal.\n6.\tValidates interpretability via visualization: retrieved images align better with true co-occurrence patterns while suppressing false positives, confirming that the method preserves semantic plausibility.", "weaknesses": "1.\tThe paper only conducts experiments on MS-COCO and VG-256 datasets, which is not sufficient to demonstrate the generalizability of the proposed method. It is recommended to include experiments on more commonly used datasets in multi-label learning to further validate the effectiveness and robustness of the approach.\n2.\tThe paper lacks discussion of recent works that also exploit label co-occurrence, e.g., SST [1] and SCPNet [2]. Although these papers address different settings, a comparative summary of how their co-occurrence modeling differs from DualPrompt is necessary to clarify novelty and avoid overlap.\n3.\tThe paper places RELATED WORKS after the experiments and just before CONCLUSION, which breaks the usual flow and forces readers to assess the method’s novelty without early context. \n4.\tThe derivation of Eq. (3) contains errors. Please carefully re-check every step.\n5.\tTable 1 reports TaI-CLIP with ResNet-101 trained on COCO Captions, but the corresponding ViT-B/16 entry under the same training condition is missing.\n6.\tThe manuscript contains numerous typographical and consistency errors that must be thoroughly corrected: in Figure 2 “false negatives” is mislabeled, “prompr” should be “prompt”, “Eq. equation” is redundant, the tables mix “x” and “×”, the caption of Figure 5 is wrongly copied from Figure 6, Section 5.5 lists “microwave, oven, toaster, sink, refrigerator” while Figure 7 shows “bottle, cup, fork, knife, spoon, bowl”, the reference “Counterfactual reasoning for multi-label image classification via patching-based training” is duplicated, and “cane” should be “can”.\n7.\tOn line 201 the symbol F^t appears in the causal path L^t → F^t → Y without any prior definition; please clarify what this feature vector represents and how it differs from F^d and F^c to avoid confusing the reader.\n\n[1] Chen, Tianshui, et al. “Structured semantic transfer for multi-label recognition with partial labels.” Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 1. 2022.\n\n[2] Ding, Zixuan, et al. “Exploring structured semantic prior for multi label recognition with incomplete labels.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0xsjcmhJmD", "reviewer": "Reviewer_C23F", "summary": "The paper targets zero-shot multi-label image classification with CLIP, noting that vanilla CLIP underperforms because it (i) focuses on the most salient object and misses others, and (ii) does not explicitly leverage label co-occurrence during pretraining or inference. The authors introduce correlative prompts (CoP) that append a target label with its frequent co-occurring labels inside the prompt, explicitly injecting co-occurrence information at inference time. They show CoP can boost recognition of less-salient objects but also induces object hallucination when correlated context appears without the target.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method provides a simple, training-free mechanism to inject label co-occurrence at inference, which meaningfully improves zero-shot multi-label performance.\n2. The approach is backbone-agnostic and complements patch-centric methods (e.g., TagCLIP), showing additive gains when combined.\n3.The study includes informative diagnostics (class-wise AP changes, true/false positives, retrieval visualizations) that clarify how prompts alter model behavior.", "weaknesses": "1.The theoretical derivation from a TDE subtraction to a simple additive fusion lacks rigorous justification; the introduced λ parameter is set to 1 without sensitivity analysis.\n2.The use of softmax across labels for multi-label probabilities and the summation of two scores raises calibration concerns and probabilistic interpretation remains unclear.\n3.The evaluation scope is limited to two datasets and broader generalization and cross-dataset portability of co-occurrence priors are not assessed.\n4.The method’s inference-time overhead should be reported (text encoding caching, similarity computation cost) for scalability considerations.\n5.The comparisons should control for image resolution, prompt ensembling, and preprocessing; decision thresholds should be explicitly defined and consistently applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JLgzFNPt2B", "reviewer": "Reviewer_PYxD", "summary": "This paper proposes DualPrompt, a simple yet effective method for improving zero-shot multi-label classification with CLIP. By incorporating co-occurrence information into the prompt design via both Discriminative Prompt and Correlative Prompt, the authors aim to leverage label dependencies during inference without training. The paper provides theoretical justification through a causal inference framework and achieves competitive performance across standard benchmarks.", "questions": "* Could you provide more detailed error analysis or qualitative cases showing the failure modes of DualPrompt, especially in cases where co-occurrence may still lead to false positives?\n\n* Have you tried evaluating the method on other commonly used multi-label datasets (e.g., VOC2007, NUS-WIDE)? If not, is there any reason for excluding them?\n\n* How sensitive is the model to the choice of co-occurring labels generated by ChatGPT? Are there failure cases when ChatGPT introduces incorrect or spurious co-occurrence?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The method is simple and elegant, requiring no fine-tuning and remaining compatible with standard CLIP backbones.\n\n* The use of ChatGPT for co-occurrence estimation is a clever and pragmatic idea, enabling training-free augmentation of prompts.\n\n* The paper is generally well-written, and the causal graph perspective helps to clarify the strengths and pitfalls of the method.", "weaknesses": "* Remaining Co-occurrence Issues Not Fully Analyzed:\nAlthough the proposed method aims to mitigate overfitting caused by co-occurrence, it is likely that some object hallucination or over-reliance on correlations remains, especially when the co-occurring object is much more prominent than the target object. However, there is no in-depth analysis or diagnostic results to explore these residual errors in detail.\n\n* Narrow Dataset Evaluation:\nThe experiments are conducted only on MS-COCO and VG-256. While VG-256 covers a wide label space, this is still limited compared to related works. Additional benchmarks such as VOC2007 or NUS-WIDE, which are widely used in multi-label classification, would strengthen the empirical evaluation and demonstrate the robustness and generalizability of the approach.\n\n* Minor Issues:\nThere are a few editorial errors, e.g., the phrase “Eq. equation” appears, which should be corrected. Such issues, while minor, slightly detract from the overall polish of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PwcX765wH9", "reviewer": "Reviewer_3XJU", "summary": "The author tries to achieve zeroshot multi-label classification based on CLIP, and explore this task with labeling the object co-occurrence in CLIP. The paper improve the prompt to achieve the goal, and also calibrate CLIP predictions to achieve SOTA results. The experimental results are verified on mutiple benchmarks.", "questions": "- In the introduction, the author mentioned that \"There are two major challenges in achieving effective zeroshot multi-label classification\", one is \"CLIP tends to capture the global features of the dominated objects and neglecting the local features on otheres\", another is \"CLIP does not explicitly leverage label correlations\". So I want to post two questions on these statements. 1). the first statement seems not like a challenge in achieving effective zeroshot multi-label classification for CLIP, since the author mentioned ViT can solve this issue while CLIP already have the ViT version, this paper's best model is also based on ViT. Why using ViT seems like a trouble in this paragraph, can you explain a bit more? 2). In the second challenge, the author shows an example of the co-occurrence probability, and demonstrate that CLIP cannot precisely model co-occurrence relationships, but it suddently comes to the conclusion that \" resulting in unfavorable recognition performance\". So why it results in unfavorable recognition performance? I think more explanation should be added here.\n\n- In the experiment section, why the training data usage is 1% in MSCOCO and 2% for VG-256? How will this number influence the effectiveness of your method?\n\n- Besides, I notice that this paper is also submitted in ICML 2025 and NeurIPS 2025 but without replying to any reviewer comments [[link](https://openreview.net/forum?id=2WLEEo4tqT)] [[link](https://openreview.net/forum?id=oCDPvJ2uyc)], could you simply summarize these questions and provide the answer?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a better prompt (discriminative prompt + correlative prompt) on the CLIP model (or similar models) to achieve better performance.\n\n- The method only needs a few training data to derive the co-occurring labels.", "weaknesses": "- The statement in the abstract is a bit vague. it would be better to add a sentence to summarize what method you used to 'keeping the positive effect while removing the negative effect caused by suspicious co-occurrence'.\n\n- The results seems promising on two datasets (MSCOCO and VG-256), but it would be great if the NUS-WIDE results can also be provided since it widely used with much larger dataset / number of labels and provide more convincing results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)