Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels
Authors: Haoyang Luo
Zheng Zhang
Yadan Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: cross-modal hashing
incomplete labels
prompt contrastive learning
Primary area: machine_vision

Abstract: In this paper, we tackle the challenge of generating high-quality hash codes for cross-modal retrieval in the presence of incomplete labels, which creates uncertainty in distinguishing between positive and negative pairs. Vision-language models such as CLIP offer a potential solution by providing generic knowledge for missing label recovery, yet their zero-shot performance remains insufficient. To address this, we propose a novel Prompt Contrastive Recovery approach, \textbf{PCRIL}, which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels. Identifying unknowns is nontrivial due to the fixed and long-tailed patterns of positive label sets in training data, which hampers the discovery of new label combinations. Therefore, we consider each subset of positive labels and construct three types of negative prompts through deletion, addition, and replacement for prompt learning. The augmented supervision guides the model to measure the completeness of label sets, thus facilitating the subsequent greedy tree search for label completion. We also address extreme cases of significant unknown labels and lack of negative pairwise supervision by deriving two augmentation strategies: seeking unknown-complementary samples for mixup and random flipping for negative labels. Extensive experiments reveal the vulnerability of current methods and demonstrate the effectiveness of PCRIL, achieving an average 12\% mAP improvement to the current SOTA across all datasets. Our code is available at https://github.com/E-Galois/PCRIL.

Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels
Haoyang Luo
Zheng Zhang"
School of Computer Science and Technology Harbin Institute of Technology. Shenzhen Shenzhen, China
Harbin Institute of Technology, Shenzhen
Peng Cheng Laboratory
Shenzhen, China
luohaoyang. waw..gglaaaattaal...ooo
darrenzz2198gmail.com
Yadan Luo
UQMM Lab, School of EECS
University of Queensland Brisbane, Australia .uluuuuuu.. edu, au
Abstract
In this paper, we tackle the challenge of generating high-quality hash codes for cross-modal retrieval in the presence of incomplete labels, which creates uncertainty in distinguishing between positive and negative pairs. Vision-languees such as CLIP offer a potential solution by providing generic knowledge for missing label recovery, yet their zero-shot performance remains insufficient. To address this, we propose a novel Prompt Contrastive Recovery approach, PCRI,, which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels. Identifying unknowns is nontrivial due the fixed and long-tailed patterns of positive label sets in training data, which hampers the discovery of new label combinations. Therefore, we consider each subset of positive labels and construct three types of negative prompts through dele- tion, addition, and replacement for prompt learning. The augmented supervision guides the model to measure the completeness of label sets, thus facilitating the subsequent greedy tree search for label completion. We also address extreme cases of significant unknown labels and lack of negative pairwise supervision by deriving samples for mixup
models
two augmentation strategies: seeking unknown-complementee and random flipping for negative labels. Extensive experiments reveal the vulnera- bility of current methods and demonstrate the effectiveness of PCRIL, achieving average 12% mAP improvement the current SOTA across all datasets. Our code is available at Ctthub.com//-Galoos/PCCCC
Introduction
Cross-modal hashing (CMH) 24
23 25 addresses the highly demanding cross- cross-modal sim-
ilarity search in both web search systems and academic domains [10] [3] By efficiently transforming multi-modal data (images, text, etc.) into a collection of compact binary codes, CMH maintains high-dimensional cross-modal semantic similarity while significantly minimizing computational and storage costs. Deep cross-modal hashing has achieved remarkable development by leveraging dual-stream networks or semantic encoding branches based on semantic labels 14 [7] However,
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MIRFlickre 25k
COCO
102
% of different pairs
% different pairs
60-
Positive Negative
Pos Positive
60-
Negative
Unknown
Unknown
Neg./Pos
Neg. /Po.
20
40 80
20
80
Unknown ratio (%)
Unknown ratio (x)
Figure 1: Incomplete labels can severely damage cross-modal similarity learning by reducing paired samples. For MIRFllickr255k annotations (left), 35% unknown classes can completely exclude all negative pairs. For MS COCO (right), even 20% labels under-annotated can fundamentally remove the negative relationships.
due to limited labour resources, fully supervised annotation becomes impractical for large-scale datasets [27 33]. A realistic compromise would be partial annotation, in which only a subset of classes are explicitly labeled, while others marked as unknown. Although facilitating adaptation to large datasets, this labeling scheme provides significantly reduced semantic supervision. Therefore, learning with partial annotation has become significant challenge in multi-label learning tasks.
Furthermore, CMH with partial labels inevitably encounters disrupted similarity learning due to pairwise uncertainty [2)22) Jointly missing classes cause incomplete labels to not only obscure class knowledge but also eliminate pairwise similarity by introducing unknown pairs. Among these, negative pairs constitute the majority and become particularly scarce with the increasing frequency of unknown labels. As shown Fig. D negative pairs are completely removed even with 35% unknown labels in Flickr dataset. Therefore, such pairwise uncertainty can severely impair similarity- pinilartty-reesrvinn hashing. However, existing CMHs desperately rely on clear positive and negative pairs to maximize supervision. It remains crucial yet unresolved how to accurately perceive potential categories and restore similarity learning in the task of CMH to prevent semantically uninformative binary codes.
To address the issues of incompleteness, some multi- image recognition methods have in- vestigated feasible solution, i.e., leveraging prior knowledge large vingun-language models [19 18 14]. Pre-trained with contrastive loss on massive image-text pairs, CLIP (28 has shown visual and textual modalities with loaded prior knowledge (modal correspondence, similarity structures, partial semantics, etc.). Although effective in many downstream tasks, they are hardly applicable for pairwise similarity recovery in CMH. Original CLIP's ability is empirically ineffective for label recovery via class-sample similarity scores. As revealed in Sec. 4.4 the original CLIP prompt yields an unsatisfactory 68% recovery precision even the easier Flickr 30% unknown case, introducing substantial noisy classes and even degrading the final performance. Therefore, CLIP label recovery for deep CMH remains under-explored.
its capability to align the global representation
In this paper, we seek to overcome the aforementioned deficiencies and propose Prompt Contrastive Recovery for eross-modal hashing with Incomplete labels (PCRIL) by considering the CLIP prior knowledge of descriptive completeness, which is the ability measure the completeness of a text caption for described image. By selecting anchor class sets, we develop a simple learnable prompt to encode selected anchor class combinations into CLIP text embeddings. Multiple negative variants are constructed via editing operations on the anchor set. A prompt contrastive recovery paradigm then imposes separation gaps between these label subsets. By learning scores conditioned their sample modalities, instance-aware class perception is enabled. Thereafter, potential labels are recovered via a tree search the learned scores. To enhance hard incomplete samples, different instances' features and unknown labels are complementarily mixed. We also introduce an adaptive negative masking strategy deal with negative pair scarcity at high missing rates. The main contributions of our work are as follows:
We propose PCRIL framework, which jointly performs semantic recovery and pairwise uncertainty elimination for efficient cross-modal hashing with incomplete labels. To the best of our knowledge, there is no prior study on CLIP-based label recovery strategies in cross-modal hashing.

---

A novel recovery architecture is proposed to recover the neglected semantic labels and pairwise similarities. Particularly, a contrastive learning objective between the anchor set and its negative variants learns instance-condittoned matching scores. A tree search process then leverages the learned scores to detect potential classes. Meanwhile, a complementary semantic augmentation and an adaptive negative masking strategy jointly enhance the similarity learning in extreme cases. Thus, PCRIL can fully restore potential labels and pairwise similarity.
Extensive experiments verify that our PCRIL can consistently outperform state-of-the-art CMH methods across range of incompleteness levels and different benchmarks. Comprehensive analyses further validate our effective recoverability for incomplete labels.
2
Related Work
2.1 Cross-modal Hashing
Cross-modal hashing (CMH) aims to encode different high-dimensional modalities into a common space of compact binary codes where modal similarity is preserved for fast and accurate retrieval. Early machine learning methods, including those by [32 21 26 3 learn common codes from
encoded features. Though simple, non-deep methods are restricted by two-step training paradigm and limited in discriminative similarity learning. Jiang el al. [4] proposed deep cross-modal hashing (DCMH), introducing a pairwise similarity matrix into deep similarity preservation for the first time. Li et al. [17] constructed a label net to project labels into a common space with modal binary codes. To close modality gaps, Gu et al. [T] adopted cross-modal feature attentions with an adversarial learning scheme for semantic discriminability and modal consistency. Zhang et al. (35] proposes to preserve multi-level knowledge in CMH with variational information bottleneck. As efficient feature learners, pretrained vision-language models have been leveraged in recent works. Tu er al. [30 introduced tranfformerbbase CLIP image encoders with selective hash optimization. Liu et al. further investigated multi-granularig cross-modal alignment based vision-language transformers. These methods rely heavily on the guidance of complete semantic labels for similarity learning, while they fail to handle the practical incompleteness problem of label annotations in large-scale datasets.
2.2 Learning with Incomplete Labels
Learning with incomplete labels involves only partially known classes, which are compromise resulting from inaccessible exhaustive supervision 77]. Existing studies regarding incomplete labels mainly focus on the image recognition task. Some works seek to recover learnability upon partial labels with modified loss functions. Bucak et al. [I] proposed to learn a modified ranking loss to alleviate the effects of false negatives while assuming all unknown labels be negative. Durand et al. [9] proposed to learn only known labels with the prior knowledge of the label ratios. Cole et al. [6] proposed solve the single positive label issue by imposing training constraints on label statistics. Another type of work investigates lost labels directly. Veit ef al. [31 resorted to learning from human re-annotations to recover original labels. Chu et al. [4] proposed variational generative model to explore data correlations with partial labels. Kim et al. [16] proposed to distinguish exceedingly large losses of false negatives and reverse them during training. By adopting vision-language models with rich prior knowledge, Sun et al. [29] proposed prompt-tuning method with separately learned positive and negative prompt parameters. Ding el al. (8 further considered constructing graph-based label structure in vision-language models to enhance image recognition. In cross-modal retrieval, the problem is however unexplored. Some methods [27] focus on the problem entirely missing labels. Ni et al. (27) and Liu al. (22) proposed shallow CMH methods that re- re-predict the labels with consistency constraints between instances and labels. However, without a fine- grained measurement of labple-label consistency, their ability for class perception is limited to only distinguish salient
ones.
Proposed Approach
3.1 Problem Definition
We focus on image-text hashing, which prioritized and fundamental. CMH with incomplete labels learns hash functions on a training set O ((f.,9,, 9)) where f. and g, are the image and text
3

---

Positive
Pan ]
Select
R
Label Set
[Leash]
Dog 105
Prooptiae
gher
Label Set
Label Set
Regative
Inknown
(Tree
[Store]
[Bike
Cat 1xg
Can
Megative Subsets
Anchar
Text Encoder
CLIP
89
murgin
Let
scores
Adapt tve Regativve Nasking (5q.9)
100
Incomplete
K
Contrastive Loss
Potential Label Tree Search
7
U
Labels
Goog
10100 00:000
U
19
01:000
S
CLIP Modal Encoders
Augrentation
100011
outside the café, the lavely
select
Features
Labels
Pairwise Sieilarity Supervision
Figure 2: Our proposed PCRIL consists of two major stages: prompt contrastive recovery and augmented pairwise similarity learning. The prompt contrastive recovery stage effectively perceives incompleteness via label prompts to learn contrastive matching scores with modal samples, recovering informative semanties. The similarity augmentations further eliminate unknown labels through complementary blending of samples and recover the scarce negative pairs using an adaptive negative masking strategy.
modalities of the i-th sample, respectively, and I is its label vector, which is the i-th column in the label matrix L E (1, 0, u) CxNN, Here, C' is the number of classes, N is the number of instances, and u denotes an unknown value. Hence, sample class could positive (1), negative (0), or unknown (u). A similarity matrix S is derived from L, where 8y = 1 iff f. and g, share at least one positive label. CMH aims to learn hash functions H' and H to encode text and image samples into binary codes 64 sign
1) and by = sign (H" (g:)) E {0, 1)d, preserving modal similarity the Hamming space. Incomplete labels can inject uncertainty in both and S. To tackle this, propose PCRIL framework in Fig. Herein, novel learnable CLIP prompt for label sets designed to recover potential positive labels in Sec. 3.2 Additionally, an unknown- hokknoow--ommllmenta sample augmentation and negative masking strategy are developed in Sec. 3.3 to deal with hard samples and negative pair scarcity, respectively.
(egnHH(( (H' (f.))
3.2 Prompt Contrastive Recovery
Prompt finetuning is promising solution for lightweight model adaptation. The typical prompting method for CLIP decorates the class tokens, either with a predefined prefix (28) or a vector of learnable tokens [9). A typical handcrafted prompt template for a single unknown class can be "A photo of CLS," where CLS is the class name. However, its fixed tokens and single-class scope hinder it from capturing inter-class complexity for label-wise recovery. In pursuit of separating labels of varied informativeness, we propose encode label sets into CLIP embeddings.
Prompt for Positive Anchors.
1e-3
Frequency
6 4
500 1000
1500
2000 2500 5000
Sorted index of label sets
Figure 3: The sorted frequency his- togram of unique positive label sets in MIRFlickr-25k samples 70% known labels. This long-tail distribution in- duces bias for learning because many rare label combinations in the dataset are associated with limited samples. and ranked the various combinations of labels in Fig. [3] As illustrated, certain dominant combinations can overshadow less frequent cases in the current CMH scheme, obscuring their impact on model fitting. This long-tail distribution of Kÿ can induce bias and disable balanced learning. Furthermore, the number of label sets available is constrained by the samples in the training dataset, which further restricts the learned correspondence. Therefore, we propose measure semantic completeness through contrastive learning paradigm instead, in which we consider an anchor set K; K3. as
Given a sample (fi; gs, 9,,4:) with incomplete labels, the known classes of this instance itive label subset Kj = {c 1 = 1} and the negative one K = {c 1 = 0}. However, directly maximizing agreement between Kÿ and modal representations insuf- ficient to capture its level of completeness. We quantified
be extracted as the pos-

---

positive instance and encode it into CLIP embeddings. For instance, a typical handcrafted template can be "A photo of some seagulls flying above the beach." " for K2 = (seagul1, beach) } and K =sseagul,, beach, sky). It is observed that class names are often surrounded by class-related prefixes and suffixes. Therefore, we further construct a learnable prompt template. The proposed prompting operation is formulated as:
P(K)) = (Phead: ((p")oEk)) Ptast) pº (uf, u..,... um CLS", of, 05 vg))
(I)
where p' is the specialized learnable prompt for the e-th class, Phend E Rd, and Prail E R
are the learnable alass-agnostic prefix and suffix, and (-) represents random permutation operation. specific example of the constructed prompt is given in Suppl. See. [A.]]
Negative Subsets and Contrastive Learning, For the selected anchor label set, dropping any positive class object would degrade its alignment with modalities. The same is true for adding negative classes. Therefore, we aim to detect potential positive labels among the unknown ones by constructing negative subsets relative to the anchor set. Three types of negative subsets are constructed by 1) deleting a class: K = K - {s}. 2) joining a negative class in K Kj K U (t), or 3) replacing a class by a negative one in K; Ki K - { U {4), where & E Kü and / E K; Note that generating a negative set by replacement is equivalent to successively performing deletion and joining. This variant is introduced to improve model robustness.
To learn completeness measurement, we define simple matching score that is compatible with the CLIP prior:
4 e'(K) (((()))) h./T,
where K is a label set, (-) and are the CLIP textual encoder and its temperature parameter, h: represents the average of modal CLIP features Eff)) and Eg(g),, which is analyzed in Sec. 4.4 The contrastive loss between a positive anchor set and negative variant is formulated as
('aa,, K.) max (8' & (Kx) + m, 0).
where K. is a negative variant of anchor set K. and m is a margin parameter that separates sets of different completeness levels. The overall contrastive loss is derived
Cetr
Ck K+))+ + Σ C'( K)) + (KC
(4)
K.CKS
K.
SEK!
EEK;
Notably, the only parameter introduced for contrastive learning is the learnable prompts.
Potential Label Tree Search (PLTS). Through con- trastive learning between label sets, knowledge in the scores is supposed to generalize surement to distinguish potential positive classes in the unknown set K L.e,, an anchor (positive) set can be negative set for another larger anchor set. Therefore, a simple greedy search is designed itive labels. The label search starts with the entire positive set K2 as node with score oe (K)). At each step, we search for an unknown class that maximizes the class set score, and merge it into the positive set. Specifically, we denote the positive and unknown label sets before the w-th iteration as (w((w) and (w), where (1(1)) = K3 and (1) = Ki At the w-th iteration, we try adding each Cu (w) into K2(w) (w) and find the one that maximizes the score. This process is formulated as
Q K3
(w)
hierarchical mea-
axyz
ayx
ayz
0=
recover potential pos-
ax
ay
az
a
K5(1):a. (5(w) - KJ (w)) U (ci) x1(1)xx.y, z. k,(()) --. (ww) u (c. c
Figure 4: An example of the potential label tree search process. Discs represent label sets.
c arg max 4 (<(K U (cw})
(5)
G. (w))((())
We then transfer the optimal et into the positive label set. Ky(w) and (y(w) are updated as (W + 1) (z(W)) U { and Kt(w + 1) ('(W) - {. This process continues with the
5

---

updated sets at iteration w + 1 to further discover potential labels. The termination condition is designed as &' (K((W)) U {ct]) < ('(K)(w")) + m which ensures monotonic increase in the T scores. Suppose it ends at -" iteration, the search stops with a final score of Φ = + (Q)),, where Q = KK is the final recovered positive set. For specific example, Fig. F demonstrates typical PLTS process. An optional choice is to further calculate a pseudo label for each of the remaining unknown classes Cu by - H ( (QU - Φ), where (xx max(0, min(1, 2 = ) is the hard sigmoid function with a linear window of m.
3.3 Augmentation Strategies for Handling Extreme Cases
Although PLTS can recover substantial positive labels, two issues can remain unresolved in highly incomplete cases. After recovery, a large portion of unknown labels can still exist. Meanwhile negative pairs can become extremely scarce at high unknown ratios. To enhance the model for these cases, we further impose the following two augmentation strategies.
Complementary Semantic Augmentation. As PLTS only recovers positive labels, the unknown values cannot be fully determined, leaving uncertainty in similarity learning. Therefore, we propose to eliminate the residual uncertainty by mixing up complementary samples. However, the symmetrical mixup (34] augmentation can be ineffective because the complementary relationship is asymmetric, i.., cases that sample f: is positive set contains an uncertain class of sample but nor vice versa. Hence, we formulate an asymmetric matrix A NxN to express sample matching scores as ō.,- = u[t.]TPl::]] 2e(l) where [[[-] sets unknown elements to and others to 0, and PI-] (N(-) sets only positive (negative) entries to 1, respectively. 6.j measures the proportion of sample i's unknown categories that are positive for sample j. This asymmetrical score evaluates the volume of semantic certainty that sample j can transfer into the semantic background of sample i. Based on the scores, an asymmetric mix-up is introduced for a sample feature f. and its complementary counterpart fi f, with label 1 = lj. where j is selected randomly from the top-K indices on the i-th row of A. The complementary mix-up represented
{ f. X (1 i - XPP(L] (1-x))111
(6)
where X" (X*) is learnable coefficient for the image (text) modality, determining the asymmetry in the complementary mix-up. The formula for the text modality is similar. Through this augmentation,
pairwise relationship sij is adjusted as deterministic values by sij = 1- ---I (1- (2/)).
dappivv Negative Masking. As illustrated in Fig. D negative pairs can disappear drastically at high unknown ratios. The existence of negative pairs is especially vulnerable yet significant in learning robust cross-modal relationships. Naive solutions such as AN (Assume Negative) ( set all unknown values to 0, introducing excessive noisy pairs that mix up with the true negative pairs and inhibit the model fitting. Therefore, we propose adaptive negative masking to restore a correct number of negative pairs. Through semantic augmentation, real- valued similarity matrix S E (I0, 1] U {u)) Dx D is constructed between unknown pairs with batch size D. By denoting u = u((( and the ratio of S 0 to S? 0 as [VSS)///P((' the similarity matrix is adjusted adaptively as
D {
SD (1 (1- U) SD + Ro (Uo
r r r <<t
where denotes the Hadamard product, R E (0, DxD is random mask that sets unknown values to 0 for a number that resets r = l,, and is a threshold close to 0 that prevents noisy similarity structure while slightly enabling AN when there are few negatives. To learn hash codes with these augmentation strategies, the overall hash optimization is defined in Appendix Sec. A.2
4 Experiments
4.1
Experiment Settings
We evaluate our method on the MIRFlickr-25k (Flickr) 12 MS COCO (COCO) 20 and NUS-
WIDE (NUS) (5) datasets. We use the frozen ViT-B-32 CLIP as backbones for all methods and set

---

Table 1: The MAP comprisons Flickr, NUS, and COCO datasets with state-of-the-art CMH methods by different known ratios (30%, 50%, and 70%). We report performance rises in red compared to the second-best results. *: cited results with their original experiment settings. Our proposed PCRIL significantly outperforms both deep and non-deep CMH methods, verifying the ability recover efficient similarity learning.
Method
30% known labels
50% known lahels
70% known lahels
1->11
DCH 33
69.8
65.9
67.8
75.7
70.2
72.9
77.5
72.1
74.8
71.9
SDMCH 26
64.3
67.2
65.8
66.0
73.9
70.0
69.5
76.0
72.8
69.5
SCRATCH
77.3
Flickr
WCHoosh
DCMH 14
63.0
65.2
64.1
67.4
70.2
68.8
71.3
62.5
74.5
62.6
62.6
72.9
68.6
SSAH 17
73.3
76.4
70.3
AGAH
63.4
61.6
7844
76.6
77.5
79.2
81.6
73.6
DCHMT [0
64.1
64.0
64.0
78.3
75.6
76.9
81.0
8000
80.5
73.8
PCRIL
(2.5
(5.4
77.0 (4.8)
854 (3.3)
79.4
$2.4
$7.5
82.2
(4.9
$1,4 (4.1)
SDMCH 26
55.7
59.9
57.8
58.9
61.2
60.0
59.3
62.2
60.7
59.5
SCRATCH C
35.5
64.1
49.8
28.9
67.4
48.2
32.6
686
50.7
49.6
NUS
SSAH 17
DCMH 14
45.3
30.4
40.6
32.4
38.4
57.1
47.8
46.7
35.5
55.3
35.9
33.1
47.9
AGAH U
46.7
49.7
48.2
58.8
49.9
$4.4
66.7
67.2
66,9
56.5
DCHMT 80
35.7
35.0
35.4
57.6
55.9
$6.7
67.3
67.4
67.1
53.1
(ours)
(4.0)
(3.1)
(3.7)
(3J0)
(3.7)
(3.5)
DCH 1R
60.9
61.1
61.0
63.0
63.4
63.2
64.2
64.9
64.5
62.9
SOMCH 2A
53.7
55.5
$4.6
57.3
56.9
S7.1
$8.5
SK.7
58.6
56.8
COCO
SSAH 17
DCMH
49.2
32.0
40.4
36.2
31.1
52.3
53.1
50.5
40.8
52.7
36.7
53.1
55.6
53.0
46,1
$1,3
41.0
AGAH
51.2
55.1
58.5
58.8
58.6
62.4
61.8
58.5
DCHMT
PORIL (OUrS)
(1.9
63.5 (2.4)
63.2 (2.3)
64.0 (1.0)
64.7 (1.3)
64.4 (1.2)
67.8 (3.6)
68.8 (3.9)
68.3 (330)
65.3 (5.44 (2.4)
Table 2: The ablation study results Flickr, NUS, and COCO datasets. B: Baseline CMH method, IU: ignoring unobserved pair relationships, AN: assuming all unknown pairs to be negative, ANM: adaptive negative masking, PCR: prompt contrastive recovery, and CSA: complementary semantic augmentation.
knoos
70%
$7.5
73.4
82.8
62.4
63.3
67.5
50.4
45.9
AN
68.9
76.6
81.5
51.1
53.8
66.2
45.8
54.1
59.8
WT ANM
75.0
78.
83.8
60.6
60.7
68.1
$9.9
614
65.1
ANM PCR
16.3
82.1
84.4
68.0
69.4
70,9
62.4
63.7
67.2
ANM PCR CSA
77.0
82.4
84.9
68.7
69.7
71.4
63.2
64.
68.3
the main hash bit as 32. Details regarding datasets, implementation, evaluation settings, and results for other bit configurations are presented in Appendix Sec. and Sec.
4.2 CMH with Incomplete Labels
To validate the model's effectiveness on CMH with incomplete labels, we illustrate mAP comparison results in Table [ Our model consistently achieves superior results across various known ratios on all benchmarks. We clarify that there is no prior deep CMH method specifically designed for incomplete labels. Therefore, we have to use regular CMH baselines by setting different missing ratios but keeping other settings equivalent. Compared to shallow methods, our method achieves higher results,
Table Ablation results for comparison on the conventional AN setting. Results on the Flickr and COCO datasets are reported.
Flickr
COCO
Method
30% known
50% known
70% known
30% known
50% known
70% known
B w/ AN CSP
68.9
76.6
81.5
45.8
54,3
59.8
76.2
82.3
46.4
59.7
B w/ AN PCR
75.0
79.4
82.9
55.7
58.2
65.8
PCR CSA
75.3
80.2
83,5
58.6
65.2

---

Table 4: Prompt construction variants compared Flickr dataset. The MAP and precisions of recovered positive labels (PRECISION) are reported. Our PCRIL can successfully marry multi-label information with CLIP prior knowledge (compared to Conventional) and yield learned prompts for instance-label matching (compared to Phrasal).
Variant
Prompt Type
MAP
PRECISSON
Learaale
Malti-label
30% known
50% known
70% known
Mean
30% known
50% known
70% known
Mean
75.0
Conventional
76.3
81.8
82.8
80.3
86.0
89.6
87.0
87.5
Ours
77.0
$2,4
84,9
81,4
87.4
89.6
92.0
89.7
Table 5: Prompt search variants compared on Flickr and NUS datasets. Compared to single-modal recovery, our proposed PLTS can perform instance-level matching to produce more precise results. The one-step all variant validates the effectiveness of our recursive label recovery in PLTS.
Dataset
Variant
MAP
PRECISION
30% knowa
50% known
70% known
Mean
30% known
70%
Mean
By image
78.2
79.2
85.3
80.9
86,2
86.6
88.1
87.0
Flickr
By text
74.3
76.6
84.4
78.4
70.1
80.2
77.2
75.8
One-step
77.4
21.0
38.4
Ours
77.0
82.4
84.9
81.4
87.4
89.6
92.0
89.7
By image
$1.3
65.4
68.5
61.7
78.4
76.0
74.9
76.4
By text
50.8
65.1
69.3
61.7
69.4
69.1
68.9
69.1
NUS
One-step ste-stepaat all
48.4
64.9
67.5
60.2
12.1
23.7
27.1
21.0
Ours
68.7
69.7
71.4
69.9
79.5
78.1
80.3
79.3
especially on Flickr, with 4.4% improvement average, validating the representation ability of our method. In comparison with deep CMH methods, our model enjoys greatly improved results on all three datasets, with a mean 8.96% mAP enhancement. Some deep CMH methods perform worse with incomplete labels especially on the NUS dataset, indicating their collapsed similarity learning. In comparison with DCHMT, which also adopts CLIPs as backbones, the proposed PCRIL obtains an average of 12.13% mAP improvement on all datasets. These results demonstrate that our model can consistently obtain superior results across unknown ratios and benchmarks, validating the effectiveness of our PCRIL in label recovery and similarity calibration for CMH.
4.3 Ablation Study
To verify the validity of each module, conduct an ablation study of the proposed method by comparing it with variants shown in Table 2 on the Flickr, NUS, and COCO datasets, and module variants exclusively baselines (IU and AN), our ANM-enhanced baseline surpasses them by 6.66% and 6. 6.09% mAP. respectively. This verifies the effectiveness of the balanced pairing strategy. By adding PCR, the model's 's performance is significantly improved by an average of 3.52%, validating its ability to detect incompleteness and recover labels. Although initial recovery by ANM and PCR substantial, by adding CSA, the model further gains consistent improvements of 0.60% average across all datasets, verifying ability of pairwise similarity augmentation. The two modules together make joint improvement of 4.12% on average, while the entire PCRIL rescues more than 10% of mAP from unknown labels. ablation studies within the AN setting (Table 3 our proposed components consistently deliver stable improvements, highlighting PCRIL's ability to overcome limitations inherent traditional training schemes. These results have comprehensively verified the effectiveness of PCRIL for CMH with incomplete labels.
the conventional AN setting. In contrast with the conventional
4.4 Model Analysis
Prompt Construction. We analyze our proposed label prompt learning by comparing it with two variants: 1) phrasal: handerafted prompt in CLIP's original template of "A photo of CN CN and CN".." constructed to directly perform recovery search, and conventional: averaging single-class prompt embeddings to acquire the label prompt embeddings. The recovery results are shown in Table 4 Compared with handcrafted prompts, our learnable prompts achieve a 6. % mAP enhancement and improve the restoration precision by 22.33%, verifying the effectiveness of

---

K rale [plants] Lite
-11
[tesple]
[[s]]
+1
PLTS Ateration:
PLTS steration:
[CI]
+1
Abbreviations
[Co]
Kp
Positive
K
[People]
[c]]
[Clouds]
Label Set
[Structune]
[P3]
[3k]
[plants]
Sky.
+
[3s]
[Sunset]
Set Scores
[transport]
[SR]
[structurc]
PLTS Steration:
Figure 5: The recovered classes and scores of 3 case images w.E.. search iterations. For brevity, only show top-3 results at all steps. The recursive recovery of potential classes results in successive increases in the set scores.
recovered recall (8)
recovered precision (%)
MIRFlickr-220 25k
NUS-WIDE
30
96-
100
28-
89
% of pairs
50-
60
P.
OF pairs
66-
p.
n.
Inage
78
Inage
40
40
U.
Text
20
22
55-
Both
epoch
epoch
unknown labels (%)
unknown labels
(a)
(b)
(c)
(d)
Figure 6: Recovery of labels and deterministic pairs. The left 2 subgraphs: the (a) recall and (b)
precision of recovered positive classes w.zl initial epochs of prompt tuning. The right 2 subgraphs: pairwise similarity recovery by (c) complementary semantic augmentation and (d) prompt contrastive recovery respectively, on the Flickr and NUS datasets. 'p', 'n', and 'u.' stands for positive, negative, and unknown, respectively. Dashed lines are corresponding results w/o applying our modules.
learned prompts. Compared with conventional prompts, our method achieves consistently improved performances, further validating the effectiveness of the structural prior utilized in our method.
Prompt Search for Recovery. We validate our prompt recovery with three variants: 1) image: label recovery search with the image modality only, 2) text: label recovery search with the text modality only, and 3) one-step all: directly recovering pseudo-labels on the initial positive label set KZ(L) without searching, which means Q (1). The results are shown in Table 5) Compared with image and text recoveries respectively, our recovery with both modalities displays about 3% and more than 10% improvements. This reveals our model's ability to evaluate the joint completeness in label prompts. Performing "one-step all" recovery, it hardly recalls true positives but still achieves acceptable mAPs, validating the latent hierarchical structure of the matching scores. It can be inferred that the prompt search works by pecking at scores for confident classes that aid completeness the most. Meanwhile, we plot how the number and precision of recovered positives change through the beginning epochs w.E.f modalities in Fig. (a-b). The precision rises with the growth of epochs while recalling more ground-trui labels, validating that the prompt gradually separates more complete and more incomplete semantic descriptors to make true positives emerge. Recovery with a single modality can introduce a drop precision, where the text modality has especially low results of 77.16% mainly due to its nature of partial reference to labels. However, recovery with both modalities surprisingly boosts the precision to achieve a more effective 89.78%.
Matching Scores Visualization. To further analyze the property of descriptive completeness, we illustrate some case images' matching scores during the PLTS process in Fig. [ In the results, the recovery of potential positive class is accompanied by an increase in the set score, indicating that the recovered label set more consistent with the instance. For example, the score increases from the original 25.38 to its best 31.76 by successively recalling sky, clouds, and sunset for the bottom-left instance. We can observe that class rankings can vary at each step, indicating that the PLTS can generate more precise measurements of multi-label complexity. Meanwhile, though not recovered as positives, the remaining scores (in orange discs) of true positives can still surpass their anchors from the last iteration, validating their effectiiveness pseudo-labels.

---

Enhancement of Pairwise Similarity. As shown in Fig. | CMH can suffer from severe pairwise uncertainty. By adopting our proposed prompt contrastive recovery and complementary semantic augmentation, respectively, the similarity structure is efficiently recovered in Fig. [ (c-d). The complementary semantic augmentation greatly alleviates the pairwise scarcity. For situations at approximately 50% unknown where negative pairs are completely lost, substantial 20% proportion of negative pairs are reconstructed. ensuring its ability to rescue similarity learning at high unknown ratios. Mcanwhile, the prompt contrastive recovery significantly improves the distribution of both positive and negative pairs via its pseudo-labeling, reducing an average of 80% uncertain pairs, verifying its strong separability for completeness of labels. Generally, these two modules ensure the method can learn CMH on sufficient data pairs with clear relationships.
Conclusion
In this paper, identified the problem of incomplete labels and the consequent collapse of similarity learning in CMH. To overcome these challenges, we introduced Prompt Contrastive Recovery for CMH with Incomplete Labels (PCRIL), novel framework that jointly recovers semantic classes and pairwise similarity. This is the first CMH method to enable prompt learning with incomplete labels. Specifically, by constructing contrastive learning with a hierarchical label prompting method, prompt contrastive recovery learns the completeness of label descriptors and detects lost labels. Moreover, complementary semantic augmentation eliminates the sparsity of semantic pairs via complementary feature blending strategy to restore similarity. An adaptive negative masking strategy is adopted to further balance the pairwise hashing. Extensive experiments on widely used benchmarks validated that PCRIL can significantly outperform state-of-the-a CMH methods with different partial levels.
Limitations. This study highlights the effectiveness of the vision-language model prior in perceiving label completeness, specifically for cross cross-modal retrieval. However, we note that it does have limitations. The prompt construction currently relies on the pretrained CLIP model with limited number of text tokens, which hinders its ability for richly labeled samples. However, annotation length exceeding the CLIP capacity is extremely rare in practice. Mcanwhile, the learning of PCR relies on sufficient multi multi-labeled samples. Single positive cases (6] may infrequently exist in real applications. We suspect that allowing unknown labels in the anchor sets could reactivate the proposed paradigm.
Acknowledgments and Disclosure of Funding
This research partially supported by Shenzhen Science and Technology Program (Grant No. (CYXX02210000928227777 National Natural Science Foundation of China (Grant No. 62372132), and the Australian Research Council (Grant No. DE240100105, DP240101814, DP230101196).
10

---

References
[1] Bucak Serhat Selcuk, Jin Rong, Jain Anil K. Multi--abel learning with incomplete class assignments // Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. 2011. 1. 2801-2808.
[2] Cao Yue, Liu Bin, Long Mingsheng. Wang Jiammin. Cross-modal hamming hashing // Proceed- ings of the European Conference on Computer Vision. 2018. 202-218.
[3] Chen Zhen-Dus Li Chuan-Xiang, Luo Xin, Nie Liqiang, Zhang Wei, Xu Xin-Shun. SCRATCH: A scalable discrete matrix factorization hashing framework for cross- moos--odaal retrieval // IEEE Transactions on Circuits and Systems for Video Technology. 2020. 30, 7. 2262-2275.
[4] Chu Hong-Min, Yeh Chih-Kuan, Wang Yu-Chiang Frank. Deep generative models for weakly- supervised multi-label classification // Proceedings of the European Conference on Computer Vision. 2018. 400-415.
[5] Chua Tat-Seng, Tang Jinhui, Hong Richang, Li Haojie, Luo Zhiping, Zheng Yantao. NUS-WIDE a real-world web image database from National University of Singapore // Proceedings of the ACM International Conference on Image and Video Retrieval. 2009, 48.
[6] Cole Elijah, Mac Aodha Oisin, Lorieul Titouan, Perona Pietro, Morris Dan, Jojic Nebojsa. Multi-label learning from single positive labels // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. 933-942.
[7] Deng Jia, Russakovsky Olga, Krause Jonathan, Bernstein Michael S, Berg Alex, Fei-Fei Li. Scalable multi-label annotation // Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2014. 3099-3102.
[8] Ding Zixuan, Wang Ao, Chen Hui, Zhang Qiang, Liu Pengzhang, Bao Yongjun, Yan Weipeng, Han Jungong. Exploring Structured Semantic Prior for Multi Label Recognition with Incom- plete Labels // Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. 2023. 3398-3407.
[9] Durand Thibaut, Mehrasa Nazamin, Mori Greg. Learning a deep convnet for multi-label classification with partial labels // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. 647-657.
[10] Fu Zheren, Mao Zhendong, Song Yan, Zhang Yongdong- Learning Semantic Relationship Among Instances for Image- Matching // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 15159-15168.
[11] Gu Wen, Gu Xiaoyan, Gu Jingzi, Li Bo, Xiong Zhi, Wang Weiping. Adversary guided asymmetric hashing for pross-modal retrieval // Proceedings of the 2019 on International Conference on Multimedia Retrieval. 2019. 159-167.
[12] Huiskes Mark Lew Michael The mir flickr retrieval evaluation Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval. 2008. 39-43.
[13] Jiang Ding, Mang. Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 2787-2797.
[14] Jiang Qing- Yuan, Li Wu-Jun. Deep cross-modal modal hashing Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 3232-3240.
[15] Kim Wonjae, Son Bokyung. Kim Ildoo. Vilt: Vision-and- lan-uuuuee transformer without convolu- tion region supervision // Proceedings of the International Conference Machine Learning. 2021. 5583-5594.
[16] Kim Youngwook, Kim Jae Myung, Akata Zeynep, Lee Jungwoo. Large loss matters in weakly supervised multi-label classification // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 14155-14165.
[17] Li Chao, Deng Cheng, Li Ning, Liu Wei, Gao Xinbo, Tao Dacheng. Self-supervised adversarial hashing networks for cross-modal retrieval // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. 4242-4251.
[18] Li Junnan, Li Dongxu, Savarese Silvio, Hoi Steven. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models // Proceedings of the International Conference on Machine Learning, 2023. 19740-19742.

---

[19] Li Jumnan, Li Dongxu. Xiong Caiming, Hoi Steven. Blip: Bootstrapping language-ing image pre-training for unified vision-language understanding and generation // Proceedings of the International Conference on Machine Learning. 2022. 12888-12900.
[20] Lin Tsung-Ys Yt, Maire Michael, Belongie Serge, Hays James, Perona Pietro, Ramanan Deva, Dollár Piotr, Zitnick C Lawrence. Microsoft coco: Common objects in context // Proceedings of the European Conference on Computer Vision. 2014. 740-755.
[21] Lin Zijia. Ding Guiguang, Hu Mingqing, Wang Jianmin. Semantics prenntiispr hashing for cross-view retrieval // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. 3864-3872.
[22] Liu Xuanwu, Yu Guoxian, Domeniconi Carlotta, Wang Jun, Xiao Guoqiang, Guo Maozu. Weakly supervised cross-modal hashing // IEEE Transactions on Big Data. 2019. 8, 2. 552-563.
[23] Liu Yishu, Wu Qingpeng, Zhang Zheng, Zhang Jingyi, Lu Guangming. Multi- Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval Proceedings of the 31st ACM International Conference on Multimedia. 2023. 893-902.
[24] Lüdddecke Timo, Ecker Alexander. Image segmentation using text and image prompts // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 7086-7096.
[25] Luo Haoyang, Zhang Zheng, Nie Liqiang. Contrastive incomplete cross-modal hashing // IEEE Transactions on Knowledge and Data Engineering. 2024. 36, 11. 5823-5834.
[26] Luo Xin, Yin Xiao-Ya, Ya, Nie Liqiang, Song Xuemeng, Wang Yongxin, Xu Xin-Shum. SDMCH: Supervised discrete manifold- ranifold-embeddde cross-modie hashing. // Proceedings of the International Joint Conferences on Artificial Intelligence. 2018. 2518-2524
[27] Ni Haomin, Zhang Jianjun, Kang Peipei, Fang Xiaozhao, Sun Weijun, Xie Shengli, Han Na. Cross-modal hashing with missing labels // Neural Networks. 2023, 165. 60-76.
[28] Radford Alec, Kim Jong Wook, Hallacy Chris, Ramesh Aditya, Goh Gabriel, Agarwal Sandhini, Sastry Girish, Askell Amanda, Mishkin Pamela, Clark Jack, others. Learning transferable visual models from natural language supervision // Proceedings of the International Conference on Machine Learning. 2021. 8748-8763.
[29] Sun Ximeng, Hu Ping, Saenko Kate. Dualcoop: Fast adaptation to multi- recognition with limited annotations // Proceedings of the Advances in Neural Information Processing Systems. 2022. 35. 30569-30582.
[30] Tu Junfeng, Liu Xueliang, Lin Zongxiang, Hong Richang, Wang Meng. Differentiable cross- modal hashing via multimodal transformers // Proceedings of the 30th ACM International Conference on Multimedia. 2022. 453-461.
[31] Veit Andreas, Alldrin Neil, Chechik Gal, Krasin Ivan, Gupta Abhinav, Belongie Serge. Learning from noisy large-scale datasets with minimal supervision // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 839-847.
[32] Xu Xing, Shen Fumin, Yang Yang, Shen Heng Tao, Li Xuelong. Learning discriminative binary codes for large-scale cross modal retrieval // IEEE Transactions on Image Processing. 2017. 26,
5. 2504-2507.
[33] Yang Hao. Zhou Joey Tianyi, Cai Jianfei. Improving multi- learning with missing labels by structured semantic correlations // Proceedings of the European Conference on Computer Vision. 2016. 835-851.
[34] Zhang Hongyi, Cisse Moustapha, Dauphin Yann N. Lopez-Paz David. Mixup: Beyond empirical risk minimization // arXiv preprint
2017.
[35] Zhang Zheng, Luo Haoyang, Zhu Lei, Lu Guangming, Shen Heng Tao. Modality -invaiatt asymmetric networks for cross-modee hashing // IEEE Transactions on Knowledge and Data Engineering. 2023. 35, 5. 5091-5104.
[36] Zhou Kaiyang, Yang Jingkang, Loy Chen Change, Liu Ziwei. Learning to prompt for vision- language models // International Journal of Computer Vision. 2022. 130, 9. 2337-2348.
12

---

Appendix
In the following sections, we provide detailed information about our model (Sec. A) and experimental settings (Sec. B). and offer more extensive validations (Sec. [ of our proposed PCRIL.
A
Model Details
A.1
Constructed Prompts
We have formulated our constructed prompts in Eq. (T). Here, describe specific example of prompt construction. Assume we have sample with a positive label set (cat, wall, house) and a negative one (dog, plane). A naive textual prompt can be formulated as 'A photo of cat, next to the wall of house.', while our prompt is constructed by selecting an anchor subset (e.g., Ca=(wall, house)) to form P(C.) (phead, phouse r Prait) with randomly shuffled class order by Then, we can directly construct two negative prompts P(C)) n) = (Phead: p PPoait and P(C)) (phead: ydog, house, poaus,,pal,, 1 -tait) where the negative sets C==(wal1] and C,=(dog, house, wall) are formulated by adding the negative class dog and removing the positive class s = house, respectively. We can also construct another negative prompt P(C.) (Pheads pPlane, p Pras) by replacing class wall with the negative one plane in C.. Both positive and negative classes are uniformly selected in our experiments. We illustrate our prompt construction and contrastive learning in Fig.
Anchor [Rench Sen
[Beach]
Sky
Pull together
Sample
Dusert Pann
(Tormination)
Tamnl Desert Horre
Negative
[Straet] Car
Margin
Sumple
[coma]]
Unknown
[Nerson]
Conbinations
Push
Csuach
Limited Sarple-Label
Delete
Distance to Target
[concl)
Iteration
Randos Select
[Came Room
Sea Kg
Eariched
Pairs
Join
[Cans]]
Margin
Sample's Positive Class Set (Incomplete)
Anchor
Replace
(s) The motivation of selecting positive anchor sets to enrich sample-label pairs.
(b) The relationship of the anchor and negative subeots.
The potental label tree search with contrastively learned label embeddings
(c)
Figure 7: The motivation and relationship of different components in our proposed prompt contrastive recovery.
A.2 Optimization
To learn hash functions, the general CMH methods 14 17 optimize the following minimum negative log-liieldood objective:
min 0 B
logg Pm (S
Φ; 0 + C
s.t.
B (E(,,,))
(8)
where L = (F) BI
(T) - BI? the quantization terms and Hu. are the visual and textual hash layers. By defining the conditioned probability of the similarity as
5(0.g) Pm((ssl0155 0%. { 1 6(0)))
Sij 1
Siy =0
where o;j = - (f)) (II* and 8()) T the objective function learns align the inner-product similarity of hash codes with the ground-truth similarity values.
However, the estimated ground-truth similarity in our work, either generated by recovered pseudo labels or by complementary blending, does not apply to the above probability due to its non-binary
13

---

Algorithm Optimization Algorithm for PCRIL
Require: Training dataset O.
Ensure: Optimal discrete hashing codes B, prompt variables OP, modal hash-net parameters 0% 0% Initialization
1:
Randomly initialize learnable parameters. Configure the learning-rate μi; the mini- batch size D; iterations EPp. Exp.ne p.net and EPhash-
2: while iter; p.nctt & not converged do
3: Sample a minibatch (fss 9s: b0
4: Sample anchor set K2 for each label and generate its negative sets k2,, K3 and Ki
5: Compute loss C and update 0":
6: end while
7: Recover labels and pseudo labels by prompt search stated in Sec. 3.2
8: while iter, < Phhash & not converged do
9:
Sample minibatch (f:: gs. :
10:
Sample and complementarily mix up instances with Eq. 6
Compute similarity supervision and randomly transfer unknown pairs as negatives by Eq.
12:
Compute loss Chash and update fu.
Update code B with B sign (H)(F)))FFFFF)))F)))) + (G))
14:
end while
property. Therefore, we formulate learnable objective by leveraging the Kullback- divergence as
min DRL (P )).()) + C gv.
s.t. pa(si)) = {
8ijn 8ij 1
81js
8ij = 0
B (-1,11)
(10)
where (-ll-) is the Kullback-Leibler divergence, and kig E [0, 1] the estimated ground-truth similarity. Taking Eq. (9) into account, the final hashing objective is derived as
min Chash B
Σ
[Bis ul + L
s.t. B
(11)
where Laim = log(1 + cou) - 8.jDij. The detailed batch-wise learning process for PCRIL is
displayed in Algorithm [
B Experimental Settings
B.1 Datasets
We evaluate our method extensively on three widely employed benchmarks for CMH, namely, MIRFlickr-25K (I2) NUS-WIDE (NUS)(5), and MS COCO (COCO) Flickr consists of 20, 015 image-text pairs that belong 24 semantic classes in which 15% annotated as positive. NUS is large-scale cross- cross-modal dataset with about 300, 000 pairs of images and texts in 80 categories. The most frequent 21 categories are selected form a labeled set where 10% are positive. COCO is cross-modal dataset consisting of about 120, 000 image-text instances associated with semantic classes, among which confirmed positives make up only 3.6% of label elements, making it rather harder dataset as shown in Fig. [ As we stated in Sec. 3.1 we randomly select given proportion of binary labels and mask them as unknown during training, and unmask them for retrieval inference.
14

---

Table 6: Data split for Flickr, NUS, and COCO in our experiments.
Dataset
Test (query)
Dataset
Train
Total
Flickr NUS
2,000
18,015
18,015
20,015
2,085
193,749
30,000
195,834
COCO
5,000
82,081
25,000
87,081
Table 7: The analysis for hyperparameter's impact the model's performance the Flickr dataset. Evaluated parameters contain the number of complementary samples (K) for each sample and the value of margin (m) used in prompt contrastive learning and PLTS process.
= 10
m
1.00
m
MAP
PRECISION (%)
K
MAP
0.25
0.826
86.76%
0.823
0.50
0.827
88.82%
10
0.824
1.00
0.824
89.61% %
0.822
2.00
0,820
90.78%
50
0.823
3.00
0.816
91.54%
100
0.825
4.00
0.817
91.29%
200
0.826
5.00
0.814
91.81%
500
0.824
B.2
Data Split
For the three evaluated datasets, we select larger training subsets compared to general CMH methods better approximate the real incomplete label scenario. The statisties of evaluated datasets summarized in Table 6 All experiments conducted have adopted this same configuration.
B.3 Implementation Details
We simply adopt publicly available ViT-B-32 based CLIP backbones and frozen weights for both image and text feature encoders of our PCRIL and other compared methods. Upon them, two 4-layer MI.Ps with the tanh function as the last activation project the 512-dimensional image and text CLIP features into 32-bit binary codes, respectively. Other layers ReLU as the activation function. The from input to output. The MLP parameters are
MLP layer dimensions are 512- 112-10024-2,048-51222 initialized with normal distributions and optimized with Adam optimizers for each modality. The learning rate is 10-4.5 4.5 for image and 10 for text.
For the label prompt encoder, encode the prompts text CLIP features and only tune the parameters in the prompts. These parameters are randomly initialized with a normal distribution. For prompt learning, we finetune the prompt and MI.P parameters while keeping the weights of CLIP encoders fixed to preserve prior knowledge.
the Adam optimizer with learning rate of 10 10-4. During training,
We set the number of learnable tokens, specifically n, m, nuy and ma, as for simplicity. We empirically set t = 0.01 as the ANP threshold and search for the best values for the margin and the top K number, which are analyzed Sec. C.2 We run all our experiments with 2 NVIDIA 2080ti GPUs.
B.4 Evaluation Settings
We evaluate the model effectiveness W.E.f. a range of known ratios. From slightly incomplete to almost untrainable, the selected ratios are 70%, 50%, and 30%. To evaluate the retrieval ability of CMH models, we leverage the mean average precision (mAP) our evaluation metric following common CMH works. When further analyzing our method in Sec. 4.4 and Sec. we also show the
precision of recovered positive labels evidence.
15

---

Missing: animal
Missing: female
Missing: water
Figure 8: The heatmap case visualization of the learned prompt net output. In every example: left: untrained prompts; right: trained prompts. Our learned prompts contrastively learn to attend to objects of potential classes compared the untrained prompts.
BASELINE(FLICKR-225
RECOVERED
Figure 9: The t-SNE visualization for the binary codes in the bascline (AN) and our PCRIL. Colors correspond classes. The compact clusters of semantically similar sample points verify our effective learning of discriminitive hash codes.
C More Experimental Analyses
C.1 More Results for CMH with Incomplete Labels
We further validate PCRIL for CMH with incomplete labels the IAPR TC-12 dataset. As shown in Table 8, stable improvements are achieved especially with fewer known labels, validating the effective label recovery and hash learning of the proposed method.
Table 8: Comparison results the IAPR TC-12 dataset.
Known ratios
30%
50%
70%
i>
mem
meun
L->
mean
SSAH DCMHT
31.5
33.4
33.1
45.3
42.3
48.3
45.3
39.1
39.2
48.0
55,9
56.3
PCRIL (ours)
45.3
45.6
45.5
49,2
48.8
49.0
56.9
55.4
56.2
C.2 Hyper-parameters
We further analyze the contrastive margin parameter and the complementary blending top-k in our work. The results demonttaee that our method is generally non-sensitive parameter changes. Although it obtained a slightly lower recovery precision, the proposed prompt contrastive recovery can achieve its best performance of 82.7% mAP at lower margin m 0.50 by recalling more positives. Higher margins prioritize precision over recall and result in drop in the overall mAP performance. When fixing m, the model achieves stable results around 82.4% MAP for different top-K values.
16

---

Table Comparison with SOTA CMH methods at the hash code lengths of 16, 32, and 64 on Flickr. Our PCRIL can perform consistently better across various code lengths.
30% known
50% known
70% known
Bits
Method
1-1
[- i
Mcan
1- I
[-+
Mean
i- I
[
Mean
AGAH
72.0
69.9
71.0
79.2
75.7
77.5
81.1
77.3
79.2
DCHMT
68.1
68.3
79.1
77.1
78.1
83.3
80,9
PCRIL
77.7
75.6
76.7
83.3
77.2
80.3
86.9
79.6
83.3
AGAH
59.8
63.4
61.6
78.4
76.6
77.5
84.1
79.2
81.7
32
DCHMT
64.1
64.0
64.1
78.3
75.6
77.0
81.0
80.0
80.5
(ours)
75.4
77.0
85.4
79.4
AGAH
63.5
64
DCHMT
72.3
71.1
71.7
77.3
75.6
76.5
78.9
76.8
77.9
PCRIL (ourS)
79.5
76.5
78.0
85.1
77.7
81.4
88.5
80.8
84.7
C.3 Completeness Attention Visualization
By treating each step of searching as a classification task, we list several case images in Fig. 8 with their feature heatmaps overlaid, which generated with the ResNet-50 image backbone. From the results, we can observe that the learnable prompts successfully steer the model prior to be sensitive to potential classes. By attending more the unlabeled object, the trained model can precisely detect incompleteness in label prompts, verifying its effectiveness in semantic detection and recovery.
C.4 Feature Visualization
We compare the t-SNE results of learned binary features with baseline outputs in Fig. 9 In the results, our CMH learning with recovered semantics can gather hash codes of similar samples into compact clusters. Our PCRIL generates semantically more discriminative codes than the baseline method, hence achieving higher performance.
C.5 CMH of Different Code Lengths
We compare method with others at 16 and 64 bits in Table From the results, PCRIL is validated to perform consistently well with mean mAP values across hash bits. Meanwhile, larger bits often produce higher average performance for CMH with incomplete labels, which is consistent with cases in fully labeled CMH.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper proposes novel prompt contrastive recovery method for cross- modal hashing with incomplete labels and validates its superior recovery ability through extensive experiments, accurately matching the main claims in the beginning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: Non-theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The architecture is fully described in the main paper. The implementation details are briefly introduced in the main paper and fully disclosed in the supplementary sections. Source code can be found in the supplementary materials for reference.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided code and instructions about data and code the supplemen- tal material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ btiaps..ccpuulic/guue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We briefly notify the key information of datasets and backbone settings in the main paper while provide full experimental details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although our results are acquired by averaging multiple runs, it's too computa- tionally expensive to run enough times and the space put error bars is too limited. Various original results need to be presented and they already occupy a significant portion of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide computation resources used in our experiments in the supplemen- tary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research in the paper fully conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Although our paper tackles the incomplete labels which is related to the robustness of models, there is direct association societal aspects.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper solves incomplete labels in cross-modal hashing and does not have these risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [No]
Justification: The CLIP model (MIT License) and the three datasets well cited and introduced our paper. However, these datasets contain internet images with custom unable find.
licenses (in CC) and licenses
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The released code well documented with codebase repository link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "bgiLNJzx5s", "reviewer": "Reviewer_FiXs", "summary": "In this paper, to solve the problem of unknown labels in the task of cross-modal retrieval, the authors aim to progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels.", "questions": "Please see aboved  weeknesses.", "limitations": "The explanation of the main contribution part should be improved. The authors should give more motivation and related theory to verify its soundness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)Compared with existing related works, the proposed method has a large performance improvement.\n(2)The author proposed a new strategy to solve the problem of label missing in thr cross-mdoal retriveval task.", "weaknesses": "(1)The meaning of the figure 3 is unclear. What is the meaning of “sorted index of label sets?” what is it relation to the positive label subset of each sample? How can we draw the conclusion that “the number of label sets is limited to the number of training samples.” from figure 3?\n\n(2)What is the motivation and theory of the Negative Subsets and Contrastive Learning?  For the first type of negative subsets, why the anchor set is changed to negative subset by deleting a positive label and the difference between $K_a^i$ and $K_d^{i,s}$ should be minimized in the loss function of equation 4?\n\n(3)In Potential Label Tree Search section, how to compute the score $Phi$? What is the theory of the termination condition? How to get this observation?\n\n(4)The content is somewhat disjointed of the positive anchors and the PLTS.\n\n(5)The ablation study setting is not reasonable. As can be seen, ANM and CSA are two augmentation strategies for handing extreme cases. The main  contribution of the paper is the Prompt Contrastive Recovery. Thus, variant version like B w/PCR, B w/PCR+CSA, B w/CSA should also be explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer `FiXs`'s Comments\nThank you for your thoughtful review and positive feedback. We are pleased that you recognize the improvements and the effectiveness of our method. Your comments are greatly appreciated and please find our point-to-point response below\n\n> Q1. The meaning of the figure 3 is unclear. What is the meaning of “sorted index of label sets?” what is its relation to the positive label subset of each sample? How can we draw the conclusion that “the number of label sets is limited to the number of training samples.” from figure 3?\n\n**(We illustrate our following points in the PDF Figure 1 for better clarity.)**\n\n**Meaning of Figure 3.** Please find our explanation to the meaning of Figure 3 in the global response. Suppose annotation $l_i = (1,1,0,u)$ indicates the presence of the first 2 classes `sky` and `sea`. The positive label set is therefore $K^i_p=(sky, sea)$. For Figure 3, the x-axis of the plot is sorted by the frequency of each set. Therefore, “sorted index of label sets” simply means label set index, sorted based on frequency.\n\nThe Figure 3 illustrates the **long-tailed distribution** of unique positive label combinations, which means there are many **rare** label combinations in the dataset associated with **limited samples**. For instance, although the above $K^i_p$ is common, the subset $\\{beach, sea\\}$ without `sky` can appear very few times. The lack of training samples of rare label sets can cause **severe bias** in typical learning systems, which only align $x_i$ with $K^i_p$. Therefore, this motivates us to consider contrastive learning on anchor sets (subsets of $K^i_p$) to help recover unknown labels.\n\n> Q2. What is the motivation and theory of the Negative Subsets and Contrastive Learning? For the first type of negative subsets, why the anchor set is changed to negative subset by deleting a positive label and the difference between $K_d^i$ and $K_a^{i,s}$ should be minimized in the loss function of equation 4?\n\nThe **motivation** behind using Negative Subsets and Contrastive Learning in our model aligns with our discussion in Q1 about handling rare label combinations. The anchor sets are selected randomly to help **enrich** the label-sample pairs. However, due to reduced positive classes, they do not perfectly align with the sample. Nonetheless, we can solve this dilemma by considering the missing of labels as relative *edit distances* to the full labels. For instance, when a positive tag is missing in the anchor, the *edit distances* to both the full and the anchor sets increase and the embedding (or specifically, learnable prompt) should be adjusted to reflect this change. This adjustment ensures that the model can better differentiate between similar but distinct items, improving its overall precision.\n\nTo clarify a potential misunderstanding in the comment: our goal is **not** to minimize the difference between $K_d^i$ (the negative subset) and $K_a^{i,s}$ (the anchor set with positive labels). Instead, our objective **maximizes** this difference. In our loss function Eq.(4), we employ a contrastive loss strategy that seeks to **enlarge the discrepancy** between embeddings of the original anchor set and those of the negative subset. This discrepancy forces the model to create more distinct embeddings for negative subsets, thereby pushing the embedding of the anchor set with positive tags to be more accurate and distinct.\n\n> Q3. In Potential Label Tree Search section, how to compute the score? What is the theory of the termination condition? How to get this observation?\n\n**PLTS Scores.** The score $\\Phi$ during PLTS is defined the same as Eq.(2) and is the same score used in Eq.(4). To compute the score $\\Phi^i(K)$ for a given label set $K$ and sample $\\pmb  x_i$, we first construct a prompt $P(K)$ according to Eq.(1), then compute its CLIP score with the instance by Eq.(2).\n\n**Termination Condition.** The termination condition $\\Phi^i(K^i_p(\\omega^*) \\cup \\{c^*_u\\}) < \\Phi^i(K^i_p(\\omega^*)) + \\frac{m}{2}$ is associated with $m$, which is the margin we used in the contrastive loss Eq.(3) to separate different levels of sample-label similarity scores, i.e., $\\Phi$.\n\nIn our global response, we explain that by optimizing Eq.(3), our method separates different label sets by a gap proportional to the edit distance. The termination condition yields completed labels with largely increased CLIP scores. In our method, we empirically set the margin *during PLTS* as $\\frac{m}{2}$ to attach more importance to recall over precision.\n\n> Q4. The content is somewhat disjointed of the positive anchors and the PLTS.\n\nAs also responded to comment 1-1, Figure 3 demonstrates the long-tailed distribution of samples' unique label combinations, which includes many rare cases and some dominating cases. The positive anchors are chosen randomly at each training step to enable coverage of a broader class combinations. For instance, the combination of (`sky`, `beach`, `sea`) is a common one while only (`beach`, `sea`) is quite rare. Our method selects random anchor sets that can effectively involve uncommon cases. As the anchor sets are randomly selected at each training step, this covers most class subsets during training. Due to this randomness, **an anchor (positive) set can be a negative set for another larger anchor set**. Therefore, each positive anchor found in the PLTS iteration $j$ is regarded as a negative set in the $(j+1)$-th iteration.\n\n> Q5. The ablation study setting is not reasonable. As can be seen, ANM and CSA are two augmentation strategies for handling extreme cases. The main contribution of the paper is the Prompt Contrastive Recovery. Thus, variant version like B w/PCR, B w/PCR+CSA, B w/CSA should also be explored.\n\nWe provide ablation results with B w/PCR, B w/PCR+CSA, and B w/CSA in Table 1 in the PDF file. Stable improvements by our proposed components have been made to the default AN setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ypml8XwPyj", "reviewer": "Reviewer_MwQQ", "summary": "The authors propose a novel approach, Prompt Contrastive Recovery for Incomplete Labels (PCRIL), for cross-modal hashing with incomplete labels in this paper. They utilize a learnable CLIP prompt to encode selected anchor class combinations and employ a contrastive learning paradigm to construct multiple negative variants of the anchor set. Additionally, they introduce tree search methods for label recovery and develop augmentation strategies to handle extreme cases of unknown labels and negative pair scarcity. Extensive experiments on various datasets validate the effectiveness of their approach.", "questions": "How does the reliance on the CLIP model's text token limit impact the overall performance of your approach? What are the specific challenges associated with needing sufficient multi-labeled samples?", "limitations": "The authors have identified and acknowledged several limitations of their work. And they have mentioned the limitation of the prompt construction relying on the pretrained CLIP model with a limited number of text tokens, but would it be beneficial for the authors to provide more details on how this limitation specifically affects the performance of their model.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates strong originality by combining a learnable CLIP prompt, contrastive learning paradigm, and tree search for label recovery in cross-modal hashing. It presents a well-founded and thoroughly tested solution, with comprehensive analysis and experiments that validate its effectiveness. The writing is clear and organized, making the methodology easy to understand. The contributions offer new insights and advancements that can benefit both researchers and practitioners in the field.", "weaknesses": "The paper could highlight its contributions by providing a detailed comparison with existing methods and including more recent studies in the related work section. A deeper theoretical analysis explaining the effectiveness of the proposed methods is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer `MwQQ`'s Comments\n\nWe sincerely appreciate your detailed review. We would thank your recognition of the originality and effectiveness of our approach, as well as your acknowledgment of our comprehensive analysis and clear presentation. Your feedback is highly valued and encouraging for our work.\n\n> Q1. The paper could highlight its contributions by providing a detailed comparison with existing methods and including more recent studies in the related work section.\n\nExisting studies in multi-label learning regarding incomplete labels mainly focus on single-modal recognition tasks. Compared with them, our method tackles missing labels in a **multi-modal problem**, i.e., Cross-Modal Hashing (CMH), in which **pairwise relationship** can also become sparse. Most of the related methods in classification tasks are hardly adopted directly into CMH due to distinct learning schemes. For instance, DualCoOp [1] learns class-level positive and negative prompts to transform their recognition task into a sample-class contrastive learning problem. The learning paradigm is designed only for the recognition task and only to enable learning with unknown labels. Instead, our method explicitly recovers potential classes and is an attempt to solve specific issues in CMH with incomplete labels, i.e., both the loss of sample-class correspondence and pairwise relationship.\n\n**Recent CMH studies** have several attempts [2-4] to solve the incomplete labels problem. However, they are all **non-deep** methods without fine-grained measurement of sample-label consistency. In contrast, our proposed method can not only consider minor cases with the selected anchor machenism, but also integrate precise multi-modal knowledge to recover the classes. Their ability for potential class discovery is limited to only distinguish salient ones.\n\n[1] Sun, X., Hu, P., & Saenko, K. (2022). Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35, 30569-30582.\n\n[2] Liu, X., Yu, G., Domeniconi, C., Wang, J., Xiao, G., & Guo, M. (2019). Weakly supervised cross-modal hashing. IEEE Transactions on Big Data, 8(2), 552-563.\n\n[3] Ni, H., Zhang, J., Kang, P., Fang, X., Sun, W., Xie, S., & Han, N. (2023). Cross-modal hashing with missing labels. Neural Networks, 165, 60-76.\n\n[4] Yong, K., Shu, Z., Wang, H., & Yu, Z. (2024). Two-stage zero-shot sparse hashing with missing labels for cross-modal retrieval. Pattern Recognition, 155, 110717.\n\n> Q2. A deeper theoretical analysis explaining the effectiveness of the proposed methods is needed.\n\nConsider a label encoder model $\\mathcal M$ that produces optimal label encodings under loss function Eq.(4). Given a sample $\\pmb  x_i$ encoded as $\\pmb  h_i$ with its positive, negative, and unknown set $K_p(0)$, $K_n(0)$, and $K_u(0)$. For simplicity, we consider the $\\omega$-th iteration of PLTS, the searched class $c_u \\in K_u(\\omega)$ is combined back to acquire $K_p(\\omega+1) = K_p(\\omega) \\cup \\{c_u\\}$. With the assumption of $\\mathcal M$'s generalizability, we can acquire $$\\Phi^i(\\mathcal M(K_p(\\omega+1))) - \\Phi^i(\\mathcal M(K_p(\\omega))) \\ge m.$$ This is true for all omega if the termination condition is associated with the original margin $m$ rather than $\\frac{m}{2}$ which we empirically choose to gain higher recall. We can further obtain $$\\Phi^i(\\mathcal M(K_p(\\omega^*))) - \\Phi^i(\\mathcal M(K_p(0))) \\ge \\omega^* m.$$ This nonnegligible gap $\\omega^*m$ implies that PLTS exploits the model $\\mathcal M$'s ability to perceive and maximize the label completeness according to the score function $\\Phi$, therefore effectively recovering potential classes.\n\n> Q3:\nHow does the reliance on the CLIP model's text token limit impact the overall performance of your approach?\n\nWe should clarify that we have already pointed out this limitation in manuscript Line 314-315. The following is a more detailed analysis for the impact of token limit.\n\nIn **Flickr25K**, the most annotated dataset among the three evaluated dataset, the sample with the most classes contains 14 distinctive class annotations, which **does not exceed** the largest capacity of 14 classes within 77 tokens of the CLIP model. For **MS COCO**, the evaluated dataset with most classes, there are totally 12 samples exceeding the capacity, taking up **less than 0.014\\%** of all samples.\n\nFurthermore, even learning on a rich-annotated dataset, substituting the CLIP backbone with, e.g., long-CLIP, may sufficiently expand the token capacity. Besides, learning with reduced labels further decreases the impact of such token limitation.\n\nIn a nutshell, the negative impact is limited and can be eliminated with some simple changes to the model. How to overcome the token limitation is an *open problem* and we hope our analysis would inspire future work in these directions.\n\n> Q4. What are the specific challenges associated with needing sufficient multi-labeled samples?\n\nWe should also clarify that we have already indicated this limitation in manuscript Line 316-317. The following is a more detailed analysis.\n\nAlthough the model can effectively recover labels even with high unknown proportions at **70%** as we illustrated in Table 1, it intrinsically relies on multi-label annotations to select non-trivial anchor sets. Without enough multi-labeled data (due to either the dataset itself or the high unknown proportion), it's difficult to utilize **the interactions between different labels**.\n\nIn real-world retrieval tasks, multi-label insufficiency is relatively rare. At present, there is comparatively little research on this open topic, which can inspire future work for incomplete labels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kzCCxB9eg8", "reviewer": "Reviewer_Sq2X", "summary": "The manuscript tackles the challenges of generating high quality hash codes for cross-modal retrieval in the presence of incomplete labels,  which creates uncertainty in distinguishing between positive and negative pairs. To address the issue, a novel Prompt Contrastive Recovery approach called PCRIL is proposed, which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels.\n\nThe proposed PCRIL framework jointly performs semantic recovery and pairwise uncertainty elimination for efficient cross-modal hashing with incomplete labels. In particular, they consider each subset of positive labels and construct three types of negative prompts through deletion, addition and replacement for prompt learning. Augmentation techniques are also derived for addressing extreme cases of significant unknown labels and lack of negative pairwise supervision. Experimental results show significant improvement in mAP of the proposed solution with respect to the current SOTA methods.", "questions": "As mentioned above, the paper needs to highlight and clearly states the motivation of some proposed techniques.\n\nThe discussion, comparison and clarification of current methods in this field need to be included.", "limitations": "From the impact, it seems the method is incremental and focuses on a relatively small scope.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea proposed in the paper seems to be novel and interesting to resolve the incompleteness for cross-modal hashing. Typically, prompt processing and contrastive learning are combined to formulate prompt constrastive recovery, which effectively detect the potential classes and enhance the similarity learning.\n(2) Moreover, the augmentation techniques are also very helpful for handling extreme cases including unknown labels and negative pairs where an asymmetric mix-up is introduced and adaptive negative masking is devised .\n(3) The experimental evaluation is detailed and sufficient by evaluating the proposed method on three datasets with hyper-parameter tuning and visualization.", "weaknesses": "(1) Some of the techniques are proposed lacking clear motivation. For instance, it is not very clear that why the mix-up augmentation is asymmetric. \n\n(2) There are already related work on contrastive learning on prompt such as \n(a) https://arxiv.org/pdf/2205.01308, \"Contrastive Learning for Prompt-Based Few-Shot Language Learners\". NAACL 2022.\n(b) https://openaccess.thecvf.com/content/CVPR2023/papers/Zhang_CLAMP_Prompt-Based_Contrastive_Learning_for_Connecting_Language_and_Animal_Pose_CVPR_2023_paper.pdf\nIt is important to compare the proposed method to these existing work to see if there are performance gains. Also it seems that compared to reference (b), the proposed method is quite similar as they are also based on prompt contrastive learning for cross-modal retrieval or hashing  except that they are applied to different tasks (pose estimation vs hashing).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Responses to Reviewer `Sq2X`'s Comments\n\nThank you for your thorough review and insightful comments. We are delighted that you found our PCRIL approach innovative and effective in resolving incompleteness in cross-modal hashing. Your appreciation of our proposed prompt contrastive learning is very encouraging and appreciated.\n\n> Q1. Some of the techniques are proposed lacking clear motivation. For instance, it is not very clear that why the mix-up augmentation is asymmetric.\n\nThe motivation of our asymmetric mix-up is to further **eliminate uncertainty** in the sample-label relationship, particularly for missing labels. The **original symmetrical mix-up augmentation is not designed for label-incomplete cases**. Due to the existence of unknown classes, cases where sample $\\pmb x_i$ complements $\\pmb x_j$ but $\\pmb x_j$ does not complement $\\pmb x_i$ can occur frequently.\n\nFor instance, $\\pmb l_i$ = (`sky`=1, `star`=0, `moon`=1, `person`=u) complements $\\pmb  l_j$ = (`sky`=1, `star`=1, `moon`=u, `person`=0) because the `moon` tag in $\\pmb l_i$ can eliminate the uncertain `moon`=u in $\\pmb l_j$. However, $\\pmb x_j$ does not fit in $\\pmb x_i$ because filling the nonexisting `person` in $\\pmb l_j$ to $\\pmb l_i$ does not change the class `person`'s value.\n\nTherefore, this observation motivates us to propose our complementary mix-up design. The asymmetrical maching score gives $\\delta_{ij}=0$ and $\\delta_{ji}=1$, which solves such issues reasonably and effectively.\n\n> Q2. There are already related work on contrastive learning on prompt such as \n(a) https://arxiv.org/pdf/2205.01308, \"Contrastive Learning for Prompt-Based Few-Shot Language Learners\". NAACL 2022. \n(b) https://openaccess.thecvf.com/content/CVPR2023/papers/Zhang_CLAMP_Prompt-Based_Contrastive_Learning_for_Connecting_Language_and_Animal_Pose_CVPR_2023_paper.pdf\nIt is important to compare the proposed method to these existing work to see if there are performance gains. Also it seems that compared to reference (b), the proposed method is quite similar as they are also based on prompt contrastive learning for cross-modal retrieval or hashing except that they are applied to different tasks (pose estimation vs hashing).\n\nWe appreciate the two works that reviewer pointed out, however, we would like to emphasize that the studied tasks are different. More distinctive challenges can appear in the retrieval task with incomplete labels, e.g., vanished pairwise similarity and low-quality sample-label correspondence. Meanwhile, both our prompt construction and contrastive learning exhibit large uniqueness compared to the provided work.\n\nThe work [1] improves language learners through clustering text examples of the same class. The method substitutes contextual demonstrations like `It is` with different prompts (e.g. `I think it is`) to produce different views of the example. The contrastive learning objective in this work is a widely-used SupCon [3] loss.\n\n- Compared to [1], our proposed method constructs **learnable** prompts for *the classes themselves*, in order to exploit the CLIP for classes' completeness knowledge.\n- Compared to their objective, our score-based contrastive margin loss, Eq.(4), involves insertion, deletion, and replacement, the three types of negative subsets, to separate class combinations of different completeness levels in terms of their CLIP scores. Such separation enables the model to discover potential positive labels in our partial-label scenario.\n\nThe work [2] proposes to leverage language information to estimate animal pose keypoints. This work is more similar to ours since they also use learnable prompts for CLIP.\n\n- However, their **single-class textual prompt** each filled with the name of one pose is similar to the learnable version of the original CLIP prompts, while our multi-class prompt construction considers both learnability and authenticity for complex multi-label samples.\n- Their contrastive learning is performed between image feature maps and their pose prompts. Ground-truth point's feature and its corresponding prompt are considered a positive pair, while others are considered negative. In our work, we randomly select subsets of positive classes as anchor sets for each sample. The positive pairs are the sample-anchor pairs, while the negative ones are the sample and the negatively modified anchor. Because the stochasticity in selecting the anchor set, our learning scheme can produce diverse pair relationships. Therefore, one significant difference is that the pair relationship in our work is dynamical and can discover label completeness knowledge through the learning process of our contrastive objective.\n\n[1] Jian, Y., Gao, C., & Vosoughi, S. (2022). Contrastive learning for prompt-based few-shot language learners. arXiv preprint arXiv:2205.01308.\n\n[2] Zhang, X., Wang, W., Chen, Z., Xu, Y., Zhang, J., & Tao, D. (2023). Clamp: Prompt-based contrastive learning for connecting language and animal pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23272-23281).\n\n[3] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., ... & Krishnan, D. (2020). Supervised contrastive learning. Advances in neural information processing systems, 33, 18661-18673.\n\n> Q3. As mentioned above, the paper needs to highlight and clearly state the motivation of some proposed techniques.\nThe discussion, comparison and clarification of current methods in this field need to be included.\n\nPlease see the responses above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cFlpNp6E1u", "reviewer": "Reviewer_zzeX", "summary": "This paper presents a novel cross-modal hashing method named PCRIL, which explores the indispensable but challenging problem of incomplete label recovery in multi-label learning. It conceives a CLIP-based prompting scheme and a complementary semantic propagation mechanism, enabling PCRIL to restore unknown labels and calibrate pairwise similarities. The paper exhibits a strong motivation, technical soundness, and a well-structured organization. Generally, the idea of the paper is interesting, especially in constructing a learnable label prompt to perceive the missing labels in cross-modal learning.", "questions": "Please refer to the weaknesses. The authors are advised to thoroughly resolve the above concerns to convince the reviewer.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper explores the indispensable but challenging problem of incomplete label recovery in multi-label learning. The authors identify this crucial problem for cross-modal hashing and propose an effective solution for this problem. The proposed method is technically sound.\n\n- The paper conceives a CLIP-based prompting scheme and a complementary semantic propagation mechanism, enabling the proposed method to restore unknown labels and calibrate pairwise similarities.\n\n- The authors design a simple learnable prompt to encode class combinations into CLIP embeddings.\n\n- The neglected semantic labels and pairwise similarities can be removed and recovered through the proposed architecture.\n\n- Extensive experiments have been conducted on MIRFlickr-25K, NUS-WIDE, and MS COCO.", "weaknesses": "- The CLIP-based prompting has been intensively studied and this paper needs to make sufficient analysis and experimental validation of its key contribution, i.e., the three types of contrastive learning, particularly in terms of their advantages over plain prompt contrastive recovery.\n\n- Can the authors analyze the effects of the ways of the prompt construction in this task?\n\n This paper contributes a new problem and some new ideas to the CMH literature. The authors should further explain the necessity of the given research topic in practical situations.\n\nThe Method section should provide more explanation of these figures. Currently, these parts are separated, making the model difficult to follow, especially given the article's innovative points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Responses to Reviewer `zzeX`'s Comments\n\nThank you for your thoughtful review. We are pleased that you found our approach novel and technically robust. Your positive feedback on our contrastive recovery method and extensive experiments is greatly appreciated. Your insights are invaluable to us.\n\n> Q1. The CLIP-based prompting has been intensively studied and this paper needs to make sufficient analysis and experimental validation of its key contribution, i.e., the three types of contrastive learning, particularly in terms of their advantages over plain prompt contrastive recovery.\n\nIn section 4.3, we have provided extensive ablation of the prompt contrastive recovery. In Table 3, we have experimentally validated our advantage over plain 'phrasal' prompt, improving greatly upon the recovery precision and the final mAP.\n\nIf contrastive learning is performed directly on the positive set, i.e., without selecting the anchor and constructing the three types of negative sets, we argue that the PLTS search can become ineffective because no margin is learned between similar but different label sets. However, substituting PLTS with a scan in the entire label space would consume substantial time.\n\n> Q2. Can the authors analyze the effects of the ways of the prompt construction in this task?\n\nIn our experiments, we construct learnable prompt for class combinations and have analyzed it through comparisons with plain phrasal prompt and conventional CLIP-like single-class prompt.\n\nThe **plain prompt** is constructed by stacking textual phrases of each class to form a prompt of the label set. Such static prompt is sensitive to the textual content it chooses and performs poor as validated in Table 3. Compared to the plain prompt, our learnable prompt can automatically adjust class-related content through contrastive learning, gaining much higher performance.\n\nThe **single-class prompt** is a learnable version of the original CLIP prompt. However, it still performs not comparable to our method. The class relationship of multi-label data is intrinsically nonlinear. Therefore, by integrating different labels in a single CLIP text prompt, the model learns to capture complex relationship among classes, fostering precise contrastive learning.\n\n> Q3. This paper contributes a new problem and some new ideas to the CMH literature. The authors should further explain the necessity of the given research topic in practical situations.\n\nOur method addresses the challenge of cross-modal hashing (CMH) with incomplete labels. CMH extracts pairwise relationship from label annotations for efficient information retrieval across various data modalities, such as images and text. This capability is crucial for practical applications like web search engines, social media, and e-commerce platforms.\n\nIn real-world scenarios, obtaining fully annotated datasets for CMH is often impractical. **User-generated tags** on websites, particularly on social media, are frequently incomplete and biased. For example, a user searching for \"gluten-free vegan dessert recipes\" may struggle to find relevant results because many recipes are simply tagged as \"dessert\" or \"vegan,\" without indicating whether they are also gluten-free. Another instance of incomplete annotations occurs in **expert systems**, where labeling is costly, such as with medical images that are rarely fully annotated despite the practical demands for medical image retrieval.\n\nOur proposed method extends the traditional research topic of incomplete labels in recognition tasks to multi-modal and retrieval contexts, making CMH more robust against incomplete annotations.\n\n> Q4. The Method section should provide more explanation of these figures. Currently, these parts are separated, making the model difficult to follow, especially given the article's innovative points.\n\nFigure 2-4 illustrates the motivation of positive anchors for contrastive learning, the general framework of our method, and the Potential Label Tree Search (PLTS) process, respectively.\n\n**Figure 3** statistically illustrates the long-tailed distribution of the sample' unique class combinations. Such phenomenon implies that there are dominating combination cases that overwhelms common CMH models and hides other infrequent ones against model fitting. This motivates us to propose our method in **Figure 2**, which learns to reconstruct sample-label correspondence through anchor subsets selected from the entire positive label.\n\nAs our main contribution, the **top-left part in Figure 2** defines a sequence of negative sets compared to the anchor. By embedding these label sets into learnable prompts, we perform contrastive learning to pull the anchor feature together with its sample while push the negative sets away from it. In this way, the contrastive objective in Eq.(3) would effectively separate similar label subsets with different completeness levels.\n\nThe learned model with such ability allows us to propose the PLTS, whose process is demonstrated in **Figure 4**. Meanwhile, **Figure 2 top part** shows the relationship of PLTS and our contrastive learning. With the margin learned among label set embeddings, the PLTS recovers potential labels via a greedy search from the sample's positive set. It enforces the searched and joined positive classes to raise the score defined in Eq.(2), therefore recovering potential labels based on the contrastive model.\n\nWe would integrate the above explanations to appropriate lines in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMCWuEl7MY", "reviewer": "Reviewer_MbLT", "summary": "Cross-modal hashing (CMH) has attracted much attention due to its low computational and storage costs while maintaining high-dimensional cross-modal semantic similarity. To address the incomplete annotation challenge of CMH, this paper proposes a novel Prompt Contrastive Recovery method, PCRIL. The method includes prompt contrastive recovery (PCR) and complementary semantic augmentation (CSA) modules. Experimental evaluation verifies the effectiveness of the proposed method. Overall, this paper has done a sufficient and concrete work. However, the novelty of the proposed method is limited, and the comparison in this experiment is also limited.", "questions": "1.There are no comparative methods within two years. Can the author perform some recent methods to show the effectiveness of the proposed method.\n2.It is recommended that the authors conduct more experiments on the IAPR TC-12 dataset to more intuitively illustrate the effectiveness of the model.\n3.Please elaborate on how to remove negative pairs from unknown classes.", "limitations": "Prompt construction currently relies on a pre-trained CLIP model with a limited number of textual labels, which hinders its ability to enrich labeled samples.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.This paper addresses the problem of incomplete annotations in cross-modal hashing, which is very common and challenging in practical applications.\n2.The prompt contrastive recovery (PCR) module proposed in this paper effectively perceives incompleteness through label prompts and can well restore the information semantics, which has been well demonstrated by experiments.", "weaknesses": "1.The authors lack comparison with some recently proposed SOTA methods, no one within two years, so the experimental results are not convincing.\n2.This paper conducted experiments on three datasets with different label known ratios, but some comparison methods provided relevant experimental results on the IAPR TC-12 dataset. The author did not provide relevant experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Responses to Reviewer `MbLT`'s Comments\n\n> Q1. The authors lack comparison with some recently proposed SOTA methods, no one within two years, so the experimental results are not convincing.\n\nWe further compare our method with:\n\n- CMHML [1], a recent method investigating CMH with incomplete labels.\n- MITH [2], a recent deep CMH method which also adopt CLIP as the backbone.\n\nThe results are shown in **Table 2 in the PDF file**. These results further demonstrate the superiority of the proposed approach against recent SOTAs and validate its effectiveness.\n\n[1] Ni, H., Zhang, J., Kang, P., Fang, X., Sun, W., Xie, S., & Han, N. (2023). Cross-modal hashing with missing labels. Neural Networks, 165, 60-76.\n\n[2] Liu, Y., Wu, Q., Zhang, Z., Zhang, J., & Lu, G. (2023, October). Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 893-902).\n\n> Q2. This paper conducted experiments on three datasets with different label known ratios, but some comparison methods provided relevant experimental results on the IAPR TC-12 dataset. The author did not provide relevant experimental results.\n\nIt's worth noting that these methods give no experimental results regarding incomplete labels. We compare our method on IAPR TC-12 with them by re-implementing their code. The results are shown in **Table 3 (PDF file)**, where our method outperforms the compared baselines significantly, especially in highly incomplete cases. This demonstrates the consistency of our method across benchmarks.\n\n> Q3. There are no comparative methods within two years. Can the author perform some recent methods to show the effectiveness of the proposed method.\nIt is recommended that the authors conduct more experiments on the IAPR TC-12 dataset to more intuitively illustrate the effectiveness of the model.\n\nAs stated above, we provide results of these experiments in the **PDF** file. The effectiveness of our method is validated.\n\n> Q4. Please elaborate on how to remove negative pairs from unknown classes.\n\nIf we understands this correctly, the question is how to remove unknown pairs. This is achieved in our work through an **Adaptive Negative Masking (ANM)** strategy.\n\nThe widely-used setting \"assume negative\" achieves this by turning all pairs with unknown relationship ($S_{ij}=u$) into negative ones ($S_{ij}=0$), which introduces substantial false negative sample pairs. The \"only known\" setting avoids pairwise learning on all unknown pairs to eliminate their influence. However, the negative relationship can disappear completely in highly unknown cases.\n\nTo remove unknown pairs while eliminating the false negative side-effect, we propose to stochastically mask the unknown entries as negative to restore a balanced ratio $r=|\\mathcal N(S^D)|/|\\mathcal P(S^D)|$ for positive and negative pairs. Empirically, $r$ is a small positive number to alleviate introducing false negatives. In such a way, we augment the final similarity supervision $S$ with less recovered negative values but robust and higher performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Reply to all reviewers

We would like to extend our gratitude to all the reviewers for their insightful comments and unanimous acknowledgement of our paper in the following aspects:
1. The task addressed by this work is both interesting and significant for real-world cross-modal hashing applications (zzeX, MbLT).
2. The proposed PCRIL method has clear motivation and strong novelty by introducing prompt-based contrastive learning to perceive incomplete classes for the task (MwQQ, Sq2X, zzeX).
3. The paper is clearly written, well-structured, and easy to understand (MwQQ). Our prompt contrastive recovery is effective in addressing the challenge of incomplete labels in cross-modal hashing (Sq2X, MbLT).
4. It shows substantial performance improvements over existing methods (FiXs) and our contributions has been validated with extensive and convincing experiments(MwQQ, Sq2X, zzeX).
5. The contributions provide new insights and advancements that benefit future research in the field (MwQQ).

It is worth noting that the **PDF file** contains explanations of the **motivations and interrelationships of each component** in our method (Figure 1), as well as **the latest experimental results** including extended ablation results, comparison with recent SOTA methods, and results on a new dataset IAPR TC-12 (from Table 2 to 4, respectively).

In addition, we hereby provide highlights to some common queries regarding our current work.

1. **Motivation from Figure 3**. For cross-modal retrieval, each multi-modal instance is associated with a label annotation within a predefined class set. Figure 3 is a statistical analysis of **unique** positive label combination patterns in all samples' annotations. This illustrates the long-tailed distribution of class combinations. For rare label cases, sample-label pairs hardly exist in the data. Our proposed contrastive anchors can enrich this relationship and help label recovery in all cases.
2. **Theory for PLTS effectiveness**. If the objective Eq.(3) is fully minimized, the scores can carry label completeness information. By separating class sets that have edit distance $D$ by a large gap $G=Dm$, The trained model guarantees to increase the CLIP score from the original incomplete label by $G$ through the potential label tree search process, which effectively perceives potential classes from the unknown set.

We have provided detailed responses to each reviewer's feedback including these questions. Please find our point-to-point responses in the individual replies.

**References** for the PDF file.

[1] Ni, H., Zhang, J., Kang, P., Fang, X., Sun, W., Xie, S., & Han, N. (2023). Cross-modal hashing with missing labels. Neural Networks, 165, 60-76.

[2] Liu, Y., Wu, Q., Zhang, Z., Zhang, J., & Lu, G. (2023). Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 893-902).

Meta-review: This paper proposes a prompt contrastive recovery approach that progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels. Several experiments on benchmark datasets show that they significantly perform favorably against several methods. Four of five reviewers rate the paper as weak/boderline accept, while one reviewer still has concerns about the limitation of the pre-trained CLIP model. After discussion, the final decision is accept, and the authors should further explore its limitations and solutions.
Decision: Accept (poster)