Title: Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels
Venue: NeurIPS 2024
Accepted: yes
Keywords: cross-modal hashing

incomplete labels

prompt contrastive learning
Abstract: In this paper, we tackle the challenge of generating high-quality hash codes for cross-modal retrieval in the presence of incomplete labels, which creates uncertainty in distinguishing between positive and negative pairs. Vision-language models such as CLIP offer a potential solution by providing generic knowledge for missing label recovery, yet their zero-shot performance remains insufficient. To address this, we propose a novel Prompt Contrastive Recovery approach, \textbf{PCRIL}, which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels. Identifying unknowns is nontrivial due to the fixed and long-tailed patterns of positive label sets in training data, which hampers the discovery of new label combinations. Therefore, we consider each subset of positive labels and construct three types of negative prompts through deletion, addition, and replacement for prompt learning. The augmented supervision guides the model to measure the completeness of label sets, thus facilitating the subsequent greedy tree search for label completion. We also address extreme cases of significant unknown labels and lack of negative pairwise supervision by deriving two augmentation strategies: seeking unknown-complementary samples for mixup and random flipping for negative labels. Extensive experiments reveal the vulnerability of current methods and demonstrate the effectiveness of PCRIL, achieving an average 12\% mAP improvement to the current SOTA across all datasets. Our code is available at https://github.com/E-Galois/PCRIL.

Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels

Abstract

In this paper, we tackle the challenge of generating high-quality hash codes for cross-modal retrieval in the presence of incomplete labels, which creates uncertainty in distinguishing between positive and negative pairs. Vision-language models such as CLIP offer a potential solution by providing generic knowledge for missing label recovery, yet their zero-shot performance remains insufficient. To address this, we propose a novel Prompt Contrastive Recovery approach, PCRIL , which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels. Identifying unknowns is nontrivial due to the fixed and long-tailed patterns of positive label sets in training data, which hampers the discovery of new label combinations. Therefore, we consider each subset of positive labels and construct three types of negative prompts through deletion, addition, and replacement for prompt learning. The augmented supervision guides the model to measure the completeness of label sets, thus facilitating the subsequent greedy tree search for label completion. We also address extreme cases of significant unknown labels and lack of negative pairwise supervision by deriving two augmentation strategies: seeking unknown-complementary samples for mixup and random flipping for negative labels. Extensive experiments reveal the vulnerability of current methods and demonstrate the effectiveness of PCRIL, achieving an average 12% mAP improvement to the current SOTA across all datasets. Our code is available at github.com/E-Galois/PCRIL.

1 Introduction

Cross-modal hashing (CMH) [2, 14, 17, 11, 23, 25] addresses the highly demanding cross-modal similarity search in both web search systems and academic domains [10, 13]. By efficiently transforming multi-modal data (images, text, etc .) into a collection of compact binary codes, CMH maintains high-dimensional cross-modal semantic similarity while significantly minimizing computational and storage costs. Deep cross-modal hashing has achieved remarkable development by leveraging dual-stream networks or semantic encoding branches based on semantic labels [14, 17]. However,

0

20

40

60

80



0

20

40

60

80

100













0

20

40

60

80



0

20

40

60

80

100













due to limited labour resources, fully supervised annotation becomes impractical for large-scale datasets [27, 33]. A realistic compromise would be partial annotation, in which only a subset of classes are explicitly labeled, while others are marked as unknown . Although facilitating adaptation to large datasets, this labeling scheme provides significantly reduced semantic supervision. Therefore, learning with partial annotation has become a significant challenge in multi-label learning tasks.

Furthermore, CMH with partial labels inevitably encounters disrupted similarity learning due to pairwise uncertainty [27, 22]. Jointly missing classes cause incomplete labels to not only obscure class knowledge but also eliminate pairwise similarity by introducing unknown pairs . Among these, negative pairs constitute the majority and become particularly scarce with the increasing frequency of unknown labels. As shown in Fig. 1, negative pairs are completely removed even with 35% unknown labels in Flickr dataset. Therefore, such pairwise uncertainty can severely impair similarity-preserving hashing. However, existing CMHs desperately rely on clear positive and negative pairs to maximize supervision. It remains crucial yet unresolved how to accurately perceive potential categories and restore similarity learning in the task of CMH to prevent semantically uninformative binary codes.

To address the issues of incompleteness, some multi-label image recognition methods have investigated a feasible solution, i.e. , leveraging prior knowledge in large vision-language models [19, 18, 15, 24]. Pre-trained with a contrastive loss on massive image-text pairs, CLIP [28] has shown its capability to align the global representation of visual and textual modalities with loaded prior knowledge (modal correspondence, similarity structures, partial semantics, etc .). Although effective in many downstream tasks, they are hardly applicable for pairwise similarity recovery in CMH. Original CLIP's ability is empirically ineffective for label recovery via class-sample similarity scores. As revealed in Sec. 4.4, the original CLIP prompt yields an unsatisfactory 68% recovery precision even on the easier Flickr 30% unknown case, introducing substantial noisy classes and even degrading the final performance. Therefore, CLIP label recovery for deep CMH remains under-explored.

In this paper, we seek to overcome the aforementioned deficiencies and propose Prompt Contrastive Recovery for cross-modal hashing with Incomplete labels (PCRIL) by considering the CLIP prior knowledge of descriptive completeness , which is the ability to measure the completeness of a text caption for its described image. By selecting anchor class sets, we develop a simple learnable prompt to encode selected anchor class combinations into CLIP text embeddings. Multiple negative variants are constructed via editing operations on the anchor set. A prompt contrastive recovery paradigm then imposes separation gaps between these label subsets. By learning scores conditioned on their sample modalities, instance-aware class perception is enabled. Thereafter, potential labels are recovered via a tree search on the learned scores. To enhance hard incomplete samples, different instances' features and unknown labels are complementarily mixed. We also introduce an adaptive negative masking strategy to deal with negative pair scarcity at high missing rates. The main contributions of our work are as follows:

Wepropose a PCRIL framework, which jointly performs semantic recovery and pairwise uncertainty elimination for efficient cross-modal hashing with incomplete labels. To the best of our knowledge, there is no prior study on CLIP-based label recovery strategies in cross-modal hashing.

· A novel recovery architecture is proposed to recover the neglected semantic labels and pairwise similarities. Particularly, a contrastive learning objective between the anchor set and its negative variants learns instance-conditioned matching scores. A tree search process then leverages the learned scores to detect potential classes. Meanwhile, a complementary semantic augmentation and an adaptive negative masking strategy jointly enhance the similarity learning in extreme cases. Thus, PCRIL can fully restore potential labels and pairwise similarity.

· Extensive experiments verify that our PCRIL can consistently outperform state-of-the-art CMH methods across a range of incompleteness levels and different benchmarks. Comprehensive analyses further validate our effective recoverability for incomplete labels.

2 Related Work

2.1 Cross-modal Hashing

Cross-modal hashing (CMH) aims to encode different high-dimensional modalities into a common space of compact binary codes where modal similarity is preserved for fast and accurate retrieval. Early machine learning methods, including those by [32, 21, 26, 3], learn common codes from encoded features. Though simple, non-deep methods are restricted by two-step training paradigm and limited in discriminative similarity learning. Jiang et al . [14] proposed deep cross-modal hashing (DCMH), introducing a pairwise similarity matrix into deep similarity preservation for the first time. Li et al . [17] constructed a label net to project labels into a common space with modal binary codes. To close modality gaps, Gu et al . [11] adopted cross-modal feature attentions with an adversarial learning scheme for semantic discriminability and modal consistency. Zhang et al . [35] proposes to preserve multi-level knowledge in CMH with a variational information bottleneck. As efficient feature learners, pretrained vision-language models have been leveraged in recent works. Tu et al . [30] introduced transformer-based CLIP image encoders with selective hash optimization. Liu et al . [23] further investigated multi-granularity cross-modal alignment based on vision-language transformers. These methods rely heavily on the guidance of complete semantic labels for similarity learning, while they fail to handle the practical incompleteness problem of label annotations in large-scale datasets.

2.2 Learning with Incomplete Labels

Learning with incomplete labels involves only partially known classes, which are a compromise resulting from inaccessible exhaustive supervision [7]. Existing studies regarding incomplete labels mainly focus on the image recognition task. Some works seek to recover learnability upon partial labels with modified loss functions. Bucak et al . [1] proposed to learn a modified ranking loss to alleviate the effects of false negatives while assuming all unknown labels to be negative. Durand et al . [9] proposed to learn only known labels with the prior knowledge of the label ratios. Cole et al . [6] proposed to solve the single positive label issue by imposing training constraints on label statistics. Another type of work investigates lost labels directly. Veit et al . [31] resorted to learning from human re-annotations to recover original labels. Chu et al . [4] proposed a variational generative model to explore data correlations with partial labels. Kim et al . [16] proposed to distinguish exceedingly large losses of false negatives and reverse them during training. By adopting vision-language models with rich prior knowledge, Sun et al . [29] proposed a prompt-tuning method with separately learned positive and negative prompt parameters. Ding et al . [8] further considered constructing a graph-based label structure in vision-language models to enhance image recognition. In cross-modal retrieval, the problem is however unexplored. Some methods [27] focus on the problem of entirely missing labels. Ni et al . [27] and Liu et al . [22] proposed shallow CMH methods that re-predict the labels with consistency constraints between instances and labels. However, without a fine-grained measurement of sample-label consistency, their ability for class perception is limited to only distinguish salient ones.

3 Proposed Approach

3.1 Problem Definition

We focus on image-text hashing, which is prioritized and fundamental. CMH with incomplete labels learns hash functions on a training set O = { ( f i , g i , l i ) } N i =1 , where f i and g i are the image and text

[Man]

[Dog]

[Cat]

Lower

scores

Higher

scores

𝑳 𝒄𝒕𝒓

+

...

...

+

Matched

Sample

Features

✓

mean

select

CLIP

Modal

Encoders

Img.

Txt.

Outside

t h e

the

man

is walking his

2 lovely dogs.

CLIP

Text

Encoder

𝒉𝒊

𝒇𝒊 /𝒈𝒊

Incomplete

[Man]

[Dog]

[ Man

]

[ Dog ]

[Leash]

[ Cat ]

[ Car ]

[Store]

[Bike ]

[Tree

]

𝑲𝒑

𝒊

𝑲𝒂

𝒊

[Dog]

[Man]

[Car]

𝑲𝒋

𝒊,𝒕

𝑲𝒅

𝒊,𝒔

𝑲𝒓

𝒊,𝒔,𝒕

Prompting

(Eq.1)

95

76

83

41

margin

Contrastive

Loss (Eq.4)

𝒇𝒊 /𝒈𝒊

𝒇 𝒊

∗

/𝒈 𝒊

∗

෡

𝒇𝒊 /ෝ

𝒈𝒊

1 0 1 U U U

U 1 0 1 U 0

1 0 U 0 1 1

U 1 0 1 1 1

m i x e d

Semantic

Augmentation

(Eq.8)

Potential Label

Negative

Subsets

Positive

Anchor Set

Negative

Unknown

𝑲𝒏

𝒊

𝑲𝒖

𝒊

select

⨂ 𝒉𝒊

0

0

1

1

U

1

1

U

1

U

0

0

1

U

1

1

𝑺

Pairwise Similarity

Supervision

Adaptive Negative

Masking (Eq.9)

𝓤

𝓤′

𝑺′

U

U

1

1

U

1

1

U

1

U

U

U

1

U

1

1

U

U

U

U

U

U

U

U

U

U

U

U

𝑹

0

0

1

1

1

0

0

1

𝑲𝒑

𝒊

𝑸

1 0 1 U U 1 0 U

modalities of the i -th sample, respectively, and l i is its label vector, which is the i -th column in the label matrix L ∈ { 1 , 0 , u } C × N . Here, C is the number of classes, N is the number of instances, and u denotes an unknown value. Hence, a sample class could be positive ( 1 ), negative ( 0 ), or unknown ( u ). A similarity matrix S is derived from L , where s ij = 1 iff f i and g j share at least one positive label. CMH aims to learn hash functions H t and H v to encode text and image samples into binary codes b t i = sign( H t ( f i )) ∈ { 0 , 1 } d and b v i = sign( H v ( g i )) ∈ { 0 , 1 } d , preserving cross-modal similarity in the Hamming space. Incomplete labels can inject uncertainty in both L and S . To tackle this, we propose our PCRIL framework in Fig. 2. Herein, a novel learnable CLIP prompt for label sets is designed to recover potential positive labels in Sec. 3.2. Additionally, an unknown-complementary sample augmentation and a negative masking strategy are developed in Sec. 3.3 to deal with hard samples and negative pair scarcity, respectively.

3.2 Prompt Contrastive Recovery

Prompt for Positive Anchors. Prompt finetuning is a promising solution for lightweight model adaptation. The typical prompting method for CLIP decorates the class tokens, either with a predefined prefix [28] or a vector of learnable tokens [36, 29]. A typical handcrafted prompt template for a single unknown class can be 'A photo of a CLS ," where CLS is the class name. However, its fixed tokens and single-class scope hinder it from capturing inter-class complexity for label-wise recovery. In pursuit of separating labels of varied informativeness, we propose to encode label sets into CLIP embeddings.

Given a sample ( f i , g i , l i ) with incomplete labels, the known classes of this instance can be extracted as the positive label subset K i p = { c | l c i = 1 } and the negative one K i n = { c | l c i = 0 } . However, directly maximizing agreement between K i p and modal representations is insufficient to capture its level of completeness. We quantified

































and ranked the various combinations of labels in Fig. 3. As illustrated, certain dominant combinations can overshadow less frequent cases in the current CMH scheme, obscuring their impact on model fitting. This long-tail distribution of K i p can induce bias and disable balanced learning. Furthermore, the number of label sets available is constrained by the samples in the training dataset, which further restricts the learned correspondence. Therefore, we propose to measure semantic completeness through a contrastive learning paradigm instead, in which we consider an anchor set K i a ⊂ K i p as

positive instance and encode it into CLIP embeddings. For instance, a typical handcrafted template can be 'A photo of some seagulls flying above the beach ." for K i a = { seagull , beach } and K i p = { seagull , beach , sky }. It is observed that class names are often surrounded by class-related prefixes and suffixes. Therefore, we further construct a learnable prompt template. The proposed prompting operation is formulated as:

where p c is the specialized learnable prompt for the c -th class, p head ∈ R m a × d f and p tail ∈ R n a × d f are the learnable class-agnostic prefix and suffix, and σ ( · ) represents a random permutation operation. A specific example of the constructed prompt is given in Suppl. Sec. A.1.

Negative Subsets and Contrastive Learning. For the selected anchor label set, dropping any positive class object would degrade its alignment with modalities. The same is true for adding negative classes. Therefore, we aim to detect potential positive labels among the unknown ones by constructing negative subsets relative to the anchor set. Three types of negative subsets are constructed by 1) deleting a class: K i,s d = K i a - { s } , 2) joining a negative class in K i n : K i,t j = K i a ∪ { t } , or 3) replacing a class by a negative one in K i n : K i,s,t r = K i a -{ s } ∪ { t } , where s ∈ K i a and t ∈ K i n . Note that generating a negative set by replacement is equivalent to successively performing deletion and joining. This variant is introduced to improve model robustness.

To learn a completeness measurement, we define a simple matching score that is compatible with the CLIP prior:

where K is a label set, E t ( · ) and τ are the CLIP textual encoder and its temperature parameter, h i represents the average of modal CLIP features E v ( f i ) and E t ( g i ) , which is analyzed in Sec. 4.4. The contrastive loss between a positive anchor set and a negative variant is formulated as

where K ∗ is a negative variant of anchor set K a and m is a margin parameter that separates sets of different completeness levels. The overall contrastive loss is derived as

Notably, the only parameter introduced for contrastive learning is the learnable prompts.

Potential Label Tree Search (PLTS). Through contrastive learning between label sets, knowledge in the scores is supposed to generalize as a hierarchical measurement to distinguish potential positive classes in the unknown set K i u , i.e. , an anchor (positive) set can be a negative set for another larger anchor set. Therefore, a simple greedy search is designed to recover potential positive labels. The label search starts with the entire positive set K i p as a node with score Φ i ( K i p ) . At each step, we search for an unknown class that maximizes the class set score, and merge it into the positive set. Specifically, we denote the positive and unknown label sets before the ω -th iteration as K i p ( ω ) and K i u ( ω ) , where K i p (1) = K i p and K i u (1) = K i u . At the ω -th iteration, we try adding each c u ∈ K i u ( ω ) into K i p ( ω ) and find the one that maximizes the score. This process is formulated as

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(1)

:

a

.

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(1)

:

x

,

y

,

z

.

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835))

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835))

∪ {GLYPH(cmap:d835)

GLYPH(cmap:d835)

∗

}

a

ax

ay

ayx

axyz

✓

✓

✘

✘

✘

✘

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835))

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835))

∪ {GLYPH(cmap:d835)

GLYPH(cmap:d835)

≠ GLYPH(cmap:d835)

GLYPH(cmap:d835)

∗

}

GLYPH(cmap:d835)

= GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835)

∗

)

GLYPH(cmap:d835) = 1

GLYPH(cmap:d835) = 2

GLYPH(cmap:d835)

∗

= 3

We then transfer the optimal c ∗ u into the positive label set. K i p ( ω ) and K i u ( ω ) are updated as K i p ( ω +1) = K i p ( ω ) ∪ { c ∗ u } and K i u ( ω +1) = K i u ( ω ) -{ c ∗ u } . This process continues with the

updated sets at iteration ω + 1 to further discover potential labels. The termination condition is designed as Φ i ( K i p ( ω ∗ ) ∪ { c ∗ u } ) < Φ i ( K i p ( ω ∗ )) + m 2 , which ensures monotonic increase in the scores. Suppose it ends at ω ∗ -th iteration, the search stops with a final score of ϕ = Φ i ( Q ) , where Q = K i p ( ω ∗ ) is the final recovered positive set. For a specific example, Fig. 4 demonstrates a typical PLTS process. An optional choice is to further calculate a pseudo label for each of the remaining unknown classes c u by l c u i = H (Φ i ( Q ∪{ c u } ) -ϕ ) , where H ( x ) = max(0 , min(1 , 1 2 + x m )) is the hard sigmoid function with a linear window of m .

3.3 Augmentation Strategies for Handling Extreme Cases

Although PLTS can recover substantial positive labels, two issues can remain unresolved in highly incomplete cases. After recovery, a large portion of unknown labels can still exist. Meanwhile, negative pairs can become extremely scarce at high unknown ratios. To enhance the model for these cases, we further impose the following two augmentation strategies.

Complementary Semantic Augmentation. As PLTS only recovers positive labels, the unknown values cannot be fully determined, leaving uncertainty in similarity learning. Therefore, we propose to eliminate the residual uncertainty by mixing up complementary samples. However, the symmetrical mixup [34] augmentation can be ineffective because the complementary relationship is asymmetric, i.e. , cases that sample f i 's positive set contains an uncertain class of sample g i but not vice versa. Hence, we formulate an asymmetric matrix ∆ N × N to express sample matching scores as δ i,j = U [ l i ] ⊤ P [ l j ] U [ l i ] ⊤ U [ l i ] , where U [ · ] sets unknown elements to 1 and others to 0 , and P [ · ] ( N [ · ] ) sets only positive (negative) entries to 1 , respectively. δ i,j measures the proportion of sample i 's unknown categories that are positive for sample j . This asymmetrical score evaluates the volume of semantic certainty that sample j can transfer into the semantic background of sample i . Based on the scores, an asymmetric mix-up is introduced for a sample feature f i and its complementary counterpart f ∗ i = f j with label l ∗ i = l j , where j is selected randomly from the topK indices on the i -th row of ∆ . The complementary mix-up is represented as

where λ v ( λ t ) is a learnable coefficient for the image (text) modality, determining the asymmetry in the complementary mix-up. The formula for the text modality is similar. Through this augmentation, pairwise relationship s ij is adjusted as deterministic values by s ij = 1 -∏ C c =1 (1 -ˆ l c i ˆ l c j ) .

Adaptive Negative Masking. As illustrated in Fig. 1, negative pairs can disappear drastically at high unknown ratios. The existence of negative pairs is especially vulnerable yet significant in learning robust cross-modal relationships. Naive solutions such as AN (Assume Negative) [6] set all unknown values to 0 , introducing excessive noisy pairs that can mix up with the true negative pairs and inhibit the model fitting. Therefore, we propose adaptive negative masking to restore a correct number of negative pairs. Through semantic augmentation, a real-valued similarity matrix S D ∈ ([0 , 1] ∪ { u } ) D × D is constructed between unknown pairs with batch size D . By denoting U = U [ S D ] and the ratio of S D ij = 0 to S D ij > 0 as r = |N [ S D ] | / |P [ S D ] | , the similarity matrix is adjusted adaptively as

where ◦ denotes the Hadamard product, R ∈ { 0 , 1 } D × D is a random mask that sets unknown values to 0 for a number that resets r = t , and t is a threshold close to 0 that prevents noisy similarity structure while slightly enabling AN when there are few negatives. To learn hash codes with these augmentation strategies, the overall hash optimization is defined in Appendix Sec. A.2

4 Experiments

4.1 Experiment Settings

We evaluate our method on the MIRFlickr-25k (Flickr) [12], MS COCO (COCO) [20], and NUSWIDE (NUS) [5] datasets. We use the frozen ViT-B-32 CLIP as backbones for all methods and set

the main hash bit as 32. Details regarding datasets, implementation, evaluation settings, and results for other bit configurations are presented in Appendix Sec. B and Sec. C.

4.2 CMHwith Incomplete Labels

To validate the model's effectiveness on CMH with incomplete labels, we illustrate mAP comparison results in Table 1. Our model consistently achieves superior results across various known ratios on all benchmarks. We clarify that there is no prior deep CMH method specifically designed for incomplete labels. Therefore, we have to use regular CMH baselines by setting different missing ratios but keeping other settings equivalent. Compared to shallow methods, our method achieves higher results,

especially on Flickr, with a 4 . 4 %improvement on average, validating the representation ability of our method. In comparison with deep CMH methods, our model enjoys greatly improved results on all three datasets, with a mean 8 . 96 %mAP enhancement. Some deep CMH methods perform worse with incomplete labels especially on the NUS dataset, indicating their collapsed similarity learning. In comparison with DCHMT, which also adopts CLIPs as backbones, the proposed PCRIL obtains an average of 12 . 13 %mAPimprovement on all datasets. These results demonstrate that our model can consistently obtain superior results across unknown ratios and benchmarks, validating the effectiveness of our PCRIL in label recovery and similarity calibration for CMH.

4.3 Ablation Study

To verify the validity of each module, we conduct an ablation study of the proposed method by comparing it with 4 variants shown in Table 2 on the Flickr, NUS, and COCO datasets, and 4 module variants exclusively on the conventional AN setting. In contrast with the two conventional baselines (IU and AN), our ANM-enhanced baseline surpasses them by 6 . 66 % and 6 . 09 % mAP, respectively. This verifies the effectiveness of the balanced pairing strategy. By adding PCR, the model's performance is significantly improved by an average of 3 . 52 %, validating its ability to detect incompleteness and recover labels. Although initial recovery by ANM and PCR is substantial, by adding CSA, the model further gains consistent improvements of 0 . 60 %on average across all datasets, verifying its ability of pairwise similarity augmentation. The two modules together make a joint improvement of 4 . 12 %on average, while the entire PCRIL rescues more than 10 %of mAP from unknown labels. In ablation studies within the AN setting (Table 3), our proposed components consistently deliver stable improvements, highlighting PCRIL's ability to overcome limitations inherent in traditional training schemes. These results have comprehensively verified the effectiveness of PCRIL for CMH with incomplete labels.

4.4 Model Analysis

Prompt Construction. We analyze our proposed label prompt learning by comparing it with two variants: 1) phrasal : a handcrafted prompt in CLIP's original template of 'A photo of CN c 1 , CN c 2 , ..., and CN c n ." is constructed to directly perform recovery search, and 2) conventional : averaging single-class prompt embeddings to acquire the label prompt embeddings. The recovery results are shown in Table 4. Compared with handcrafted prompts, our learnable prompts achieve a 6 . 1 % mAP enhancement and improve the restoration precision by 22 . 33 %, verifying the effectiveness of

+[X]

+[St]

+[R]

+[W]

+[Pl]

+[Sk]

[Clouds]

[ Male ]

[Plants]

[ Life ]

[ Tree ]

𝑲𝒑

𝒊

+[X]

+[St]

Abbreviations

[P] =  [People]

[R] =  [River ]

[T] =  [ Tree ]

[W] =  [Water ]

[X] =  [Transport]

+[St]

+[P]

+[P]

+[C]

41.91

42.88

42.81

42.75

43.49

43.37

43.32

44.03

43.91

43.64

[People]

𝑲𝒑

𝒊

+[Pl]

+[W]

+[S]

+[Pl]

+[S]

+[T]

34.47

35.53

35.32

35.14

36.36

36.31

35.97

37.12

37.00

36.36

PLTS iteration:   1           2           3

PLTS iteration:    1            2            3

…

…

…

…

…

…

+[Cl]

+[Ss]

+[Sk]

[Structure]

𝑲𝒑

𝒊

+[Cl]

+[Ss]

+[Ss]

+[T]

+[T]

+[Pl]

25.38

28.87

27.70

27.08

30.84

29.90

29.75

31.76

31.20

30.63

PLTS iteration:     1           2           3

…

…

…

[Ca] =  [ Car  ]

[Cl] =  [Clouds]

[Pl] =  [Plants]

[Sk] =  [ Sky  ]

[Ss] =  [Sunset]

[St] =  [Structure]

𝑲𝒑

𝒊

Positive

Set Scores



































(a)











































(b)





































(c)











































(d)

learned prompts. Compared with conventional prompts, our method achieves consistently improved performances, further validating the effectiveness of the structural prior utilized in our method.

Prompt Search for Recovery. We validate our prompt recovery with three variants: 1) image: label recovery search with the image modality only, 2) text: label recovery search with the text modality only, and 3) one-step all: directly recovering pseudo-labels on the initial positive label set K i p (1) without searching, which means Q = K i p (1) . The results are shown in Table 5. Compared with image and text recoveries respectively, our recovery with both modalities displays about 3 % and more than 10 %improvements. This reveals our model's ability to evaluate the joint completeness in label prompts. Performing 'one-step all" recovery, it hardly recalls true positives but still achieves acceptable mAPs, validating the latent hierarchical structure of the matching scores. It can be inferred that the prompt search works by peeking at scores for confident classes that aid completeness the most. Meanwhile, we plot how the number and precision of recovered positives change through the beginning epochs w.r.t. modalities in Fig. 6 (a-b). The precision rises with the growth of epochs while recalling more ground-truth labels, validating that the prompt gradually separates more complete and more incomplete semantic descriptors to make true positives emerge. Recovery with a single modality can introduce a drop in precision, where the text modality has especially low results of 77 . 16 %mainly due to its nature of partial reference to labels. However, recovery with both modalities surprisingly boosts the precision to achieve a more effective 89 . 78 %.

Matching Scores Visualization. To further analyze the property of descriptive completeness, we illustrate some case images' matching scores during the PLTS process in Fig. 5. In the results, the recovery of a potential positive class is accompanied by an increase in the set score, indicating that the recovered label set is more consistent with the instance. For example, the score increases from the original 25 . 38 to its best 31 . 76 by successively recalling sky , clouds , and sunset for the bottom-left instance. We can observe that class rankings can vary at each step, indicating that the PLTS can generate more precise measurements of multi-label complexity. Meanwhile, though not recovered as positives, the remaining scores (in orange discs) of true positives can still surpass their anchors from the last iteration, validating their effectiveness as pseudo-labels.

Enhancement of Pairwise Similarity. As shown in Fig. 1, CMH can suffer from severe pairwise uncertainty. By adopting our proposed prompt contrastive recovery and complementary semantic augmentation, respectively, the similarity structure is efficiently recovered in Fig. 6 (c-d). The complementary semantic augmentation greatly alleviates the pairwise scarcity. For situations at approximately 50 %unknown where negative pairs are completely lost, a substantial 20 %proportion of negative pairs are reconstructed, ensuring its ability to rescue similarity learning at high unknown ratios. Meanwhile, the prompt contrastive recovery significantly improves the distribution of both positive and negative pairs via its pseudo-labeling, reducing an average of 80 % uncertain pairs, verifying its strong separability for completeness of labels. Generally, these two modules ensure the method can learn CMH on sufficient data pairs with clear relationships.

5 Conclusion

In this paper, we identified the problem of incomplete labels and the consequent collapse of similarity learning in CMH. To overcome these challenges, we introduced Prompt Contrastive Recovery for CMH with Incomplete Labels (PCRIL), a novel framework that jointly recovers semantic classes and pairwise similarity. This is the first CMH method to enable prompt learning with incomplete labels. Specifically, by constructing contrastive learning with a hierarchical label prompting method, prompt contrastive recovery learns the completeness of label descriptors and detects lost labels. Moreover, complementary semantic augmentation eliminates the sparsity of semantic pairs via a complementary feature blending strategy to restore similarity. An adaptive negative masking strategy is adopted to further balance the pairwise hashing. Extensive experiments on widely used benchmarks validated that PCRIL can significantly outperform state-of-the-art CMH methods with different partial levels.

Limitations. This study highlights the effectiveness of the vision-language model prior in perceiving label completeness, specifically for cross-modal retrieval. However, we note that it does have limitations. The prompt construction currently relies on the pretrained CLIP model with a limited number of text tokens, which hinders its ability for richly labeled samples. However, annotation length exceeding the CLIP capacity is extremely rare in practice. Meanwhile, the learning of PCR relies on sufficient multi-labeled samples. Single positive cases [6] may infrequently exist in real applications. We suspect that allowing unknown labels in the anchor sets could reactivate the proposed paradigm.

Acknowledgments and Disclosure of Funding

This research is partially supported by Shenzhen Science and Technology Program (Grant No. RCYX20221008092852077), National Natural Science Foundation of China (Grant No. 62372132), and the Australian Research Council (Grant No. DE240100105, DP240101814, DP230101196).

References

Bucak Serhat Selcuk, Jin Rong, Jain Anil K . Multi-label learning with incomplete class assignments // Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. 2011. 2801-2808.

Cao Yue, Liu Bin, Long Mingsheng, Wang Jianmin . Cross-modal hamming hashing // Proceedings of the European Conference on Computer Vision. 2018. 202-218.

Chen Zhen-Duo, Li Chuan-Xiang, Luo Xin, Nie Liqiang, Zhang Wei, Xu Xin-Shun . SCRATCH: A scalable discrete matrix factorization hashing framework for cross-modal retrieval // IEEE Transactions on Circuits and Systems for Video Technology. 2020. 30, 7. 2262-2275.

Chu Hong-Min, Yeh Chih-Kuan, Wang Yu-Chiang Frank . Deep generative models for weaklysupervised multi-label classification // Proceedings of the European Conference on Computer Vision. 2018. 400-415.

Chua Tat-Seng, Tang Jinhui, Hong Richang, Li Haojie, Luo Zhiping, Zheng Yantao . NUS-WIDE: a real-world web image database from National University of Singapore // Proceedings of the ACM International Conference on Image and Video Retrieval. 2009. 48.

Cole Elijah, Mac Aodha Oisin, Lorieul Titouan, Perona Pietro, Morris Dan, Jojic Nebojsa . Multi-label learning from single positive labels // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. 933-942.

Deng Jia, Russakovsky Olga, Krause Jonathan, Bernstein Michael S, Berg Alex, Fei-Fei Li . Scalable multi-label annotation // Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2014. 3099-3102.

Ding Zixuan, Wang Ao, Chen Hui, Zhang Qiang, Liu Pengzhang, Bao Yongjun, Yan Weipeng, Han Jungong . Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 3398-3407.

Durand Thibaut, Mehrasa Nazanin, Mori Greg . Learning a deep convnet for multi-label classification with partial labels // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. 647-657.

Fu Zheren, Mao Zhendong, Song Yan, Zhang Yongdong . Learning Semantic Relationship Among Instances for Image-Text Matching // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 15159-15168.

Gu Wen, Gu Xiaoyan, Gu Jingzi, Li Bo, Xiong Zhi, Wang Weiping . Adversary guided asymmetric hashing for cross-modal retrieval // Proceedings of the 2019 on International Conference on Multimedia Retrieval. 2019. 159-167.

Huiskes Mark J, Lew Michael S . The mir flickr retrieval evaluation // Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval. 2008. 39-43.

Jiang Ding, Ye Mang . Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. 2787-2797.

Jiang Qing-Yuan, Li Wu-Jun . Deep cross-modal hashing // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 3232-3240.

Kim Wonjae, Son Bokyung, Kim Ildoo . Vilt: Vision-and-language transformer without convolution or region supervision // Proceedings of the International Conference on Machine Learning. 2021. 5583-5594.

Kim Youngwook, Kim Jae Myung, Akata Zeynep, Lee Jungwoo . Large loss matters in weakly supervised multi-label classification // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 14156-14165.

Li Chao, Deng Cheng, Li Ning, Liu Wei, Gao Xinbo, Tao Dacheng . Self-supervised adversarial hashing networks for cross-modal retrieval // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. 4242-4251.

Li Junnan, Li Dongxu, Savarese Silvio, Hoi Steven . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models // Proceedings of the International Conference on Machine Learning. 2023. 19730-19742.

Li Junnan, Li Dongxu, Xiong Caiming, Hoi Steven . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation // Proceedings of the International Conference on Machine Learning. 2022. 12888-12900.

Lin Tsung-Yi, Maire Michael, Belongie Serge, Hays James, Perona Pietro, Ramanan Deva, Dollár Piotr, Zitnick C Lawrence . Microsoft coco: Common objects in context // Proceedings of the European Conference on Computer Vision. 2014. 740-755.

Lin Zijia, Ding Guiguang, Hu Mingqing, Wang Jianmin . Semantics-preserving hashing for cross-view retrieval // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. 3864-3872.

Liu Xuanwu, Yu Guoxian, Domeniconi Carlotta, Wang Jun, Xiao Guoqiang, Guo Maozu . Weakly supervised cross-modal hashing // IEEE Transactions on Big Data. 2019. 8, 2. 552-563.

Liu Yishu, Wu Qingpeng, Zhang Zheng, Zhang Jingyi, Lu Guangming . Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval // Proceedings of the 31st ACM International Conference on Multimedia. 2023. 893-902.

Lüddecke Timo, Ecker Alexander . Image segmentation using text and image prompts // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 7086-7096.

Luo Haoyang, Zhang Zheng, Nie Liqiang . Contrastive incomplete cross-modal hashing // IEEE Transactions on Knowledge and Data Engineering. 2024. 36, 11. 5823-5834.

Luo Xin, Yin Xiao-Ya, Nie Liqiang, Song Xuemeng, Wang Yongxin, Xu Xin-Shun . SDMCH: Supervised discrete manifold-embedded cross-modal hashing. // Proceedings of the International Joint Conferences on Artificial Intelligence. 2018. 2518-2524.

Ni Haomin, Zhang Jianjun, Kang Peipei, Fang Xiaozhao, Sun Weijun, Xie Shengli, Han Na . Cross-modal hashing with missing labels // Neural Networks. 2023. 165. 60-76.

Radford Alec, Kim Jong Wook, Hallacy Chris, Ramesh Aditya, Goh Gabriel, Agarwal Sandhini, Sastry Girish, Askell Amanda, Mishkin Pamela, Clark Jack, others . Learning transferable visual models from natural language supervision // Proceedings of the International Conference on Machine Learning. 2021. 8748-8763.

Sun Ximeng, Hu Ping, Saenko Kate . Dualcoop: Fast adaptation to multi-label recognition with limited annotations // Proceedings of the Advances in Neural Information Processing Systems. 2022. 35. 30569-30582.

Tu Junfeng, Liu Xueliang, Lin Zongxiang, Hong Richang, Wang Meng . Differentiable crossmodal hashing via multimodal transformers // Proceedings of the 30th ACM International Conference on Multimedia. 2022. 453-461.

Veit Andreas, Alldrin Neil, Chechik Gal, Krasin Ivan, Gupta Abhinav, Belongie Serge . Learning from noisy large-scale datasets with minimal supervision // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 839-847.

Xu Xing, Shen Fumin, Yang Yang, Shen Heng Tao, Li Xuelong . Learning discriminative binary codes for large-scale cross-modal retrieval // IEEE Transactions on Image Processing. 2017. 26, 5. 2494-2507.

Yang Hao, Zhou Joey Tianyi, Cai Jianfei . Improving multi-label learning with missing labels by structured semantic correlations // Proceedings of the European Conference on Computer Vision. 2016. 835-851.

Zhang Hongyi, Cisse Moustapha, Dauphin Yann N, Lopez-Paz David . Mixup: Beyond empirical risk minimization // arXiv preprint arXiv:1710.09412. 2017.

Zhang Zheng, Luo Haoyang, Zhu Lei, Lu Guangming, Shen Heng Tao . Modality-invariant asymmetric networks for cross-modal hashing // IEEE Transactions on Knowledge and Data Engineering. 2023. 35, 5. 5091-5104.

Zhou Kaiyang, Yang Jingkang, Loy Chen Change, Liu Ziwei . Learning to prompt for visionlanguage models // International Journal of Computer Vision. 2022. 130, 9. 2337-2348.

Appendix

In the following sections, we provide detailed information about our model (Sec. A) and experimental settings (Sec. B), and offer more extensive validations (Sec. C) of our proposed PCRIL.

A Model Details

A.1 Constructed Prompts

We have formulated our constructed prompts in Eq. (1). Here, we describe a specific example of prompt construction. Assume we have a sample with a positive label set { cat , wall , house } and a negative one { dog , plane }. A naive textual prompt can be formulated as 'A photo of a cat , next to the wall of a house .', while our prompt is constructed by selecting an anchor subset ( e.g., C a ={ wall , house }) to form P ( C a ) = ( p head , p house , p wall , p tail ) with randomly shuffled class order by σ . Then, we can directly construct two negative prompts P ( C 1 n ) = ( p head , p wall , p tail ) and P ( C 2 n ) = ( p head , p dog , house , p wall , p tail ) where the negative sets C n ={ wall } and C n ={ dog , house , wall } are formulated by adding the negative class t = dog and removing the positive class s = house , respectively. We can also construct another negative prompt P ( C 3 n ) = ( p head , p plane , p cat , p tail ) by replacing class wall with the negative one plane in C a . Both positive and negative classes are uniformly selected in our experiments. We illustrate our prompt construction and contrastive learning in Fig. 7.

[Beach]

[ Sea ]

[Beach]

[ Sky ]

[ Sea ]

[Street]

[ Car ]

[Person]

𝑲𝒑

𝒊

𝑲𝒂

𝒊

Unknown

Random

Select

Positive

Negative

…

[Beach]

[ Sky ]

[ Sea ]

[Child]

[ Man ]

[Woman]

[Beach]

[ Sea ]

[Bird]

[Room]

Anchor

Limited

Sample-Label

Pairs

[Beach]

[ Sea ]

[Beach]

Sample

𝒉𝒊

S a m p l e s

Label

Combinations

Anchor

Delete

[Beach]

[ Sea ]

[ Car

]

Join

[Beach]

[ Car

]

Replace

Sample's Positive

Class Set (Incomplete)

Sample

𝒉𝒋

[Camel]

[Camel]

[Room ]

[Camel]

[Desert]

[Camel]

[ Sea ]

[Camel]

[Desert]

[Man]

Search

Path

Distance

to Target

𝑚

Contrastive

Margin

(Termination)

Iteration

Minimum

[Camel]

[Desert]

[Horse]

(a) The motivation of selecting positive anchor sets to enrich sample-label pairs.

(b) The relationship of the anchor and negative subsets.

(c) The potential label tree search with contrastively learned label embeddings.

A.2 Optimization

To learn hash functions, the general CMH methods [14, 17] optimize the following minimum negative log-likelihood objective:

where L q = ∥ H v ( F ) -B ∥ 2 F + ∥ H t ( G ) -B ∥ 2 F is the quantization terms and H v,t are the visual and textual hash layers. By defining the conditioned probability of the similarity as

where ϕ ij = 1 2 ( H v ( f i )) ⊤ ( H t ( g j )) and δ ( ϕ ) = 1 1+ e -ϕ , the objective function learns to align the inner-product similarity of hash codes with the ground-truth similarity values.

However, the estimated ground-truth similarity in our work, either generated by recovered pseudo labels or by complementary blending, does not apply to the above probability due to its non-binary

Algorithm 1 Optimization Algorithm for PCRIL

Require:

Training dataset O .

Ensure:

Optimal discrete hashing codes B , prompt variables θ p , modal hash-net parameters θ v , θ t .

1: Initialization Randomly initialize learnable parameters. Configure the learning-rate µ ; the mini-batch size D ; iterations EP p.net and EP hash .

2: while iter i < EP p.net &not converged do

3: Sample a minibatch ( f i , g i , l i ) D i =1 ;

4: Sample an anchor set K i a for each label l i and generate its negative sets K i,s d , K i,t j , and K i,s,t r ;

5: Compute loss L ctr and update θ p ;

6: end while

7: Recover labels and pseudo labels by prompt search stated in Sec. 3.2;

8: while iter j < EP hash &not converged do

9: Sample a minibatch ( f i , g i , l i ) D i =1 ;

10: Sample and complementarily mix up instances with Eq. (6);

11: Compute similarity supervision and randomly transfer unknown pairs as negatives by Eq. (7);

12: Compute loss L hash and update θ v,t ;

13: Update code B with B = sign( H v ( F ) + H t ( G ))

14: end while

property. Therefore, we formulate a learnable objective by leveraging the Kullback-Leibler divergence as

where D KL ( ·∥· ) is the Kullback-Leibler divergence, and ˆ s ij ∈ [0 , 1] is the estimated ground-truth similarity. Taking Eq. (9) into account, the final hashing objective is derived as

̸

where L sim ij = log(1 + e ϕ ij ) -ˆ s ij ϕ ij . The detailed batch-wise learning process for PCRIL is displayed in Algorithm 1.

B Experimental Settings

B.1 Datasets

We evaluate our method extensively on three widely employed benchmarks for CMH, namely, MIRFlickr-25K (Flickr)[12], NUS-WIDE (NUS)[5], and MS COCO (COCO)[20]. Flickr consists of 20 , 015 image-text pairs that belong to 24 semantic classes in which 15 % are annotated as positive. NUS is a large-scale cross-modal dataset with about 300 , 000 pairs of images and texts in 80 categories. The most frequent 21 categories are selected to form a labeled set where 10 % are positive. COCO is a cross-modal dataset consisting of about 120 , 000 image-text instances associated with 80 semantic classes, among which confirmed positives make up only 3 . 6 %of label elements, making it a rather harder dataset as shown in Fig. 1. As we stated in Sec. 3.1, we randomly select a given proportion of binary labels and mask them as unknown during training, and unmask them for retrieval inference.

B.2 Data Split

For the three evaluated datasets, we select larger training subsets compared to general CMH methods to better approximate the real incomplete label scenario. The statistics of our evaluated datasets are summarized in Table 6. All experiments conducted have adopted this same configuration.

B.3 Implementation Details

We simply adopt publicly available ViT-B-32 based CLIP backbones and frozen weights for both image and text feature encoders of our PCRIL and other compared methods. Upon them, two 4-layer MLPs with the tanh function as the last activation project the 512 -dimensional image and text CLIP features into 32 -bit binary codes, respectively. Other layers use ReLU as the activation function. The MLP layer dimensions are 512-1,024-2,048-512-32 from input to output. The MLP parameters are initialized with normal distributions and optimized with Adam optimizers for each modality. The learning rate is 10 -4 . 5 for image and 10 -3 for text.

For the label prompt encoder, we encode the prompts as text CLIP features and only tune the parameters in the prompts. These parameters are randomly initialized with a normal distribution. For prompt learning, we use the Adam optimizer with a learning rate of 10 -4 . During training, we finetune the prompt and MLP parameters while keeping the weights of CLIP encoders fixed to preserve prior knowledge.

We set the number of learnable tokens, specifically n , m , n a , and m a , as 2 for simplicity. We empirically set t = 0 . 01 as the ANP threshold and search for the best values for the margin m and the top K number, which are analyzed in Sec. C.2. We run all our experiments with 2 NVIDIA 2080ti GPUs.

B.4 Evaluation Settings

We evaluate the model effectiveness w.r.t. a range of known ratios. From slightly incomplete to almost untrainable, the selected ratios are 70 %, 50 %, and 30 %. To evaluate the retrieval ability of CMH models, we leverage the mean average precision (mAP) as our evaluation metric following common CMH works. When further analyzing our method in Sec. 4.4 and Sec. C, we also show the precision of recovered positive labels as evidence.

Missing:

female

Missing:

water

Missing:

animal

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.



.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

C More Experimental Analyses

C.1 More Results for CMH with Incomplete Labels

We further validate PCRIL for CMH with incomplete labels on the IAPR TC-12 dataset. As shown in Table 8, stable improvements are achieved especially with fewer known labels, validating the effective label recovery and hash learning of the proposed method.

C.2 Hyper-parameters

We further analyze the contrastive margin parameter m and the complementary blending topk in our work. The results in Table 7 demonstrate that our method is generally non-sensitive to parameter changes. Although it obtained a slightly lower recovery precision, the proposed prompt contrastive recovery can achieve its best performance of 82.7% mAP at a lower margin m = 0 . 50 by recalling more positives. Higher margins prioritize precision over recall and result in a drop in the overall mAP performance. When fixing m , the model achieves stable results around 82.4% mAP for different topK values.

C.3 Completeness Attention Visualization

By treating each step of searching as a classification task, we list several case images in Fig. 8 with their feature heatmaps overlaid, which are generated with the ResNet-50 image backbone. From the results, we can observe that the learnable prompts successfully steer the model prior to be sensitive to potential classes. By attending more to the unlabeled object, the trained model can precisely detect incompleteness in label prompts, verifying its effectiveness in semantic detection and recovery.

C.4 Feature Visualization

We compare the t-SNE results of learned binary features with baseline outputs in Fig. 9. In the results, our CMH learning with recovered semantics can gather hash codes of similar samples into compact clusters. Our PCRIL generates semantically more discriminative codes than the baseline method, hence achieving higher performance.

C.5 CMHof Different Code Lengths

We compare our method with others at 16 and 64 bits in Table 9. From the results, PCRIL is validated to perform consistently well with mean mAP values across hash bits. Meanwhile, larger bits often produce higher average performance for CMH with incomplete labels, which is consistent with cases in fully labeled CMH.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper proposes a novel prompt contrastive recovery method for crossmodal hashing with incomplete labels and validates its superior recovery ability through extensive experiments, accurately matching the main claims in the beginning.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Non-theoretical paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The architecture is fully described in the main paper. The implementation details are briefly introduced in the main paper and fully disclosed in the supplementary sections. Source code can be found in the supplementary materials for reference.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided code and instructions about data and code in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We briefly notify the key information of datasets and backbone settings in the main paper while provide full experimental details in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Although our results are acquired by averaging multiple runs, it's too computationally expensive to run enough times and the space to put error bars is too limited. Various original results need to be presented and they already occupy a significant portion of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide computation resources used in our experiments in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in the paper fully conforms with the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Although our paper tackles the incomplete labels which is related to the robustness of models, there is no direct association to societal aspects.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper solves incomplete labels in cross-modal hashing and does not have these risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [No]

Justification: The CLIP model (MIT License) and the three datasets are well cited and introduced in our paper. However, these datasets contain internet images with custom licenses (in CC) and licenses we were unable to find.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The released code is well documented with a codebase repository link.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCross-modal hashing (CMH) has attracted much attention due to its low computational and storage costs while maintaining high-dimensional cross-modal semantic similarity. To address the incomplete annotation challenge of CMH, this paper proposes a novel Prompt Contrastive Recovery method, PCRIL. The method includes prompt contrastive recovery (PCR) and complementary semantic augmentation (CSA) modules. Experimental evaluation verifies the effectiveness of the proposed method. Overall, this paper has done a sufficient and concrete work. However, the novelty of the proposed method is limited, and the comparison in this experiment is also limited.\n\nSTRENGTHS:\n1.This paper addresses the problem of incomplete annotations in cross-modal hashing, which is very common and challenging in practical applications.\n2.The prompt contrastive recovery (PCR) module proposed in this paper effectively perceives incompleteness through label prompts and can well restore the information semantics, which has been well demonstrated by experiments.\n\nWEAKNESSES:\n1.The authors lack comparison with some recently proposed SOTA methods, no one within two years, so the experimental results are not convincing.\n2.This paper conducted experiments on three datasets with different label known ratios, but some comparison methods provided relevant experimental results on the IAPR TC-12 dataset. The author did not provide relevant experimental results.\n\nQUESTIONS:\n1.There are no comparative methods within two years. Can the author perform some recent methods to show the effectiveness of the proposed method.\n2.It is recommended that the authors conduct more experiments on the IAPR TC-12 dataset to more intuitively illustrate the effectiveness of the model.\n3.Please elaborate on how to remove negative pairs from unknown classes.\n\nLIMITATIONS:\nPrompt construction currently relies on a pre-trained CLIP model with a limited number of textual labels, which hinders its ability to enrich labeled samples.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel cross-modal hashing method named PCRIL, which explores the indispensable but challenging problem of incomplete label recovery in multi-label learning. It conceives a CLIP-based prompting scheme and a complementary semantic propagation mechanism, enabling PCRIL to restore unknown labels and calibrate pairwise similarities. The paper exhibits a strong motivation, technical soundness, and a well-structured organization. Generally, the idea of the paper is interesting, especially in constructing a learnable label prompt to perceive the missing labels in cross-modal learning.\n\nSTRENGTHS:\n- This paper explores the indispensable but challenging problem of incomplete label recovery in multi-label learning. The authors identify this crucial problem for cross-modal hashing and propose an effective solution for this problem. The proposed method is technically sound.\n\n- The paper conceives a CLIP-based prompting scheme and a complementary semantic propagation mechanism, enabling the proposed method to restore unknown labels and calibrate pairwise similarities.\n\n- The authors design a simple learnable prompt to encode class combinations into CLIP embeddings.\n\n- The neglected semantic labels and pairwise similarities can be removed and recovered through the proposed architecture.\n\n- Extensive experiments have been conducted on MIRFlickr-25K, NUS-WIDE, and MS COCO.\n\nWEAKNESSES:\n- The CLIP-based prompting has been intensively studied and this paper needs to make sufficient analysis and experimental validation of its key contribution, i.e., the three types of contrastive learning, particularly in terms of their advantages over plain prompt contrastive recovery.\n\n- Can the authors analyze the effects of the ways of the prompt construction in this task?\n\n This paper contributes a new problem and some new ideas to the CMH literature. The authors should further explain the necessity of the given research topic in practical situations.\n\nThe Method section should provide more explanation of these figures. Currently, these parts are separated, making the model difficult to follow, especially given the article's innovative points.\n\nQUESTIONS:\nPlease refer to the weaknesses. The authors are advised to thoroughly resolve the above concerns to convince the reviewer.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript tackles the challenges of generating high quality hash codes for cross-modal retrieval in the presence of incomplete labels,  which creates uncertainty in distinguishing between positive and negative pairs. To address the issue, a novel Prompt Contrastive Recovery approach called PCRIL is proposed, which progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels.\n\nThe proposed PCRIL framework jointly performs semantic recovery and pairwise uncertainty elimination for efficient cross-modal hashing with incomplete labels. In particular, they consider each subset of positive labels and construct three types of negative prompts through deletion, addition and replacement for prompt learning. Augmentation techniques are also derived for addressing extreme cases of significant unknown labels and lack of negative pairwise supervision. Experimental results show significant improvement in mAP of the proposed solution with respect to the current SOTA methods.\n\nSTRENGTHS:\n(1) The idea proposed in the paper seems to be novel and interesting to resolve the incompleteness for cross-modal hashing. Typically, prompt processing and contrastive learning are combined to formulate prompt constrastive recovery, which effectively detect the potential classes and enhance the similarity learning.\n(2) Moreover, the augmentation techniques are also very helpful for handling extreme cases including unknown labels and negative pairs where an asymmetric mix-up is introduced and adaptive negative masking is devised .\n(3) The experimental evaluation is detailed and sufficient by evaluating the proposed method on three datasets with hyper-parameter tuning and visualization.\n\nWEAKNESSES:\n(1) Some of the techniques are proposed lacking clear motivation. For instance, it is not very clear that why the mix-up augmentation is asymmetric. \n\n(2) There are already related work on contrastive learning on prompt such as \n(a) https://arxiv.org/pdf/2205.01308, \"Contrastive Learning for Prompt-Based Few-Shot Language Learners\". NAACL 2022.\n(b) https://openaccess.thecvf.com/content/CVPR2023/papers/Zhang_CLAMP_Prompt-Based_Contrastive_Learning_for_Connecting_Language_and_Animal_Pose_CVPR_2023_paper.pdf\nIt is important to compare the proposed method to these existing work to see if there are performance gains. Also it seems that compared to reference (b), the proposed method is quite similar as they are also based on prompt contrastive learning for cross-modal retrieval or hashing  except that they are applied to different tasks (pose estimation vs hashing).\n\nQUESTIONS:\nAs mentioned above, the paper needs to highlight and clearly states the motivation of some proposed techniques.\n\nThe discussion, comparison and clarification of current methods in this field need to be included.\n\nLIMITATIONS:\nFrom the impact, it seems the method is incremental and focuses on a relatively small scope.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel approach, Prompt Contrastive Recovery for Incomplete Labels (PCRIL), for cross-modal hashing with incomplete labels in this paper. They utilize a learnable CLIP prompt to encode selected anchor class combinations and employ a contrastive learning paradigm to construct multiple negative variants of the anchor set. Additionally, they introduce tree search methods for label recovery and develop augmentation strategies to handle extreme cases of unknown labels and negative pair scarcity. Extensive experiments on various datasets validate the effectiveness of their approach.\n\nSTRENGTHS:\nThe paper demonstrates strong originality by combining a learnable CLIP prompt, contrastive learning paradigm, and tree search for label recovery in cross-modal hashing. It presents a well-founded and thoroughly tested solution, with comprehensive analysis and experiments that validate its effectiveness. The writing is clear and organized, making the methodology easy to understand. The contributions offer new insights and advancements that can benefit both researchers and practitioners in the field.\n\nWEAKNESSES:\nThe paper could highlight its contributions by providing a detailed comparison with existing methods and including more recent studies in the related work section. A deeper theoretical analysis explaining the effectiveness of the proposed methods is needed.\n\nQUESTIONS:\nHow does the reliance on the CLIP model's text token limit impact the overall performance of your approach? What are the specific challenges associated with needing sufficient multi-labeled samples?\n\nLIMITATIONS:\nThe authors have identified and acknowledged several limitations of their work. And they have mentioned the limitation of the prompt construction relying on the pretrained CLIP model with a limited number of text tokens, but would it be beneficial for the authors to provide more details on how this limitation specifically affects the performance of their model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-17"}

comments: SUMMARY:
In this paper, to solve the problem of unknown labels in the task of cross-modal retrieval, the authors aim to progressively identifies promising positive classes from unknown label sets and recursively searches for other relevant labels.

STRENGTHS:
(1)Compared with existing related works, the proposed method has a large performance improvement.
(2)The author proposed a new strategy to solve the problem of label missing in thr cross-mdoal retriveval task.

WEAKNESSES:
(1)The meaning of the figure 3 is unclear. What is the meaning of “sorted index of label sets?” what is it relation to the positive label subset of each sample? How can we draw the conclusion that “the number of label sets is limited to the number of training samples.” from figure 3?

(2)What is the motivation and theory of the Negative Subsets and Contrastive Learning?  For the first type of negative subsets, why the anchor set is changed to negative subset by deleting a positive label and the difference between $K_a^i$ and $K_d^{i,s}$ should be minimized in the loss function of equation 4?

(3)In Potential Label Tree Search section, how to compute the score $Phi$? What is the theory of the termination condition? How to get this observation?

(4)The content is somewhat disjointed of the positive anchors and the PLTS.

(5)The ablation study setting is not reasonable. As can be seen, ANM and CSA are two augmentation strategies for handing extreme cases. The main  contribution of the paper is the Prompt Contrastive Recovery. Thus, variant version like B w/PCR, B w/PCR+CSA, B w/CSA should also be explored.

QUESTIONS:
Please see aboved  weeknesses.

LIMITATIONS:
The explanation of the main contribution part should be improved. The authors should give more motivation and related theory to verify its soundness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17