Deep Unsupervised Hashing via External Guidance
Authors: Qihong Song
XitingLiu
Hongyuan Zhu
Joey Tianyi Zhou
Xi Peng
Peng Hu
Venue: ICML 2025 poster (2025)
Keywords: Deep Unsupervised Hashing
External Guidance
Contrastive Learning
Image Retrieval
Primary area: general_machine_learning->representation_learning
TL;DR: We propose a novel method called Deep Unsupervised Hashing via External Guidance (DUH-EG), which incorporates external textual knowledge as semantic guidance to enhance discrete representation learning.

Abstract: Recently, deep unsupervised hashing has gained considerable attention in image retrieval due to its advantages in cost-free data labeling, computational efficiency, and storage savings. Although existing methods achieve promising performance by leveraging inherent visual structures within the data, they primarily focus on learning discriminative features from unlabeled images through limited internal knowledge, resulting in an intrinsic upper bound on their performance. To break through this intrinsic limitation, we propose a novel method, called Deep Unsupervised Hashing with External Guidance (DUH-EG), which incorporates external textual knowledge as semantic guidance to enhance discrete representation learning. Specifically, our DUH-EG: i) selects representative semantic nouns from an external textual database by minimizing their redundancy, then matches images with them to extract more discriminative external features; and ii) presents a novel bidirectional contrastive learning mechanism to maximize agreement between hash codes in internal and external spaces, thereby capturing discrimination from both external and intrinsic structures in Hamming space. Extensive experiments on four benchmark datasets demonstrate that our DUH-EG remarkably outperforms existing state-of-the-art hashing methods.

Deep Unsupervised Hashing via External Guidance
Qihong Song" 12 Xiting Liu Hongyuan Zhu
Joey Tianyi Zhou 56
Xi Peng
Peng Hu 12
Abstract
Image Data
Recently, deep unsupervised hashing has gained considerable attention in image retrieval due to its advantages in cost-free data labeling, computa- tional efficiency, and storage savings. Although existing methods achieve promising performance by leveraging inherent visual structures within the data, they primarily focus on learning discrimi- native features from unlabeled images through limited internal knowledge, resulting in an intrin- sic upper bound their performance. To break through this intrinsic limitation, we propose a novel method, called Deep Unsupervised Hash- ing with External Guidance (DUH-EG), which incorporates external textual knowledge as seman- tic guidance enhance discrete representation learning. Specifically, our DUH-EG: i) selects representative semantic nouns from external textual database by minimizing their redundancy, then matches images with them to extract more discriminative external features; and ii) presents novel bidirectional contrastive learning mecha- nism to maximize agreement between hash codes in internal and external spaces, thereby capturing diserimination from both external and intrinsic structures in Hamming space. Extensive exper- iments on four benchmark datasets demonstrate that our DUH-EG remarkably outperforms exist- ing state-of-the-an hashing methods. The code is available at: https//github..oo/X..c -ee//neennnn SCU/2025-ICML-DHHEE
Matching Prebabilily
0.429
0.571
Corresponding Nouns in W WordNet
Mismalch
External Nouns
mustang
airbus
American egir
Matching Probabilily
Match
Figure 1. The matching result among three images from the MSCOCO dataset. The category of the two images above (with blue border) is the airplane, while the remaining one (with yellow border) is categorized as bird. We illustrate bility based on features of image data and external nouns extracted by the CLIP (ViT-B/16) model (Radford et al., 2021), respectively. Specifically, for two images with a higher matching probability, if they belong different categories (indicating a mismatch), their matching probability is marked in red. Otherwise, the probability is marked in green.
matching proba-
1. Introduction
High retrieval efficiency and low storage cost are essen- tial for large-scale information retrieval tasks. Hashing methods map high- nigh-dimensional data into compact binary codes, significantly reducing storage space and accelerating computation through XOR operation, while preserving the discrimination in the discrete representations. As result, has received increasing attention from both academia and industry. Existing hashing methods can be broadly cate- gorized into supervised and unsupervised hashing. While supervised hashing methods achieve promising performance with well-labeled data, large-scale data annotation is labor- intensive, costly, and even infeasible in practice. By contrast, unsupervised approaches leverage unlabeled data to learn binary representations, thereby avoiding cost tos-prohibe data annotation and drawing more practical interest. The key to unsupervised hashing lies in constructing intrinsic data structures without class labels.
'The College Computer Science, Sichuan University Chengdu 610065, China "The State Key Laboratory of Inte- grated Services Network, Xidian University, Xi' an 710071, China Georgia Institute of Technology, USA *PR & CFAR, **STAR 'Centre for Frontier AI Research (CFAR), Ageney for Science, Technology Research (A* (A*STAR), Singapore Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore NNaiinnl Key Laboratory of Fundamental Algorithms and Models for Engineering Numerical Simulation, Sichuan University, China. Correspondence to: Peng Hu <penghu.m ml @ gmail.com>.
To address this challenge, most unsupervised hashing meth- ods explore mining inherent information from unlabeled data to facilitate hash code learning, such as data reconstruc- tion (Do et al., 2016; Song et al., 2018; Zieba et al., 2018),
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Deep Unsupervised Hashing via External Guidance
External Foalure Construction
Hash Codes Generation
Bidirectional Contr eslive Learning
t Regresentattve Foints
Normel Points
Af Tist Hash Codes
For Features
Pre-tnined
CLIP
Sumilar Estemal Knowlnige Pairs
A
e,e
Dissimilar Exteraal Knowledge Parts
A
Knowladgo
External
Pairwise
T
Nouns
An
Augt Hish Codes
Ner
Bird
Selected Discrimintive Poids
Eyge
Afrbus
Kitten
Eriemal Festuns
Weighted
Test Hashing Net
A
Text Harh Codes
Pairwise
$
Summalion
Ar Aug? Hash Codes
truined
N
CLIP
Hamming Space
N
Image Postures
Input Image
Know Jedge
Viewi of (Augt)
Backbore
Image Hashing Net
Aig! Hash Codes
Data Augientation
Shared
Shardd
View2 of image (Aug2))
Backbore
Image Hashing
A
Argg2 Hach Codes
Attrast
Repel
M Hash Codes
Net
APt
Augi Hash Codos
Am Ag2 Hash Codes
AT
Positive Sumple
Negative Somple
Figure 2. The overall framework of the proposed DUH-EG method. To be specific, in the left part, we construct the external features e corresponding to the visual feature of input images and identify similar (positive) and dissimilar (negative) external knowledge pairs, marking them with different colors. Then, we generate hash codes of the external feature h. and two augmented views h" and h respectively. After that, we employ a bidirectional contrastive learning loss to maximize the agreement across the positive samples as shown in the right part. Note that use distinet colors represent different clusters of normal points in the left part.
similarity preserving (Tu et al., 2020; Shen et al., 2018; Yang et al., 2019; Huang et al., 2016; Shen et al., 2019; Gu et al., 2019), contrastive learning (Luo et al., 2021b; Lin et al., 2022: Yu et al., 2022), etc. Data reconstruction methods typically utilize generative models to Iearn binary bottleneck representations by reconstructing input data, encouraging hash codes to preserve as much intrinsic information from input data as possible (Dai et al., 2017: Shen et al., 2020; Cao et al., 2018). However, such methods may preserve some disiiiicreeeeetive information for reconstruction in the Hamming space, such as extraneous background, lead- ing to sacrifice in hash code discriminability (Qiu et al., 2021). In contrast, similarity-preserving methods (Luo et al., 2021a; Dong et al., 2020) employ pre- pre-trained models or clus- tering techniques to extract underlining similarity structures, such as pairwise similarity (Yang et al., 2018; Ma et al., 2024) or pseudo-labels (Hu et al., 2017; Zhang et al., 2017), from unlabeled inputs guide hash function learning, thus enhancing the discriminability of the discrete representa- tions. Inspired by the great success achieved by contrastive learning (Chen et al., 2020), recent studies (Qiu et al., 2021; Ma et al., 2022; Luo et al., 2021b; Lin et al., 2022; Yu et al., 2022) have integrated it to maximize mutual informa- tion between positive pairs in Hamming space, constructed from data augmentation (Qiu et al., 2021), similarity struc- ture (Wei et al., 2024), pseudo-labels (Luo et al., 2021c), resulting in stronger supervision and more discriminative
hash codes. Despite these advancements, these methods, which rely solely on internal visual structure, may face an in- herent limitation on semantic guidance, potentially resulting in suboptimal performance.
To be specific, the exclusive use of internal supervision may introduce semantic ambiguity, thereby leading to an inherent performance ceiling. For instance, as illustrated in Figure 1, we extracted visual features from three images using pre-trained CLIP (ViT-B/16) model (Radford et al., 2021), and calculated the matching probabilities based on normalized similarity scores. Surprisingly, an airplane im- age has higher matching probability with bird image than with another airplane image (0.571 v.s. 0.429). Fortu- nately, by associating the images with external nouns (e.g. "airbus", "mustang", and "American (American.egrett)) we achieved clearer distinctions and more accurate matching probabili- ties (0.545 v.s. 0.455). In brief, leveraging external textual information could thus provide richer and more precise guid- ance for hash learning, enhancing the discriminability of discrete codes. However, it is challenging to find repre- sentative external nouns that discriminatively depict each image without class labels. Intuitively, we could retrieve re- lated nouns from a word database, such as WordNet (Miller, 1995). but excessive synonyms may be introduced and domi- nate textual semanties, leading to semantic homogeneity and discrimination loss. Additionally, integrating textual infor-
2

---

Deep Unsupervised Hashing via External Guidance
mation as external guidance presents challenges in aligning image-text semantics across different modalities, as well as reconciling internal and external knowledge for consistent semantic guidance.
2018; Zieba et al., 2018) and similarity preserving (Yang et al., 2018; Huang et al., 2016; Shen et al., 2019). For data reconstruction, various studies (Do et al., 2016; Song et al., 2018: Zieba et al., 2018; He et al., 2024) employ generative models such as variational auto-encoders (VAEs) (Kingma & Welling, 2014) and generative adversarial net- works (GANs) (Goodfellow et al., 2014) to reconstruct im- ages and produce hash codes that preserve intrinsic infor- mation implicit in inputs. For example, (Shen et al., 2020) proposes a Wasserstein auto-encoder variant that uses code- driven adjacency graphs to guide image reconstruction. Sim- ilarly, (Cao et al., 2018) leverages images synthesized by Pair Conditional Wasserstein GAN (PC- WGAN) to augment training data, enabling the hash model to learn hash codes that respect semantic pairwise relationships. For similarity preserving, some methods mine semantic similarity struc- tures from pre- trained models to learn similarity similarity-reesering preserving hash codes (Yang et al., 2018; Tu et al., 2020; Shen et al., 2018; Ma et al., 2024). By contrast, other methods lever- age clustering techniques to generate pseudo-labels that facilitate similarity-preserving preserving learning in the Hamming space (Huang et al., 2016; Shen et al., 2019; Wei al., 2023). For instance, (Luo et al., 2021a) employs deep clus- tering and similarity exploration to build pairwise semantic similarity structures from both local and global perspectives. Likewise, (Qiu et al., 2024) applies hierarchical clustering in the Euclidean tangent space to extract pseudo hierarchical semantics, thereby providing additional training supervision. Recently, contrastive learning has been introduced to learn hash representations by maximizing instance discrimination, achieving promising performance (Luo al., 2021b; Yu et al., 2022; Wei et al., 2024). However, almost all of these methods seek to explore the intrinsic information implicit in the input data, leading to an inherent performance ceiling.
To address the challenges, we propose Deep Unsupervised Hashing with External Guidance (DUH-EG) to enhance the semantic similarity guidanee for hash learning. as shown in Figure 2. Specifically, we first extract text features of each external noun using pre-trained vision-language model, clustering them to select representative features, thereby mitigating semantic homogeneity caused by excessive syn- onyms. DUH-EG then extracts image features using the vision-language model, matching them with the representa- tive textual features to integrate external features for each image. To balance internal and external information, we present a bidirectional contrastive learning loss align tex- tual and visual views for each image while maximizing the agreement across different augmented views. However, tra- ditional contrastive learning roughly treats different images as negative pairs, inevitably sampling some intra-class views as negatives, i.e., false negatives. To eliminate the adverse impact of false negative pairs, we select latent positive pairs from the negatives by using external features, enabling the model to focus semantic positives rather than only dif- ferent views of the same sample. The main contributions of this work are summarized as follows:
We propose a novel approach that incorporates exter- nal textual guidance to facilitate discrete representation learning, overcoming the inherent limitations of inter- nal visual structures.
An efficient, automated, and non-generative method is presented to select representative semantic nouns from accessible external textual database, integrating the external textual features external guidance of the corresponding images.
Improvement: Differing from existing peeeaartty-rrservingg methods, our DUH-EG introduces the corresponding tex- tual features as external knowledge for each image used in training with efficient, automated, and non- non-generative ap- proach. Furthermore, rather than generating leeuls-labels, DUH-EG balances the internal and external knowledge of the images by leveraging a contrastive learning strategy that aligns the visual and textual features of each training image. Additionally, textual features are utilized to identify poten- tial felse-negative pairs, thereby maximizing the consistency of features within the same class.
A bidirectional contrastive learning loss is developed to preserve discrimination in hash codes by maximizing mutual information between semantic positive pairs instead of different views, thereby enhancing effective- ness and robustness.
Extensive experiments on the four widely- bench- marks, i.e., CIFAR-10, MSCOCO, demonstrate that our approach remark- ably outperforms state-of-the-ar unsupervised hashing methods.
Flickr25k, and
2.2. Learning with External Guidance
In recent years, there has been a growing interest in incor- porating external guidance to enhance model performance. External resources, such as noun phrases (Li et al., 2024), co- occurrence relations of entities (Yang et al., 2023), instance- aware information (Yan et al., 2023), and large language models (Wang et al., 2023), have been increasingly used
2. Related Work
2.1. Deep Unsupervised Hashing
Deep unsupervised hashing approaches can primarily be cat- egorized into data reconstruction (Do et al., 2016; Song al.,
3

---

Deep Unsupervised Hashing via External Guidance
to enrich understanding and improve reasoning capabilities. This typically involves reusing existing guidance and in- tegrating external guidance to uncover latent and implicit representations, providing more comprehensive guidance information and improving model performance. For exam- ple, in the task of learning with noisy labels, (Wang et al., 2023) utilize large language models to assess the categories of training samples and generate confidence scores, which guide the fine-tuning of pre-trained language models. In 3D scene generation, (Wu et al., 2024) extract the relative posi- tional relationships and associated probabilities between the paired objects from existing indoor scene data. They incor- porate this external guidance to mitigate the ambiguity in object shapes and layouts in sketches, ultimately enhancing the diversity of the generated content. Similarly, in image classification and object detection, (Shen et al., 2022) select noun phrases from external textual sources, such category names and text titles, and append them to the original text data. These augmented textual inputs enhance the visual conceptual information in the data, improving the ability of the model to recognize and classify objects. Although these methods have demonstrated promising results with external guidance, their effectiveness in the context of hash learning remains an open question.
hash codes. Simultaneously, we eliminate false negatives by selecting latent positive pairs using external features, en- abling the model to focus on semantic positives. In addition, for ease of differentiation, we denote the aligned external feature and two differently augmented views of æ; as ei v and vī. respectively. The hashing networks applied to the augmented views and external features as H,, and He respectively, where H. consists of a pre-trained backbone followed by a hashing layer, while He consists solely of hashing layer to project the external features into L-bit hash codes. In our DUH-EG, the outputs of each hash function are defined as h, = (v) Ov), h = Ho θ,) and h = He, θ²) where O and O2 are the learnable pa- rameters in the hashing networks. h h and he are L-bit hash codes (1-1,+1)") corresponding two augmented views and an external textual view, respectively. Thus, to maximize agreement across each view of the redefined posi- tive pairs, the bidirectional contrastive learning loss function is formulated as:
LBCL = Cs Co (h Co (h)) (1)
3.2. External Feature Construction
In this phase, we first select all nouns from WordNet (Miller, 1995) as an external noun set. Following a prompt template subset recommended by CLIP I (contains 7 templates in total), each noun is then constructed into several natural phrases (texth we p=11 where w corresponds to the w-th sam- ple in the noun set, p refers to the index of prompt templates, and Np the number of prompt templates. Subsequently, we utilize a pre-trained CLIP model (Radford et al., 2021) to extract textual features t for each constructed phrase. To integrate the textual features of each word into a unified representation, we fuse these features through
3. Method
3.1. Overview
Let X' = - represents an unlabeled dataset that contains {N,} image samples. The goal of deep unsuper- vised hashing to learn a hash function H: x - h, e (-1,+134 that encodes the i-th sample of images æ; as L-bit compact discrete representation h, for efficient image retrieval. Following common practice in most previous hashing methods (Qiu et al., 2021; Ma et al., 2024; Luo et al., 2021b), we assume that the hash codes are composed of +1, ie., ha E {-1,+1}4.
N
ty
t "
w W f1, NNe.
N₂
p=1
The core challenge of unsupervised hashing is to construct semantic guidance from input data without relying on se- mantic labels. However, the semantic guidance extracted solely from the images is constrained by their inherent vi- sual structure. To overcome this limitation, we propose DUH-EG, which introduces external knowledge to improve the semantic similarity guidance for hash learning. To il- lustrate our DUH-EG, we categorize this approach into two distinct phases: External Feature Construction (Section 3.2) and Bidirectional Contrastive Learning (Section 3.3). In the first phase, the proposed method screens representative external nouns and extracts external features that match well with the visual features. For the subsequent stage, we present a bidirectional contrastive learning loss to maximize agreement between textual and visual views of the same image, thus encapsulating the instance discrimination into
where N, indicates the number of samples in the noun set. Next, to select representative semantic nouns that discrim- inatively characterize each image, we apply the K-means clustering to the fused textual features and generate N₂ clus- ters. For the k-th cluster, we select textual features that are relatively distant from the cluster center C₁ and treat them as representative subset { w= with Nrep features, while the remaining features are denoted as (t (onor) Near
( (twekk E [Erep] if dis cx) > Tı, (Ew,hh E [[nor] if dis(T cu) S Ti,
(3)
[Tw) = ((Ere))
Neen ComorrNnor }, Jw=i:
ImageNet... .ageNet.ipynb
4

---

Deep Unsupervised Hashing via External Guidance
where luck indicates that ty belongs to the k-thh cluster, dis, C&) denotes the distance between each word and its cluster center in the public embedding space, and Ti is distance threshold. In addition, to preserve informa- tive features of each cluster center, we predict the softmax probability Pur that each remaining word foor belongs to a cluster center Cj: as
the discrete representation of textual and visual views for each image while maximizing the mutual information be- tween the two augmented views. However, traditional con- trastive learning only treats different views of the same im- age as positive pairs, as in Equation (7), inevitably neglect- ing the potential semantic similarity among other samples and misidentifying some intra-class samples as negatives.
exp(cos((er (tmmr cx)) , Σ/= exp(eos(t. ttrr ,c,))
(4)
N.
Σ j exp( (h) A
L(a, b) = N,
log
Σ k= exp(
For each cluster, we select textual features with the highest softmax probability relative to the cluster center repre-
where a. E (U1, e (a + b), and T2 is a temperature parameter controlling the scaling of similarity.
sent it, forming new subset [feen) Newn.
/w=1- We combine (E Nrep and (Emr Neen to form a final representative Tw=1
/w=1 [( Tw=", (E Tw=") Neer). }-
To reduce the detrimental effect of false negative pairs, we redefine positive and negative pairs by finding external fea- ture pairs with relatively high similarity. To be specific, we first treat the hash codes of external features (h.) and two augmented views (h and h2²) that are derived from the same image as positive sample pairs. After that, to select potential positive pairs from the negatives, we calculate the cosine similarity between all external features and determine the semantic positive and negative pairs as
subset {Egmm) 1 Noom
Turm1
Subsequently, we use the pre-trained CLIP model to ex- tract image features i; and compute their counterpart from (Eorm) } Neom Juol for each image data T₁. To be specific, we cal- culate the softmax probability between the image features and each text feature, and treat the probability as weight. Hence, through applying weighted summation, the external feature e, corresponding to each image can be calculated as
Neam
j E {inea} if cos(e,, e₁) S T2, ( E {ipost if cos (e; eÿ)= > T2, {ineg} U {ipos} = (1,2, ... ,N.,,
8i D poom
(5)
(8)
exp(coo((s, is, Teom) I/T))
81 Neom (ii (com) )/T)
(6)
where T2 indicates a similarity threshold. If the similarity between sample pair e; and ej exceeds this threshold, we consider the j-th image æj (j E {f )) corresponding to the i-th image Z;. Conversely, if the similarity does not exceed the threshold, the image pair is considered as negative sample pair (j E (fneeg). }). Based on the redefined positive pairs, we modify Equation (7) to
where 85,00 indicates the softmax probability between the image features i; and each natural phrase feature Eoom, and Ti is the temperature parameter.
positive sample pair
3.3. Bidirectional Contrastive Learning
To leverage external features, we propose a bidirectional contrastive learning loss that maximizes mutual information between images and their corresponding textual features. Specifically, we apply random transformations to each im- age a, generating two augmented views ot and v/. which aligned with the external feature e,, Next, we project both the augmented views and the external text features into a Hamming space using two distinct hashing networks, He and He. producing I.-bit hash codes: h h," and h,. To facilitate the computation of the Hamming distance dis (hi, h2) between two hash codes, we adopt the inner product: dish (h1, h2) { (h1, l22)). This formu- lation ensures that the similarity measure, sim(h,, h2)
1 N:
Eje(hon.)
exp(
(A. M
Σ log Ls(a, b) = N
(9)
(13 .k
1k=1 exp(
4. Experiments
4.1. Experiment Setup
4.1.1. DATASETS
CIFAR-10 (Krizhevsky & Hinton, 2009) - -Datast con- tains 60,000 photographs divided into ten categories, with 6,000 images in each class. Following the proto- col of previous studies (Qiu et al., 2021; Wang et al., 2022; Qiu et al., 2024), we randomly pick 1,000 pho- tographs from each class as the query set, for a total of 10,000 images. We take the remaining images as the retrieval set and randomly choose 500 images per class as the training set from the retrieval set.
27. [ + h2), is positively correlated with the inner product, allowing it to quantify similarity effectively in the Hamming space.
After generating the hash codes, we employ a bidirectional contrastive learning loss (as illustrated in Figure 2) to align
5

---

Deep Unsupervised Hashing via External Guidance
CIFAR-10
NUS-WIDE 16 bits 32 bits 64 bits
Flickr25k
MSCOCO
Method
References
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
CNN
CIBHash (Qiu et al., 2021)
ICCA211
0.590
0.622
0.641
0.790
0.807
0.815
0.737
0.760 0.775
DATE (Luo et ul., 2021b)
ACMMM21
0.577
0.629 0.647
0.798
0.810
0.815
0.822
0.841
0.844
McCoQ (Wang al., 2022)
AAA122
0.629
0.641
0.651
0.802
0.822
0.832
0.813
0.817
0.827
DSCH (Lin al., 2022)
AAA122
0.624
0.644 0.670
0.762
0.780 0.786
0.817
0.827
0.828
NSH (Yu et al., 2022)
UCCA122
0.706
0.733
0.756
0.758
0.811 0.824
0.746
0.774
0.783
CGHash (Song al., 2023)
ACMMM23
0.795
0.803
0.817
0.814
0.833 0.841
0.768
0.782
0.791
DDCH (Wei al., 2023)
PR23
0.611
0.648
0.658
0.781
0.798
0.808
0.721
0.759
0.779
HAMAN (Ma et al., 2022)
TCSVT23
0.806
0.825
0.834
0.796
0.813
0.826
0.722
0.775
0.787
HiHPQ (Qiu et al., 2024)
AAA124
0.633
0.658
0.671
0.799
0.821 0.826
0,807
0.826
0.830
HARR (Ma al., 2024)
TOMM24
0.520
0.536
0.575
0.807
0.826
0.841
0.818
0.830 0.838
0.748
0.789
0.816
LGH (Zhao et al., 2024)
ICMR24
0.846
0.862
0.874
0.815
0.828
0.837
0.796
0.814
0.827
HHCH((Wei et al., 2024)
TIP24
0.631
0.657
0.681
0.797
0.820
0.828
0.825
0.838
0.842
0.775
0.798
0.810
VGG-16
Ours
0.887
0.888
0.885
0.852
0.854
0.856
0.856
0.858
0.856
0.833
0.840
0.842
VGG- 16
Ours
0.889
0.889
0.891
0.852
0.855
0.856
0.856
0.858
0.856
0.842
0.846
0.847
VIT
UDBH (Guo et al. 2023)
TCSVT23
0.775
0.779
0.783
0.829
0.840
0.849
0.845
0.850
0.860
DDCH (Wei et al., 2023)
PR23
0.882
0.923
0.935
0.816
0.866
0.875
FSCH (Cao et al., 2023)
TCSVT23
0.876
0.912
0.926
0.812
0.832
0.844
0.815
0.838
0.849
0.760
0.787
0.799
*HHCH (Wei et al., 2024)
TIP24
0.881
0.907
0.912
0.812
0.829
0.838
0.815
0.854
0.866
0.805
0.849
0.853
CLIP
Ours
0.931
0.934
0.933
0.849
0.855
0.855
0.874
0.887
0,892
0.854
0.878
0.887
bLiPbest-22
Ours
0.939
0.940
0.940
0.849
0.856
0.856
0.874
0.887
0.892
0.862
0.881
0.888
Table Mean Average Precision (MAP) comparison among different state-of-the-art deep unsupervised hashing methods on four benchmark datasets with hash code lengths varying from 16 to 64. Note: i) **' indicates that the source code for the paper is available. and we used the CLIP (ViT-B/16) model as its hackbone. ii) The "hest-T2" represents the optimal MAP results conresponding to different values of T2 (as illustrated in Figure 5b and indicated with star markers).
NUS-WIDE (Chua et al., 2009) Dataset includes 269.648 photos in 81 different categories. We concen- trate on the top 21 categories following (Zhao et al., 2024; Cao et al., 2023; Ma al., 2024), from which we randomly select 100 images per category create query set (2.100 photographs in total), and the remain- ing images are served as the retrieval set. Additionally, 10,500 randomly chosen images (500 images per class) from the retrieval set are used as the training set.
4. 1.2. IMPLEMENTATION DETAILS
In Section 3.3, we employ a VGG-16 network (Simonyan & Zisserman, 2015) pre-trained on ImageNet- K and a pre- trained CLIP (ViT-B/16) model (Radford et al., 2021) as the backbone of H.y respectively. The hashing layer of He consists of a two-layer multi-layer perceptron (MIP) with dimensions [F [F-512-L], where F represents the out- put dimension of the pre-trained backbones, while L de- notes the length of the generated hash codes, selected from (16, 32, 64). To be specific, F was set to 4096 for the pre-trained VGG-16 model, and to 512 for the pre-trained CLIP model. In addition, we adopted a non-cyclic cosine annealing strategy for the learning rate at each epoch t.
Flickr25k (Huiskes& & Lew, 2008) 25,000 multi-label photos with 24 classes are included in the dataset. After eliminating unlaheled data, following the protocol used in previous studies (Ma et al., 2022; Cao et al., 2023; Ma et al., 2024). 2,000 images are randomly chosen as the query set, while the remaining images are utilized as the retrieval set. In the retrieval set, 5,000 images are randomly chosen for training.
Ir Irmin + (Irmax - -rrin) X D'eepp
(10)
where Irmin indicates the minimum learning rate and Ir refers to the maximum learning rate. The lrimm is denoted as:
MSCOCO (Lin al., 2014)- 123,287 samples are contained in the dataset, and subset of 122,218 im- ages from 80 categories is used in our approach. In this subset, similar to (Cao et al., 2023), we randomly select 5,000 images as the query set, and use the remaining images for retrieval, with 10,000 of them further used for training.
[t/Twarm (1 (1+cos cos
t T Twarm:
Ir Ir'emp
(t-Teru)) Trax
(11)
t T Taam-
Tamm
Let Trax denotes the total number of epochs and Twam rep- resents the warm-up epoch. In our training procedure, we fixed ITmin = 1 x 10 Urmax = x 10 44 Tmax = 60, T Twam- = 10, N₂- 800, Ti = 0.9 and T2 0.97.
6

---

Deep Unsupervised Hashing via External Guidance
2022; Qiu et al., 2024), we adopt MAP@ 1000 for CIFAR-
10 dataset and MAP@ for NUS-WIDEE Flickr25k and MSCOCO datasets while evaluation. Table 1 illustrates the MAP results of our proposed method DUH-EG on four im- age retrieval datasets with hash code lengths varying from 16 to 64. It is clear that our method consistently achieves the best performance compared with the state-of-the- meth- ods on four benchmark datasets. To be specific, in the Con- volutional Neural Network (CNN) part, we adopt VGG- 16 model pre-trained on ImageNet as the backbone and compare its performance with other baseline models. Our DUH-EG achieves average improvements of 3.39%, 3.03%, 2.53% and 4.04% on CIFAR- 10, WIS-, Flickr25k and MSCOCO datasets compared to the most competitive methods LGH (Zhao et al., 2024). CGHash (Song al.. 2023) and DATE (Luo et al., 2021b), separately. In Vision Transformer (ViT) part, we adopt pre-trained CLIP (VIT- B/16) model as the backbone and evaluate its performance against other ViT-based methods. Our method achieves an average increase of 2.95%, 1.71%. 4.70% and 2.95% on CIFAR-10, NUS-WIDE, Flickr25k and MSCOCO datasets compared with the top-performing methods DDCH (Wei et al., 2023), UDBH (Guo et al., 2023) and HHCH (Wei et al., 2024), respectively. Moreover, the improvements are particularly evident in the case of a short hash code on the four datasets.
CHREL
MECOU
BIERO
BOIVGOLSI
(a) P-R curves on CIFAR-10
(b) P-R curves on NUS- WIDE
TENCII
HUKI
(c) P-R curves on Flick125k (d) P-R curves on MSCOCO
Figure 3. Results of Precision- Recall curves with 64-bit hash codes on four benchmark datasets.
C.08
TUHPF)
prodecon
TEKC
HETPO
T9004 UUP
LG2
Number of Retrieved Samples
Numbee Retneved Samples
(a) P@N on CIFAR-10
(b) P@N on NUS-WIDE
Clitrash
4.2.2. PRECISION- RECALL AND TOPN-
DETE FROOOII
2411 Fr00ann
To evaluate retrieval quality, we follow (Wei et al., 2024;
Ma et al., 2024; Cao et al., 2023; Duan et al., 2025) to ap- ply the following additional metries: i) Precision- Recaal (P-R) curves and ii) TopN-Precision (P@N) curves different number of retrieved samples. We report the P-R curves for the 64-bit hash codes in Figure 3. Obviously, DUH-EG outperforms all the compared methods by large margins on CIFAR-10 10 dataset. Additionally, on NUS-WIDE Flickr25k and MSCOCO datasets, DUH-EH achieves higher precision at lower recall rates. For P@N N curves illustrated in Figure 4, DUH-EG demonstrates significantly superior performance compared to other methods across four benchmark datasets. Notably, on the CIFAR-10 dataset, as the number of re- trieval samples increases, DUH-EG exhibits a much slower rate of precision degradation relative to the other methods. This highlights that by incorporating external guidance, our DUH-EG can generate high-quality hash codes.
Nomber Iherrseed Sumples
Namber Hatranved Semples
(c) N on Flickr25k
(d) P@N on MSCOCO
Figure 4, Results of JopN-Precision curves with bit hash codes on four benehmark datascts.
4.2. Comparisons with State-of-The-Art
4.2. MEAN AVERAGE PRECISION
In this section, we conduct comparison between DUH- EG and 14 state-of-the-art deep unsupervised hashing ap- proaches on the four image retrieval datasets with hash code lengths varying from 16 to 64. The approaches include CIBHash (Qiu et al., 2021), DATE (Luo et al., 2021b), MeCoQ (Wang et al., 2022), DSCH (Lin et al.,
2022), NSH (Yu et al., 2022), CGHash (Song al.,
4.3. Ablation Studies
2023), DDCH (Wei et al., 2023), HAMAN (Ma et al.,
2022), HiHPQ (Qiu et al., 2024), HARR (Ma et al.,
In this section, we conduct ablation studies to validate the indispensability of methods used in DUH-EG on all four benchmark datasets. Specifically, as shown in Table 2, we evaluate the effect of the SIM and AUG modules while us- ing external features introduced in Section 3.2. Here, the SIM module represents utilizing the mechanism illustrated
2024), LGH (Zhao et al., 2024), HHCH (Wei et al., 2024), UDBH (Guo et al., 2023) and FSCH (Cao et al., 2023). To assess the quality of the hash codes that are produced, we employed the standard statistic Mean Average Precision (MAP). In accordance with (Qiu et al., 2021; Ma et al.,
7

---

Deep Unsupervised Hashing via External Guidancee
Module
CIFAR-10
NUS-WIDE
Flickr25k
MSCOCO
SIM AUG
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
#1
0.922
0.926 0.925
0.827
0.833 0.835
0.845
0.860
0.871
0.812
0.849
0.864
#2
0.922 0.925 0.925
0.832
0.838
0.839
0.859
0.870
0.878
0.855
0.874
0.883
#3
<
0.939 0.940
0.940
0.848
0.854
0.855
0.864
0,880
0.886
0.831
0.862
0.874
#4
<
0.939 0.940
0.940
0.849
0.856 0.856
0.874
0.887
0.892
0.862
0.881
0.888
Table 2. Ablation studies the SIM and AUG modules with different bit lengths across four benchmark datasets.
External Knowledge
CIFAR-I0
NUS-WIDD
Flickr25k
MSCOCO
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
IMGAUG
0.922
0.923 0.929
0.828
0.839
0.844
0.839
0.856
0,866
0.733
0.785
0,819
IMGK
0.924
0.927
0.929
0.836
0.846
0.846
0.840
0.857
0.864
0.739
0.794
0.820
EK IMGCLS
0.935
0.936
0.937
0.845
0.850
0.851
0.844
0.852
0.858
0.858
0.877
0.883
0.934
0.937
0.936
0.838
0.848
0.849
0.842
0.859
0.867
0.748
0.798
0.823
EFC
0.939
0.940
0.940
0.849
0.856
0.856
0,874
0.887
0.892
0.862
0.881
0.888
Table 3. Ablation studies on the IMGAUG, IMGK, IMGCLS, EK and EFC methods with different bit lengths across four benchmark datasets.
in Equation (8) identify the potential positive pairs and negative pairs in bidirectional contrastive learning loss by the similarity threshold (T2). Moreover, the AUG module refers to incorporating two augmented views of images in bidirectional contrastive learning loss and maximizing mu- tual information between the discrete representation of these two views. Table 2 illustrates the MAP result of different configurations on four benchmark datasets. In particular, when considering module SIM and AUG individually, the comparison results of (#1 v.s. #2) demonstrate notable con- tribution of the SIM module to performance improvement. Especially on the MSCOCO dataset, the performance of the 16-bit hash codes increases with gain of 5.30%. For the AUG module, comparing the MAP result between #I and #3 clearly indicates that enabling the AUG module leads to significant enhancement in performance across all four datasets. In addition, configuration #4 demonstrates that the simultaneous introduction of both the SIM and AUG modules leads to the best performance.
representative semantic nouns from external knowledge, while EK does not apply any such filtering and instead gen- erates external features based all available nouns. For IMGK and IMGCLS, non-augmented image features are used as internal visual knowledge. In the case of IMGK, the method replaces the external knowledge used in EK with internal visual knowledge. Similarly, for IMGCLS, this ap- proach replaces the external knowledge used in EFC with in- ternal visual knowledge, and ensures that the ratio of clusters to total features is consistent between internal and external knowledge. As illustrated in Table 3, the performance of EK is significantly higher compared to IMGAUG, IMGK and IMGCLS. This improvement is particularly evident when evaluated on the MSCOCO dataset, where EK demonstrates an average gain of 11.37%. Additionally, EFC achieves op- timal performance across all four datasets. When compared to the EK method, the improvement of the EFC combi- nation is particularly notable, especially on the Flickr25k dataset. This not only verifies the effectiveness of introduc- ing external knowledge and filtering representative semantic nouns, but also demonstrates the advantages of using exter- nal knowledge compared to exhaustively mining internal visual information.
Furthermore, to validate the facilitating effect of the exter- nal feature construction method mentioned in Section 3.2, we compare the performance of five different methods: IM- GAUG, IMGK, IMGCLS, EK and EFC in Table 3. To be specific, IMGAUG performs contrastive learning using only augmented image features extracted by the CLIP model, without incorporating any external knowledge, which means IMGAUG only maximizes the agreement between two aug- mented views of the same image. contrast, EK and EFC incorporate external knowledge into contrastive learning, treating external textual features as additional guidance for the corresponding images. Notably, EFC follows the exter- nal feature construction method (in Section 3.2) to screen
4.4. Parameter Analysis
To investigate the impact of the distance threshold T, and the similarity threshold T2, on performance, we evaluate the MAP results of our proposed DUH-EG method under various values of Ti and T2, with hash codes length equal to 64. As shown in Figure 5, we denote the MAP performance on different datasets using various colors and markers. To be specific, we apply red circles to represent CIFAR-10, blue
8

---

Deep Unsupervised Hashing via External Guidance
0.98, 0.97, and 0.95, respectively. For consistency while comparing with other approaches (as shown in Table 1), we set 72 to common value of 0.97 across all datasets. This setting is indicated by the red vertical line in Figure 5b.
0.93
CIFAR-10
NUS WIDE
3.02
3.05
FLICKR25K
5. Conclusion
0.54
MSCOCO
In this paper, we present a novel deep unsupervised hash- ing method that integrates external textual information as semantic guidance to address the limitations of inherent vi- sual structures and enhance discrete representation learning. Unlike existing works, our method offers two key contribu- tions: i) To prevent excessive synonyms from dominating the semantic features, we propose a simple yet effective method for selecting representative semantic nouns, thereby ensuring diversity and discriminability of external textual features. ii) We present bidirectional contrastive learn- ing loss that maximizes the alignment between semantic positive pairs, rather than just between views of the same image, improving robustness and effectiveness. By incorpo- rating external textual information as guidance, we improve matching similarity among images, facilitating more accu- rate hash learning. We validate the effectiveness of our method through extensive experiments four benchmark datasets, demonstrating its superior performance compared to 14 state-of of-te-of-thh-arr approaches.
0.0
0.00
0.56
Distance threshold Ti
(a) Distance threshold 73 used Equation (3)
0.90
CITAR-10
0.85
NUS-WIDE
MAP
0.80
FLICKR25K
Fladd Value 0,9 while compartin othei mathods Table
MSCOCO
0.75
0.90
0.92
0.91
0.96
0.98
1.00
Similarity threshold T2
(b) Similarity threshold T2 used Equation (8)
Figure 5. Parameter analysis with 64-bit hash codes four bench- mark datasets.
A Acknowledgments
squares for NUS-WIDE, orange triangles for Flickr25k, and green inverted triangles for MSCOCO. Additionally, we use star markers to indicate the optimal MAP results corresponding to different values of parameters.
This work was supported in part by the National Natural Sci- ence Foundation of China (NSFC) under Grants 62472295, 62176171, and U24B20174; the Fundamental Research Funds for the Central Universities under Grants CJ202303 and CJ202403; the Sichuan Science and Technology Plan- ning Project under Grant 24NSFTD0130, the TCL Science and Technology Innovation Fund; the EDB Space Tech- nology Development Programme under Project S22-19012 STDP; and the SCU-LuZhou Science and Technology Co- operation Program under Grant 2023CDLZ-16.
To effectively filter the representative semantie nouns based on the extracted text features, apply distance threshold T3 to determine the point relatively farther from its cluster center, and treat them as representative features as defined in Equation (3). To further determine the best representative noun screening threshold, we investigate the impact of the distance threshold Tj in Figure 5a. The results show that the optimal similarity threshold is 0.9 on four benchmark datasets. To enhance the effectiveness of the InfoNCE loss, we apply a similarity threshold T2 to distinguish positive and negative sample pairs utilized in the InfoNCE loss (de- fined in Equation (8)). As illustrated in Figure 5b, we also investigate the impact of the similarity threshold T2. Since the cosine similarity between each pair of external features concentrated higher values, even slight variations in T2 can significantly influence the determination of false nega- tive pairs in contrastive learning. Therefore, we measured the optimal threshold value within a narrow range from 0.9 to 1.0 to ensure the precise selection of these pairs. It can be observed that the optimal values of T3 for the CIFAR- 10, NUS- WIDE, Flickr25k, and MSCOCO datasets are 0.99,
Impact Statement
This paper proposes a method that aims to advance hashing image retrieval by integrating external text knowledge from WordNet. This approach is evaluated on publicly available image datasets that do not pose security or privacy risks. However, the performance of our method is partially de- pendent on the distribution of both the training data and the external knowledge. If there exists a domain shift be- tween them, the model's effectiveness may be compromised. Therefore, to enhance its reliability in specific application domains like medicine and biology, it advisable to intro- duce external textual knowledge that closely aligns with the target domain.

---

Deep Unsupervised Hashing via External Guidance
References
Hu, Q., Wu, J.. Cheng, J., Wu, L., and Lu, H. Pseudo label based unsupervised deep discriminative hashing for image retrieval. In ACM International Conference Multimedia, pp. 1 1584-1590, 2017.
Cao, H., Huang, L., Nie, J., and Wei, Z. Unsupervised deep hashing with fine-gran ree-raadedd similaiityprrsserve con- trastive learning for image retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 34(5): 4095-4108, 2023.
Huang, C., Loy, C. C., and Tang, X. Unsupervised learning of discriminative attributes and visual representations. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5175-5184, 2016.
Cao, Y. Liu, B., Long, M., and Wang. J. Hashgan: Deep learning to hash with pair conditional wasserstein gan. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1287-1296, 2018.
Huiskes, M. J. and Lew, M. S. The mir flickr retrieval evaluation. In Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval, pp. 39 43, 2008.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A sim- ple framework for contrastive learning of visual represen- tations. In Proceedings of the International Conference on Machine Learning, pp. 1597-1607, 2020.
Kingma, D. P. and Welling, M. Auto- variational bayes. In In Proceedings of the 2nd International Confer- ence Learning Representations, 2014.
Chua, T.-S., Tang, J., Hong. R., Li, H., Luo, Z.,, and Zheng, Y. Nus-wide: a real-world web image database from na- tional university of singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval, pp. 1-9, 2009.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Li, Y., Hu, P., Peng, D., Lv, J.., Fan, J., and Peng, X. Image clustering with external guidance. In Forty- first Interna- tional Conference on Machine Learning, 2024.
Dai, B.. Guo, R., Kumar, S., He, N., and Song, L. Stochastic generative hashing. In Proceedings of the International Conference on Machine Learning, pp. 913-922, 2017.
Lin, Q., Chen, X., Zhang, Q., Cai, S., Zhao, W., and Wang, H. Deep unsupervised hashing with latent semantic com- ponents. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7488 7496, 2022.
Do, T.-T., Doan, A.-D., and Ngai-ManCheung. Learning to hash with binary deep neural network. In European Conference on Computer Vision, pp. 219-234, 2016.
Dong, X., Liu, L.., Zhu, L., Cheng, Z.., and Zhang, H. Un- supervised deep k-means hashing for efficient image re- trieval and clustering. IEEE Transactions on Circuits and Systems for Video Technology, (18)33266-321) 2020.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014.
Duan, S., Sun, Y., Peng, D., Liu, Z., Song, X., and Hu, P. Fuzzy multimodal learning for trusted cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Luo, X., Wu, D., Chen, C., Ma, J., and Deng, M. Deep unsupervised hashing by global and local consistency. In IEEE International Conference on Multimedia and Expo, pp. 1-6, 2021a.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Luo, X., Wu, D., Ma. Z., Chen, C., Deng, M., Huang, J., and Hua, X.SS. A statistical approach to mining semantic similarity for deep unsupervised hashing. In ACM In- ternational Conference on Multimedia, pp. 4306 4314, 2021b.
Gu, Y., Wang, S., Zhang, H., Yao, Y., Yang, W., and Liu, L. Clustering-driver unsupervised deep hashing for image retrieval. Neurocomputing, 368:1 14-123, 2019.
Luo, X., WU, D., Ma, Z.., Chen, C., Deng, M., Ma, J., Jin, Z.. Huang, J., and Hua, X.SS Cimon: Towards high- quality hash codes. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 902-908, 2021c.
Guo, J.-M., Prayuda, A. W. H.. Prasetyo, H., and Seshathiri, S. Deep learning-based image retrieval with unsupervised double bit hashing. IEEE Transactions Circuits and Systems for Video Technology, 33(1 1):7050-7065, 2023.
Ma, 7., Luo, X., Chen, Y., Hou, M., Li, Deng, M., and Lu, G. Improved deep unsupervised hashing with fine- -rainga semantic similarity mining for multi-label image retrieval. Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1254 1260, 2022.
He, C., Zhu, H., Hu, P., and Peng, X. Robust variational con- trastive learning for partially unallnndd clustering. In Proceedings of the 32nd ACM International Confer- ence on Multimedia, pp. 4167-41766 2024.
10

---

Deep Unsupervised Hashing via External Guidance
Ma, Z., Wang, S., Luo, X., Gu, Z., Chen, C.,, Li, J., Hua, X., and Lu, G. Harr: Learning discriminative and high- quality hash codes for image retrieval. ACM Transactions on Multimedia Computing, Communications and Appli- cations, 2033):1-23, 2024.
Song, Z., Su, Q., and Chen, J. Unsupervised hashing with contrastive learning by exploiting similarity knowledge and hidden structure of data. In ACM International Con- ference on Multimedia, pp. 6350-6335 2023.
Tu, R.-C,, Mao, X.-L., and Wei, W. MIs3rduh: Deep un- supervised hashing via manifold based local semantic similarity structure reconstructing. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 066263372222 2020.
Miller, G. A. Wordnet: lexical database for english. Com- munications of the ACM, 38(1 38111):39-41 1995.
Qiu, Z., Su, Q. Ou, Z., Yu, J., and Chen, C. Unsuper- vised hashing with contrastive information bottleneck. In Proceedings of the International Joint Conference on Artificial Intelligence. pp. 959-965, 2021.
Wang, J., Zeng. Z., Chen, B., Dai, T., and Xia, S. Contrastive quantization with code memory for unsupervised image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2468-2476, 2022.
Qiu, Z., Liu, J., Chen, Y., and King, I. Hihpq: Hierarchical hyperbolic product quantization for unsupervised image retrieval. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 461 4 4622, 2024.
Wang, S., Tan, Z.., Guo, R., and Li, J. Noise-robust fine- tuning of pretrained language models via external guid- ance. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In Proc. ICML, in Proceedings of Machine Learning Research, volume 139, pp. 8748-8763, 2021.
Wei, R., Liu, Y., Song, J., Xie, Y., and Zhou, K. Deep debiased contrastive hashing. Pattern Recognition, 139: 109483, 2023.
Wei, R., Liu, Y., Song, J., Xie, Y., and Zhou, K. Explor- ing hicrarchical information in hyperbolic space for self- supervised image hashing. IEEE Transactions on Image Processing, 13:1768-1788 2024.
Shen, F., Xu, Y., Liu, L., Yang, Y., Huang, Z., and Shen, H. Unsupervised deep hashing with amnilarity-adaplie and discrete optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12)333343 2018.
Wu, Z., Feng, M., Wang, Y., Xie, H., Dong, W., Miao, B., and Mian, A. External knowledge enhanced 3d scene generation from sketch. In European Conference on Computer Vision, pp. 286-304, 2024.
Shen, S., Li, C., Hu, X., Yang, J., Xie, Y., Zhang, P., Gan, Z., Wang, L., Yuan, L., Liu, C., Keutzer, K., Darrell, T., Rohrbach, A., and Gao, J. K-lite: Learning transferable visual models with external knowledge. In Advances in Neural Information Processing Systems, pp. 15558- 15573, 2022.
Yan, R., Xie, L,, Shu, X., Tang, J., and Zhang, L. Progressive instance-aware feature learning for compositional action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8): 10317-10330, 2023.
Yang, E., Deng, C., Liu, T., Liu, W., and Tao, D. Semantic structure-basee unsupervised deep hashing. In Proceed- ings of the International Joint Conference on Artificial Intelligence, pp. 1064-1070, 2018
Shen, Y., Liu, L., and Shao, L. Unsupervised binary rep- resentation learning with deep variational networks. In- ternational Journal of Computer Vision, 127111) 1628, 2019.
Yang, E., Liu, T., Deng, C., Liu, W., and Tao, D. Distill- hash: Unsupervised deep hashing by distilling data pairs. In IEEE Conference Computer Vision and Pattern Recognition, pp. 2946-22 -255,,295,.5 2019.
Shen, Y., Qin, J., Chen, J., Yu, M., Liu, L., Zhu, F., Shen, F., and Shao, L. Auto-encoding twin-bottleneck hashing. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2815-285555 2824, 2020.
Yang, Li, Q., Li, W., Liu, M., Li, X., and Liu, A. External knowledge dynamic modeling for image-text retrieval. In Proceedings of the 3/ ACM International Conference on Multimedia, pp. 5330-5338, 2023.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd Inter- national Conference Learning Representations, 2015.
Song. J., He, T., Gao, L., Xu, X., Hanjalic, A., and Shen, H. T. Binary generative adversarial networks for image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
Yu, J., Shen, Y., Wang, M., Zhang, H., and Torr, P. H. Learning to hash naturally sorts. In Proceedings of the International Joint Conference on Artificial Intelligence, pp.
2022.

---

Deep Unsupervised Hashing via External Guidance
Zhang, H., Liu, L.,, Long, Y., and Shao, L. Unsupervised deep hashing with pseudo labels for scalable image trieval. IEEE Transactions on Image Processing, 27(4): 2017.
Zhao, C., Ling, H., Lu, S., Shi, Y., Chen, and Li, Improve deep hashing with language guidance for supervised image retrieval. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pp. 137-145, 2024.
Zieba, M., Semberecki, P., El-Gaaly, T., and Trzcinski, T. Bingan: Learning compact binary descriptors with reg- ularized gan. In Advances in Neural Information Process- ing Systems, 2018.
12

---

Deep Unsupervised Hashing via External Guidance
A. Additional Experiment
phasis not only induces redundancy in the external features but also attenuates the weight associated with the primary discriminative term ("cat sleep"). This leads to an excessive allocation of weight to the shape feature when constructing the external features of the image by Equation (5), rather than to the object itself. Conversely, when the EFC method is employed, the Top5 matched nouns highlight specific objects relevant to the image and accurately capture the essence of the depicted scene. Therefore, by utilizing the EFC method, more precise external features can be con- structed.
A.I. Visualization
CIFAR
DUH-EG P@ 10
-10
HHCH P@ 10
Query
Image
FLICKR
DUH-EG P@ 10
25K
wothhuu EFC
Top5 with EFC
HHCH Pθ 10
Query
1
Round_shape
1
Cal sleep
Image
2
Ovaid
2
Toroid
NUS-
DUH-EG P@ 10
WIDE
3
Hair hall
3
Black catechu
HHCH P@ 10
4
Cal sleep
4
Raised doughnut
Query
Image
5
Back circle
5
Pouf
MS COCO
DUH-EG Pê
HHCH P@
Query Image
Figure 6. The Top10 retrieval results of our proposed DUH-EG and HHCH (Wei al., 2024) four benchmark datasets with 64 bits hash codes. The red box indicates the incorrect retrieval results, while the green box represents correct ones. Note that for multi-label datasets Flickr25k, NUS-WIDE and MSCOCO, correct retrieval result indicates that the retrieved image shares at least one label with the query image.
Figure 7. The Top5 matched external nouns (with the Top5 soft- max probability computed by Equation (6)) corresponding to an image from the Flickr25k dataset. The left panel presents the Top5 nouns without using the EFC method defined in Section 4.3, while the right panel displays the Top5 nouns obtained when the EFC method is applied. We highlight the noun that best matches the given image in blue.
We illustrate the Top10 retrieval results on CIFAR-10 10, NUS- WIDE, Flickr25k and MSCOCO datasets with 64 bits hash codes in Figure 6, respectively. On four benchmark datasets, our DUH-EG outperforms the HHCH hashing method (Wei et al., 2024), showing higher number of correct samples in the Top10 retrieval list. Moreover, we visualize the Top5 nouns matched by two different screening methods corre- sponding to the same image in Figure As defined in Section 4.3, we filter representative semantic nouns through the EFC method and match them with image data, as de- scribed in Section 3.2. From Figure 7, it can be observed that, in the absence of the EFC method, the Top5 matched nouns include "Round shape", "Ovoid", "Hair ball", "Cat sleep" and "Back circle". These terms primarily focus on shape-related attributes of the image content. Such an em-
A.2. Performance
Vocabulary
Word Num
Pre Pre-processing Time
ImageNet
21,843
99.53s
WordNet
117,797
476.79s
ConceptNet 134,364
531.12s
GloVe
317,756
1182. 1s
Table 4. The number of words and pre- processens time for different external knowledge sources.

---

Deep Unsupervised Hashing via External Guidance
Vocabulary Set
10
NUS-W NUS-WIDE
Flickr25k
MSCOCO
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
ImageNet WordNet
0.940 0.942
0.941
0.849
0.854 0.855
0.871
0.885
0.888
0.864
0.885
0.892
0.939 0.940
0.940
0.849
0.856 0.856
0.874
0.887
0.892
0.862
0.881
0.888
ConceptNet
0.938
0.940
0.940
0,850
0.853
0.855
0.875
0.885
0.890
0.865
0.886
0.893
Glo
0.932
0.936
0.935
0.846
0.852 0.852
0.870
0.882 0.887
0.842
0.863 0.869
Table 5. The MAP performance while using different external vocabulary sets.
Pre-trained Model
CIFAR-10
NUS-WIDE
Flickr25k
MSCOCO
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
16 bits 32 bits 64 bits
CLIP (ViT-B/16)
0.939 0.940
0.940
0.849
0.856 0.856
0.887 0.892 0.869
0.874
0.862 0.848
0.881 0.868 0.887
0.888
CLIP (ViT-B/32)
0.920 0.929 0.920
0.855
0.856 0.860
0.887 0.888
Table 6. The MAP performance while using different pro-trained models.
To evaluate the computational cost and scalability of our approach, conducted experiments using external vocabu- lary sets of varying sizes (as shown in Table 4), derived from four representative sources: (1) noun category labels from ImageNet; (2) nouns extracted from WordNet; (3) noun entries from the ConceptNet knowledge graph; and (4) the full vocabulary from GloVe embeddings. Table 4 reports the pre-processing time required by our method for each of the above sources. As our approach relies CLIP to extract textual features for all words in the vocabulary, the pre-processing time increases substantially with vocabulary size. This observation underscores the importance of ex- ploring more efficient textual feature extraction strategies, which we identify as promising direction for future work. is important to note that once the external knowledge is pre-processed, both training and inference stages proceed identically to baseline methods, incurring additional com- putational overhead.
CIFAR-IO and MSCOCO datasets varies significantly be-
tween the two variants, confirming the significant impact of the pre-trained model choice. These findings suggest that similar effects can be expected when adopting other multi- modal backbones, as the selection of the textual encoder directly influences the semantic richness of the extracted external text features and thus the retrieval effectiveness of the proposed framework.
Moreover, to assess scalability, we further evaluated model performance across the same four external vocabulary sources, with results presented in Table 5. Our External Feature Construction (EFC) module (see Section 3.2) ef- fectively reduces redundancy in external textual features, maintaining consistent performance across diverse external sources. These results demonstrate the robustness and scala- bility of our method in handling varying degrees of external knowledge integration.
Additionally, our method leverages Bidirectional Con- trastive Learning (in Section 3.3) align textual and visual representations of images, making the quality of external knowledge introduced via the textual modality key factor in the overall performance of the hashing network. To in- vestigate this, we trained our model using textual features from two different pre-trained CLIP models: VIT-B/16 and ViT-B/32. As shown Table 6, the MAP performance on
2

---

Deep Unsupervised Hashing via External Guidance
B. Optimization Procedure
Algorithm 2 The process of Section 3.3
Input: Training images X' { ) External features constructed in Algorithm E = fer):= Image hash function Hy, Text hash function He. Similarity threshold 72, Training epochs T. Output: Optimized hash function Hy.
To present the method flow introduced Section more clearly and systematically, we provide a detailed descrip- tion of the DUH-EG learning process in Algorithm and Algorithm 2. In Algorithm 1 we detail the construction of external features for each image, while Algorithm 2 illus- trates the Bidirectional Contrastive Learning process.
Initialize O and Og randomly:
for = 1 to N; do
for 1 to N; do Compute similarity: cos(es,
/x cos(-,)): A similarity function measuring the cosine similarity between vectors. */ if cos(e,, 0,) T2 then
Algorithm The process Section 3.2
Input: Training images X'
{ External nouns N (nw | Text encoder E,, Image encoder E, Distance threshold Zi.
treat the j-th sample forming negative pair with the i-th sample;
* E,, E;: Encoders from the pre-trained CLIP model.
else
treat the j-th sample as forming a positive pair with the i-th sample:
Output: Constructed external features & = {ei
Construct natural phrases: nhw (leath ) p=
for t = 1 to T do
/x where N₂ - in practice.
Generate two differently augmented views of images:
Extract textual features: ty E, (textt]);
Vi = {v! and V/2 = { 1==:
Generate noun features Ew by Equation (2);
Generate hash codes: h. = H. Ov), h
Perform clustering: K-means(f..) - k
H..u?, O.) and h = Hele, O.):
/* C₁ means the cluster center the k-th cluster.
12
Compute LAC,. with Equation (1);
Update O, and O with the Adam optimizer.
for = to N₁ do
/* Denote by as turk if to k-th cluster.
+/
Compute distance: dis(tw.k..
x* - dis(;,)): A function measuring
normalized distance between vectors.
if dis(ttu cx) > Ti then L tw E ( w=
else
L Ew E (Euer) Nnar. Jw=l:
10
Compute the softmax probability of tror corresponding to each cluster center Pwrk = S(Pmor, .c.) by Equation (4);
/x - S: The softmax function used to compute probability distributions
*/
12 Assign (t Neen; Purky k) sort(Pw.k
ki k((nor) arg max.x. Pw, Eoor) [1[1;
If S(Dnor, cx) + Pw, k), Enor Teen Jw */
Assign representative semantic subset: (Bee)) [ (Frep) Neep w=1 fcom Neam. w=l 14 Extract image features: i, = (x();;
Jusn U
15 Construct external features e; = Noom by Equation (5).
3


Reviews:
[{"review_id": "74YngeMxas", "reviewer": "Reviewer_2wPY", "summary": "This paper proposes a novel deep unsupervised hashing method, Deep Unsupervised Hashing with External Guidance (DUH-EG), to enhance image retrieval by incorporating external textual knowledge as semantic guidance. The method selects representative semantic nouns from an external textual database, aligns images with them to extract more discriminative external features, and employs a bidirectional contrastive learning mechanism to maximize agreement between hash codes in internal and external spaces. Experiments on CIFAR-10, NUS-WIDE, FLICKR25K, and MSCOCO demonstrate that DUH-EG significantly outperforms state-of-the-art unsupervised hashing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful feedback and recognition of our contributions. Below are our point-to-point responses:\n\n**Q1: How does the selection of external textual features affect the model's robustness across datasets?**\n\n**R1:** In our work, we focus on widely used benchmarks, such as CIFAR-10, NUS-WIDE, FLICKR25K, and MSCOCO, which primarily contain images of common objects and everyday scenes. To ensure generalization, we use a general vocabulary (e.g., WordNet) as our external knowledge source. Specifically, we extract textual features via CLIP and align them with visual features, promoting semantic consistency among similar images in the learned discrete representations.\n\nHowever, due to the presence of synonyms, textual features from the general vocabulary often introduce redundancy. To address this, we propose an External Feature Construction (EFC) method (Section 3.2) to cluster semantically similar nouns and select representative textual features, thereby reducing redundancy caused by excessive synonyms. This process enhances the semantic richness of the aggregated external textual features obtained via weighted summation (Eq. 5) and improves their representational capacity.\n\nAs demonstrated in our ablation experiments (Table 3), the EFC method improves model performance. Furthermore, as shown in Fig. 7 (Appendix), applying EFC reduces the number of high-weighted nouns with similar meanings, validating its effectiveness in redundancy reduction.\n\n---\n\n**Q2: Would using domain-specific external knowledge further enhance performance?**\n\n**R2:** Yes, using domain-specific external knowledge can further enhance performance. To evaluate this, we conducted an ablation study using six external knowledge sources (ImageNet, ConceptNet, WordNet_N, WordNet_V, GloVe, and Medical) on the MSCOCO dataset, which belongs to the natural image domain. \n\n- ImageNet represents 21,843 noun labels from the ImageNet dataset.  \n- ConceptNet and WordNet_N contain noun data.  \n- WordNet_V consists of verb data.  \n- GloVe includes words spanning multiple parts of speech.  \n- Medical contains vocabulary from the medical domain.  \n\nExperimental results in Table 1 show that models trained with general noun vocabulary (e.g., ImageNet, ConceptNet, WordNet_N) perform better, despite differences in vocabulary size. In contrast, models trained on verb-based (WordNet_V) or mixed-POS vocabularies (GloVe) exhibit lower performance, with medical vocabulary yielding the lowest performance.\n\nIn summary, for a target dataset (e.g., MSCOCO) in the natural image domain, using domain-relevant external knowledge (e.g., general noun vocabulary) effectively enhances model performance. Thus, matching external knowledge with the target dataset’s domain improves performance.\n\n**Table 1: The ablation studies on the influence of different external knowledge on specific dataset MSCOCO.**\n\n| **External Knowledge** | **Word Num** | **MSCOCO (16 bits, 32 bits, 64 bits)** |\n|------------------------|-------------|----------------------------------------|\n| **Medical**           | 35,500      | 0.807 / 0.835 / 0.846                 |\n| **GloVe**             | 317,756     | 0.842 / 0.863 / 0.869                 |\n| **WordNet_V**         | 11,531      | 0.825 / 0.850 / 0.863                 |\n| **WordNet_N**         | 117,797     | 0.862 / 0.881 / 0.888                 |\n| **ImageNet**          | 21,843      | 0.864 / 0.885 / 0.892                 |\n| **ConceptNet**        | 134,364     | 0.865 / 0.886 / 0.893                 |\n\n**Q3: Can this approach be extended to other modalities beyond images and text?**\n\n**R3:** Yes, our approach could be extended to other modalities, but certain preconditions apply. Specifically, our method aligns external textual features with visual representations via CLIP, making it well-suited for unsupervised image hashing. While extending to other modalities (e.g., audio using models like Wav2CLIP) is promising, its effectiveness depends on the availability of modality-specific external knowledge and corresponding pre-trained models.", "claims_and_evidence": "The paper claims that incorporating external textual guidance improves hash learning by overcoming the limitations of internal visual structures. This claim is supported by extensive experiments, where DUH-EG consistently outperforms existing methods across different datasets and hash code lengths.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem. The paper follows standard benchmarks and metrics, such as Mean Average Precision (MAP), Precision-Recall (P-R) curves, and Top-N Precision. The inclusion of multiple datasets and comparison with strong baselines provides a comprehensive evaluation.", "theoretical_claims": "No theoretical claims or proofs are presented; the work is empirically driven.", "experimental_designs_or_analyses": "The experimental design is rigorous, with a comprehensive evaluation across multiple datasets and comparison with state-of-the-art methods. The ablation studies provide insights into the contributions of different proposed components.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in deep unsupervised hashing and contrastive learning, while introducing external knowledge as a novel enhancement.", "essential_references_not_discussed": "No. The paper has covered a wide range of related works.", "other_strengths_and_weaknesses": "The strengths of the paper include the novelty of integrating external textual knowledge, strong experimental validation, and a well-designed bidirectional contrastive learning mechanism. The primary weakness is the lack of deeper analysis of how different external knowledge sources affect performance.", "other_comments_or_suggestions": "(1) Clarify the selection criteria for external textual features and how they generalize across datasets.\n(2) Provide additional ablation studies on the influence of different external knowledge sources.\n(3) Discuss potential limitations, such as dependency on pre-trained vision-language models like CLIP.", "questions_for_authors": "(1) How does the selection of external textual features affect the model's robustness across datasets?\n(2) Would using domain-specific external knowledge further enhance performance?\n(3) Can this approach be extended to other modalities beyond images and text?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wU0y51c8bO", "reviewer": "Reviewer_vrcG", "summary": "The paper identifies a crucial bottleneck in unsupervised image hashing, i.e., the limitation of insufficient knowledge guidance solely relying on the visual structures. To remedy this, semantic representatives are selected from external textual databases, serving as external guidance for the image modality via a bidirectional contrastive loss. Leveraging textual external knowledge, they designed a novel contrastive loss to avoid false negative pairs. Experiments on various datasets and model architectures have validated the consistent effectiveness of the method in mining and preserving semantic similarity for unsupervised hashing.\n\n## update after rebuttal\nThe authors' rebuttal adequately addressed these concerns. As a result, I keep my score as 4 (recommend acceptance).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and constructive suggestions. Below are our point-to-point responses:\n\n**Q1: More recent works could be included in the related work of deep unsupervised hashing.**\n\n**R1:** Thank you for your suggestion. We will update the related work section to incorporate more recent studies on deep unsupervised hashing, ensuring a more comprehensive and up-to-date literature review.\n\n---\n\n**Q2: The paper is clear in its language, but it’s better to check some mathematical operations with different names (e.g., 'sim' and 'cos' in Eq.(3), (4), (6), (8), and Lines 256-258) which may have the same meanings and cause ambiguity.**\n\n**R2:** We appreciate your careful examination of our mathematical operations. To eliminate potential ambiguity, we will revise the notation in Eq.(3), (4), (6), and (8) of the original paper, as well as in Lines 256–258, ensuring consistency in mathematical operations.\n\n---\n\n**Q3: The authors may limit the use of boldface abbreviations only for necessary cases.**\n\n**R3:** Thanks for your suggestion. We will refine the formatting to use boldface abbreviations only where necessary, improving readability and clarity.\n\n---\n\n**Q4: Why does the similarity threshold $T_1$ specifically affect the Flickr25k dataset when its value is high?**\n\n**R4:** The similarity threshold $T_1$ plays a crucial role in selecting representative textual features with lower similarity to the cluster center. As shown in Eq.(5) of the original paper, the final fused external feature e is computed from the external textual feature set $\\left(\\bar{\\{t}}_w^{com}\\right)$. Thus, reducing redundancy in $\\left(\\bar{\\{t}}_w^{com}\\right)$ is essential for ensuring well-matched external textual knowledge for the original images.\n\nIf $T_1$ is set too high, some representative textual features may be filtered out during the process described in Eq.(4), weakening the discriminative power of the final fused external feature **e**. Since the representational capacity of **e** is influenced by $\\left(\\bar{\\{t}}_w^{com}\\right)$, different datasets may respond differently to variations in $T_1$. In the case of Flickr25k, a higher $T_1$ can significantly impact model performance, likely due to the dataset’s inherent characteristics, making it more sensitive to the removal of certain textual features.\n\n---\n\n**Q5: Why does a reduction in $T_2$ have a more significant impact on the model compared to $T_1$?**\n\n**R5:** Unlike $ T_1$, which filters representative textual features, the primary role of the similarity threshold $T_2$ is to determine positive and negative sample pairs in Eq.(9). Fluctuations in $T_2$ directly affect the model’s ability to correctly distinguish between potential positive and negative pairs. \n\nEven a slight reduction in $T_2$ can cause some negative sample pairs to be misclassified as positive ones. This misclassification directly affects the discrimination of the learned hash codes, ultimately leading to a decline in overall model performance.", "claims_and_evidence": "Yes. Claims regarding proposed approaches are well-verified by extensive ablation results w.r.t. all 4 evaluated datasets and multiple hash code lengths.", "methods_and_evaluation_criteria": "The proposed method is clear and reasonable with unique consideration of external textual characteristics. The evaluation criteria are also reasonable as the evaluated datasets vary in scales and the adopted evaluation metrics are widely-used for the hashing task.", "theoretical_claims": "The paper does not contain proofs/theoretical claims.", "experimental_designs_or_analyses": "The paper’s method is verified comprehensively and analyzed thoroughly via extensive experiments to establish its validity.", "relation_to_broader_scientific_literature": "The work is a novel improvement in unsupervised semantic image hashing, while being potential to be applied in hashing retrieval of other modalities that can be associated with textual knowledge. The paper’s method is closely related to contrastive learning and clustering approaches.", "essential_references_not_discussed": "Key references are well-discussed.", "other_strengths_and_weaknesses": "Strengths:\n1. As summarized, the authors identify a knowledge bottleneck in unsupervised image hashing retrieval. The proposed method is well-motivated and novel by analyzing of the key challenges and potential problems from learning with external guidance.\n2. The experiments compare state-of-the-art methods with datasets of different scales and complexities, while presenting extensive analyses on both the learning approaches and the external feature construction.\n3. The paper effectively presents the proposed method with generally excellent clarity.\n\nWeaknesses:\n1. More recent works could be included in the related work of deep unsupervised hashing.", "other_comments_or_suggestions": "1. The paper is clear in its language but it’s better to check some mathematical operations with different names (e.g., 'sim' and 'cos' in Eq.(3),(4),(6),(8), and Line 256-258) which may have the same meanings and cause ambiguity.\n2. The authors may limit the use of boldface abbreviations only for necessary cases.", "questions_for_authors": "1. Why does the similarity threshold T_1 specifically affect the Flickr25k dataset when its values is high?\n2. Why does a reduction in T_2 have a more significant impact on the model compared to T_1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8HxrYuVu80", "reviewer": "Reviewer_w54n", "summary": "This paper proposes a deep unsupervised hashing framework (DUH-EG) designed for image retrieval. Unlike traditional unsupervised hashing methods that rely solely on intrinsic visual structures, DUH-EG leverages external textual guidance extracted from a lexical database (i.e., WordNet) and processed via a pre-trained CLIP model. The method comprises two components: (1) an external feature construction module that selects representative semantic nouns using clustering and filtering, and (2)) a bidirectional contrastive learning loss that aligns hash codes from two augmented views of an image with its corresponding external textual feature. Extensive experiments on four widely-used benchmarks (i.e., CIFAR-10, NUS-WIDE, FLICKR25K, and MSCOCO) show considerable improvements in MAP over state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and constructive suggestions. Below are our point-to-point responses:\n\n**Q1: A discussion on the computational costs and scalability issues.**\n\n**R1:** To analyze computational costs and scalability, we evaluated our approach using external vocabulary sets of varying sizes from four sources: 1) Noun category labels from ImageNet; 2) Nouns extracted from WordNet; 3) Nouns from the ConceptNet knowledge graph; and 4) The full vocabulary from GloVe.\n\nThe experimental results are shown in Table 1, presenting the pre-processing time of our method for each external knowledge source. Since our approach extracts textual features for all words using CLIP, the pre-processing time increases as the vocabulary size grows significantly. Therefore, exploring more efficient strategies for extracting external textual features is a promising direction for our future research. Notably, after external knowledge preprocessing, our training and inference are the same as the existing methods with no additional time added.\n\nRegarding scalability, we evaluated the model performance using four external vocabulary sources with different sizes, as shown in Table 2. Our External Feature Construction (EFC) method (in section 3.2 of the original paper) effectively mitigates redundancy in external textual features, ensuring robust performance even when different external knowledge sources are used. This highlights our method's adaptability and scalability.\n\n**Q2: Relying on external pre-trained models (e.g., CLIP) and WordNet might limit the method's applicability in scenarios where such resources are constrained.**\n\n**R2:** Our method leverages the pre-trained CLIP to incorporate textual features from WordNet as external knowledge for unsupervised hashing. However, we would like to clarify that our approach does not strictly rely on these specific resources. In resource-constrained scenarios, lightweight or domain-specific pre-trained multimodal models and textual knowledge bases could be used as alternatives.\n\n**Q3: Why was $T_2$ fixed at 0.97 for all datasets despite varying optimal values (Fig. 3b)?**\n\n**R3:** To maintain a consistent set of hyperparameters across all datasets, we fixed $T_2=0.97$ in comparison experiments, thereby reducing parameter tuning costs. If dataset-specific $T_2$ values were used, the performance would improve slightly, as indicated by the star-marked points in Fig. 3b.\n\n**Q4: What is the training/inference time of DUH-EG compared to baselines, especially with CLIP?**\n\n**R4:** We freeze the parameters of CLIP and train only the subsequent hashing layers. This makes our training process more efficient than updating all parameters of CLIP. For inference, our method consists of two steps: (1) feature extraction using CLIP and (2) hashing the features via the trained hashing layers. Moreover, the hashing layers consist of only two fully connected layers [$F$--$512$--$L$] with activation functions, which introduce negligible additional overhead (about 0.3M params). We provide a detailed overview of the computational costs associated with our method in Table 3, which demonstrates the efficiency of our method compared with CLIP.\n\n**Q5: Will code and pretrained models be released?**\n\n**R5:** Yes. We will release them after the acceptance of the paper. \n\n\n**Table 1: The number of words and pre-processing time for different external knowledge.**\n| External Knowledge | Word Num | Pre-processing Time |\n|------------------------|-------------|-------------------------|\n| ImageNet | 21,843 | 99.53s |\n| WordNet | 117,797 | 476.79s |\n| ConceptNet | 134,364 | 531.12s |\n| GloVe | 317,756 | 1182.11s |\n\n\n**Table 2: The MAP performance using different external knowledge.**\n| External Knowledge | CIFAR-10 (16, 32, 64 bits) | NUS-WIDE (16, 32, 64 bits)| FLICKR25K (16, 32, 64 bits) | MSCOCO (16, 32, 64 bits) |\n|------------------------|--------------------------------|--------------------------------|--------------------------------|------------------------------|\n| ImageNet | 0.940 / 0.942 / 0.941 | 0.849 / 0.854 / 0.855 | 0.871 / 0.885 / 0.888 | 0.864 / 0.885 / 0.892 |\n| WordNet | 0.939 / 0.940 / 0.940 | 0.849 / 0.856 / 0.856 | 0.874 / 0.887 / 0.892 | 0.862 / 0.881 / 0.888 |\n| ConceptNet | 0.938 / 0.940 / 0.940 | 0.850 / 0.853 / 0.855 | 0.875 / 0.885 / 0.890 | 0.865 / 0.886 / 0.893 |\n| GloVe | 0.932 / 0.936 / 0.935 | 0.846 / 0.852 / 0.852 | 0.870 / 0.882 / 0.887 | 0.842 / 0.863 / 0.869 |\n\n**Table 3: Training/inference time, FLOPs, and params of DUH-EG and CLIP on an NVIDIA 3090 GPU.**\n| Model | Training Time / epoch | Inference Time / sample| GFLOPs| Params|\n|------------|----------------------------------------------------|----------------------------|------------------|-------------|\n| DUH-EG | 105.27s (without pre-processing of external knowledge) | 4.1227s | 11.2703 | 57.56M |\n| CLIP | The CLIP was not trained on the 3090 GPU | 4.1223s | 11.2700 | 57.26M |", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "1. This paper introduces a two-phase approach. In the first phase, textual features for nouns are fused and clustered to select a diverse set of representative semantic features. In the second phase, these external features are aligned with internal image representations via a specially designed bidirectional contrastive loss. This loss not only maximizes mutual information between different augmented views but also redefines positive pairs by using external guidance to mitigate false negatives.\n2. The evaluation is primarily based on MAP metrics across four image retrieval datasets. In addition, the authors provide ablation studies, sensitivity analyses for the key thresholds (T1 and T2), and supplementary metrics (P-R and P@N curves) to reinforce the empirical performance.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "1. The authors evaluate DUH-EG on four standard benchmarks, comparing against 14 state-of-the-art unsupervised hashing methods. The comprehensive comparison and the use of diverse backbones (i.e., VGG-16 and CLIP) strengthen the empirical study.\n2. Detailed ablation experiments dissect the contributions of individual modules (i.e., SIM and AUG) and various configurations (e.g., using internal visual knowledge vs. external textual knowledge). The parameter analysis on thresholds T1 and T2 further validates the robustness of the proposed method.\n3. The experimental analysis is thorough. However, a discussion regarding the computational overhead introduced by the external feature extraction and the sensitivity to pre-trained models (like CLIP) would provide additional clarity.", "relation_to_broader_scientific_literature": "The paper positions its contributions well within the literature on unsupervised hashing, contrastive learning, and multimodal representation learning. It advances the field by combining cross-modal external guidance with contrastive learning, addressing a known limitation of internal visual structures.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of using external textual guidance to overcome intrinsic limitations of visual data is novel and interesting.\n2. Extensive experiments with competitive baselines across multiple datasets demonstrate the efficacy of the proposed method.\n3. The detailed ablation studies and parameter analyses provide a clear breakdown of the contributions of each component.\n4. The method and experiments are described with sufficient clarity, making the approach understandable.\n\nWeaknesses:\n1. A discussion on the computational costs and scalability issues associated with integrating external guidance would be beneficial.\n2. Relying on external pre-trained models (e.g., CLIP) and WordNet might limit the method's applicability in scenarios where such resources are constrained.", "other_comments_or_suggestions": "Typo: Page 6, \"token the remaining images\" -> \"took the remaining images\".", "questions_for_authors": "1. Why was T2 fixed at 0.97 for all datasets despite varying optimal values (Fig. 3b)? What about performance if dataset-specific T2 is used?\n2. What is the training/inference time of DUH-EG compared to baselines, especially with CLIP?\n3. Will code and pretrained models be released?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Js51glwExu", "reviewer": "Reviewer_m7gz", "summary": "This paper introduces DUH-EG, a deep unsupervised hashing framework that integrates external textual information as semantic guidance to overcome the limitations of relying solely on internal visual structures. The method first constructs external features by extracting and clustering textual features (derived from WordNet nouns and a pre-trained CLIP model) to select representative semantic nouns. It then employs a bidirectional contrastive learning loss to align hash codes generated from two augmented views of an image and its corresponding external textual view. Extensive experiments on standard image retrieval benchmarks (CIFAR-10, NUS-WIDE, FLICKR25K, and MSCOCO) demonstrate that DUH-EG outperforms several state-of-the-art unsupervised hashing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and suggestions. To enhance readability, we will refine the description of the bidirectional contrastive loss in the revised manuscript accordingly.\n\n**Q1: How does DUH-EG scale when the external textual database is significantly larger, and what are the computational implications?**\n\n**R1:** Our method enhances scalability by selecting representative textual features and aggregating them into a single external feature **e** using Eq.(5) of the original paper. This ensures scalability when handling a significantly larger textual database.\n\nRegarding computational cost and scalability, Table 1 and Table 2 analyze computational cost, showing that while pre-processing time increases with larger external noun databases, MAP performance remains stable, demonstrating scalability. Additionally, the MAP performance remains largely unchanged, indicating that our method exhibits good scalability. However, in the inference phase, since only the image hashing network is used to generate discriminative hash codes, the size of the textual database has no impact on inference efficiency.\n\n---\n\n**Q2: Have you evaluated the impact of noisy or irrelevant external textual features on the hash code quality?**\n\n**R2:** Yes. We highlighted the positive impact of filtering redundant external textual features using the External Feature Construction (EFC) method (Section 3.2) in the supplementary material. As shown in Fig. 7, after filtering, the feature \"Cat_sleep\" had the highest weight. Without EFC, irrelevant features like \"Round_shape,\" \"Ovoid,\" and \"Hair_ball\" had higher weights than \"Cat_sleep.\" The ablation study in Table 3 confirms that irrelevant features degrade the quality of the learned hash codes.\n\n**Q3: Can you provide further justification for the selection of the similarity thresholds $T_1$ and $T_2$ across different datasets? Are these hyperparameters sensitive to dataset characteristics?**\n\n**R3:** \n1. To ensure consistency across datasets, we fixed $T_1$ at 0.90 and $T_2$ at 0.97 when comparing with other methods in Table 1 of the original paper.\n2. No, they are not sensitive to datasets, so we use the same parameters in all datasets.\n\n**Q4: How does the choice of pre-trained models (e.g., using alternatives to CLIP) affect performance? Would similar improvements be expected with other multi-modal models?**\n\n**R4:** \n1. Since our method uses Bidirectional Contrastive Learning to align an image's textual and visual representations, the quality of external knowledge introduced through the textual modality affects the hashing network’s performance. To validate this, we trained our model with textual features from CLIP (ViT-B/16) and CLIP (ViT-B/32). As shown in Table 3, the MAP performance differs significantly on CIFAR-10 and MSCOCO, confirming the impact of the pre-trained model choice.\n2. Yes, similar improvements would be expected with other multi-modal models, as evidenced by the results of CLIP (ViT-B/16) and CLIP (ViT-B/32).\n\n**Table 1: The number of words and pre-processing time for different external knowledge.**\n\n| External Knowledge | Word Num | Pre-processing Time |\n|--------------------|----------|---------------------|\n| ImageNet           | 21,843   | 99.53s              |\n| WordNet            | 117,797  | 476.79s             |\n| ConceptNet         | 134,364  | 531.12s             |\n| GloVe              | 317,756  | 1182.11s            |\n\n**Table 2: The MAP performance while using different external knowledge.**\n\n| External Knowledge | CIFAR-10 (16, 32, 64 bits) | NUS-WIDE (16, 32, 64 bits) | FLICKR25K (16, 32, 64 bits) | MSCOCO (16, 32, 64 bits) |\n|--------------------|---------------------------|----------------------------|----------------------------|--------------------------|\n| ImageNet           | 0.940 / 0.942 / 0.941     | 0.849 / 0.854 / 0.855      | 0.871 / 0.885 / 0.888      | 0.864 / 0.885 / 0.892    |\n| WordNet            | 0.939 / 0.940 / 0.940     | 0.849 / 0.856 / 0.856      | 0.874 / 0.887 / 0.892      | 0.862 / 0.881 / 0.888    |\n| ConceptNet         | 0.938 / 0.940 / 0.940     | 0.850 / 0.853 / 0.855      | 0.875 / 0.885 / 0.890      | 0.865 / 0.886 / 0.893    |\n| GloVe              | 0.932 / 0.936 / 0.935     | 0.846 / 0.852 / 0.852      | 0.870 / 0.882 / 0.887      | 0.842 / 0.863 / 0.869    |\n\n**Table 3: The MAP performance while using different pre-trained models.**\n\n| Pre-trained Model   | CIFAR-10 (16, 32, 64 bits) | NUS-WIDE (16, 32, 64 bits) | FLICKR25K (16, 32, 64 bits) | MSCOCO (16, 32, 64 bits) |\n|---------------------|---------------------------|----------------------------|----------------------------|--------------------------|\n| CLIP (ViT-B/16)     | 0.939 / 0.940 / 0.940     | 0.849 / 0.856 / 0.856      | 0.874 / 0.887 / 0.892      | 0.862 / 0.881 / 0.888    |\n| CLIP (ViT-B/32)     | 0.920 / 0.929 / 0.920     | 0.855 / 0.856 / 0.860      | 0.869 / 0.887 / 0.888      | 0.848 / 0.868 / 0.887    |", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence. The paper claims that incorporating external textual guidance can enrich semantic supervision and lead to more discriminative hash codes. These claims are supported by the comprehensive experimental results, such as comparison experiments, ablation studies, and parameter analysis.", "methods_and_evaluation_criteria": "Yes, both the proposed methods and evaluation criteria make sense for the problem at hand. Specifically, the proposed method is well motivated and clearly described. Leveraging a pre-trained CLIP model to extract and fuse textual features, followed by a clustering-based selection mechanism, is a creative strategy to overcome synonym redundancy. The use of a bidirectional contrastive loss—extending traditional contrastive learning to align not only different augmented views but also external textual views—is appropriate for enhancing semantic consistency. Evaluation metrics such as MAP, Precision-Recall curves, and TopN-Precision are standard and suitable for assessing image retrieval performance.", "theoretical_claims": "There is no any proofs for theoretical claims. The paper primarily focuses on algorithmic and empirical contributions rather than deep theoretical proofs. The derivation of the bidirectional contrastive loss and the rationale for the external feature construction are presented clearly. The formulation is standard within the contrastive learning literature and appears correct.", "experimental_designs_or_analyses": "Yes, I checked the soundness of the experimental designs and analyses. The experimental design is robust:\n1) The authors evaluate on multiple benchmark datasets.\n2) Comparisons are made against 14 state-of-the-art unsupervised hashing methods.\n3) Ablation studies and parameter sensitivity analyses provide insights into the contribution of each component.", "relation_to_broader_scientific_literature": "The paper is well situated within the broader context of unsupervised hashing and contrastive learning. It builds on recent advances in using internal data structures for hash code learning while making a novel contribution by integrating external textual information. The work is compared with and references several key studies in both unsupervised hashing and multi-modal representation learning, clearly delineating its contributions relative to prior art.", "essential_references_not_discussed": "No. This paper have cited the essential related works.", "other_strengths_and_weaknesses": "Strengths:\n1) Novel integration of external textual guidance with unsupervised hashing.\n2) Comprehensive experimental validation across multiple benchmark datasets.\n3) Clear ablation studies and parameter analyses that illustrate the effectiveness of individual components.\n\nWeaknesses:\n1) Limited discussion on the scalability and computational cost of extracting and processing external textual features.\n2) Lack of deeper analysis on why external guidance particularly improves the discrimination of hash codes.\n3) Some methodological details (e.g., sensitivity to the choice of pre-trained models) could be elaborated further.", "other_comments_or_suggestions": "1) A more detailed analysis of computational overhead would be valuable, especially when scaling to larger external noun databases.\n2) Consider discussing potential impacts of noisy or less relevant textual features.\n3) Minor editorial improvements and clarifications in the description of the bidirectional contrastive loss could help improve readability.", "questions_for_authors": "1) How does DUH-EG scale when the external textual database is significantly larger, and what are the computational implications?\n2) Have you evaluated the impact of noisy or irrelevant external textual features on the hash code quality?\n3) Can you provide further justification for the selection of the similarity thresholds (T1 and T2) across different datasets? Are these hyperparameters sensitive to dataset characteristics?\n4) How does the choice of pre-trained models (e.g., using alternatives to CLIP) affect performance? Would similar improvements be expected with other multi-modal models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a deep unsupervised hashing method, named DUH-EG, which improves image retrieval by incorporating external textual knowledge. It selects key semantic nouns from an external source to guide feature learning and uses bidirectional contrastive learning to align internal and external representations. This dual approach boosts discriminative power and outperforms existing methods on benchmark datasets, which can achieve SOTA retrieval performance. Four reviewers have carefully checked the motivations, technical details, method analysis, and experimental validations. All of our reviewers agree to accept this paper. After the post-rebuttal discussion, the authors’ point-to-point rebuttal addressed most of these concerns so that no significant issues remain. This paper is acceptable in its current form.
Decision: Accept (poster)