Hierarchical Encoding Tree with Modality Mixup for Cross-modal Hashing
Authors: Zhiping Xiao
Junyu Luo
Hang Zhou
Yusheng Zhao
Xiao Luo
Pengyun Wang
Wei Ju
Siyu Heng
Ming Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Cross-modal Hashing
Unsupervised Hash Retrieval
Cross-modal Retireval
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Cross-modal retrieval is a fundamental task that aims to learn semantic correspondences across different data modalities, such as visual and textual modalities. Unsupervised hashing methods can efficiently manage large-scale data and can be effectively applied to cross-modal retrieval studies.However, existing methods typically fail to fully exploit the hierarchical semantic structure within text and image data, where instances naturally organize into multi-level communities of varying granularity. Moreover, the commonly-used direct modal alignment cannot effectively bridge the semantic gap between these two modalities. To address these issues, we introduce a novel Hierarchical Encoding Tree with Modality Mixup (HINT) method, which achieves effective cross-modal retrieval by extracting hierarchical cross-modal relations. HINT constructs a cross-modal encoding tree guided by hierarchical structural entropy and generates proxy samples of text and image modalities for each instance from the encoding tree. Through the curriculum-based mixup of proxy samples, HINT achieves progressive modal alignment and effective cross-modal retrieval. We also conduct cross-modal consistency learning to achieve global-view semantic alignment between text and image representations. Extensive experiments on a range of cross-modal retrieval datasets demonstrate the superiority of HINT over state-of-the-art methods.

Published as a conference paper at ICLR 2026
HIERARCHICAL ENCODING TREE WITH MODALITY MIXUP FOR CROSS-MODAL HASHING
Zhiping Xiao ,unyu Luo Hang Zhou², Yusheng Zhao?, Xiao Luo Pengyun Wang", Wei Ju', Siyu Heng", Ming Zhang*!
1 Paul G. Allen School of Computer Science and Engineering, University of Washington
2 State Key Laboratory for Multimedia Information Processing,
School of Computer Science, PKU-Anker LLM Lab, Peking University
3 Department of Statistics and Operations Research & School of Data Science and Society, University of North Carolina at Chapel Hill
Department of Statisties, University of Ausconsin-Madisee
5 Data Science Institute, University of Chicago
6 Department of Biostatistics, New York University
patxiaoluw edu, luo. junyu@uut com, edu.cc
xiao. luo@wi sc. edu,
ABSTRACT
Cross-modal retrieval is a fundamental task that aims to learn semantic corre- spondences across different data modalities, such visual and textual modalities. Unsupervised hashing methods can efficiently manage large-scale data and can be effectively applied cross-modal retrieval studies. However, existing meth- ods typically fail to fully exploit the hierarchical semantic structure within text and image data, where instances naturally organize into multi-level communities of varying granularity. Moreover, the commonly-usen direct modal alignment cannot effectively bridge the semantic gap between these two modalities. To ad- dress these issues, we introduce novel Hierarchical Encoding Tree with Modal- ity Mixup (HINT) method, which achieves effective cross-modas retrieval by ex- tracting hierarchical cross-modal relations. HINT constructs a cross- moss-modal en- coding tree guided by hierarchical structural entropy and generates proxy samples of text and image modalities for each instance from the encoding tree. Through the curriculum-baie based mixup of proxy samples, HINT achieves progressive modal alignment and effective cross-modal retrieval. We also conduct cross-modal con- sistency learning to achieve global-view semantic alignment between text and im- age representations. Extensive experiments on range of cross- modall retrieval datasets demonstrate the superiority of HINT over state-of-the- methods.
INTRODUCTION
Cross-modal retrieval aims to measure the semantic similarity between different modalities, using retrieval methods such as approximate nearest neighbors (ANNs) search (Zhu et al., 2023; Zhen al., 2019). Cross-modal retrieval has significant application value, such as retrieval-augmentte generation (RAG) (Li et al., 2024b; Cui et al., 2024) and search engines (Song et al., 2024; Chen et al., 2017). With the rapid development of large-scale vision- languaee datasets, cross-modal re- trieval has attracted increasing attention. Therefore, researchers have turned to hashing-based cross- modal retrieval (Cao et al., 2017: Yan et al., 2020), which achieves efficient storage and indexing by substituting bit-wise operations for computationally prohibitive pairwise distance comparisons. Hashing- cross-modal retrieval methods map high-dimensiooal semantic vectors from differ- modalities into unified Hamming space (binary space) (Luo al., 2023; Huang et al., 2024), enabling similarity comparison and ANNs.
1 Correspondiig authors.

---

Published as a conference paper at ICLR 2026
Cross-modal modal hashing has garnered significant attention from the community (Zhang et al., 2024b; Sun et al., 2024; Luo et al., 2025). It includes supervised and unsupervised methods. Supervised cross-modal hashing (Shen et al., 2024; Ma et al., 2024; Liu et al., 2019a; Lu et al., 2019) learn hash codes using labeled data. However, due to the expensiveness and scarcity of cross-modal annotations in real-world scenarios (Wang et al., 2023), researchers have shifted their focus to unsupervised methods that do not rely on annotations. Unsupervised cross-modal hashing (Liang et al., 2024; Li al., 2024a; Zhang et al., 2018) leverage pair-wise cross-modal data, exploiting the similarity between samples and employing mechanisms such as contrastive learning (Hu et al., 2022) and adversarial learning (Li et al., 2019) to guide hash learning.
Unsupervised mooss-modal hashing has made promising progress (Liang et al., 2024; Li et al., 2024a; Zhang et al., 2024b; Tu et al., 2023), but still suffers from the following issues: The first challenge is the lack of hierarchical semantic structure. Without semantic annotations (e.g.. category labels), previous unsupervised methods can only leverage paired data (Hu et al., 2022) (e.g., image-text pairs) as supervision, which provides flat (i.e., no hierarchical structure, all data points are on the same level) and sparse signals. However, real-world text and image data naturally exhibit a hi- erarchical semantic structure, containing numerous local communities. The instances within each community have similar semantics, while the semantic differences across communities are substan- leads to insufficient exploration of community
tial. The absence of wtwrrciiaal-iifomationn-miiiiiii relationships, hindering the learning of generalizable hash codes. Furthermore, this challenge is compounded by the ineffective alignment of heterogeneous modalities. Existing methods (Hu et al.. 2022; Zhang et al., 2024b; Tu et al., 2023) employ different encoders to project data from various modalities and optimize towards a common objective. Nevertheless, due the inherent het- erogeneity across modalities (e.g., manifested in structure and semantics), direct alignment can pose a high learning difficulty and lead to suboptimal performance. Therefore, it necessary to conduct hierarchical cross-modal learning and in a progressive manner.
To address the aforementioned is- sues, we propose a novel approach called Hierarchical Coding Tree with Modality Mixup (HINT) for hashing-basse based cross-modal retrieval. The core idea of HINT lies in con- structing a cross- moosl encoding tree that recovers hierarchical semantic structures and mines local semantic communities. Specifically. guided by the hicrarchical structural entropy (Li et al., 2018: Zou et al., 2023), we con-
Sparse Supervision
Hierarchical Encoding Tree
Harzratical Struciural Eneroay
Image
Text
Local Communines
Figure 1: HINT transforms sparse cross-modal supervision (left) into a meaningful hierarchical encoding tree (right), which reveals local semantic communities for robust cross- modal alignment.
struct the encoding tree from the enhanced cross modal relationship graph. The encoding tree has dense connections within local communities and sparse connections between communities. By uti- lizing the cross-modal encoding tree, we mitigate the performance degradation associated with flat and sparse cross-modal connections. Next, we synthesize proxy samples in different modalities for each sample based on the encoding tree. Through curriculum-based mixup on these proxy samples, we achieve progressive modality alignment, circumventing the challenging task of directly aligning heterogeneous modalities. We also achieve semantic alignment from a global perspective by opti- mizing the consistency of the semantic distributions of the proxy samples in different modalities. Extensive experiments on benchmark datasets demonstrate the superior performance of HINT.
The main contributions of this paper are: Hierarchical Modeling Approach. We connect the en- coding tree with cross-modal hashing problems. Specifically, we construct cross-modal encoding tree to explore the cross modal relationships and uncover local semantic communities hierarchically. @ Novel Integrated Framework. Based on the hierarchical encoding tree, we extract cross-modke proxy samples. Leveraging the proxy samples, we design a curriculum-bast anism for effective cross-moda hash learning. These components synergistically form an end-to-end framework where each enables and enhances the others. On the other hand, we achieve global-view consistency learning through the distribution alignment. @ Competitive Performance. Comprehen- sive experiments demonstrate that HINT achieves state-of-the-art performance.
modality mixup mech-
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Cross-modal Retrieval a fundamental task for bridging data of different modalities (Lee et al., 2024; Chen et al., 2024; Li et al., 2023; Krojer al., 2022; Radford al., 2021; Ge al., 2024). Due to the diverse distributions and structures of texts and images, it is necessary to map them effectively into a unified representation space to calculate the semantic similarity between samples (Ding al., 2016b; Liu et al., 2019b). With this unified representation, we can employ the approximate nearest neighbors (ANNs) (Zhu al., 2023; Zhen et al., 2019) methods for similarity search. Recently, researchers turn to cross-modal hashing methods to enhance efficiency for both storage costs and computational costly retrieval processes (Xu al., 2017; Jiang Li, 2019; Wang al., 2024b).
Unsupervised Cross-modal Hashing (Liang et al., 2024; Li et al., 2024a; Zhang et al., 2024b; Wang et al., 2024b) utilizes data correlation information to project cross-modal data into a unified binary space (Huang et al., 2024; Wu et al., 2018). Due to the expensive and difficult acquisition of labeled cross-modal data, supervised methods sometimes face challenges in the real-world (Hu et al., 2022). Therefore, unsupervised cross-modal hashing has attracted widespread attention (Zhou et al., 2014; Gao et al., 2023; Mikriukov et al., 2022). Researchers also use adversarial networks (Li et al., 2019) and contrastive learning (Hu et al., 2022) handle cross-modal hash learning. These methods generally rely on sparse text-image relationships, lacking local community mining. We explore the hierarchical cross-modal relationship and learn more generalizable hash representations through modality mixup and cross-modal consistency learning.
Cross-modal Relationship Modeling an essential topic in multi-modal research (Oh et al., 2023; Wang et al., 2024a; Li et al., 2024c; Liang et al., 2022; Huang et al., 2021). Some methods employ similarity modeling within modalities (Zhang et al., 2018), such constructing graph structures for images and texts separately using the Wasserstein metric. The tree-based methods (Ge et al., 2021; Chen et al., 2022) are also introduced for cross- modal relationship modeling. Recent hierarchical methods (Jin et al., 2021; Zhang et al., 2025) construct predefined multi-level structures using exter- nal tools or preset granularities for supervised retrieval tasks. Other works (Sun et al., 2023) adopt architectural hierarchies through progressive projection networks for hashing.
PRELIMINARY
In this work, we consider cross-modal hash retrieval problems. The objective is to map samples from both modalities into the shared Hamming space, enabling efficient cross- moosl retrieval. Specifi- cally, let the visual vector space be D [f." T==1 and the text vector space be D' {f Ti=1
which is encoded by standard pre-trained visual and text encoders. We have N image-text pairs without label information. We employ neural networks ()) and d'(-) to map each visual and text feature vector f and f into the Hamming space as:
6y = sign (Ø" (f:'), bj sign (0' (f!))
(1)
where by and by are L-length hash codes, i.e., bj (-1,+114 E {u, (). and sign(-) is the sign function. The hash codes could be used for subsequent efficient retrieval. Therefore, we need to minimize the Hamming distances between semantically similar samples across modalities while maximizing the distances between dissimilar ones. The Hamming distance is calculated as d(b;, bj) }( (b; bj)). where L is the code length and (..)) denotes the inner product.
METHODOLOGY
4.1 FRAMEWORK OVERVIEW
Sparse cross-modal connections in unsupervised scenarios pose challenges for modality alignment and cross-modal retrieval. The core idea of HINT is to establish a cross-modal encoding tree to re- cover the hierarchical structure across modalities and enhance the connections among local semantic clusters. Specifically, as illustrated in Figure 2, our HINT comprises three main components: 1 Hi- erarchical encoding tree construction. Guided by the hierarchical structural entropy. optimize the enhanced cross cross-modal relationship graph to obtain the encoding tree. @ Cross-modas hash learn- ing with modality mixup. To minimize the heterogeneous gap between two modalities, HINT builds
3

---

Published as a conference paper at ICLR 2026
Visual Feakure
Modal Prezies
Learning
Visual Encoder
Coding Tree
Visual
Progressive
Hash Codes
Modality Mixup
hhaak
Condense
Vasal Mocaliy
Text
Origin
Proxies
the loptop of owner.
Text Encader
Crom
Modal Proxies
Visual
Alignment
Text Modality
Feature
Consistency Learning.
Figure 2: Overview of HINT, which constructs hierarchical encoding tree from sparse cross-modal relationships. It then synthesizes modality proxies and performs progressive modality mixup and global-view consistency learning.
proxy samples for different modalities and progressively aligns them through a curriculum-base modality mixup mechanism (gradually transitioning from intra-modal to tross-modal alignment). Proxy-based consistency learning. We optimize the distribution of cross-modal proxy samples, achieving global-level alignment.
4.2 HIERARCHICAL CODING TREE CONSTRUCTION
To address the challenges posed by flat and sparse cross-modal connections, we construct a hierar- chical encoding tree in
aahhaceeanddcondenss manner, shown in Figure 3. First, we enhance the intra-modal connections within the relation graph. Then, guided by the structure entropy (Li al., 2018; Zou al., 2023), condense the relation graph to obtain the hierarchical encoding tree. The encoding tree exhibits a hierarchical community structure, facilitating the hash learning.
Enhance. In unsupervised cross-modal retrieval, primarily rely cross-modal pairwise rela- tions. We first construct a inter-modal relation graph Ginter (V, Einter J. where V D U D
and Einter {f. f i-1- Since cross-modal pairs only provide sparse supervision signals in unsupervised scenarios, we strategically employ KNN to enhance intra-modal relationships. This enables to capture fine-grained local similarities and form cohesive bottom-level semantic com- munities, which serve as a robust foundation for subsequent hierarchical modeling. Since Ginser is sparse and inadequate for cross- moosl learning, we enrich the intra-mode relationships construct tightly-knit low-level communities. Specifically, we turn to the cosine similarity within modality by ((s,) = cos cos(f: - )f), E (u, f). We choose cosine similarity as it focuses on semantic directional alignment by normalizing vector magnitudes, which is crucial for cross-modal feature comparison. We then construct the intra-modal relationship graph Gintra (V, Eintra) } based on the similarity matrix by KNN manner:
= ff} lj E topk (f., (,, {
where l is set to 3 according the hyperparameter study in Section 5.2. This choice balances information gain and noise robustness: larger & introduces noisy relationships, while smaller fails capture sufficient local semantics. Then, merge the intra-modal relationships Gintre and the inter-modal relationships Ginter to obtain the cross-modal relationship graph Geross (V, Eintra U E.nter).
Condense. To simplify the cross-modal semantics and construct the hierarchical relationship, we extract the encoding tree T from Geroez with the guidance of structural entropy, as illustrated in the center part of Figure 2. We first introduce the 1-D structural entropy as: d,
d,
((()=- vol (G) log vol (g)
(3)
where d, denotes the degree for node v and vol(-) denotes the sum degree for nodes in G. Then, we introduce the hierarchical structural entropy for the T as:
E (9)
=
Σ GET
Jo vol information encoding lenkage
log
V.
(4)
(g)
Va-
efficiency
4

---

Published as a conference paper at ICLR 2026
@ittaalGGaphh
Enhance
Intra-modai XNN
9 inttr
9 croes
Condense O Guided by Structurai Entropy Merge
Optimized Shooding Tree T
Compress
Figure 3: The construction pipeline of the encoding tree, involving graph enhancement and entropy- guided condensing operations.
Intuitively, the factor 2o captures the information leakage of community a, while the logarithmic
V.- reflects the encoding efficiency of the hierarchy. In Eq. 4, G is Geross is a node in T, Ta is the subtree rooted at o, Ta- is the subtree rooted a''s parent node, ga is the number of intra-modal relation links originating from the subtree Ta. V. is the sum of degrees in subtree Ta and V.- the sum of degrees in subtree Ta-. Eq. generalizes Eq. 3 hierarchical communities and reduces to Eq. 3 when the hierarchy collapses to flat partition.
term log Vu_
Guided by structural entropy, we convert the cross-modal relation graph into an encoding tree through Merge and Compress, as shown in Figure 3. Firstly, we perform node Merge operation to generate a binary encoding tree. The node Merge operation merges nodes that belong to the same parent node. These nodes may be highly semantically similar, and merging them can reduce the overall structural entropy of the cross- cross-modal encoding tree. The node Merge operation is defined as: T' = Merget (a, () we check all nodes if it can introduce decrease in FTT (9).
Secondly, Compress operation is performed to optimize the encoding tree, mainly targeting adjacent nodes at different levels, and constructing local clusters. This is achieved by attempting move the child encoding tree with a as the root to its parent node's parent node, thereby enabling compression of the cross-modal semantic graph. After encoding tree compression, if there no child encoding trees connected to a parent node, this parent node can be contracted to its parent node. The Com- press operation is defined the Compress operation if it can introduce entropy decrease. Overall, we optimize the cross-modal encoding following greedy principle. Specifically, traverse the tree nodes in breadth-first search manner. We attempt the aforementioned operations, and if they can decrease the structural en- tropy, execute the operation. While this greedy approach does not guarantee global optimality, provides an efficient and practical solution that converges to stable structures with clear hierarchical communities, as validated by our experiments. The cross-modal encoding tree after optimization is defined as T arg min (ET (G)), where T is the optimized encoding tree. The resulting tree naturally captures semantic granularity transitions, with upper nodes representing broad categories, mid-level nodes capturing fine- fine-grained concepts, and leaf nodes corresponding to specific instances. T exhibits better cross-mas mooal semantic properties. Specifically, they encompass more comprehen- sive local semantic motifs while mitigating the connections within high-density communities. These characteristics facilitate subsequent discriminative hash code learning and consistency learning.
T Compresst (a, B3 Similarly, we check the nodes and conduct
4.3 STRUCTURE-GUIDER CROSS-MODAL. HASH LEARNING
After obtaining the optimized cross-modal encoding tree T we use it for unsupervised hash leamn ing. Compared to existing unsupervised cross-modal hashing works (Hu et al., 2022; Liu et al., 2017: Zhang et al., 2018; 2024b; Tu et al., 2023), our method can exploit local semantic communi- ties, avoiding the bias caused by individual samples. Simultaneously, we jointly model both image and text modalities, mapping the vectors from distinct modalities into unified binary space.
5

---

Published as a conference paper at ICLR 2026
N MT(
Proxy Construction. For each sam- ple fi we sample its neighboring nodes A (ff() with the same modal- ity and A N with the opposite modality the cross-modal encod- ing tree T*, and obtain the proxy samples via:
Local Communities
A
R
Mixup
mi
m
E 8' (f)), (EN*(H)
Some Modallt
m
Curriculum Alignment
MMD
Differet Modility
Figure 4: The Modality Mixup pipeline: generating pondality-specifee proxies, using their MMD to guide curriculum-basee mixup, and producing byriz.
m,
cross
E W (fn) jEN-(A)
0 (f)),
(5)
where f. E (v, is the vector, (-) is the hash model we introduced in Eq. 1. The neighbor counts W (fr') and (Wf(fi) are naturally determined by the tree structure 7 and vary across nodes. Therefore, mame for a text sample f.¹ (aggregating text neighbors) is distinct from m mame for a visual sample f. (aggregating visual neighbors), as they are derived from different sets of mopelity-specifie neighbors. A similar distinction applies m, mross, The modal proxy samples can be viewed as a mediator between the two modalities, consisting of semantically similar nodes from the opposite modality, exhibiting better semantic robustness.
Modality Mixup. Leveraging proxy samples, we introduce mixup mechanism (Oh et al., 2023; Verma al., 2019; Huang et al., 2022) for progressively learning. Specifically, as Figure 4, grosss are mixup generate the hash codes:
0.8
m mi game and m.
0.6
muiz = sign(
m same
I+ m (TOSs),
X
(6)
10
20
Epoch
A = MMD (p (m same BB) , pm cross (B)o,,
Figure 5: Evolution of À during learning MIRFlickr-25K.
where À is to measure the distribution difference of different modalities, and p is the cosine distance metrics, we sample in mini-batch B and calculate MMD using the Gaussian ker-
(Long et al., 2015; Tolstikhin al., 2016). Notably, À is not manually tuned hyperparameter but a data-driven metric computed via MMD, which dynamically quantifies the modality gap during training. This adaptive weighting mechanism implements curriculum learning strategy (Hu et al., 2025). The theoretical foundation lies in manifold regularization (Verma et al., 2019): by interpo- lating in feature space between semantically purified proxy samples, we encourage smooth decision boundaries and force both modalities to find shared semantic space where their interpolation mains meaningful for hash learning. As Figure 5, the evolution of empirically demonstrates this progressive alignment process.
Cross-modal Hash Learning. We employ the mixed hash code brii for hash learning. In the learning process, sample in batches, and the other samples in the batch samples, enhancing the discriminative power of the hash codes. The objective function for cross- modal hash learning is:
negative
N
exp ((f bmbzt/T) \/T)
Chash =
Σ log [8] exp (ff: bmnit//T)
where * E (u, t) and 7 is the temperature parameter, which is set to 0.3 according to Section 5.2.
In HINT, utilize the cross-modal encoding tree to guide hash learning. Since cross-modal encoding tree has more comprehensive connections on local base groups, can help align hash codes more robust semantics. Meanwhile, the cross-modal encoding tree is sparser between communities, facilitating discriminability between groups and achieving discriminative hash codes.
Theoretical Discussion. This paragraph will discuss the limiting behavior of the cross-modal hash- ing loss Lhash and demonstrate how it enables effective retrieval. In Eq. 7, the inner product (f. braie) serves similarity measure between the feature f and the hash code b buiie and

---

Published as a conference paper at ICLR 2026
the temperature controls the scale of the measured similarity. Since is relatively small in imple- mentation, the following theorem follows Oh et al. (2023) and shows the convergence property of Lhash-
Lemma 4.1. For sufficiently large N and batch size [B), we have
lim r-00
N Shash F. b 11 +1I b min Rp
min kef
| b (8)
The right-hand side of Eq. can be used to measure the difference between the distances between positive pairs in comparison with the hardest negative pairs that are closest to the positive pair. The- orem 4.1 implies that minimizing Lhash is equivalent to minimizing the triplet loss, and the smaller triplet loss implies more transferable representation and more effective retrieval. This theorem shows how Chash contributes to the cross-modal retrieval performance. The detailed proof is available in the Appendix B.
4.4 PROXY-BASED CONSISTENCY LEARNING
Due to the heterogeneous gap between modalities, it is necessary to introduce an additional global- view modal aligmment mechanism to achieve better alignment and enhance the generalization ability.
Semantic Consistency Learning. We leverage the modal praxy samples for modality alignment learning, assuming that the original samples and their praxy samples should have similar semantic positions and similar distribution. Specifically, the modality semantics in a global view can be represented as the similarity distribution between samples and other samples within the same batch:
p(f;) I (f: (f: Ifj E
(9)
where B- includes the opposite modality instances within the same mini-batch, and p(-) is the cosine similarity function. Our objective is achieved by optimizing the KL divergence:
[8]
Loon Σ (DKL (p
| (m:rose))
(10)
where [B] is the batch size, (f(') and (mzroos)) are the semantic distributions of the i-th sample and its cross-modal proxy sample, respectively. By optimizing the consistency learning Ccons achieve modality alignment learning at high level by leveraging the semantic-stable proxy samples.
Summary. Our method constructs hierarchical encoding tree by unsupervised cross-modal mining and simultaneously leverages the encoding tree for mods- hash learning and semantic con- sistency learning. During the testing phase, the hierarchical encoding tree and proxy samples are not used. We directly employ the corresponding hash model to generate its hash code. This de- sign ensures efficient retrieval with minimal computational overhead during inference time. Due to the non-differentiability of the sign(-) function, it is challenging to optimize the overall objective. Therefore, we adopt tanh(-) as a surrogate during the training process. The whole algorithm is summarized in Algorithm 1 and Appendix A. The computational complexity and time efficiency discussed in the Appendix C.
EXPERIMENT
5.1 EXPERIMENTAL SETTINGS
! Datasets. In order to comprehensively evaluate the performance of HINT, we conduct experi- ments on three popular public datasets: MIRFlickr-25K (Huiskes & Lew, 2008), NUS-WIDE (Rasi- wasia et al., 2010). and MS-COCO (Lin et al., 2014). The detailed information is available in Appendix E.2. @ Baselines. We compare our method HINT with 15 baselines from related fields, including the latest state-of-the-an works. Details in Appendix E. @ Implementation Details. To ensure fair comparison, we implement our method based on the latest SOTA works (Zhang et al., 2024b; Hu et al., 2022). Details in Appendix E.3.
5.2 RESULTS
Hamming Ranking. The experiments on cross-modee retrieval benchmarks show that the proposed HINT consistently outperforms baseline approaches across different code lengths (16- (16-128 bits). Key
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of MAP performance (%) across various cross-modal hashing methods.
MIRFlickr-25K
NUS-WIDE
MS-COCO
Methods
16
32
64
128
16
32
64
128
16
32
64
128
Image
Text
CVH
62.0
60.8
59.4
58.3
48.7
49.5
45.6
41.9
50.3
50.4
47.1
42.5
LSSH
59.7
60.9
60.6
60.5
44.2
45.7
45.0
45.1
48.4
52.5
54.2
55.1
CMFH
55.7
55.7
55.6
55.7
33.9
34.3
33.9
36.6
36.9
37.0
36.5
61.2
55.7
59.8
63.5
53.9
54.9
57.6
MTFH
50.7
51.2
29.7
27.2
32.8
39.9
29.3
29.5
FOMH
57.5
64.0
69.1
65.9
30.5
30.5
30.6
31.4
37.8
51.4
57.1
60.1
DCH
59.6
60.2
62.6
63.6
39.2
42.2
43.0
43.6
42.2
42.0
44.6
46.8
DGCPN
65.1
68.3
71.8
72.4
60.1
61.8
63.1
64.0
55.6
56.9
57.8
58.0
UCHSTM
70.1
71.5
72.4
72.3
62.5
63.5
64.6
64.4
55.8
57.2
57.6
57.3
UCCH
71.6
72.6
72.8
73.2
62.1
62.3
64.0
64.5
56.0
56.2
56.6
57.4
UDDH
71.4
72.9
74.0
74.6
63.7
64.2
65.1
65.9
56.8
57.8
59.0
59.9
HuggingHash+
71.6
73.2
74.3
74.5
63.9
64.8
65.6
66.4
57.1
58.3
59.4
60.5
DEMO
71.8
73.3
73.4
74.3
64.6
64.8
66.2
66.4
57.5
57.8
58.6
60.5
GCRH
71.0
72.2
72.7
73.3
63.9
65.3
65.5
56.8
57.0
57.8
59.6
VTM-UCH
71.9
73.6
66.0
66.6
57.6
58.2
58.8
HINT
72.9
74.4
75.1
75.5
65.1
65.5
66.5
67.3
58.5
59.5
60.4
61.1
Text Image
CVH
62.9
61.5
59.9
58.7
47.0
47.5
44.4
41.2
50.6
50.8
48.6
42.9
LSSH
60.2
59.8
59.8
59.7
47.3
48.2
47.1
45.7
49.0
52.2
54.7
56.0
CMPH
55.3
55.3
55.3
55.3
30.6
30.6
30.6
30.6
34.6
34.6
34.6
34.6
FSH
57.6
60.7
63.5
66.0
56.9
60.4
65.1
66.6
53.7
52.4
56.4
57.3
MTFH
51.4
52.4
51.8
39.9
41.0
33.5
37.4
33.4
FOMH
58.5
64.8
71.9
30.2
30.4
30.0
30.6
36.8
48.4
55.9
59.5
DCH
61.2
62.3
65.3
66.5
37.9
44.4
45.9
42.1
42.8
45.4
47.1
DGCPN
65.3
68.2
71.2
71.5
60.5
62.6
63.7
64.4
55.0
56.6
57.8
57.7
UCHSTM
69.5
71.1
71.3
72.3
63.2
64.3
65.1
65.2
55.5
56.7
57.8
57.3
UCCH
70.3
71.2
72.0
72.1
62.5
63.7
65.0
65.2
56.4
57.3
57.2
58.1
UDDH
70.5
71.6
72.8
73.5
64.5
65.2
66.0
66.6
56.6
57.5
58.5
59.4
HuggingHash+
70.7
72.0
73.2
73.8
64.8
65.7
66.5
67.0
56.9
57.9
59.0
60.1
DEMO
70.8
71.9
72.2
72.8
65.4
65.5
66.9
67.1
57.2
57.9
58.3
59.7
GCRH
70.1
71.0
71.3
71.8
64.5
64.7
66.0
66.2
56.5
57.3
57.4
58.8
VTM-UCH
71.4
72.1
72.4
73.0
65.6
65.7
66.9
67.3
57.4
58.0
58.3
59.6
HINT
72.0
73.1
74.0
74.6
66.0
66.6
67.3
67.8
58.2
59.0
59.8
60.8
Image
Imcage
Image
Image
Image
Image
tesan
0A
Droctaan
0n
Rocall
Top Ranked Sangles
Top Rocked Sunpies
Rocall
Top Ratlied Saniles
Top Rasked Sumples
Image 10 20hibx
Imagr @ 122fiits
Image 122hbi
Image R1ilbb
Image 121dhit
Imagge dF 10hetx
EIMT
1630
Propaiin
Rooal
017
Recall
Thp Ranked Samplies
Tog Bostled Sompies
Recall
Top Ratioed Somples
Top Rasked Semples
Figure 6: Hash lookup performance with 128-bit and 16-bit codes on the MIRFlickr-255 dataset. The Precision-Reddt curves, Precision-N curves, and Recall-N curves are shown from left to right.
findings show that: 1 Deep unsupervised hashing approaches typically outperform traditional ones, Supervised methods struggle when labeled data is limited, The method shows improved per- formance challenging sub-tasks like Text= retrieval, and O The hierarchical modeling ap- proach proves more effective than other deep cross-modal methods for achieving progressive cross- modal alignment. Additionally, HINT demonstrates strong robustness against noisy data, maintain-
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation studies. The component columns "KNN, Tree, Curr, Con" respectively denote
intra-modal KNN, hierarchical encoding tree, curriculum- mixup, and proxy-based learning.
Components
MIRF-25K
NUS-WIDE
MS-COCO
Methods
KNN
Tree
Curr
Con
I-T
T-xI
I-T
T-I
I-+T
T->
HINT VI
73.2
72.0
64.0
65.1
57.9
58.5
HINT V2
73.8
72.8
65.2
65.7
59.1
58.9
HINT V3
74.2
73.6
65.9
66,4
60.0
59.5
HINT V4
75.1
74.1
67.0
67.3
60.7
60.2
HINT
75.5
74.6
67.8
61.1
60.8
UCCH
DEMO
HINT
Visual
Text
Figure 7: The I-SNE projection of hash codes from different modalities. Among competing methods, HINT shows the best ability of modal alignment.
ing superior performance even with 10% corrupted pairs. Detailed noise robustness analysis and results are provided in Appendix C.6. As shown in Table performance generally improves with increasing bit length as longer hash codes provide larger Hamming space for encoding more infor- mation, though with diminishing returns higher lengths. The more pronounced improvements on Imae tasks can be attributed to text feature sparsity and lower initial quality compared to visual features. HINT's hierarchical structure and proxy-based neighborhood aggregation provide more substantial benefits for the inherently more challenging tasks.
Hash Lookup. To comprehensively analyze HINT's performance, we evaluate Precision-Recae Precision-N, and Recall-N curves with 128-bit and 64-bit codes MirFlickr-25K, From the results Figure 6, HINT performs better comparison these baselines across all metrics, aligning with the MAP scores from Hamming ranking. In summary, HINT exhibited optimal performance in cross-modal hash retrieval. Experiments for other code lengths are in Appendix C.I.
Visualization. We provide t-SNE visualization analysis of HINT's performance using 128-bit hash codes on the MirFlickr-25K dataset, comparing with DEMO and UCCH, distinguishing dif- ferent modalities with distinct colors. As shown in Figure HINT demonstrates superior ability to map representations from different modalities into a unified hash space, exhibiting higher align- ment between text and visual modalities. The visualization suggests that HINT effectively aligns modalities and learns hash codes with generalization capabilities.
Ablation Study. We compare the following variants of HINT: VI, which only uses text-image pairs without consistency learning (Ccon), where by bini is obtained solely from the opposite modality; V2, which uses both text-image pairs and intra-modal KNN without Lcons where 6" by is obtained by averaging opposite modality and KNN samples; V3, which construets the hierarchical encoding tree with sample selection but without curriculum-basee based progressive alignment and without con- sistency loss Term: V4, which uses Chash consistent with the full model but excludes L.m- As shown in Table 2, the full model achieves optimal performance, with hierarchical encoding tree and progressive alignment (V3 and V4) yielding the most improvements. We also explored itera- tive tree updates but found static construction provides better pereiiecyy-erffommncc trade-off, with detailed analysis in Appendix C.8. Additional experiments comparing different similarity metrics for tree construction demonstrate cosine similarity's 's superiority over L./L2 distances, with detailed analysis in Appendix C.7. Additional ablation studies available in Appendix C.2.
Sensitivity Analysis. We analyze the hyperparameters and r. As shown in Figure 8a, increasing K from to 3 improves performance on both retrieval tasks, validating the benefits of enhanced

---

Published as a conference paper at ICLR 2026
0.00
0.00
Teagee tea misagg
Image Toxt
20.2
Toxt Imsage
0.77
Actummy 184
10.0
20.2
0.74
0.74
40.2
0.71
0.71
20.0
30.2
Length
(a) Sensitive analysis of K and r. HINT demonstrates (b) Stability analysis of HINT. Error bars indicate robustness hyperparameters.
standard deviation.
Figure 8: Sensitivity and stability analyses on the MIRFlickr-2511 dataset.
cross-modal relationships. However, further increasing K to 5 introduces noisy relationships and decreases performance. Our experiments demonstrate HINT's remarkable stability, with perfor- mance fluctuation remaining within narrow 2% margin when varying from 0.1 0.5. The model consistently outperforms baselines across most parameter settings. Based on these findings, we set K- and 0.3 as the default values in our experiments.
Stability Analysis. We conducted 5 independent runs with different seeds. As shown in Figure 8b, the results show that HINT exhibits remarkable stability, with performance variations consistently remaining below 1% standard deviation across different code lengths. Details in Appendix C.9.
Time Efficiency. We provide a comprehensive efficiency analysis covering model complexity, train- ing cost, and retrieval speed. As shown in Table 5. HINT achieves superior MAP performance with training efficiency comparable to DEMO and UDDH, demonstrating an excellent efficiency- performance trade-off. For retrieval efficiency, HINT and all compared hashing methods use identi- cal Hamming distance computation on binary codes, thus sharing the same retrieval speed. Table demonstrates that hash-based retrieval achieves 26-31 x speedup over dense vector approaches. De- tails in Appendix C.5 and C.4.
CONCLUSION
This paper investigates the problem of efficient cross-modal retrieval through cross-modal hashing. We propose HINT that leverages hierarchical structural entropy to guide the construction a cross- modal encoding tree, which has tightly connected local clusters. By incorporating progressive mixup for proxy-based alignment and consistency learning from global perspective, we enhance the gen- eralization capability of the produced hash codes. In the end, we demonstrate the effectiveness of our HINT via comprehensive experiments on benchmark datasets.
ETHICS STATEMENT
Our research adheres to the ICLR Code of Ethics. All datasets used in this study are publicly available. The code and related materials will be appropriately released to ensure transparency and reproducibility of our work.
ACKNOWLEDGMENT
Ming Zhang and Junyu Luo supported by grants from the National Natural Science Foundation of China (NSFC Grant Number 62276002).
REFERENCES
Zhangjic Cao, Mingsheng Long. Jianmin Wang, and Qiang Yang. Transitive hashing network for heterogeneous multimedia retrieval. In Proc. of Association for the Advancement of Artificial Intelligence, volume 31, 2017.
Dapeng Chen, Min Wang, Haobin Chen, Lin Wu, Jing Qin, and Wei Peng. Cross-modas retrieval with heterogeneous graph embedding. In Proc. of ACM International Conference on Multimedia, pp. -200, 2022.
10

---

Published as a conference paper at ICLR 2026
Haonan Chen, Zhicheng Dou, Kelong Mao, Jiongnan Liu, and Ziliang Zhao. Generalizing conversa- tional dense retrieval via LLM-cognition data augmentation. In Proceedings of the 62nd Anmual MMeting of the Association for Computational Linguistics ( (Volume I: Long Papers), 2024.
Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia. Ame: Attention guided multi-modal correlation learning for image search. In Proc. of Computer Vision and Pattern Recognition, 2017.
Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural net- works: Better and robust node embeddings. Advances in neural information processing systems, 33:19314- 19326, 2020.
Wanqing Cui, Keping Bi, Jiafeng Guo, and Xueqi Cheng. MORE: Multi-mOdal REtrieval aug-
mented generative commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Guiguang Ding, Yuchen Guo, Jile Zhou, and Yue Gao. Large-scale cross-modality search via col- lective matrix factorization hashing. IEEE Transactions on Image Processing, 2016a.
Kun Ding, Bin Fan, Chunlei Huo, Shiming Xiang, and Chunhong Pan. Cross-modal hashing via rank-order preserving. IEEE Transactions on Multimedia, 2016b.
Haozhi Fan and Yuan Cao. Vision-guided text mining for unsupervised cross-modal hashing with community similarity quantization. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 39, pp. 2843-2851, 2025.
Zijun Gao, Jun Wang, Guoxian Yu, Zhongmin Yan, Carlotta Domeniconi, and Jinglin Zhang. Long- tail cross modal hashing. In Proc. of Association for the Advancement of Artificial Intelligence, 2023.
Xuri Ge, Fuhai Chen, Joemon M Jose, Zhilong Ji, Zhongqin Wu, and Xiao Liu. Structured multi- modal feature embedding and alignment for image-sentence retrieval. In Proc. of ACM Interna- tional Conference on Multimedia, pp. 5185-5193, 2021.
Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 3shnet: Boosting image-sentence retrieval via visual semantic-spatial self-highlighhing. Information Pro- cessing & Management, 61(4):103716 2024.
Kaiming He, Haoqi Fan, Yuxin Wa, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738 2020.
Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, and Xi Peng. Unsupervised con- trastive cross-modal hashing. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2022.
Xiaoxing Hu, Kaicheng Yang, Ziyong Feng, Qi Ming, Zonghao Guo, Xiang An, Junchi Yan, and Xue Yang. Proclip: Progressive vision-language alignment via Ilm-based embedder. arXiv preprint arXiv:2510.18795 2025.
Hailang Huang, Zhijie Nie, Ziqiao Wang, and Ziyu Shang. Cross-modal and uni-modal soft-label alignment for image-text retrieval. In Proc. of Association for the Advancement of Artificial Intel- ligence, volume 38, pp. 18298-18300 2024.
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng- Learning with noisy correspondence for cross-modal matching, Proc. of Advances in Neural Information Processing Systems, 2021.
Zhipeng Huang, Jiawei Liu, Liang Li, Kecheng Zheng, and Zheng-Jun Zha. Modality mixup and invariant decomposition for rgb-infrared person re-identification. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 1044, 2022.
Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In Proc. of ACM interna- fional conference Multimedia information retrieval, 2008
11

---

Published as a conference paper at ICLR 2026
Qing- Yuan Jiang and Wu-Jun Li. Discrete latent factor model for cross-modal hashing. IEEE Transactions on Image Processing, 2019.
Weike Jin, Zhou Zhao, Pengcheng Zhang, Jieming Zhu, Xiuqiang He, and Yueting Zhuang. Hierar- chical cross-modal graph consistency learning for video-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on research and development in information retrieval, pp. 1114-1124, 2021.
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Im- age retrieval from contextual descriptions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
Shaishav Kumar and Raghavendra Udupa. Learning hash functions for cross-view similarity search. In Proc. of International Joint Conference on Artificial Intelligence, 2011.
Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, and Sungroh Yoon. Interactive text-to-image retrieval with large language models: A plug-and-play approach. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Angsheng Li and Yicheng Pan. Structural information and dynamical complexity of networks. IEEE Transactions Information Theory, 2016.
Angsheng Li, Xianchen Yin, Bingxiang Xu, Danyang Wang, Jimin Han, Yi Wei, Yun Deng, Ying Xiong, and Zhihua Zhang. Decoding topologically associating domains with ultra-low resolution hi-c data by graph structural entropy. Nature communications, 2018.
Chao Li, Cheng Deng, Lei Wang, De Xie, and Xianglong Liu. Coupled cyclegan: Unsupervised hashing network for cross-modal retrieval. In Proc. of Association for the Advancement of Artifi- cial Intelligence, 2019.
Fengling Li, Bowen Wang, Lei Zhu, Jingjing Li, Zheng Zhang, and Xiaojun Chang. Cross-domain transfer hashing for efficient cross-modal retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 2024a.
Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, and Desmond Elliont. Understanding retrieval robustness for retrieval-augmented image captioning. arXiv preprint arXiv:2406.02265, 2024b.
Yunxin Li, Baotian Hu, Yuxin Ding, Lin Ma, and Min Zhang. A neural divide-and-conquer reason- ing framework for image retrieval from linguistically complex text. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Zhi Li. Yifan Liu, and Yin Zhang. Back-modality: Leveraging modal transformation for data aug- mentation. Proc. of Advances in Neural Information Processing Systems, 2024c.
Meiyu Liang, Junping Du, Zhengyang Liang, Yongwang Xing, Wei Huang, and Zhe Xue. Self- supervised multi-modal knowledge graph contrastive hashing for cross-modal search. In Proc. of Association for the Advancement of Artificial Intelligence, volume 38, pp. 13744-13753, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Proc. of Advances in Neural Information Processing Systems, 2022.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Proc. of Euro- pean Conference on Computer Vision, 2014.
Hong Liu, Rongrong Ji, Yongjian Wu, Feiyue Huang, and Baochang Zhang. Cross-modality binary code learning via fusion similarity hashing. In Proc. of Computer Vision and Pattern Recognition, 2017.
Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and Shih-Fu Chang. Supervised hashing with kernels. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2074-2081. IEEE, 2012.
12

---

Published as a conference paper at ICLR 2026
Xin Liu, Zhikai Hu, Haibin Ling, and Yiu-ming Cheung. Mtfh: A matrix tri fi-factorization hash- ing framework for efficient cross- moosl retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019a.
Xuanwu Liu, Guoxian Yu, Carlotta Domeniconi, Jun Wang, Yazhou Ren, and Maozu Guo. Ranking- based deep cross moosl hashing. In Proc. of Associaatio for the Advancement of Artificial Intel- ligence, 2019b.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proc. of International Conference Machine Learning. PMLR, 2015.
Xu Lu, Lei Zhu, Zhiyong Cheng, Jingjing Li, Xiushan Nie, and Huaxiang Zhang. Flexible online multi- modal hashing for large-scale multimedia retrieval. In Proc. of ACM International Confer- ence on Multimedia, 2019.
Junyu Luo, Yusheng Zhao, Xiao Luo, Zhiping Xiao, Wei Ju, Li Shen, Dacheng Tao, and Ming Zhang. Cross-domain diffusion with progressive alignment for efficient adaptive retrieval. JEEE Transactions on Image Processing, 2025.
Xiao Luo, Haixin Wang. Daqing Wu, Chong Chen, Minghua Deng, Jianqiang Huang, and Xian- Sheng Hua. A survey on deep hashing methods. ACM Transactions on Knowledge Discovery from Data, 2023.
Zeyu Ma, Siwei Wang. Xiao Luo, Zhonghui Gu, Chong Chen, Jinxing Li, Xian-Sheng Hua, and Guangming Lu. Harr: Learning discriminative and high-quality hash codes for image retrieval. ACM Transactions on Multimedia Computing, Communications and Applications, 20(5):1-23, 2024.
Georgü Mikriukov, Mahdyar Ravanbakhsh, and Begüm Demir. Unsupervised contrastive hashing for cross-modal retrieval in remote sensing, In Proc. of International Conference Acoustics, Speech, and Signal Processing. IEEE, 2022.
Changdae Oh, Junhyuk So, Hoyoon Byun, YongTack Lim, Minchul Shin, Jong-June Jeon, and Kyungwoo Song. Geodesic multi-modal mixup for robust fine-tuning. In Proc. of Advances in Neural Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of International Conference Machine Learning, 2021.
Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In Proc. of ACM International Conference on Multimedia, 2010.
Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. Supervised discrete hashing. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 37-45, 2015.
Xiaobo Shen, Yinfan Chen, Weiwei Liu, Yuhui Zheng. Quan-Sen Sun, and Shirui Pan. Graph con- volutional multi-label hashing for cross-modal retrieval. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Mingyang Song, Liping Jing, and Yi Feng. Match more, extract better! hybrid matching model for open domain web keyphrase extraction. In Findings of the Association for Computational Linguistics ACL. 2024, pp. 17-27,202.
Yuan Sun, Zhenwen Ren, Peng Hu, Dezhong Peng, and Xu Wang. Hierarchical consensus hashing for cross-modal retrieval. IEEE Transactions on Multimedia, 26:824-8366 2023.
Yuan Sun, Jian Dai. Zhenwen Ren, Yingke Chen, Dezhong Peng, and Peng Hu. Dual self-paced moosl hashing. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 15184-15192, 2024.
13

---

Published as a conference paper at ICLR 2026
Iyya O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf. Minimax estimation of max- imum mean discrepancy with radial kernels. Proc. of Advances Neural Information Processing Systems, 2016.
Kong-Cheng Tu, Xian-Ling Mao, Qinghong Lin, Wenjin Ji, Weize Qin, Wei Wei, and Heyan Huang. Unsupervised cross-modal hashing via semantic text mining. IEEE Trunsuctions Multimedia, 2023.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference on machine learning, pp. 6438-6447, PMLR, 2019.
Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tun- ing multi-mode token-level prompt alignment across modalities. Proc. of Advances in Neural Information Processing Systems, 2024a.
Haixin Wang, Hao Wu, Jinan Sun. Shikun Zhang, Chong Chen, Xian- SannShhng Hua, and Xiao Luo. Idea: An invariant perspective for efficient domain adaptive image retrieval. Advances in Neural Information Processing Systems, (6657256-57277 2023.
Jinpeng Wang, Ziyun Zeng. Bin Chen, Yuting Wang, Dongliang Liao, Gongfu Li, Yiru Wang, and Thu Xia. Hugs bring double benefits: Unsupervised cross-modal hashing with multi- granularity aligned transformers. International Journal of Computer Vision, pp. 1-33, 2024b.
Jun Wang. Sanjiv Kumar, and Shih-Fu Chang. Semi-supervised hashing for scalable image retrieval. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp- 3424 3431. IEEE, 2010.
Rukai Wei, Yu Liu, Heng Cui, Yanzhao Xie, and Ke Zhou, Graph contrasive-and-reonstttttteee hashing for unsupervised cross-modal retrieval. Data Science and Engineering, pp. 1-17, 2025.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proc. of Computer Vision and Pattern Recognition, 2018.
Xing Xu, Fumin Shen, Yang Yang. Heng Tao Shen, and Xuelong Li. Learning discriminative binary codes for acgee cross-modal retrieval. JEEE Transactions on Image Processing, 2017.
Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Jun Yu, Hao Zhou, Yibing Zhan, and Dacheng Tao. Deep graph-neighbor coherence preserving network for unsupervised cross- hashing. In Proc. of Association for the Advancement of Artificial Intelligence, 2021.
Bin Zhang, Yue Zhang, Junyu Li, Jiazhou Chen, Tatsuya Akutsu, Yiu-Ming Cheung, and Hongmin Cai. Unsupervised dual deep hashing with semantic-index and content-code for cross-modal retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Donglin Zhang, Zhiwen Wang, Zhikai Hu, and Xiao-Jun Wu. Htvr: Hierarchical text-to-video retrieval based on relative similarity. Pattern Recognition, pp. 112145,
Fan Zhang, Xian Kian-Sheng Hua, Chong Chen, and Xiao Luo. DEMO: A Statistical Perspective for Efficient Image-Text Matching. arXiv preprint arXiv:2405. 1496, 2024b.
Jian Zhang, Yuxin Peng, and Mingkuan Yuan. Unsupervised generative adversarial cross-modal hashing. In Proc. of Association for the Advancement of Artificial Intelligence, 2018.
Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In Proc. of Computer Vision and Pattern Recognition, 2019.
Jile Zhou, Guiguang Ding, and Yuchen Guo. Latent semantic sparse hashing for cross-modal sim- ilarity search. In Proc. of ACM SIGIR conference on Research & development in information retrieval, 2014.
14

---

Published as a conference paper at ICLR 2026
Lei Zhu, Chaoqun Zheng. Weili Guan, Jingjing Li, Yang Yang. and Heng Tao Shen. Multi-modal hashing for efficient multimedia retrieval: A survey. IEEE Transactions on Knowledge and Data Engineering, 2023.
Dongcheng Zou, Hao Peng, Xiang Huang, Renyu Yang, Jianxin Li, Jia Wu, Chunyang Liu, and Philip S Yu. Se-gsl: A general and effective graph structure learning framework through structural entropy optimization. In Proc. of ACM Web Conference. 2023.
15

---

Published as a conference paper at ICLR 2026
A ALGORITHM
We present the optimization algorithm of our method in Algorithm The algorithm first constructs a cross-modal relationship graph and optimizes the hierarchical encoding tree. Then during training, it performs modality mixup and consistency learning to generate effective hash codes through back- propagation.
Algorithm Optimization Algorithm of HINT
Require: Visual modality DO; Text modality D Ensure: Hashing model ø (-) and o'(+);
1: Construct the cross cross-modal relationship graph Gcrossi
2: Optimize the hierarchical encoding tree T"
3: for each epoch do
4: for each batch do
5: Sample B", B from D Dt:
6: Construct proxy samples m moos using Eq. 5;
7: Perform modality mixup and generate hash code b with T by Eq. 6;
8: Calculate the Lhash and Leon
9: Update parameters through back back- proppagpopaannn;
10: end for
11: end for
B PROOF OF LEMMA 4.1
Proof of Lemma 4.1. Denote N₁ and Mr denote the index sets of visual vector and text vector, respondingly. We assume the batch number [B] is equal to to reduce the notation burden and by definition,
exp (ifr b.ode)))
exp ((fr" b.oke) //rr)e//))))))
Chash
Σ iEA.
log
Σ REM,
log
Σ .N exp (Ifi, blnies/7)
N exp ((fi bynins //r
(11)
We focus on the first term of the right-hand side of the above equation. By the proof of Proposi- tion D.2. in Oh et (2023).
exp (ff b/ube)/T)
lim +-++
Σ log
N
E| exp (ff: bynit)/
IEN
(12)
lim 7-01
Σ
max maxfmaxx (max (fxx ".ft".p bmiie brth 13.0).
N
E..
Since m misum
same and mitrus cruss are continuous random variables, then
m 1+ m cross || = T.) =1
[ |2 = I.) P sign
same
(13)
and
mir
min
I|
b maiz 1
jzi
= max i
[1. + |
mima 1|
2 briiay
11 1 || + +(f" B.mia
(14)
-2 (f. Bypi 374
b bmiin
which implies that
exp (ff: bumiz
lim +
Σ SEN.
log
min 343
b
EJM exp (ff: byeit)
(15)
11

---

Published as a conference paper at ICLR 2026
Table Additional ablation studies with different code lengths.
MIRFIickr-22K
Methods
Image
Text
Text >Image
16bit
32bit
64bit
128bit
16bit
32bit
64bit
128bit
HINT VI
71.5
72.4
72.6
73.2
70.5
71.1
71.7
72.0
HINT V2
71.8
72.8
73.0
73.7
70.8
71.4
72.2
72.8
HINT V3
72.0
73.4
73.8
74.2
71.5
72.4
73.2
73.6
HINT V4
72.6
73,9
74.8
75.2
71.5
72.8
73.9
74.3
Full Model
72.9
74.4
75.1
75.5
72.0
73.1
74.0
74.9
Table Retrieval time cost (ms) varies with code length.
16 Bit
Bit
Bit
64 Bit
Bit
128 Bit
Hash Code
16.7
18.0
19.4
19.9
21.8
22.2
Dense Vector
441.4
491.0
543.0
602.3
657.7
696.6
Speed Up
26.5x
27.2x
28.0x
30.2x
30.1x
31.4x
C ADDITIONAL EXPERIMENTAL RESULTS
C.I ADDITIONAL EXPERIMENTS FOR HASH LOOKUP
To comprehensively evaluate the performance of the proposed HINT, we present precision-reaat curves, precision-N curves, and recall-N curves the MirFlickr-25K dataset with code lengths of 32 and 64 bits. As shown in Figure 10, 9, our method consistently outperforms other approaches, which is consistent with the corresponding MAP score based on Hamming ranking. Additionally, we compute the precision and recall rates of the top-N retrieved results, demonstrating HINT's persistent advantage. Our method achieves superior performance in cross-modal hash retrieval.
C.2 ADDITIONAL ABLATION EXPERIMENTS
To evaluate the effectiveness of these components, we introduce four variants:
HINT V1, which only employs image-text pairs for cross-modal hash learning without the global-view consistency learning module;
HINT V2, which combines image-text pairs with intra-modal KNN for cross-modal hash learning, also without the consistency learning module;
HINT V3, which uses the average of two modalities as the learning target and builds hierarchical encoding tree with sample selection, but without curriculum-ber progressive modal alignment (Equation 6), also without the global-view consistency learning module;
HINT V4, which adopts the full model's Lhash but excludes consistency learning.
As shown in Table 3, results demonstrate that our complete HINT achieves optimal perfor- mance, confirming the importance of each component. Furthermore, our ablation study reveals that hierarchical encoding trees and progressive alignment yield significant improvements (V2 and V3), validating our motivation.
C.3 COMPUTATIONAL COMPLEXITY
We mainly discuss the computational complexity the additionally introduced encoding tree con- struction. Assuming the number of data points is N (including samples from different modalities), the time cost for calculating the similarity matrix is O(N2), the cost for constructing the KNN graph is O(N). and the cost for optimizing the cross- moosl encoding tree is O(N log N) (Li & Pan, 2016). It is worth noting that the most time-consuming similarity matrix calculation can be
17

---

Published as a conference paper at ICLR 2026
Table 5: Model complexity and training efficiency comparison on MIRFlickr-255K with 128-bit.
Method
Params (M)
Total Time (min)
MAP (%)
UCHSTM
52.5
182
72.3
UCCH
88.0
58
73,2
UDDH
28.9
45
74.6
HuggingHash+
40.1
211
74.5
DEMO
86.2
59
74.3
HINT
84.8
61
75.5
Image Text @64bits
Image
Text @64bits
Image + Text @64bits
0.95
0.5
HINT
HINT
HINT
DEMO UCCH
DEMO UCCH
DEMO
0.90
0.4
UCCH
Precision
Precision
0.85
Recall
0.3
0.7
0.80
0.2
0.75
0.1
0.6
0.0
0.70
0.25
0.50
0.75
1.00
0
2000
4000
0
2000
4000
Recall
Top Ranked Samples
Top Ranked Samples
Text Image @64bits
Text Image @64bits
Text + Image @64bits
0.95
0.5
HINT
HINT
HINT
DEMO
DEMO
DEMO
UCCH
0.90
UCCH
0.4
UCCH
0.8
Precision
Precision
0.85
Recall
0.3
0.7
0.2
0.75
0.1
0.0
0.70
0.75
1.00
2000
4000
2000
4000
Recall
Top Ranked Samples
Top Ranked Samples
Figure 9: Hash lookup performance with 64 bits codes the MIRFlickr-25K dataset. The precision- recall curves, Precision-N curves, and recall-N curves are shown from left right.
accelerated by parallel computing. Moreover, the encoding tree construction is only performed once at the beginning of the training, so the additional computational complexity and time consumption brought to the overall training process are negligible.
C.4 TIME EFFICIENCY
We conducted a speed test between HINT and dense vector retrieval on an Intel Xeon CPU E5-2697 v4 (2.30GHz), as illustrated Table The speed test is conducted on the MIRFlicker-25K dataset with retrieval database of 100 items. We perform 103 runs and report the average retrieval speed cost (ms). Hash methods excel in enabling efficient and scalable image retrieval, especially for large- scale datasets, due fast Hamming distance computation. In contrast, existing pre-trained models only output dense vectors, resulting in slower computation. Table 4 compares the efficiency of hash codes and dense codes generated by model and pre-trained model various bit lengths. The
18

---

Published as a conference paper at ICLR 2026
Image Text @32bits
Image
Text @32bits
Image Text @32bits
0.9
0.95
0.5
HINT
DEMO UCCH
DEMO
DEMO
0.4
UCCH
UCCH
Precision
Precision
0.85
Recall
0.7
0.80
0.2
0.75
0.0
0.70
0.25
0.50
0.75
1.00
0
2000
4000
0
2000
4000
Recall
Top Ranked Samples
Top Ranked Samples
Text + Image @32bits
Text Image @32bits
Text Image @32bits
0.9
0.95
0.5
HINT
HINT
HINT
DEMO
DFMO
DEMO
UCCH
0.90
UCCH
0.4
UCCH
0.8
Precision
Precision
0.85
Recall
0.3
0.80
0.2
0.75
0.1
0.6
0.70
0.25
0.50
0.75
1.00
0
2000
4000
0
2000
4000
Recall
Top Ranked Samples
Top Ranked Samples
Figure 10: Hash lookup performance with 32 bits codes on the MIRFlickr-25K dataset. The precision-recall curves, precision-N curves, and recall-N curves are shown from left right.
results clearly demonstrate that hash codes achieve significantly faster retrieval speeds than deep feature codes, confirming their superiority, particularly in large-scale image retrieval scenarios.
C.5 COMPUTATIONAL EFFICIENCY ANALYSIS
While HINT introduces additional computational steps through the hicrarchical encoding tree, we have implemented several optimizations to ensure practical efficiency:
Controlled Time Complexity: The hierarchical encoding construction has complexity of O(N log*N) and is executed only once during the initial training phase. Our experiments on MIRFlickr-25K show that tree construction takes <3 minutes (single GPU), representing <5% of total training time. Compared to existing methods like DEMO and UCCH, HINT does not significantly increase the overall training duration.
Memory-Effiiiene Design: The encoding tree is stored using compressed relation triplets (parent- child-edge weight) instead of maintaining complete similarity matrices. Furthermore, the hash code generation phase completely decoupled from the encoding tree, eliminating the need to load tree structures during inference and conserving deployment resources.
Practical Scalability: The encoding tree's one-time construction and reusability make it partic- ularly suitable for large-scale applications. This design choice significantly amortizes the initial computational investment across multiple training sessions.
These results demonstrate that HINT achieves superior performance while maintaining competitive training efficiency through its optimized design. The comprehensive model complexity and training efficiency comparison with existing methods is shown in Table 5.
19

---

Published as a conference paper at ICLR 2026
Table 6: Noise robustness comparison on MIRFlickr-25K (128 bit) with 10% corrupted pairs.
Method
I-TT
T-
I-+T(109n))
T~1((10%))
UCCH
73.2
73.2
70.7
72.6
DEMO
74.3
74.3
71.4
73.3
HINT
75.5
73.8
74.4
72.9
Table Performance comparison of different similarity metries on MIRFlickr-25K.
Metric
I-T
T-xI
Cosine
75.5
74.6
1.2
74.8
74.2
L11
73.5
72.8
C.6 NOISE ROBUSTNESS ANALYSIS
To evaluate HINT's robustness against noisy data, we conducted experiments by randomly corrupt- ing 10% of text-image pairs in the MIRFlickr-25K dataset. The results demonstrate HINT's superior noise resilience through both architectural design and experimental validation:
Architectural Robustness: HINT's hierarchical encoding tree provides two-level noise adaptation:
The tree construction process inherently suppresses individual outliers by aggregating lo- cal semantic communities. Proxy samples, generated through neighbor feature averaging, effectively smooth out the impact of noisy samples within local communities.
The cross-modal consistency learning module (Lcon) constrains the influence of outliers hash space mapping by enforcing semantic distribution alignment between proxy and original samples from a global perspective.
As shown in Table 6, HINT consistently maintains higher performance under noisy conditions, with minimal degradation compared to baseline methods. This demonstrates that the encoding tree's hierarchical structure effectively identifies and mitigates the interference of mismatched pairs in cross-modal alignment.
C.7 ADDITIONAL ANALYSIS OF SIMILARITY METRICS
We conducted experiments comparing different similarity metrics for construction MIRFlickr-25K. The results demonstrate that cosine similarity achieves optimal performance due three key advantages:
Directional Consistency: Cosine similarity focuses semantic directional alignment by nor- malizing vector magnitudes
Loss Function Alignment: The training objective relies on inner product similarity, which aligns with cosine similarity computation
Feature Space Compatibility: Hamming distance is not suitable since features not yet bina- rized during tree construction
The results show cosine similarity's s 1.5-2.0% performance advantage over alternative metrics, con- firming that feature directional alignment is more crucial than absolute distance for tree construction.
C.8 ITERATIVE TREE CONSTRUCTION ANALYSIS
Iterative structural optimization promising direction (Chen al., 2020). Our experiments reveal that static tree construction strategy achieves better balance between efficiency and performance:
Empirical Results: We compared HINT with HINT-ITER (dynamic tree updates every epochs) across multiple datasets:
20

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison between static and iterative tree construction.
Method
MIRFlickr-25K
MS-COCO
Training Time (h)
I-T
T-1
I-T
T-I
HINT
75.5
74.6
61.1
60.8
2.1
HINT-ITER
75.0
74.7
60.5
60.4
2.9
Table 9: Stability analysis of HINT across different code lengths on MIRFlickr-25K dataset. Results show mean MAP scores + standard deviation over five runs.
Task
32 bits
64 bits
96 bits
128 bits
T+T
72.910.81
75.1 74.410099 73.1
75.5 74.0:10.72
T-I 72.010.75
74.6:L0.96
The results show that HINT-ITER achieves comparable but slightly lower performance while requir- ing more training time. Two fundamental reasons explain this phenomenon:
Robust Initial Structure: Our one-time tree construction leverages hierarchical structural entropy recover semantically coherent communities, providing stable foundation for proxy sample generation. Iterative refinement struggles to further improve this already optimized structure.
Stability-Aware Alignment: The curriculum-basee mixup mechanism and consistency learning rely on stable neighborhood relationships to progressively align modalities. Frequent tree updates disrupt this process, similar to how unstable negative samples degrade contrastive learning (He et al., 2020).
While our current approach suits existing tasks, we acknowledge potential benefits dynamic struc- tures for specific scenarios (e.g., evolving data streams), which we leave for future work.
C.9 STABILITY ANALYSIS
To rigorously assess the stability of HINT, we conducted extensive experiments with five indepen- dent using different random seeds. Table 9 presents the mean performance and standard devia- tions across different code lengths for both Image-to-Texx (I-T) and Text-to-lmage (T-+I) retrieval tasks on the dataset.
The results demonstrate that HINT maintains consistent performance with remarkably low vari- across different code lengths. The standard deviations consistently remain below 1% for both retrieval directions, indicating strong robustness to random initialization. This stability can be at- tributed our hierarchical encoding tree structure and curriculum-bust progressive alignment strat- egy, which provide reliable guidance for hash code learning regardless of initialization conditions.
C.10 TAIL CLASS PERFORMANCE ANALYSIS
To evaluate HINT's robustness under class imbalance, we conducted tail class analysis on MIRFlickr-25K dataset. We categorized the 24 classes into three groups based sample counts: Head (8 classes), Medium (10 classes), and Tail (6 classes). As shown Figure 11, HINT demon- strates consistent advantages across all class groups, with particularly significant improvements on Tail classes. This superior performance can be attributed to HINT's hierarchical structure: the en- coding tree leverages cross cross-modal connections to augment sparse intra-modal neighborhoods for tail classes, while the hierarchical community structure ensures these connections remain semantically coherent rather than introducing noise from head classes.
21

---

Published as a conference paper at ICLR 2026
Image Text
Text Image
0.50
HINT DENO
HINT
0.73
0.70
DEMO
UCON
0.78
0.76
0.74
0.74
0.72
0.72
0.70
0.70
0.50
Mend
Medium
Thil
0.60
Head
Medium
Thill
Class Group
Class Group
Figure 11: Performance comparison Head, Medium, and Tail class groups MIRFlick--25K (128-bit). HINT demonstrates consistent advantages across all groups, with more significant im- provements on Tail classes compared baselines.
D ADDITIONAL DISCUSSION
D.1 RATIONALE FOR HIERARCHICAL MINING OF RELATIONSHIPS
Hierarchical semantic structures are inherent in real-world data. Visual and textual content naturally form multi-level conceptual taxonomies. For instance, a general category like "Objects" can be decomposed into "Animals" and "Vehicles". "Animals" can be further subdivided into "Domestic" and "Wild", with "Domestic" containing specific instances like "Cats" and "Dogs" (e.g., "Maine Coon", "Siamese"). Similarly, "Vehicles" might branch into "Ground" and "Air" transport, with "Ground" including "Cars" and "Trucks""
Prevailing unsupervised cross-modal hashing methods often rely flat representations of image- text pair relationships. Such flat structures exhibit limitations in capturing these intrinsic hierarchical dependencies. Specifically, they may:
Treat the semantic dissimilarity between disparate pairs (e.g.. "Cat-Dog" vs. "Cat-Car") undiffer- entiatedly, failing recognize varying degrees of relatedness based hierarchical proximity.
Implicitly assume transitive relationships, an assumption that does not consistently hold for com- plex semantic relationships across differrnnt levels of abstraction.
Struggle to ensure that instances within sub-category (c.g.. "Maine Coon" and "Siamese" under "Cats") are represented being semantically closer each other than instances from distant categories (e.g.. "Cars").
Our proposed HINT addresses these limitations through the Hierarchical Encoding Tree, which explicitly discovers and models inherent semantic hierarchies by optimizing structural entropy. This hierarchical approach offers several advantages:
It facilitates deeper exploration of semantic relationships that extend beyond direct, observed pairings, uncovering latent community structures.
It enables the generation of more generalizable hash codes that are grounded in these discovered semantic communities, rather than isolated instances.
It promotes smoother and more effective cross- moosl aligmment through the of proxy samples derived from semantic neighborhoods and curriculum learning strategy, thereby more effectively bridging the semantic gap berween modalities.
By capturing these multi-level containment relationships, HINT can learn hash codes that better flect real-world semantic structures, leading to improved retrieval performane. Current flat model- ing approaches, by contrast, which treat "Cat-Dog" and "Cat-Car" similarity differences with equal weight, miss this crucial hierarchical semantic information, significantly impeding their ability to learn generalizable hash codes that align with complex real-world semantic organizations.
22

---

Published as a conference paper at ICLR 2026
E ADDITIONAL IMPLEMENTATION DETAILS
E. BASELINE DETAILS
The introduction of the baseline methods is as follows:
CVH (Kumar & Udupa, 2011) transforms the learning process into a manageable caature-based hashing problem.
LSSH (Zhou et al., 2014) is an effective iterative strategy, which explores the correlations between multi-modal representations and bridges the semantic gaps in the latent semantic space.
CMFH (Ding et al., 2016a) exploites modsl decomposition to establish strong connections.
FSH (Liu et al., 2017) alternates optimization to learn consistent binary codes and hash functions.
MTFH (Liu et al., 2019a) produces modality-specifi binary codes with varying lengths while ensuring the comparability of heterogeneous data.
FOMH (Lu et al., 2019) integrates self-weighted and flexible multi-modal fusion strategy, en- abling robust fusion even when missing modalities.
DCH (Xu et al., 2017) utilizes an efficient optimization algorithm to produce the optimized bit recursively.
DGCPN (Yu et al., 2021) is a graph neighborhood approach, which explore the relationships be- tween data points and their neighbors through graph neighborhood approach, thereby enhancing the accuracy of data similarity measurement.
UCHSTM (Tu et al., 2023) employs a custom-designed similarity loss to rectify any ill-defined similarities in the instance similarity matrix.
UCCH (Hu et al., 2022) uses contrastive learning to enforce alignment between different modal- ities and unified binary representations, focusing leveraging discriminative information from all pairs.
UDDH (Zhang et al., 2024a)ís dual deep hashing architecture that combines semantic indexing with content codes for cross-modal retrieval.
HuggingHash+ (Wang et al., 2024b) a transformer- based uulti-granullrity learning framework for unsupervised cross-modal hashing.
DEMO (Zhang et al., 2024b) utilizes multi-view augmentation to represent each image, followed by parameterized distribution divergence to ensure robust similarity structures.
GCRH (Wei al., 2025) is a contrasiveeand--rconsttue
framework combining global graph contrastive learning with local graph reconstruction to reduce the modality gap.
VTM-UCH (Fan & Cao, 2025) utilizes vision-guided text mining by combining vision-language models with community detection for refining hash code distribution.
E.2 DATASET DETAILS
We conduct experiments three widely used public datasets:
MIRFlickr-25K (Huiskes & Lew, 2008) contains 25, 000 text-image pairs. Each text sample is embedded using the Bag-of- -fordd (BoW) strategy.
NUS-WIDE (Rasiwasia et al., 2010) comprises 269, 498 text-image pairs with multiple labels from 81 categories, where each text is embedded into 1000-dimensional vector via BoW.
MS-COCO (Lin et al., 2014) includes 123, 287 text-image pairs with multiple labels from 80 categories. Each text sample is embeded into 2026-dimenli BoW vector.
Following the problem settings of the latest baseline (Zhang et al., 2024b), each dataset is divided into a query set and retrieval set. During the training process, only text-image pair information is accessible without label information.
23

---

Published as a conference paper at ICLR 2026
E.3 IMPLEMENTATION DETAILS
Our method is implemented based on the latest baseline (Zhang et al., 2024b). For training, we main- tain consistency with baseline approaches by selecting 10,000 samples pre-extracted visual and text feature vectors, which are mapped to the Hamming space through two- layer MLPs with a dimension of 512. The implementation is done using Py PyTorch framework, with all experiments conducted on single NVIDIA A40 GPU.
training set. We utilize
E.4 EVALUATION METRICS
Mean Average Precision (MAP) is a comprehensive metric widely used to evaluate retrieval per- formance in cross-modal hashing research (Wang et al., 2010; Liu et al., 2012; Shen et al., 2015). The MAP score has a range of () to 1, where higher values indicate better retrieval performance. It works by calculating the average precision for each query, followed the queries in the test set. This provides a single-figure measure that reflects system performance across all relevant documents and all recall levels. MAP considers both precision and recall aspects of the retrieval system, making particularly suitable for evaluating hashing-based retrieval systems where we are concerned with the overall ranking quality of results.
taking the mean across
THE USE OF LARGE LANGUAGE MODELS
In this article, we use LLM for language polishing and retrieval of the latest research works, We confirm that we take full responsibility for the contents written in this paper.
24


Reviews:
[{"review_id": "OI0YoSQy4A", "reviewer": "Reviewer_qurP", "summary": "This paper proposes a novel unsupervised cross-modal retrieval framework named HINT (Hierarchical Encoding Tree with Modality Mixup). Specifically, HINT constructs a cross-modal encoding tree guided by hierarchical structural entropy, which organizes visual and textual representations into hierarchical communities. Each node in the encoding tree captures local semantic relations, while the overall tree structure preserves global semantic hierarchy. Based on this tree, the method synthesizes proxy samples for both modalities through a modality mixup strategy, enabling progressive alignment via curriculum learning. Experimental results show the proposed method can achieve good results.", "questions": "See the weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1) The manuscript is well organized and easy to follow. The motivation, methodological design, and experimental setup are clearly presented, making the overall contribution understandable and coherent.\n\n2) The proposed HINT framework achieves good and consistent results across multiple cross-modal retrieval benchmarks. \n\n3) The appendix provides detailed additional analyses, including implementation details and supplementary experiments.", "weaknesses": "1) The contribution section uses overly strong and promotional language (e.g., “New Perspective,” “Coherent Framework,” “Outstanding Performance”), which is not fully justified by the presented methodology or experimental evidence. The proposed approach is conceptually sound, but the degree of novelty and improvement appears incremental rather than fundamentally transformative. The authors are encouraged to adopt a more objective tone and support such claims with stronger quantitative and qualitative evidence.\n\n2)  The paper states that connecting the encoding tree with cross-modal hashing offers a new perspective. However, the idea of representing cross-modal relations in a hierarchical structure is not entirely new and has been discussed in previous studies (e.g., [ref1]) on hierarchical representation learning and cross-modal graph encoding. The proposed method mainly extends existing hierarchical modeling techniques rather than introducing a fundamentally different formulation or conceptual insight.\n\n3) The paper only provides a brief time cost analysis in Table 4, 5 , without comparing the retrieval efficiency with existing cross-modal hashing methods (only 3). Moreover, as hashing-based models are typically valued for their efficiency, the absence of parameter-scale or computational complexity comparisons (e.g., model size, FLOPs, or training cost) weakens the empirical completeness of the work. \n\n4) The comparative methods used in the experiments appear to be outdated, with most baselines coming from earlier studies (mainly up to 2023 or before). Recent advances in cross-modal hashing and retrieval from 2025 are not included.\n\n5) The proposed modality mixup simply performs a linear interpolation between same-modality and cross-modality proxy samples. This operation lacks theoretical grounding on why such a linear combination can lead to better cross-modal alignment. Moreover, the parameter λ is only heuristically adjusted during training, and the method does not ensure semantic consistency when mixing features from heterogeneous modalities. As a result, the mixup process may blur modality-specific information rather than truly enhancing cross-modal representation learning.\n\n6) While the paper presents a well-structured framework, its overall novelty appears limited. Most components—such as hierarchical encoding, proxy construction, and mixup-based alignment—are adaptations or recombinations of existing techniques.\n\n\n[ref1]:  Jin W, Zhao Z, Zhang P, et al. Hierarchical cross-modal graph consistency learning for video-text retrieval[C]//Proceedings of the 44th International ACM SIGIR Conference on research and development in information retrieval. 2021: 1114-1124.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o9BrLd5iWu", "reviewer": "Reviewer_kUAR", "summary": "This paper proposes an unsupervised cross-modal hashing method, HINT, for efficient retrieval. It addresses the problem that existing methods often overlook the inherent hierarchical semantic structure of data and face difficulties in directly aligning different modalities. HINT constructs a hierarchical encoding tree guided by structural entropy to capture local semantic communities. It introduces a curriculum-based modality mixup mechanism using proxy samples generated from the tree to achieve progressive modal alignment. It employs a consistency learning objective to align the global semantic distributions between modalities. Experiments on three benchmark datasets demonstrate that HINT outperforms state-of-the-art methods.", "questions": "1. How is the number of neighbors determined for proxy sample construction? \n2. The structural entropy minimization process is greedy. Is the greedy approach practically sufficient to get close to a global optimum?\n3. The curriculum learning schedule seems to be determined automatically. Do you think, for certain tasks, it would be possible to introduce some form of manual control to guide this learning process?\n4. For future work, do you think integrating knowledge from pre-trained vlm into the construction of the hierarchical encoding tree would be a promising direction?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation is clear. The paper accurately identifies a key problem in unsupervised cross-modal hashing: the lack of hierarchical semantic modeling and the difficulty of direct modal alignment.\n2. Using a hierarchical encoding tree to mine the semantic structure of cross-modal data is an insightful contribution.\n3. The use of structural entropy to guide the tree construction is a principled and technically sound approach.\n4. The introduction of proxy samples is a smart way to provide more robust signals for hash learning by smoothing out potential noise from individual samples through leveraging local semantic communities.\n5. The paper is well-structured and easy to follow. The narrative flows logically from problem definition to methodology and experimental validation. The figures are helpful.", "weaknesses": "1. The construction of the hierarchical encoding tree relies on an initial KNN graph, which could be sensitive to noise or data sparsity.\n2. Proxy samples are constructed by averaging neighbors. It might be interesting to discuss whether other aggregation strategies, such as weighted averaging or an attention mechanism, could yield further improvements.\n3. The encoding tree is constructed once before training. While efficient, a discussion on the potential limitations of this static structure when dealing with dynamic or streaming data would be beneficial.\n4. The evolution of the $\\lambda$ value in modality mixup is interesting. A deeper analysis of what factors drive this specific convergence pattern would offer more profound insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYXHWuSIOQ", "reviewer": "Reviewer_Y8zT", "summary": "This paper addresses unsupervised cross-modal hashing retrieval by proposing HINT, which constructs a hierarchical encoding tree guided by structural entropy to mine local semantic communities and overcome the limitations of flat sparse connections in existing methods. The approach consists of three main components: hierarchical encoding tree construction based on structural entropy, curriculum-based modality mixup strategy, and proxy-based consistency learning. Experiments on benchmarks demonstrate that HINT achieves optimal performance. The work connects encoding trees with cross-modal hashing problems, and alleviates the difficulty of direct heterogeneous modality alignment through progressive alignment strategy.", "questions": "1.\tDuring the modality mixup stage, both m same and m cross proxies are generated. Could you explain why the combination of these two proxies is needed, rather than directly using cross-modal neighbors from the encoding tree to generate target hash codes?\n2.\tFor the Text-to-Image retrieval task, the performance improvement is more pronounced compared to Image-to-Text. Could you analyze the reasons for this asymmetry from the method design perspective? Is it related to certain characteristics of text features compared to image features?\n3.\tThe proxy-based design is reminiscent of prototype learning. Could you discuss the similarities and differences between proxies in HINT and prototypes in prototype learning, both conceptually and functionally?\n4.\tRegarding future work, the paper mentions extending to more modalities such as audio and video. What new challenges would encoding tree construction face in multi-modal scenarios?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces structural entropy and encoding tree concepts into cross-modal hashing, providing a fresh perspective on understanding cross-modal relationships from a graph structure viewpoint, which is relatively uncommon in this field.\n2.\tThe curriculum-based modality mixup mechanism is well-designed, dynamically adjusting weights between same-modal and cross-modal features via MMD, embodying an easy-to-hard learning strategy that aligns with cross-modal learning characteristics.\n3.\tThe method achieves consistent performance improvements across three mainstream datasets, demonstrating reasonable generalization capability, particularly strong performance on the more challenging Text-to-Image direction.\n4.\tThe theoretical analysis section proves hash loss convergence to triplet loss, providing theoretical support for the method's effectiveness. This combination of theory and practice is commendable.", "weaknesses": "1.\tThe encoding tree construction relies on initial feature representation quality. The paper uses features extracted from pre-trained models but does not discuss whether structural entropy optimization can still effectively recover hierarchical relationships when semantic structure in the initial feature space is unclear.\n2.\tThe proxy sample construction uses simple feature averaging, assuming neighbors have similar semantics, but semantic similarity among neighbors may vary significantly across different levels of the encoding tree. Have you considered weighted aggregation based on hierarchy or distance?\n3.\tThe encoding tree is constructed statically. While the appendix mentions dynamic update experiments, the explanation for why a static tree suffices is relatively simple.\n4.\tThe paper limits the method to unsupervised scenarios, but partial annotations often exist in practical applications. How HINT could be extended to semi-supervised settings, or how to leverage limited annotation information to improve encoding tree construction, deserves further consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "geyBX81vnt", "reviewer": "Reviewer_LKH2", "summary": "This paper proposes HINT (Hierarchical Encoding Tree with Modality Mixup) for cross-modal hashing. The core contribution of HINT is the construction of a cross-modal encoding tree, guided by hierarchical structural entropy, to recover implicit semantic communities and hierarchical relationships. Based on this tree, the method generates same-modality and cross-modality \"proxy samples\" for each instance. Subsequently, a curriculum-based modality mixup strategy is employed to progressively align these proxy samples, thereby gradually bridging the modality gap. The framework is further enhanced with a cross-modal consistency learning objective to ensure global semantic alignment. Extensive experiments on several standard cross-modal retrieval datasets demonstrate that HINT outperforms current state-of-the-art methods.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper identifies a critical limitation of existing unsupervised cross-modal hashing methods—their reliance on \"flat\" and sparse image-text pair signals, which ignores the hierarchical semantic structures prevalent in real-world data. Introducing hierarchical modeling into this domain is an intuitive and valuable direction, offering inspirational value.\n\n2. The proposed method is methodologically sound. The framework, which integrates hierarchical structure discovery, proxy sample generation, progressive alignment, and global consistency constraints, forms a logically coherent pipeline. Each component is well-defined and works synergistically toward the final objective.", "weaknesses": "1. A core limitation is that the hierarchical encoding tree is constructed only once at the beginning of training and remains static. This process is highly dependent on the quality of the initial features. Sub-optimal or biased features could lead to an erroneous tree, irreversibly compromising the entire subsequent learning process.\n\n2. It is mentioned that The \"Merge\" and \"Compress\" operations are based on “if they can decrease the structural entropy”。Since optimizing structural entropy is an NP-hard problem, this greedy strategy is likely to converge to a local optimum, potentially limiting the quality of the learned hierarchy.\n\n3. In line 183, the description of Eq. (4) mentions terms T_α- and T_α, but these symbols do not appear in the equation itself. The authors should revise this for clarity and consistency.\n\n4.  The generation of proxy samples (Eq. 6) via simple neighbor averaging assumes that local neighborhoods are semantically clean and coherent. However, in real-world data, especially near class boundaries, neighbors may come from different fine-grained categories (e.g., a \"Shepherd dog\" neighboring a \"wolf\"). Averaging these features could lead to \"semantic drift,\" introducing noise rather than robust signals. The paper lacks an analysis of this neighborhood noise.\n\n5. HINT smooths the learning signals by building communities. However, this might also have a side effect: it might \"average out\" those very valuable difficult negative samples (i.e., samples that are semantically close but belong to different classes). For example, in a neighborhood community of an \"Alaskan Husky\" image, there might be an image of an \"Alaskan Malamute\". If they are averaged into a proxy sample, will this weaken the model's ability to learn fine-grained distinctions?\n\n6. Real-world datasets often exhibit significant class imbalance. For tail classes with few samples, the KNN-based neighborhoods can be sparse, unreliable, or even incorrectly connected to head classes. It is unclear whether HINT's tree construction and proxy generation mechanisms would exacerbate or mitigate this problem. The paper does not report on the model's retrieval performance on such tail classes.\n\n7. The paper's readability could be improved by clarifying several points:\n    - The term \"curriculum-based mixup\" is introduced without sufficient explanation or citation on its first appearance.\n    - In line 98, the meaning of \"common\" in \"common visual and text encoders\" is ambiguous and should be specified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)