An End-To-End Graph Attention Network Hashing for Cross-Modal Retrieval
Authors: Huilong Jin
Yingxue Zhang
Lei Shi
Shuang Zhang
Feifei Kou
Jiapeng Yang
Chuangying Zhu
Jia Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: Cross-modal Retrieval
Graph Attention Network
Hash Algorithm
CLIP
Transformer
Primary area: graph_neural_networks

Abstract: Due to its low storage cost and fast search speed, cross-modal retrieval based on hashing has attracted widespread attention and is widely used in real-world applications of social media search. However, most existing hashing methods are often limited by uncomprehensive feature representations and semantic associations, which greatly restricts their performance and applicability in practical applications. To deal with this challenge, in this paper, we propose an end-to-end graph attention network hashing (EGATH) for cross-modal retrieval, which can not only capture direct semantic associations between images and texts but also match semantic content between different modalities. We adopt the contrastive language image pretraining (CLIP) combined with the Transformer to improve understanding and generalization ability in semantic consistency across different data modalities. The classifier based on graph attention network is applied to obtain predicted labels to enhance cross-modal feature representation. We construct hash codes using an optimization strategy and loss function to preserve the semantic information and compactness of the hash code. Comprehensive experiments on the NUS-WIDE, MIRFlickr25K, and MS-COCO benchmark datasets show that our EGATH significantly outperforms against several state-of-the-art methods.

An End-to-End Graph Attention Network Hashing for Cross-Modal Retrieval
Huilong Jin', Yingxue Zhang", Lei Shi Shuang Zhang Feifei Kou
Jiapeng Yang", Chuangying Zhu", Jía Luo"
1 College of Engineering, Hebei Normal University
College of Computer and Cyber Security, Hebei Normal University
State Key Laboratory of Media Convergence and Communication, Communication University of China
'Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education
5School of Computer Science, Beijing University of Posts and Telecommunications Guangxx Key Laboratory of Trusted Software, Guilin University of Electronic Technology Chongging Research Institute, Beijing University of Technology *Correspondence: Leiky_shi@cuu cn, zshuang@hebtu, edu cn
Abstract
Due to its low storage cost and fast search speed, cross-modal retrieval based on hashing has attracted widespread attention and widely used in real-world applications of social media search. However, most existing hashing methods are often limited by uncomprehensive feature representations and semantic asso- ciations, which greatly restricts their performance and applicability in practical applications. To deal with this challenge, in this paper, we propose an end-to-end graph attention network hashing (EGATH) for cross-modal retrieval, which can not only capture direct semantic associations between images and texts but also match semantic content between different modalities. We adopt the contrastive language image pretraining (CLIP) combined with the Transformer to improve understanding and generalization ability in semantic consistency across different data modalities. The classifier based on graph attention network is applied to obtain predicted labels to enhance cross- modal feature representation. We construct hash codes using an optimization strategy and loss function to preserve the semantic information and compactness of the hash code. Comprehensive experiments the NUS-WIDE MIRFlickr25K, and MS-COCO benchmark datasets show that our EGATH significantly outperforms against several state-of-the-an methods.
Introduction
With the rapid development of information technology and social networks, variety of modal data, including text, images, audio, and video, have gradually integrated into our daily lives. Due to the differences in the amount of information and expression of different modalities, how to retrieve information related to another modality one modality has become a research hotspot the field of information retrieval [I] 2 3 Traditional methods rely on manually designed feature extraction and similarity measurements, but it is difficult to adequately capture the complexity and diversity of data when dealing with large-scale tigh-dimensional data. Cross-modal hashing retrieval 4 [ 6 due to its fast retrieval speed and high storage efficiency. has received widespread attention.
The existing tross-mode hashing methods mainly divided into two types: supervised meth- [T) [2] and unsupervised methods [[3] [5)) Unsupervised methods learn models by exploring data structures and distributions without relying labeled data. It has low
ods [ 110
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

retrieval accuracy in complex cross-modal scenarios due to a lack of semantic alignment. In contrast, supervised methods leverage labeled data to establish the associations between different modalities and therefore accurately capture the similarities between different modalities, thus improving the accuracy of retrieval. Despite the wide application of supervised methods in cross-modal retrieval, there are some challenges that still remain. Traditional feature extraction techniques tend to focus on local features, which can hinder the capture of global information in images, ultimately affecting semantic consistency across modalities. Additionally, the high computational complexity associated with constructing label co-occurrence matrices limits their applicability to large-scale labeled datasets. It is also crucial for the hashing module to preserve the semantic information of the original data while maintaining compactness.
To address these challenges, we propose novel supervised cross-modal hashing method based on graph attention network, termed end-to-end graph attention network hashing (EGATH) method. Our approach integrates the contrastive language image pretraining (CLIP) and transformer models to facilitate the global extraction of features from both images and texts. We implement a label classifier module utilizing graph attention networks (GAT) to enrich the semantic depth of feature representations while reducing the computational complexity associated with labeling. In addition, we constructed two functions to control the range of hash code values: cosine similarity function to ensure the compactness and semantic preservation of hash codes, and a ternary loss function to bolster the robustness of hash codes. The main contributions of our work are as follows:
By combining CLIP and transformer technologies. an end-to-end architecture is implemented that significantly improves the model's ability to capture global features of multi modal data, thus ensuring semantic consistency of images and text. Unlike other existing works, where CLIP and transformer are both used for text and image to extract features. We utilize CLIP coupled with transformer to extract features for image data, and feature extraction via a transformer for text, which can realize lightweight network.
Predicted labels are combined with the feature extracted from the feature modules of image net and text net to enhance feature representation. We utilize GAT as label classifier to explore the hidden information in the label to predict labels, which can directly model the label graph to dig the correlation between labels and has higher flexibility than other label classification using preset weights for feature processing.
MIR- Flickr25K, and MS-COCO demonstrating clear performance advantages. Experimental results show that our method outperforms current state-of ofte-offthhe-rt cross- modsl hashing methods.
Our EGATH was evaluated on three widely recognized benchmark datasets:
2 Methodology
2.1 Notations and Problem Definition
In this paper, we introduce the notation used as follows. Suppose we have a training dataset O = (:), n], where x; RRxxd represents the data of the 7th image sample and y. I gixdr represents the data of the 7th text sample. Here, n, dj, and dr represent the number of samples, the dimension of image features, and the dimension of text features, respectively. We use F, Rd, and F, E dR to denote the extracted features of images and texts. Given label set L= { E [1, NT}. where L is a set of labels composed of label I, and i ranges from 1 to N, N represents the number of label categories. The label co-occurrence matrix is constructed based on the frequency of label co-occurrence in the dataset, and after thresholding, the adjacency matrix A obtained. The labelled glove 117) vector W and adjacency matrix A E Rd is constructed as input to GAT, where dj represents the dimension of the labeled feature. This paper aims to obtain compact hash codes B, E (-1,1) 1) and By E (-1, 1) with h representing the length of the hash codes. The tanh(-) function and sign(-) function are used to constrain the values of the hash codes.
2.2 Network Architecture of EGATH
The EGATH framework depicted in Figure encompasses two streams, designed for learning the hash functions for image and text modalities. For image data, we utilize CLIP coupled with a transformer to extract features, while for text data, we initially represent it as bag-of-words (BoW) vector, followed by feature extraction via a transformer. To enhance the discriminative power of the representations for both types of data, we employ classifier module based on GAT to integrate the

---

Clasificaion
ELIPTNI
GAllrameer
Simbartty Leuraing
Labd Sandiiccioo l0s
Wal aod
Title kes
For
Adl New
Optezeztas
Trage
Tandirse Tradie
DWw.
DEw
Trunff
Closiieer
Pilayo
Lest
tite dhooping dlii grea
File Torr
Walline
ORL 1.11
Our proposed EGATH framework consists of three main modules: a feature extraction module, which employs the CLIP and transformer encoders to extract the global features of each modality; 2) a GAT classification module, which uses a graph attention network to dig deeper into the structural and semantic information of the labels; 3) hash code module, which employs two functions and an optimization strategy to generate hash code that preserves semantic information.
Figure 1:
co-occurrence information of labels into the word embeddings. This process, facilitated by dynamic attention mechanism, ensures that the learned word embeddings can reflect the interrelationships between labels, which are then used to predict the labels of instances in different modalities. In this manner, the predicted labels not only enrich the semantic representation of features but also enhance their distinctiveness. Thanks to the capabilities of the GAT classifier module, it effectively bridges the information gap when dealing with and integrating information from two distinct types of data, fostering deeper semantic consistency and enhanced representational power between different modalities. Finally, under the hash code loss function, we obtain hash codes that preserve more relevant semantic information, thereby improving the quality of retrieval.
2.3 Feature Extraction
In the EGATH model, the feature extraction part primarily consists of two components: an image network and a text network.
2.3.1 CLIPTrans
We adopt the CLIP to map images and text into shared vector space to understand the semantic associations between the modalities. The input image is denoted x,, and the specific feature extraction process can be expressed as follows:
fi = CLIP(X;;
While the image features extracted by the CLIP model already contain rich information, their dimensions are not suitable for subsequent processing steps. To resolve this issue, use transformer encoder to perform dimension expansion and alignment on the features. The specific dimension expansion and alignment process is expressed as follows:
QKT
Attention (iciiionnQAKA K. V) softmax
V
(2)
Q S=
= WK
f f
(3)
The terms Wº Rdxds WK Rdddd, and W Rdddd represent the weight matrices that, when multiplied by the feature fI, yield the queries, keys, and values. The self-attention output provides

---

feature representation enriched with contextual information.
(x) max(0, 1 + b1) + b2
(4)
Fy = (FN W output + boutput
(5)
The output of the self-attention layer is processed by feedforward network, where W1, W and W ouiput are the weight matrices for the linear and output layers of the feedforward network, respectively. bi. b2, and Douppur are the corresponding bias terms, and F, is the dimension of the image features after adjustment and alignment.
2.3.2 TransFC
In our text network, we utilize the transformer model to extract textual features, resulting in high- dimensional feature representation denoted by fr. Given the high dimensionality the extracted features, a fully connected layer is introduced to adjust the feature dimensions appropriately, thereby enhancing the model's capability to capture complex features and patterns.
2.4 GAT Classifier
In this section, we will delve into the label classifier module based on GAT. GAT incorporates co- occurrence information of labels into word embeddings and uses the learned embeddings to predict labels across modalities. The predicted labels are further combined with feature representations to enhance their discriminative properties.
We construct an adjacency matrix based on the co-occurrence frequency of labels. Firstly, we analyze the number of times label pairs appear in the training set to obtain the matrix C, where Ciş denotes the number of occurrences of label and label, j together. Initially, we sum across the rows or columns of the co-occurrence matrix, and the marginal probability of label i can be represented as:
N
Ci
where N; is the total occurrence of label i, including its co-occurrence with all other labels, then the conditional probability matrix D calculated, where Dij represents the probability of label j occurring given label has occurred. The specific conditional probability can be expressed as follows:
Cjj N
D
We set threshold Ç to eliminate those marginal connections that may caused only by noise. The main operation is as follows:
A. 0, if Dij <<
(8)
otherwise
According to Aij in it is known that the set of neighbouring points j of data point is (()) (j
Aij 7 0, i S n). Inspired by the work [19], For each label pair (i. j) calculate the attention coefficient is expressed as follows:
exp (LeakyyyeeLJUU (aT [Wg: Wg:l))
= ExEN(O) exp (LeakyReLU (a" [Wg: || Wgx]))
where W a learnable weight matrix, a the attention vector that is used compute the correlation between pairs of labels, g: and g, denotes the glove vector of label i and
Using the computed attention coefficients, the feature vectors of the labels are weighted and summed to update the features of the labels, which are represented as follows:
H Σ ai Wg,
(10)
JEN(()
It is known from GAT that the output of the feature representation of the last layer is H,,, S, where the predicted scores of the labels are represented as follows:
S, H
(11)

---

where represents I or T, f. represents the representation of a single image text, and the multi classification loss can be expressed as follows:
Lisss Σ (s + +1 S.) log(1 oS.)))
(12)
where N represents the number of labels in the sample, summing over all N labels to calculate the total loss of the sample, S represents the true value of the sample labels, (((( the predicted score after applying the sigmoid function, mapping the original score to the (0,1) interval, log is used compute the cross-entropy, sending the predicted label results the hash layer for the next operation.
2.5 Hash Code Module
In this section, delve deeper into the hash code module in the model. The primary function of this module is to map continuous features to discrete hash codes. This module achieves effective encoding of multimodal features and labels by combining the tanh(-) and sign(-) functions, as well as a strategy that combines triplet loss with cosine similarity.
2.5.1 Hash Function
During the training process, the phenomena of gradient vanishing and unbalanced distribution of hash codes can occur. We introduce a tanh(-) function after the hash coding layer, which effectively compress the input values into the range of 11 to Meanwhile, tanh(-) provides relatively smooth gradient information, which is conducive to the updating and optimization of network parameters during the training process, defined as follows:
B, tanh(W, 7 + by)
(13)
By = tanh W T by:)
(14)
where W, and WT are the weights for mapping image and text features to the hash space, and by and by are the bias terms. Although the output of the hash layer is compressed to certain range, remains continuous binary values. In order to obtain discrete binary hash codes for retrieval, the sign()) threshold function is applied to convert the continuous values into binary values, generating the final required hash code. The sign(.) function is defined as sign(x), and the following the mathematical representation involved in the hash code module:
{
if 0
sign(x)
(15)
if <
where represents the values of By and Br. The processed hash codes are distinguished into positive and negative values, establishing a consistent hash space across different modalities of data, thereby obtaining discrete binary hash codes.
2.5.2 Loss Function
We use combination of triplet loss 20 and cosine similarity to optimize the encoding process of the hash code. enabling the model to generate more accurate hash codes. Specifically, the cosine similarity is used to calculate the similarity between two hash vectors, and then the triplet loss function is used guide the model training so that the cosine similarity of the positive sample pair as close to as possible, while the cosine similarity of the negative sample pair is less than a preset threshold, with the cosine similarity defined as follows:
(a. b)
sim b)
(16)
|a|]
|b]
where, and b represent the hash code vectors of images and texts, respectively, with denoting the dot product, and ||a]], [[b]] representing the norm of the hash code vectors. The triplet loss function is designed with three sample points: an anchor (Anchor) a negative sample (Negative) n. The anchor and the positive sample are highly similar, while the negative sample is dissimilar to the anchor. The main purpose of the triplet loss is to bring the anchor
positive sample (Positive) p. and
5

---

and positive sample closer together while pushing the anchor and negative sample further apart, thereby preserving similarity in the representation space. The specific loss function is expressed as follows:
L = [[f(a) f(p)]l2
(17)
L2 = [[f(a) f(n)]l²
(18)
Ceii max(L; (L= - a.
(19)
Among them L₁ represents the squared distance calculated between the anchor a and the positíve
sample p. while Ly represents the squared distance calculated between the anchor a and the negative
sample n. The parameter is used to specify the minimum distance difference between the positive and negative samples.
2.6 Optimization
In this section, we discuss the optimization problem. mainly focusing on the optimization of hash codes and the program. We adopt the approach of GCDH [2T] for iterative optimization of hash codes, introducing auxiliary variables R. Q, P. and B. Here, R represents the mapping from the feature space to the hash code space, mapping high-dimensional feature vectors to low-dimensional hash codes. P represents the target hash code with constraints, Q is used to ensure consistency between the target hash code and the current hash code, balancing the relationship between the two, and B represents the current hash code.
Hash learning is typically non-convex optimization problem that cannot be solved for global optimum in a single step. Therefore, it requires an iterative approach to gradually approximate the optimal solution. Initially, based on the current hash encoding B, optimization from features hash code mapping is performed to find more suitable hash function. The update strategy is as follows:
R = (L L XI) LL
(20)
The update of R may cause a deviation between the hash representation of the data and the label, therefore, it is necessary to update B again to correct this deviation:
B sign [LR + TPP Q/7) + (aB - BT))
(21)
P is typically updated to better meet the constraints that B should adhere to. It can be viewed as an ideal state, and the hash code B should be as close as possible to this state. B is usually derived from the singular value decomposition of the hash code, so first, B is subjected to singular value decomposition, expressed as follows:
B = UNVT
(22)
where and V are orthogonal matrices, containing the left and right singular vectors, respectively. We use the vector data from these matrices to update P. with the main construction expressed as follows:
P= VT
(23)
where Up and Vp are the columns selected from U and V that correspond to the largest singular values, and p is the diagonal matrix that includes the selected largest singular values. Q is introduced to gradually correct the deviation between B and P. When B equals P, the update of Q is shown as follows:
Q Q= Q + (B
(24)
Throughout the optimization process, the updates of P and Q are carried out alternately. First, P is updated reflect the constraints on the hash code B, then Q is adjusted correct the deviation from the target. This alternating update strategy enables the model to find the optimal hash code while satisfying constraints.
3 Experiments
In this section, we evaluate the performance of the proposed EGATH framework on three benchmark datasets using state-of-the-art methods. Then, we present a detailed discussion on ablation studies to investigate each component of our model.

---

Table 1: Experimental results on the three datasets.
MIRFlickr25K
NUS-WIDE
MS-COCO
Task
Method
16bit
32bit
64bit
16bit
32bit
64bit
16bit
32bit
64bit
DISRH
0.6632
0.6873
0.6987
0.5271
0.5582
0.6015
0.5257
0,5454
0,5646
JDSH
0.7276
0.7426
0.7468
0.6536
0.6601
0.6900
0.5928
0.6348
0.6517
DCHMT
0.8177
0.8221
0.8261
0.6711
0.6812
0.6932
0.6450
0.6331
0.6647
MLCAH
0.7960
0.8080
0.8150
0.6440
0.6410
0.6430
0.5700
0.5620
0.5620
I-T
CDTH
0.7317
0.7461
0.7477
0.6596
0.6613
0.6700
0,5853
0.6411
0.6573
UCCH
0.7606
0.7620
0.7674
0.6718
0.6738
0.6891
0.6039
0.6249
0.6398
SCAHN
0.8123
0.8131
0.8336
0.6588
0.6621
0.6669
0.6727
0.7108
0.7528
DSPH
0.7935
0.8141
0.8363
0.6851
0.6996
0.7092
0.6466
0.6666
0.6757
EGATH
0.8411
0,8562
0.8637
0.7191
0.7345
0.7480
0.7259
0.7688
0.7945
DISRH
0,6710
0.6958
0.7043
0.5575
0.5680
0.5952
0.5590
0.5591
0,5519
JDSH
0.7304
0.7326
0.7481
0.6439
0.6640
0.6921
0.5888
0.6510
0.6635
DCHMT
0.8007
0.8021
0.8065
0.6852
0.6963
0.7009
0.6298
0.6176
0.6616
MI.AHH
0.7940
0.8050
0.8050
0.6620
0.6730
0.6870
0.5440
0.5470
0.5940
T-1
CDTH
0.7315
0.7464
0.7503
0.6788
0.6815
0.6910
0.5846
0.6427
0.6573
UCCH
0.7343
0.7342
0.7410
0.6740
0.6812
0.6945
0.6023
0.6258
0.6371
SCAHN
0.7890
0.7971
0.8186
0.6718
0.6803
0.6980
0.7183
0.7504
0.8093
DSPH
0.7928
0.8038
0.8149
0.6957
0.7147
0.7256
0.6473
0.6656
0.6774
EGATH
0.8064
0.8185
0.8293
0.7270
0.7437
0.7539
0.7247
0.7729
0,8015
3.1 Datasets
To evaluate the proposed methods, conducted experiments three widely used cross-modal datasets, namely, MiRFlickr25K (22]. NUS-WIDE [23], and MS-COCO [24]. The specific division of these datasets is outlined in Appendix B.
3.2 Implementation Details
In this paper, we use CLIP and transformer as feature extractors for images, transformer and Fully Connected Layer as feature extractors for text, and GAT as the label classifier bridge the information gap between modalities. The input for GAT is the 300-dimensional glove vectors pre-trained on the Wikipedia dataset and the adjacency matrix between labels. We carefully set some hyperparameters a,3, and k for auxiliary learning. Through experimental analysis, we examined the sensitivity of these parameters. Finally, we set the batch size to 64 and employed the Adam [25] optimization strategy for the main optimization, with weight decay set to 0.0005 and adopting a method of dynamically adjusting the learning rate.
Our EGATH method is implemented in Pytorch (26,, and the experiments were conducted on server equipped with an NVIDIA GeForce RTX 3080 with 40GB of RAM.
3.3 Baselines and Evaluation Metrics
There are two cross modal hash retrieval tasks in this article (from text image, from image to text), and we compared our method with eight advanced cross modal hash methods: mainly about
DCHMT 277, JDSH (28). MLCAH (2). UCCH [30]. DSPH (II, CDTH (32). SCAHN [33 and DISRH 34 To ensure fairness, we reran the code for the comparative experiments. For papers that
did not provide code, we directly used the results reported in the papers, ensuring consistent dataset splits. In this study, we used three evaluation metrics to measure the performance of retrieval: mean average precision (mAP), the precision- curve). Detailed descriptions of these metrics are shown in Appendix C.
curve (PR curve). and top-K K Precision curve (top-K
7

---

1-1 MURESARAA Sahats
1-T @ Mi biitts
T-1 0F MIRIIEZAK Schtts
T-1 6 MIRFSCIRK bdbrts
0.9
1041
Phoddoo
1003
Poedioo
0.6
0.6
0.6
02
04
1.0
Rocall
Tocall
Recall
Toxiil
1-T NUSSIDE Schits
64h05
NUSWIDE 32hits
T- NUSWIDE GAbtts
1.0
1.0
1.0
1.0
0.9
Rom
0.0
0.9
0.0
088
08
018
Presimm
Trecskon
07
0.6
066
06
066
0.5
055
0.3
03-
0.3
60
022
014
0.6
005
1.0
00
012
0.8
00.5
1,0
00
02
014
010
1.0
00
02
014
116
08
1.0
Focall
Perall
Recall
Fazall
[-T @ MS00CO 37bis
|-T MSCOCO 64bbis
T- (8) MSC000 325
T-1 @ MS00CC 6tb0s
1.0
10.
103
0.
09
0.5
0.9
085
Proziem
0.8
Pmmtsoe
08
0.6
0.7
07
00
0.4
044
0.4
0.4
00
02
0.4
0.6
08
100
0.0
0.6
0.5
1.0
0.0
02
0.4
0.6
01.0
1.0
00
02
04
0.6
08
1.0
Rocill
Rocall
Honill
Romil
Figure 2: PR curves on the three datasets
3.4 Performance Comparison
To verify the effectiveness of the proposed EGATH method, we conducted performance comparisons on three datasets. All results are shown in Table where "I T" indicates image-to-text retrieval, and "T + I" represents text-to-image retrieval.
The optimal results from the experiment are shown in bold. Compared to the results of DSPH and DCHMT, especially the effect on MSCOCO dataset has significant enhancement, which is due to the following reasons. The MSCOCO dataset with rich label information provides more contextual information for GAT, which effectively improves the accuracy of label prediction and the consistency of cross-modal features, and indicates that our method has special advantage in dealing with data-rich datasets with diverse labels. For the coco dataset in SCAHN 64bit is higher than our results, which due to the fact that SCAHN uses cross-media and intra-modal contrast against hashing mechanisms to enhance the differentiation of different modal semantic representations. Therefore, it achieves better performance in multi-label type datasets.
We introduce the PR curve in Figure 2 It can be observed that our model achieves high precision scores all recall levels. This outperformance is due the fact that EGATH significantly improves semantic discriminability between modalities and modsl consistency through feature networks, which enables the model more accurately identify and match relevant items in cross-modal data. In addition, GAT is used in EGATH to deeply mine the latent semantic information in the labels, which further enhances the model's performance in the feature representation and matching process.
Figure [3 demonstrates the performance of EGATH with different Top-K retrieval results. The K values include 50, 100, 200, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, and 5000, and these experimental data mainly cover small-to large-scale retrieval tasks to evaluate the effectiveness of different methods under various retrieval scales. BGATH is consistently maintains high accuracy rate under any top-K ordering. The trend in the figure shows that EGATH tends to be stable under different data sizes and K values. By combining aspects such as semantic matching and feature representation between different modalities, thus performing better in large-scale retrieval.
3.5 Ablation Study
To validate the effectiivness of our contributions, designed two ablation experiments as follows:
8

---

I-T (8 MIRFI 3006
1-1 O1 SURTIIEZEK 0the
T-1 MERRIWAE 37Nts
T-1 O1 MIRUade22K Ciarts
040
090
082
030
Prosden
2001
0,00
Predsio
040
0.00
070
0.75
0.70
020
0100
0/0
1000
2000
300
1100
1000
1000
2000
3000
4000
5000
1100
2000
3001
(000
2011
3000
2000
3000
4000
5000
top-K
1xp-K
trp.k
1-2 02 NUSWIDE 12hats
[-[ NUSWIDI Gibets
T-1 @ NUSWIDEE Schtss
T-1 @8 NUSWIDE 0124ts
0.75
0:00
0270
005
Reddon
0075
0070
mmm
600
6.0
0.00
0120
0.00
050
0150
050
3000
4000
2000
2000
3000
5000
Xxp-X
ugg.
kop-K
txp-X
1-T M6C000
MSC0CO 17N1
0.00
0.00
0.5
0.55
020)
0.5
0.0
0.5
0.80
0130
1991
0.00
10m
Prestgo
0000
0.00
0.00
020
060
030
6.0
0.5
1000
100
1000
1000
0.50
1000
2000
5000
2000
3000
1000
1000
100
txp-K
tnp-K
1xp-K
trp-K
Figure 3: Top-K curves the three datasets
This experiment removes the GAT classifier module, with all other modules being the same as in EGATH.
EGATH-G:
EGATH-T: This experiment removes the triplet loss function based on cosine similarity, with all other modules being the same as EGATH.
The main comparative results are presented in Table 2 The performance of the EGATH-G model notably poor when handling imbalanced and sparse label data, underscoring the importance of the graph attention module. In the EGATH-T model, despite the absence of the hash code optimization strategy, the model retains some performance. However, compared to the complete EGATH model, its performance declines significantly, particularly under complex and diverse query conditions. The inclusion of the graph attention module effectively addresses label data challenges and enhances overall retrieval performance. Furthermore, the hash code optimization strategy contributes to generating more robust hash codes, which is crucial for improving retrieval accuracy and efficiency. The results from the ablation experiments clearly demonstrate that each component positively influences the overall performance.
In summary, these ablation experiments not only validate the effectiveness of the graph attention module and hash code optimization strategy but also showcase their significant roles in improving the performance of cross- cross-modal retrieval systems.
Table Ablation study on the three datasets.
FIKKlick25KK
NUS- HUS-WIDE
MS-COCO
Task
Method
16bit
32bit
64bit
16bit
32bit
64bit
16bit
32bit
64bit
EGATH-G
0.8145
0.8354
0.8464
0.7072
0.7294
0.7429
0.6937
0.7333
0.7583
T
EGATH-T
0.8348
0.8521
0.8622
0.7135
0.7297
0.7435
0.7029
0.7438
0.7736
EGATH
0.8411
0.8562
0.8637
0.7191
0.7345
0.7480
0.7259
0.7688
0.7945
BGATH-G
0.7965
0,8102
0.8184
0.7163
0.7386
0.7477
0.6991
0.7372
0.7649
T EGATH-T
0.7990
0.8123
0.8224
0.7165
0.7390
0.7483
0.7058
0.7494
0.7785
EGATH
0.8064
0.8185
0.8293
0.7270
0.7437
0.7539
0.7247
0.7729
0.8015

---

1-1
T-1
[--7
T-xl
0.00
0220
0.76
077
0.70
0.74
12:00
0250
0.72
C2 UL 046
05 00 65 3M
01 O2 02
066 00 07
0.31
01
03
000 01
120
Figure 4: The influence of haper-parameterr
3.6 Parameter Sensitivity
In this study, we set several parameters for experimental optimization: a=0.3, ==0.5. To analyze these parameters, we chose validate three datasets using uniform 64-bit hash code. When adjusting the values of different parameters, the results of the mAP also fluctuate, confirming the rationality of our parameters for the model. The specific results are shown in Figure
From this figure, we can see that the performance of cross cross-modal search improves with the increase of a. When a is equal 0.3, the optimal results achieved for the mAP values all three datasets, so we set the value of a to 0.3. When J increases from 0.01 to 0.5, the performance metrics gradually improve, indicating that y helpful for the regularisation effect. However, when y is increased the performance starts to deteriorate, probably because the regularisation is too strong, which limits the learning ability of the model, so we the value of j to 0.5.
4 Conclusion
This paper proposes EGATH, which uses the CLIP model and transformer architecture to extract features from multimodal data, enhancing the semantic consistency between different modalities in an end-to-end manner. Furthermore, by integrating GAT, we strengthen the model's understanding of data structure of different modalities, treating them as a set of interdependent object classifiers to deeply mine and learn the intrinsic graph structure features of the data. Labels are processed as word embeddings, and the predicted labels are obtained from these classifiers, enhancing cross-modal feature representation. We delve into the hidden features and relationships within the graph structure, achieving effective encoding and representation of data, and comprehensive experiments demonstrate the effectiveness of this method. Future work could explore the extension of EGATH to other types of multimodal data, such as audio and text, and investigate its scalability with larger datasets. Notably, this method is limited by the high computational complexity of the graph attention network, which constrains the model's scalability under large-scale label sets. Although cross-mode hashing retrieval enhances the convenience of information access and the potential for cross-cultural exchange, also brings risks of privacy breaches and bias amplification. Therefore, strengthening data security and fairness management essential ensure its positive societal impact.
Acknowledgments
This work is supported by the Science and Technology Project of Hebei Education Department (No. CXY2024050), Fundamental Research Funds for the Central Universities CO0,2033RO08), the National Natural Science Foundation of China (No. U22B2028, No.72104016) the 8th Young Elite Scientists Sponsorship Program by CAST 200.20222NR00001 Open Research Subject of State Key Laboratory of Intelligent Game (No. 24K22)--24)2 the Natural Science Foundation of Chongqing, China (No. C...B2222NNSC- MSX0391), the Foundation of Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education (N0.EII202240006), and the Beijing Natural Science Foundation (No.9242003).
Guangxi Key Laboratory of Trusted Software (No.KX202306
10

---

References
[1] Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2251-2260, 2020.
[2] Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maillet Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17 (4):1-23, 2021.
[3] Feifei Zhang, Mingliang Xu, Qirong Man, and Changsheng Xu. Joint attribute manipulation and modality alignment learning for composing text and image to image retrieval. In Proceedings of the 28th ACM international conference on multimedia, pages 3367-3376, 2020.
[4] Kaiyi Luo, Chao Zhang, Huaxiong Li, Xiuyi Jia, and Chunlin Chen. Adaptive marginalized semantic hashing for unpaired cross-modal retrieval. IEEE Transactions on Multimedia, 2023.
[5] Junpeng Tan, Zhijing Yang. Jielin Ye, Ruihan Chen, Yongqiang Cheng, Jinghui Qin, and Yongfeng Chen. Cross-modal hash retrieval based on semantic multiple similarity learning and interactive projection matrix learning. Information Sciences, 149:11,,71, 2023.
[6] Jian Zhang, Yuxin Peng, and Mingkuan Yuan. Sch-gan: Semi-supervised cross- modal hashing by generative adversarial network. IEEE transactions on cybernetics, 50(2):489-500 502, 2018.
[7] Yongxin Wang, Wei Zhan, Zhen-Duo Chen, Xin Luo, and Xin-Shun Xu. Multiple information embedded hashing for large-scale cross-moda retrieval. IEEE Transactions Circuits and Systems for Video Technology, 2023.
[8] Zijia Lin, Guiguang Ding, Mingqine Hu, and Jianmin Wang. Semsanies-preserinng hashing for cross-view retrieval. In Proceedings the IEEE conference computer vision and pattern recognition, pages 3864-3872, 2015.
[9] Dongging Zhang and Wu-Jun Li. Large-scale supervised multimodal hashing with semantic correlation maximization. In Proceedings of the AAAI conference on artificial intelligence, volume 28, 2014.
[10] Jile Zhou, Guiguang Ding, and Yuchen Guo. Latent semantic sparse hashing for cross-modal similarity search. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 415-424, 2014.
[11] Xin Liu, Zhikai Hu, Haibin Ling, and Yiu-ming Cheung. Mtfh: A matrix tri-factorization hashing frame- work for efficient cross-modal retrieval. IEEE transactions puttern analysis and muchine intelligence, 43(3):964-99) 98166499911111 2019.
[12] Donglin Zhang, Xiao-Jun Wu, Tianyang Xu, and He-Feng Yin. Dah: Discrete asymmetric hashing for efficient cross-media retrieval. IEFF Transactions on Knowledge and Data Engineering, 1365)81365-1378, 2021.
[13] Tuan Hoang, Thanh-Tome Do, Tam V Nguyen, and Ngai-Man Cheung. Unsupervised deep cross-modality spectral hashing. IEEE Transactions Image Processing, 29:8391-8406 2020.
[14] Rong-Cheng Tu, Jie Jiang, Qinghong Lin, Chengfei Cai, Shangxuan Tian, Hongfa Wang, and Wei Liu. Unsupervised cross-modal hashing with inttalit-itteracctoon IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[15] Peng-Fei Zhang, Yang Li, Zi Huang, and Xin-Shun Xu. Aggregation-based graph convolutional hashing for unsupervised cross- cross-modal retrieval. IEEE Transactions on Multimedia, 04:466-479, 2021.
[16] Hong Liu. Rongrong Ji. Yongjian Wu, Feiyue Huang, and Baochang Zhang. Cross-modality binary code learning via fusion similarity hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7380-7388, 2017.
[17] Jeffrey Pennington. Richard Socher. and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532- 1543, 2014.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.

---

[19] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1090: 2017.
[20] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815 823, 2015.
[21] Cong Bai, Chao Zeng, Qing Ma, and Jinglin Zhang, Graph convolutional network discrete hashing for cross-modal retrieval. IEEE Trunsactions on Neurul Networks and Learning Systems, 2022.
[22] Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In Proceedings the 1st ACM international conference Multimedia information retrieval, pages 39-43, 2008.
[23] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojic Li, Zhiping Luo, and Yantao Zheng. Nus-wide: real-world web image database from national university of singapore. Proceedings of the ACM international conference on image and video retrieval, pages 1-9, 2009.
[24] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[25] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[27] Junfeng Tu, Xueliang Liu, Zongxiang Lin, Richang Hong, and Meng Wang. Differentiable cross-modal hashing via multimodal transformers. In Proceedings of the 30th ACM International Conference on Multimedia, pages 453-461, 2022.
[28] Song Liu, Shengsheng Qian, Yang Guan, Jiawei Zhan, and Long Ying. Joint-modal modal distribution based similarity hashing for large-scale unsupervised deep cross-modal retrieval. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 1379- 1388, 2020.
[29] Xinhong Ma, Tianzhu Zhang, and Changsheng Xu, Multi-level correlation adversarial hashing for cross- modal retricval. IEEE Transactions on Multimedia, 22(12):300131144 2020.
[30] Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin- -ing Zhao, and Xi Peng. Unsupervised contrastive cross-modal hashing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3877-3889, 2022.
[31] Yadong Huo, Qibing Qin, Jiangyan Dai, Lei Wang, Wenfeng Zhang, Lei Huang, and Chengduan Wang. Deep semantic-aware proxy hashing for multi-label cross- modal retricval. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[32] Fengling Li. Bowen Wang. Lei Zhu, Jingjing Li, Zheng Zhang, and Xiaojun Chang. Cross-domain transfer hashing for efficient cross-modal retrieval. IEEE Transactions Circuits and Systems for Video Technology, 2024.
[33] Mciyu Liang, Yawen Li, Yang Yu, Xiaowen Cao, Zhe Xue, Ang Li, and Kangkang Lu. Structures aware fine rreegrained contrastive adversarial hashing for cross- media retrieval. JEEE Transactions on Knowledge and Data Engineering, 2024.
[34] Shupeng Su, Zhisheng Zhong, and Chao Zhang. Deep joint-semantics reconstructing hashing for large- scale unsupervised cross-modal retrieval. In Proceedings of the IEFF//VF international conference on computer vision, pages 3027-3035, 2019.
[35] Lei Shi, Jiapeng Yang, Pengtao Lv, Lu Yuan, Feifei Kou, Jia Luo, and Mingying Xu. Self-derived knowledge graph contrastive lcarning for recommendation. In ACM Multimedia 2024.
[36] Yongxin Wang. Xin Luo, Liqiang Nie, Jingkuan Song, Wei Zhang, and Sin-Shun 2 Xu. Batch: A scalable asymmetric discrete cross mooss-modal hashing. IEEE Transactions on Knowledge and Data Engineering. 33 (11):3507-3519, 2020.
12

---

[37] Lei Shi, Junping Du, Gang Cheng, Xia Liu, Zenggang Xiong, and Jia Luo. Cross-media search method based complementary attention and generative adversarial network for social networks. International Journal of Intelligent Systems, 37(8):4393-446, 2022.
[38] Qing- Yuan Jiang and Wu-Jun Li. Deep modsl hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3232-3240, 2017.
[39] Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, and Dacheng Tao. Self-supervised adversarial hashing networks for cross- moodll retrieval. Proceedings of IEEE conference on computer vision and pattern recognition, pages 4242 4251, 2018.
[40] Guiguang Ding, Yuchen Guo, and Jile Zhou. Collective matrix factorization hashing for multimodal data. In Proceeding of the IEEE conference on computer vision and puttern recognition, pages 2075-2082, 2014.
[41] Shaishav Kumar and Raghavendra Udupa. Learning hash functions for cross- cross-view similarity search. In Twenty-secone international joint conference on artificial intelligence, 2011.
[42] Lei Shi, Guangjia Song, Gang Cheng, and Xia Liu. A user-based aggregation topic model for understanding user's preference and intention in social network. Neurocomputing, 413:1 13, 2020.
[43] Wei Chen, Yuxuan Liu, Zhao Zhang, Fuzhen Zhuang. and Jiang Zhong. Modeling adaptive inter-task feature interactions via sentiment-aware contrastive learning for joint aspect-sentiment prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 17781-17789, 2024.
[44] Wei Chen, Yiqing Wu, Zhao Zhang, Fuzhen Zhuang, Zhongshi He. Ruobing Xie, and Feng Xia. Fairgap: Fairness- aware recommendation via generating counterfactual graph. ACM Transactions Information Systems, 42(4):1 25, 2024.
[45] Lei Shi, Jia Luo, Chuangying Zhu, Feifei Kou, Gang Cheng, and Xia Liu. A survey media search based user intention understanding social networks. Information Fusion, 91:566 -81566581,,,6 2023.
[46] Yongbiao Chen, Sheng Zhang, Fangxin Liu, Zhigang Chang, Mang Ye, and Zhengwei Qi. Transhash: Transformer- based hamming hashing efficient image retrieval, Proceedings of the 2022 international conference on multimedia retrieval, pages 127-136, 2022.
[47] Jiaxing Li, Wai Keung Wong, Lin Jiang, Xiaozhao Fang, Shengli Xie, and Yong Xu. Ckdh: Clip- based knowledge distillation hashing for cross- modal retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[48] Xinyu Xia, Guohua Dong, Fengling Li, Lci Zhu, and Xiaomin Ying. When clip meets cross-mode hashing retrieval: A new strong baseline. Information Fusion, 1019101,,, 2023.
[49] Heng Yu, Shuyan Ding, Lunbo Li, and Jiexin Wu. Self-attentive clip hashing for unsupervised cross-modal retrieval. In Proceedings of the 4th ACM International Conference on Multimedia in Asia, pages 1-7, 2022.
13

---

Appendix / supplemental material
A
Related Work
In this section, we introduce existing work related to our research, focusing primarily on the two forms of cross-modal hash retrieval.
A.1 Supervised Cross-Modal Hashing
Supervised hashing methods 8 9 10 12 35 mainly utilize the semantic associations between diffcrent modalities and manually annotated label information to learn a function that can be mapped to common hash space, enabling effective similarity search of data from different modalities in this hash space. Among these, semantic correlation maximization (SCM) p is a supervised multimodal hash method that uses all supervision information for training. generates more distinctive hash codes by learning hash functions bit by bit and achieves high performance. Semantic preserving hashing (SePH) ( is a supervised cross- view hashing method that mainly uses the semantic similarity of training data as supervision information and converts it into a probability distribution. Then, it approximates the semantic similarity between training data and the hash codes he learned by minimizing the KI. divergence. Batch learning of asymmetric discrete cross-modal hashing (BATCH) 36 uses collective matrix factorization learn the common latent space of labels and different modalities and establishes a connection between the common latent space and hash codes with an asymmetric strategy by minimizing the distance-distange difference problem.
Although the above methods have achieved good results, with the integration of deep neural networks, many supervised deep moosl hashing methods have also achieved good performance (37]. Deep cross-modal hashing (DCMH) (38 proposes an end-to-end learning framework that integrates fcature learning and hash code learning into one framework, enabling direct learning of discrete hash codes without relaxation. Self-supervised adversarial hashing (SSAH) (39] is an early attempt to incorporate adversarial learning into cross modal hashing. It uses two adversarial networks to maximize the relevance between different modalities and employs self-supervised semantic network to discover high-level semantic information.
A.2 Unsupervised Cross-Modal Hashing
Regarding unsupervised cross-modal hashing, this method does not rely on manually annotated data. Instead, it learns the mapping relationships between different modalities by analyzing the intrinsic structure of the data, mapping data from different modalities common Hamming space. The goal is bring similar samples closer together the space, thereby improving retrieval performance. Collective matrix factorization hashing (CMFH) (40) uses collective matrix decomposition on data from different modalities of the same instance and adopts a latent factor model to learn unified hash codes. Cross-vier -iiss hashing (CVH) [4] generates common hash codes hased on the intra-modal and inter-modal similarities of multimodal data, effectively solving cross-view search problems.
Beyond the aforementioned shallow methods, the introduction of deep learning technologies has brought new Unsupervised contrastive hashing (UCCH) 300 introduces contrastive learning into hash retrieval and narrows the gap between contrastive learning and hashing with a new momentum optimizer. Transformer- based hamming hashing for efficient image retrieval (TransHash) [46] is the first method address the deep hash learning problem without the support of convolutional neural networks (CNNs), using dual- strl-mtream framework based the vision transformer (ViT) for extracting features from images and texts, and utilizes dynamic similarity matrices to learn compact binary codes. With the introduction of the CLIP model, have begun using this technology for extracting features from images and texts.
possibilities for retrieval[22] [
4+
more methods 47
[88 49
Table 3: Partition of Datasets
Dataset
MIRFIickr25K
NUS-WIDE
MS-COCO
Size
25,000
269,648
123,289
Label
24
81
80
Query
2000
2100
5000
Train
10000
10500
10000
Text
1386D
1000D
2026D
B Datasets
As shown in Table
the statistics of each dataset used in paper.
14

---

MIRFlickr25K is dataset that contains 25,000 images, each with multiple text lags. In addition, each image is annotated with at least one of 24 categories. In experiments, selected only those image-text pairs that have least 20 tags as our experimental data.
consists of 269,648 web images with text tags, cach annotated with least one of 81 categories. We selected 186,577 image-text pairs belonging to the ten most common categories.
MS-COCO includes 123,289 images, each annotated with least one of 80 categories. In our experiments, samples without instances in the text samples were removed.
C Evaluation Metries
Mean average precision (mAP)) A standard criterion to evaluate retrieval accuracy. For each query. the average precision (AP) is first calculated, which is the average of precision values at different recall lovels. Then, by averaging the AP values across all queries, the mAP value obtained, reflecting the specific performance of the retrieval.
Precision-recet curve (PR curve): method that measures the performance of model different threshold settings, providing comprehensive view model's performance across various thresholds, and precisely depicting the relationship between precision and recall.
Top-K precision curve (top-K curve): This metric aims to evaluate the accuracy of retrieval in the top N results, to measure the effect of retrieval among different numbers of samples.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the main contributions of the paper, proposing novel moosllmddal hashing retrieval method aimed at enhancing semantic association follows: We adopt the contrastive language image pretraining (CLIP) combined with the transformer to improve understanding and generalization ability in semantic consistency across different data modalities. The classifier based on graph attention network is applied to obtain predicted labels to enhance cross-modal feature representation. We construct hash codes using optimization strategy and loss function to preserve the semantic information and compactness of the hash code. Theoretical analysis and experimental results support these claims, validating the effectiveness this method.
and feature representation across different modalities. The primary contributions
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the
results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
In the conclusion section of the paper, we have provided a detailed explanation of the method's limitations. The primary limitation lies in the graph attention network module within the model. Since this module requires calculating attention weights between nodes, each node must perform complex interactions with others as the number labels increases. significantly raising the c
Justification:
computational complexity of the attention mechanism. This increase in complexity is particularly pronounced with large label sets, presenting challenges to the model's scalability and operational efficiency.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust results are violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If a'pppiicalle, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
In Sections 2.3 to 2.6 of the paper, provide detailed description of each step involved in the framework, including feature extraction, the graph attention network classification module, the hash code module, and optimization, ensuring that each component follows a clear and progressive logic. All theorems, formulas, and proof's are outlined in these sections with primary proof summaries, and numbered to maintain coherence and rigor in the argumentation.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross resseeeffeeeedd
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Dnes the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our proposed EGATH method constructs a new model framework, with detailed introduction of its core components provided in Section 2.2, "Network Architecture of EGATH." This model consists of three main parts: feature extraction, the graph attention network classifier, and the hash code module. Section 3, "Experiments." we provide a comprehensive description of the experimental setup, parameter configurations, and other key details to ensure reproducibility of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The source code pertaining to our paper is accessible on GitHub at the following link: , com/ begi www.reeeereeeaaeee /EGATH.
The three datasets used in this study are listed below with their download links: MIRFlickr25K:
http://press Sttresss.laaa..om liacs.
NUS WIDE; attps://1ms comp
nus. mus.edu. http:////o..oooooo/////////////s
MS-COCO:
sttps://cocodataset. org http://ooooo//oooaaaaoo.ooooo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines httt:p:://nips.cc/putt
guides/Codedee CodeSubmiss stt.osuubbisssioooooooy. for more details.
While we encourage the release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g.. for a new open- source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines pottpa://nipe cc/public/ for more details.
guides/ S/ooeeobmiissioonnnnnn
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detail
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In this paper, we use CLIP and transformer as feature extractors for images. transformer and Fully Connected Layer as feature extractors for text. and GAT as the label classifier to bridge the information gap between modalities. The input for GAT is the 300 300-dimensional glove vectors pre-trained on the Wikipedia dataset and the adjacency matrix between labels. We carefully set some hyperparameters 3,3, and k for auxiliary learning. Through experimental analysis, we examined the sensitivity of these parameters. Finally, we set the batch size to 64 and employed the Adam optimization strategy for the main optimization, with weight decay set to 0.0005 and adopting method of dynamically adjusting the learning rate. Our EGATH method is implemented in Pytorch, and the experiments were conducted on a server equipped with an NVIDIA GeForce RTX 3080 with 40GB of RAM.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The performance of our model is evaluated using both precision-recall curve and top-K curve. The PR curve illustrate the trade-off between precision and recall at various threshold levels, providing insight into the model's ability correctly identify positive samples, especially in the context of imbalanced datasets. Additionally, relevant items within the top-K predictions, offering a practical perspective on its effectiveness in real-world applications. These metrics serve as significant indicators of the statistical validity of our experimental results.
top-K curve assess the model's 's performance in retrieving
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variabiliity that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the emror bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For cach experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The BGATH method this paper is implemented in Pytorch, and experiments were conducted on a server equipped with an NVIDIA GeForce RTX 3080 GPU (40GB RAM). The paper clearly specifies the compute resources required to reproduce the experiments, including the type of compute device and memory size, enabling other researchers reproduce the experiments under similar conditions.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether full research project required more compute than experiments reported the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
The code for our paper is follows: http://p///gttt..ooooooo/meeeaea [EGATH
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurTPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make
preserve anonymity (e.g., if there is special consideration due laws or regulations their jurisdiction).
10. Broader Impacts
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Question:
Answer: [Yes]
19

---

Justification: Cross-modal hashing retrieval technology enhances the convenience of information access and the potential for cross-cultural exchange, but it also brings risks of privacy breaches, ethical concerns, and potential bias amplification. These societal impacts, while promoting information equality, require careful handling of data sccurity and fairness to ensure positive outcomes in technology applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational rescarch and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example. is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should he released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing resources used in this paper, including the comparative experi- ments and the three datasets, have been properly cited. The main download links for the datascts are as follows: MIRFlickr25K: http:://prest: liacs ww//aarflicck///roooooooo htw.a///w/.o..oooooooo/////////a
mus.p.nus..e html] NUS- WIDE; https: :/1ms/ comp. nus, edu wtw.uu..g//wpp..
nuswide/NUS: Nt///NUSSWWDS html MS-COCO: https://cocooaaassttt org/ #dowmload
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should he provided. For popular datasets, pwppraai thoode......... /oometttttta has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
existing datasets that are re-packaged. both original license and license of derived asset (if changed) should provided.
this information not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the dctails of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor rescarch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial suhmissions, not include any information that would break anonymity applica- ble), such as the institution conducting the review.
21


Reviews:
[{"review_id": "geXuaF6obC", "reviewer": "Reviewer_5SGA", "summary": "This paper proposes a method for cross-modal hash retrieval, unlike previous methods, the EGATH proposed in this paper combines multiple techniques with each other, which can effectively improve the performance of retrieval. At the same time, graph attention network is introduced to further process the information inherent in labels, which can enable better mining of potential semantic associations between different modalities. In addition, EGATH also introduces hash code optimization strategies to further preserve the semantic relationships of hash codes.\nIn conclusion, the method proposed by the authors and utilizing graph attention networks as a label classifier seems to be feasible.", "questions": "1. The authors need add some details about the graph attention network module, how to process the components.\n2. In this paper, does the use of CLIP and Transformer and GAT models involve high model complexity? Does this affect the training performance of the model?\n3. The author need add comparative experiments on Graph Attention Networks to further illustrate the importance of this module.", "limitations": "This paper emphasizes the limitations of the proposed methodology.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper is clearly written and easy to understand.\n2.\tThis paper proposes a novel approach for the application of graph attention networks in cross-modal retrieval.\n3.\tThe paper has strong empirical evidence to support the effectiveness of the proposed method.", "weaknesses": "1. the paper does not provide enough detail on some specific components of the framework.\n2. Some of the baselines could be added to this paper as appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\nQ1. the paper does not provide enough detail on some specific components of the framework\n\nR1:We would add more enough detail on the framework. we propose a new end-to-end cross-modal hash retrieval based on graph attention network, which contains three main components: feature extraction, GAT-based classification module and hash learning module. Firstly, in the feature extraction module, an image feature learning network that combines CLIP and Transformer with each other and a text feature learning network based on the attention mechanism of Transformer are used to obtain the feature representations of images and texts, and secondly, in the GAT-based tag prediction classifier, we integrate the co-occurrence information of tags into word embeddings, specifically, calculate the co-occurrence frequency of the labels and construct the adjacent matrix, take the initial feature vector and adjacent matrix of the labels as inputs, mine the implicit semantic information present in the labels through the dynamic attention mechanism, and compute the attention weights between the labels, so as to weight the average and update the embedding of the labels, and the updated embedding of the labels not only contains the information of the labels themselves, but also integrates the relational information between the labels, which is used to guide the feature learning of images and texts. Finally, the hash code loss function and optimum strategy enable the hash code to retain more semantic information, thus obtaining a discriminative and compact cross-modal uniform hash representation.\n\nQ2. Some of the baselines could be added to this paper as appropriate\n\nR2:  We have added four new baselines. The experiment result shows that our proposed method has much better performance than the four baselines. \n| Task | Method      | MIR FLICKr-25K |     |     | NUSWIDE |     |     | MS COCO |     |     |\n|-------|-------------|-------------------|----|--- |------------|----|-----|-----------|----|----|\n|      |             | 16bit          | 32bit | 64bit | 16bit   | 32bit | 64bit | 16bit   | 32bit | 64bit |\n|      | CDTH(2024)   | 0.7317| 0.7461|\t0.7477| 0.6596|\t0.6613| 0.6700|\t 0.5853| 0.6411| 0.6573|\n|      | SCAHN(2024) |0.8123 | 0.8131 | 0.8336 |\t0.6588 | 0.6621 | 0.6669 |\t0.6727 | 0.7108 | 0.7528 |\n| I→T  | CMGCAH(2023)| 0.7901| 0.8030|  0.8150| 0.6213| 0.6440| 0.6462  | - | -  | - |\n|      | GASKN(2024) | 0.7610| \t0.7720| \t0.7830| **0.7200**| \t0.7260|0.7470|\t0.7210| 0.7360| 0.7410|\n|      | **EGATH(Ours)**  | **0.8411**| **0.8562**| **0.8637**| 0.7191| **0.7345**| **0.7480**| **0.7259**| **0.7688**| **0.7945**| \n|      | CDTH(2024)   |  0.7315|\t0.7464 | 0.7503 | 0.6788| 0.6815 | 0.6910 | 0.5846 | 0.6427| 0.6573|\t\n|      | SCAHN(2024) |  0.7890 |\t0.7971| 0.8186|\t0.6718| 0.6803| 0.6980| 0.7183|\t 0.7504| **0.8093**|\n| T→I  | CMGCAH(2023)| 0.7823| 0.7932|\t0.8045| 0.6782|\t 0.6801| 0.6844| -| -|\t-|\n|      | GASKN(2024) |  0.7450 |\t0.7560| 0.7720|\t0.7080| 0.7170| 0.7450| 0.7090| 0.7200| 0.7280|\n|      | **EGATH(Ours)**  | **0.8064** | **0.8185**  | **0.8293**  | **0.7270**  | **0.7437**| **0.7539** | **0.7247**  | **0.7729** | 0.8015 |\n\n[1] Cross-domain transfer hashing for efficient cross-modal retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 2024\n\n[2] Structures awarefine-grained contrastive adversarial hashing for cross-media retrieval. IEEE Transactions on Knowledge and Data Engineering, 2024\n\n[3]Cross-modal generation and pair correlation alignment hashing. IEEE Transactions on Intelligent Transportation Systems, 24(3):3018-3026,2023\n\n[4]Unsupervised multimodal graph contrastive semantic anchor space dynamic knowledge distillation network for cross-media hash retrieval. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 4699-4708\n\n Questions:\n\nQ1. The authors need add some details about the graph attention network module, how to process the components.\n\nR1: We would added the following detail in our final manuscript.\n\nFirstly, the GloVe vectors and adjacency matrices of the labels are taken as inputs, and the information of the neighbour nodes is aggregated through the attention mechanism of GAT, which maps the input features to a new feature space by linearly transforming the input features using the weighting matrix. Secondly, each pair of labels is spliced and the attention score is computed to enable the model to assign higher weights to important node information, which in turn updates the feature representation of each label by weighted aggregation of information from neighbour nodes using the attention score. Finally, the updated label features are fused with the input features (image or text features), and the similarity is calculated by matrix multiplication to obtain the final category prediction.\n\nQ2. In this paper, does the use of CLIP and Transformer and GAT models involve high model complexity? Does this affect the training performance of the model.\n\nR2:  Constructing matrices for labels does increase computational complexity than traditional method, while this doesn’t affect the training performance of the model. Actually, using GAT as a label classifier to predict labels can provide more context information. Moreover, combining the predicted labels with features of the extracted image and text can enhance the representation ability of feature representation, at the same time, improve the consistence and matching effect between images and text.\n\nQ3.The author need add comparative experiments on Graph Attention Networks to further illustrate the importance of this module.\n\nR3:We conducted further tests for this module in the added comparison experiments, specifically using the GASKN model from the 2024 ICDE conference. The experimental results show that the addition of the graph network module significantly improves retrieval accuracy, which validates the effectiveness of graph networks in improving cross-modal retrieval performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3FvRe2InvV", "reviewer": "Reviewer_FNxK", "summary": "In this paper, the authors present an E2E graph attention network based hashing method for cross-modal retrieval task. The proposed method adopts CLIP and Transformer to extract the features of images or texts. In addition, it also uses a classifier based on graph attention network to predict labels to enhance cross-modal feature representation. The authors demonstrate the effectiveness of the proposed method by experiments.", "questions": "1.\tThe ones in the weaknesses.\n2.\tThe authors use GAT to predict labels. Actually, the samples have labels during training. Is it necessary for the model to predict the labels? In addition, from the framework, I cannot understand how the model makes use of the predicted labels.", "limitations": "The authors have addressed the limitations.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe idea seems technically sound.\n2.\tThere are some contributions to this field, example the combination of CLIP, Transformer and Graph Attention Network.", "weaknesses": "1.\tThe contribution is not significant. Actually, some hashing methods have explored the use of CLIP, Transformer or graph attention network.\n2.\tThe results are not convincing. Only one method proposed in 2023 is used as a basline for comparison. To my knowledge, many methods of this field are proposed in each year. The authors should compare the proposed method with more recent works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\nQ1. The contribution is not significant. Actually, some hashing methods have explored the use of CLIP, Transformer or graph attention network.\n\nR1: Thank you for your comments. The main contributions of our work are as follows.\nBy combining CLIP and Transformer technologies, an end-to-end architecture is implemented that significantly improves the model’s ability to capture global features of multi modal data, thus ensuring semantic consistency of images and text. Different from other existing works, where CLIP and Transformer are both used for text and image to extract features. While we utilize CLIP coupled with transformer to extract features for image data, and feature extraction via a transformer for text, which can realize lightweight network.\nPredicted labels is used to combined with the feature extracted from the feature modules of image net and text net to enhance feature representation, which also reduce the noise in the original labels during training. We utilize GAT as a label classifier to explore the hidden information in the label to predict labels, which can directly model the label graph to dig the correlation between labels and has higher flexibility than other label classification using preset weights for feature processing.\nOur EGATH was evaluated on three widely recognized benchmark datasets: NUS-WIDE, MIRFlickr25K, and MS-COCO, proving that it has obvious advantages in performance. Experimental results on these datasets show that our EGATH is superior to the current advanced cross-modal hash methods.\n\nQ2. The results are not convincing. Only one method proposed in 2023 is used as a baseline for comparison. To my knowledge, many methods of this field are proposed in each year. The authors should compare the proposed method with more recent works.\n\nR2: Thank you for your comments. We have added four new baselines, which includes the GASKN in 2024 ICDE, CDTH in 2024 IEEE TSVT, SCAHN in 2024 IEEE TKD, and CMGCAH in 2023 IEEE TITS. The experiment result shows that our proposed method has much better performance than the four baselines. More detailed are as follows.\n# Table 1 Additional experimental data\n\n| Task | Method      | MIR FLICKr-25K |     |     | NUSWIDE |     |     | MS COCO |     |     |\n|-------|-------------|-------------------|----|--- |------------|----|-----|-----------|----|----|\n|      |             | 16bit          | 32bit | 64bit | 16bit   | 32bit | 64bit | 16bit   | 32bit | 64bit |\n|      | CDTH(2024)   | 0.7317| 0.7461|\t0.7477| 0.6596|\t0.6613| 0.6700|\t 0.5853| 0.6411| 0.6573|\n|      | SCAHN(2024) |0.8123 | 0.8131 | 0.8336 |\t0.6588 | 0.6621 | 0.6669 |\t0.6727 | 0.7108 | 0.7528 |\n| I→T  | CMGCAH(2023)| 0.7901| 0.8030|  0.8150| 0.6213| 0.6440| 0.6462  | - | -  | - |\n|      | GASKN(2024) | 0.7610| \t0.7720| \t0.7830| **0.7200**| \t0.7260|0.7470|\t0.7210| 0.7360| 0.7410|\n|      | **EGATH(Ours)**  | **0.8411**| **0.8562**| **0.8637**| 0.7191| **0.7345**| **0.7480**| **0.7259**| **0.7688**| **0.7945**| \n|      | CDTH(2024)   |  0.7315|\t0.7464 | 0.7503 | 0.6788| 0.6815 | 0.6910 | 0.5846 | 0.6427| 0.6573|\t\n|      | SCAHN(2024) |  0.7890 |\t0.7971| 0.8186|\t0.6718| 0.6803| 0.6980| 0.7183|\t 0.7504| **0.8093**|\n| T→I  | CMGCAH(2023)| 0.7823| 0.7932|\t0.8045| 0.6782|\t 0.6801| 0.6844| -| -|\t-|\n|      | GASKN(2024) |  0.7450 |\t0.7560| 0.7720|\t0.7080| 0.7170| 0.7450| 0.7090| 0.7200| 0.7280|\n|      | **EGATH(Ours)**  | **0.8064** | **0.8185**  | **0.8293**  | **0.7270**  | **0.7437**| **0.7539** | **0.7247**  | **0.7729** | 0.8015 |\n\nREFERENCE\n\n[1]\tFengling Li, Bowen Wang, Lei Zhu, Jingjing Li, Zheng Zhang, and Xiaojun Chang. Cross-domain transfer hashing for efficient cross-modal retrieval. IEEE Transactions on Circuits and Systems for VideoTechnology, 2024.\n\n[2]\tMeiyu Liang, Yawen Li, Yang Yu, Xiaowen Cao, Zhe Xue, Ang Li, and Kangkang Lu. Structures awarefine-grained contrastive adversarial hashing for cross-media retrieval. IEEE Transactions on Knowledgeand Data Engineering, 2024.\n\n[3]\tWeihua Ou, Jiaxin Deng, Lei Zhang, Jianping Gou, and Quan Zhou. Cross-modal generation and paircorrelation alignment hashing. IEEE Transactions on Intelligent Transportation Systems, 24(3):3018–3026,2023.\n\n[4]\tYang Yu, Meiyu Liang, Mengran Yin, Kangkang Lu, Junping Du, and Zhe Xue. Unsupervised multimodal graph contrastive semantic anchor space dynamic knowledge distillation network for cross-media hash retrieval. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 4699–4708.IEEE, 2024.\n \nQuestion:\n\nQ1. The authors use GAT to predict labels. Actually, the samples have labels during training. Is it necessary for the model to predict the labels? In addition, from the framework, I cannot understand how the model makes use of the predicted labels.\n\nR1: Thank you for your comments. Compared with labels during training, the predicted labels can explore the information hidden in labels during training to enhance feature representation. Moreover, labels during training have a lot of noise and mismatches, while predicted labels can effectively handle noise to avoid the mismatches. The model integrates the co-occurrence information from the labels during training into the word embeddings to form the predicted labels (text and images). Then, model combined the predicted labels with the feature extracted from the feature modules of image net and text net. The whole process can not only bridge the information gap between text and image but also enhance the deep characteristics of them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mIF4xYWj88", "reviewer": "Reviewer_V2kH", "summary": "The paper mainly tries to address the issues about uncomprehensive feature representation and semantic associations of the existing cross modal retrieval works. In the paper, EGATH, an end-to-end graph attention network hash is proposed. EGATH adopts CLIP to improve the generalization ability in semantic consistency across different data modalities. The experiments are performed on three well-known benchmarks, which show the superior performance of the method on cross-modal retrieval tasks.", "questions": "See weakness", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a novel end-to-end Graph Attention Hash Network for cross-modal retrieval. By combining CLIP and Transformer, the method can capture global features and ensure semantic consistency of multi-modal data.\n\n2. The contributions of the work are clearly stated. The organization is good. The experimental results are convincing. Some ablation studies are also conducted. The network architecture looks beautiful and clear.", "weaknesses": "1. Some grammar, spelling, and symbol errors need carefully check.\n\n2. For the title, ‘Network Hash’ seems uncommon. Please check it.\n\n3. It is suggested to check the use of mathematical symbols. For example, it is difficult to distinguish vector, matrix, elements, etc in the paper.\n\n4. In (10), where is $i$?\n\n5. It is suggested to add an Algorithm table to summarize the training/optimization process.\n\n6. Figure 4 is too small to look.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses & Questions:\n\nQ1.Some grammar, spelling, and symbol errors need carefully check.\n\nR1: Sorry for this, we will carefully check to ensure that there are no grammatical, spelling, and symbol errors in the final version.\n\nQ2. For the title, ‘Network Hash’ seems uncommon. Please check it\n\nR2: Sorry for this,‘Network Hash’would be revised with ‘Hashing Network’ in the final version.\n\nQ3. It is suggested to check the use of mathematical symbols. For example, it is difficult to distinguish vector, matrix, elements, etc in the paper.\n\nR3: Sorry for this, we will carefully check carefully checked to ensure that the difference between vector, matrix and elements in the final version.\n\nQ4. In (10), where is $i$?\n\nR4: Sorry for this. $i$ is the foot mark of $\\hat{S}$ in Eq.(9) and also  $\\hat{S}$ in Eq.(10).Correct form would be revised in our final manuscript.\n\nQ5. It is suggested to add an Algorithm table to summarize the training/optimization process.\n\nR5: Thank you for this insightful comment. Algorithm table has been added in PDF attachment with Table 2.\n\nQ6. Figure 4 is too small to look.\n\nR6：Sorry for this, Figure 4 would be revised in our final version. The version that has been changed is in the PDF attachment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pgCP0XiEcc", "reviewer": "Reviewer_L29L", "summary": "In this paper, a cross-modal hash retrieval method based on graph attention networks is proposed. The main framework of the authors' research contains three key components: feature extraction, graph attention classifier and hash code module. An end-to-end architecture is realized by combining CLIP and Transformer techniques, and in this work, while using GAT as a label classifier, the model is able to better capture and utilize the semantic information of the labels and dig deeper into the graph structure features of the data and their interconnections. A hash optimization strategy and loss function are used in the hash code module to further optimize the hash code. It is shown through extensive experiments that the authors' proposed method achieves better performance in the field of cross-modal hash retrieval.", "questions": "1. I suggest that the authors elaborate on the results of the experiments in the section of the results of the comparison of the data obtained in detail\n2. Could the graphs in Figures 2 and 3 in section 3.4 of the paper be improved?\n3. The way of constructing matrices for labels mentioned in the paper increases the computational complexity, so in the method of this paper, constructing the adjacency matrix of labels, which involves steps such as thresholding, will this be more complex than the traditional method? And what is the difference of this approach?", "limitations": "The authors have discussed the limitations of the proposed method.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This thesis provides a new perspective on cross-modal hash retrieval by using more novel techniques combined with each other in the framework composition.\n2. The use of GAT as a tag classifier is designed to facilitate better capture of semantic information of tags while achieving more desirable experimental results. \n3. The components of the framework in this paper are shown through extensive experiments, proving that each component is indispensable.", "weaknesses": "1. The paper could delve deeper into the details of how the CLIP and Transformer models are integrated. \n2. Further explanation is needed to elucidate the advantages of the Graph Attention Network compared to other methods.\n3. Some typos exist in mathematical formulas. For example: ① In Line 202, $U$ and $V$ should be $\\mathbf{U}$ and $\\mathbf{V}$. ② \"$.$\" should be \"$\\cdot$\" in Eq. (14). ③ Many mathematical formulas lack necessary punctuations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n\nQ1.The paper could delve deeper into the details of how the CLIP and Transformer models are integrated.\n\nR1: Thank you for your comments. We would added more enough detail on the process of how the CLIP and Transformer models are integrated as follows. In the feature extraction module, an image feature learning network that combines CLIP and Transformer with each other and a text feature learning network based on the attention mechanism of Transformer are used to obtain the feature representations of images and texts.\nR1: Thank you for your comments. More detail are added as follows. \n\nQ2.Further explanation is needed to elucidate the advantages of the Graph Attention Network compared to other methods.\n\nR2: Using GAT to predict labels have many advantages. Compared with labels during training that is static, predicted labels with GAT is dynamic, which can be updated according to specific tasks and data sets to model the relationship between labels explicitly and improve the representation of label features. In new samples, where there is no clear label, the label definition is not clear, or a sample contains multiple labels, the graph structure of GAT can help to infer the label of the sample, so as to obtain better inference and classification. Using a graph structure, GAT reduces the impact of label noise on training data by smoothing operations, while being able to predict labels and reduce label mismatches in original sample.\n\nQ3.Some typos exist in mathematical formulas. For example: ① In Line 202, “U and V” should be “U and V” . ② \".\" should be \".\" in Eq. (14). ③ Many mathematical formulas lack necessary punctuations.\n\nR3: Sorry for this, full text would be carefully checked to ensure that the typos\nin mathematical formulas. “U and V” would be changed with “U and V”, \".\" would be changed with \".\".Punctuations of mathematical formulas would be added in our final manuscript.\n\nQuestions:\n\nQ1. I suggest that the authors elaborate on the results of the experiments in the section of the results of the comparison of the data obtained in detail\n\nR1: Thank you for your comments. More detail are added as follows. \n     In order to further investigate the performance of our EGATH method, we used three evaluation metrics: mAP, PR curve and Top-k curve, as can be seen from the tables in the paper, we compared with the state-of-the-art seven methods on three datasets, the underlined values in the table are the best results among the compared methods, Table 1 illustrates the results of the mAP for the different lengths of the hash codes (16-bit, 32-bit, 64-bit). bit) of mAP, we can get the following observations\nFirstly, our method achieves better experimental results on the three datasets, with 3.4%, 3.8% and 12.4% improvement on MIR Flickr25K, NUSWIDE, and MSCOCO, respectively, compared to the best results DSPH and DCHMT among the compared methods, especially the effect on MSCOCO dataset has a significant enhancement, which is due to the following reasons The MSCOCO dataset with rich label information provides more contextual information for GAT, which effectively improves the accuracy of label prediction and the consistency of cross-modal features, and indicates that our method has a special advantage in dealing with data-rich datasets with diverse labels.\n    Secondly, the corresponding PR curves can be seen in Fig. 2, from which we can observe that EGATH performs better and obtains higher precision scores at all recall levels. EGATH improves semantic differentiation and consistency across modalities through feature networks, and improves potential semantic associations through tag classification based on graph-attention networks, thus facilitating the cross-media retrieval performance.\n    Finally, in Fig. 3 we compare EGATH with other methods under different Top-k search results, including 50, 100, 200, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, and 5000, and these experiments cover retrieval tasks ranging from small to large scale in order to evaluate the different methods at various retrieval scales. The experimental results show that the EGATH algorithm outperforms other existing algorithms for the cross-media retrieval tasks I→T and T→I under any top-K ordering, especially with higher MAP values for larger K values. This advantage is attributed to the fact that the EGATH method better preserves semantic information during the generation of hash codes and thus performs better in large-scale retrieval.\n\nQ2. Could the graphs in Figures 2 and 3 in section 3.4 of the paper be improved?\n\nR2: Sorry for this. We would improve Figures 2 and 3 in our final manuscript. The version that has been changed is in the PDF attachment.\n\nQ3.The way of constructing matrices for labels mentioned in the paper increases the computational complexity, so in the method of this paper, constructing the adjacency matrix of labels, which involves steps such as thresholding, will this be more complex than the traditional method? And what is the difference of this approach?\n\nR3: Constructing matrices for labels does increase computational complexity than traditional method, which is relatively small. However using GAT to predict labels have many advantages. Compared with labels during training that is static, predicted labels with GAT is dynamic, which can be updated according to specific tasks and data sets to model the relationship between labels explicitly and improve the representation of label features. In new samples, where there is no clear label, the label definition is not clear, or a sample contains multiple labels, the graph structure of GAT can help to infer the label of the sample, so as to obtain better inference and classification. Using a graph structure, GAT reduces the impact of label noise on training data by smoothing operations, while being able to predict labels and reduce label mismatches in original sample.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We appreciate the time and effort you have put into reviewing our paper titled “[An End-To-End Graph Attention Network Hash for Cross-modal Retrieval].” We value your feedback and have addressed each of the comments raised. Below, we provide detailed responses to the reviewers' comments.   For the relevant figures, tables, and algorithms, please view the attached PDF file.

Meta-review: This paper adopts the CLIP combined with the Transformer to improve understanding and generalization ability in semantic consistency across different data modalities. Several experiments on benchmark datasets show that they significantly perform favorably against several methods. Three of four reviewers rate the paper as weak accept/accept, while one reviewer still has concerns about the contributions and state-of-the-art performance. After discussion, the final decision is accept, and the authors should further highlight innovations and improve experiments.
Decision: Accept (poster)