# Quadruplet-Based Deep Cross-Modal Hashing

Authors: Huan Liu; Jiang Xiong; Nian Zhang; Fuming Liu; Xitao Zou
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Recently, benefitting from the storage and retrieval efficiency of hashing and the powerful discriminative feature extraction capability of deep neural networks, deep cross-modal hashing retrieval has drawn more and more attention. To preserve the semantic similarities of cross-modal instances during the hash mapping procedure, most existing deep cross-modal hashing methods usually learn deep hashing networks with a pairwise loss or a triplet loss. However, these methods may not fully explore the similarity relation across modalities. To solve this problem, in this paper, we introduce a quadruplet loss into deep cross-modal hashing and propose a quadruplet-based deep cross-modal hashing (termed QDCMH) method. Extensive experiments on two benchmark cross-modal retrieval datasets show that our proposed method achieves state-of-the-art performance and demonstrate the efficiency of the quadruplet loss in cross-modal hashing.


---

## Body

## 1. Introduction

With the advent of the era of big data, there are surging massive multimedia data on the Internet, such as images, videos, and texts. These data usually exist in diversified modalities, for example, there may exist a textual data and an audio data describing a video data or an image data. As data from different modalities may have compact semantic relevance, cross-modal retrieval [1, 2] is proposed to retrieve semantic similar data from one modality while the querying data is from a distinct modality. Benefitting from the high efficiency and low cost, hashing-based cross-modal retrieval (cross-modal hashing) [3–6] has drew extensive attention. The goal of cross-modal hashing is to map the modal heterogeneous data into a common binary space and ensure that semantic similar/dissimilar cross-modal data have similar/dissimilar hash codes. Cross-modal hashing methods can usually achieve superior performance; nonetheless, most of existing cross-modal hashing methods (such as cross-modal similarity sensitive hashing (CMSSH) [7], semantic correlation maximization (SCM) [8], semantics-preserving hashing (SePH) [9], and generalized semantic preserving hashing (GSPH) [10]) are based on handcrafted feature learning, which cannot effectively capture the heterogeneous relevance between different modalities and thus may result in inferior performance.In the last decade, deep convolutional neural networks [11, 12] have been successfully utilized in many computer vision tasks, and therefore, some researchers also deploy it in cross-modal hashing, such as deep cross-modal hashing (DCMH) [13], pairwise relationship guided deep hashing (PRDH) [14], self-supervised adversarial hashing (SSAH) [15], and triplet-based deep hashing (TDH) [16]. Cross-modal hashing methods with deep neural networks efficiently integrate the hash representation learning and the hash function learning into an end-to-end framework, which can capture heterogeneous cross-modal relevance more effectively and thus acquire better cross-modal retrieval performance.To date, most deep cross-modal hashing methods utilize the pairwise loss (such as [13–15]) or the triplet loss (such as [16]) to preserve semantic relevance during the hash representation learning procedure. Nevertheless, the pairwise loss- and triplet loss-based hash methods suffer from a weak generalization capacity from the training set to the testing set [17, 18], as shown in Figure 1(a). On the contrary, quadruplet loss is proposed and has been utilized in image hashing retrieval [17] and person reidentification [18], and in these works, it has been proved that the quadruplet loss-based model can enhance the generalization capability. Therefore, cross-modal hashing combines quadruplet loss as a natural solution to enhance the performance of cross-modal hashing, as shown in Figure 1(b).Figure 1
(a) Triplet loss-based cross-modal hashing methods suffer from a weak generalization capacity from the training set to the testing set because the test instances belong to the category and cannot be mapped into compact binary codes (see the lower-right corner). (b) Triplet loss-based cross-modal hashing methods can project the test instances, which belong to the category , into compact binary space (see the lower right corner).
(a)(b)To this end, in this paper, we introduce quadruplet loss into cross-modal hashing and propose a quadruplet-based deep cross-modal hashing method (QDCMH). Specifically, QDCMH firstly defines a quadruplet-based cross-modal semantic preserving module. Afterwards, QDCMH integrates this module, hash representation learning, and hash code generation into an end-to-end framework. Finally, experiments on two benchmark cross-modal retrieval datasets are conducted to validate the performance of the proposed method. The main contributions of our proposed QDCMH include the following:(i)
We introduce quadruplet loss into cross-modal retrieval and propose a novel deep cross-modal hashing method. To the best of our knowledge, this is the first work to introduce quadruplet loss into cross-modal hashing retrieval.(ii)
We conduct extensive experiments on benchmark cross-modal retrieval datasets to investigate the performance of our proposed QDCMH.The remainder of this paper is organized as follows. Section2 elaborates our proposed quadruplet-based deep cross-modal hashing method. Section 3 presents the learning algorithm of QDCMH. Section 4 is the experimental results and the corresponding analysis. Section 5 concludes our work.


## 2. Proposed Method

In this section, we elaborate our proposed quadruplet-based deep cross-modal hashing (QDCMH) method with the following sections: notations, quadruplet-based cross-modal semantic preserving module, feature learning networks, and hash function learning. Figure2 presents the flowchart of our proposed QDCMH, which cooperates quadruplet-based cross-modal semantic preserving module, hash representation learning, and hash codes generation into an end-to-end framework. In our proposed QDCMH method, we assume that each instance has two modalities, i.e., an image modality and a text modality, but they can be easily applied to multimodalities.Figure 2
Flowchart of the proposed quadruplet-based deep cross-modal hashing (QDCMH) method. QDCMH encompasses three steps: (1) a quadruplet-based cross-modal semantic preserving module, (2) a classical convolutional neural network is used to learn image-modality features and the TxtNet in SSAH [15] is adopted to learn the text-modality features, and (3) an intermodal quadruplet loss is utilized to efficiently capture the relevant semantic information during the feature learning process and a quantization loss is used to decrease information loss during the hash codes generation procedure. (a) Quadruplet Vq,Tp,Tn1,Tn2, which utilizes an image instance Vq to retrieve three text instances: Tp, Tn1, and Tn2. Vq and Tp have at least one common labels, while Vq and Tn1, Vq and Tn2, and Tn1 and Tn2 are three pairwise instances and the two instances in each pairwise have no common label. (b) Quadruplet Vq,Tp,Tn1,Tn2, which utilizes a text instance Tq to retrieve three image instances: Vp, Vn1, and Vn2. Tq and Vp have at least one common labels, while Tq and Vn1, Tq and Vn2, and Vn1 and Vn2 are three pairwise instances and the two instances in each pairwise have no common label.
(a)(b)


### 2.1. Notations

Assume that the training data comprisesn image-text pairs, i.e., the original image features V∈Rn×dv and the original text features T∈Rn×dt. Besides, there is a label vector associated with each image-text pair and label vectors for all training instances constitute a label matrix L∈Rn×dl. dv and dt are the corresponding original dimensions of image features and text features, respectively, and dl is the total number of class categories. If image-text pair Vi,Ti attaches to the jth category, then Lij=1, otherwise Lij=0. The quadruplet Vq,Tp,Tn1,Tn2 denotes that Vq is a query instance from the image modality, and Tp,Tn1,Tn2 are three retrieval instances from the text modality, where Vq and Tp have at least one common categories, while Vq and Tn1, Vq and Tn2, and Tn1 and Tn2 are three pairwise instances and the two instances in each pairwise have no common label.With the known quadrupletVq,Tp,Tn1,Tn2, the target of our proposed QDCMH is to learn the corresponding hash codes BVq,BTp,BTn1,BTn2, where BVq,BTp,BTn1,BTn2 are the hash codes of instances Vq,Tp,Tn1,Tn2, respectively. To learn the above hash codes, we first learn the hash representations FVq,GTp,GTn1,GTn2 from the quadruplet Vq,Tp,Tn1,Tn2 with deep neural networks, where FVq=fVq,θV and GTp=gTp,θT are the hash representations of instance Vq and Tp, respectively. f.,θV and g.,θT are the hash representation learning functions for the image modality and the text modality, respectively. θV and θT are the parameters of deep neural networks to extract features for the image modality and for the text modality, respectively. Secondly, we can utilize the following sign function to approximately map the hash representations into the corresponding hash codes, i.e., BVq=signFVq and BTp=signGTp. In the same way, we can learn the hash codes of quadruplet Tq,Vp,Vn1,Vn2. For convenience, we denote the hash codes of all training image-text pairs, the hash representations of all training image instances, and the hash representations of all training text instances as B∈−1,1n×k, F∈Rn×k, and G∈Rn×k, respectively, where k is the length of hash codes:(1)y=1,ifx>=0,x∈R,−1,ifx<0,x∈R.


### 2.2. Quadruplet-Based Cross-Modal Semantic Preserving Module

In cross-modal hashing retrieval, given an image instanceVi and a text instance Tj, it is intractable to preserve the semantic relativity during the hash code learning procedure as the huge semantic gap across modalities. To solve this, DCMH [13] defines pairwise loss to map similar/dissimilar image-text pairs into similar/dissimilar hash codes. TDH [16] utilizes triplet loss to learn similar hash codes for similar cross-modal instances and generate distinct hash codes for semantic irrelevant cross-modal instances. Both pairwise loss and triplet loss can preserve the relevance in the original instance space; however, pairwise loss- and triplet loss-based hashing methods often suffer from a weaker generalization capability from the training set to the testing set [17, 18]. To solve this problem, in this section, a quadruplet-based cross-modal semantic preserving module is proposed to boost the generalization capability and better preserve the semantic relevance for cross-modal hashing.For a quadrupletVq,Tp,Tn1,Tn2, we should keep the semantic relevance unchanged during the hash representation learning, i.e., FVq should be similar to GTp, FVq should be distinct to GTn1 and GTn2, and GTn1 should be dissimilar with GTn2. Thus, we can define the following quadruplet loss for cross-modal hashing:(2)JquadrupletI⟶TFVq,GTp,GTn1,GTn2=∑Vq,Tp,Tn1max0,FVq−GTp22−FVq−GTn122+α1+∑Vq,Tp,Tn1,Tn2max0,FVq−GTp22−GTn1−GTn222+α2,where Vq is a query instance from the image modality, Tp, Tn1, and Tn2 are three retrieval instances from the text modality, and Vq and Tp are semantic similar. While Vq and Tn1, Vq and Tn2, and Tn1 and Tn2 are three pairwise instances, and the two instances in each pairwise have distinct semantics. Equation (2) denotes that the distance of hash representations of similar cross-modal pairwise instances should be smaller than that of dissimilar pairwise instances (both from intermodalities and from intramodalities) with a positive margin α1orα2. This can ensure that similar cross-modal instances have similar hash representations while dissimilar instances have distinct hash representations. By this quadruplet loss, the cross-modal semantic relevance can be preserved during the hash representation learning stage.Similarly, given a quadrupletTq,Vp,Vn1,Vn2, we can have the following cross-modal quadruplet loss:(3)JquadrupletT⟶IGTq,FVp,FVn1,FVn2=∑Tq,Vp,Vn1max0,GTq−FVp22−GTq−FVn122+α3+∑Tq,Vp,Vn1,Vn2max0,GTq−FVp22−FVn1−FVn222+α4,where Tq is a query instance from the text modality, Vp, Vn1, and Vn2 are three retrieval instances from the image modality, GTq, FVp, FVn1, and FVn2 are hash representations for instances Tq, Vp, Vn1, and Vn2, respectively, and α3 and α4 are two positive margins. Equation (3) is distinct to equation (2) as the modality of query instance and the modality of retrieval instances are inverse.


### 2.3. Hash Representation Learning and Hash Code Learning

For each quadruplet from training set, it is easy to learn their hash representations and fully protect the semantic similarity with the above quadruplet-based cross-modal semantic relevance preserving module, so we have the following hash representation learning loss:(4)Jrepresentation=1nI⟶TJquadrupletI⟶TFVq,GTp,GTn1,GTn2+βnT⟶IJquadrupletT⟶IGTq,FVp,FVn1,FVn2,where nI⟶T is the number of quadruplets for utilizing image to retrieve text, nT⟶I is the number of quadruplets for utilizing text to retrieve images, and β is a hyperparameter to balance the two parts.Additionally, to learn high-quality hash codes, we generate hash codes from the learned hash representations with the sign function in equation (1), and the final hash codes matrix for all training image-text pairs are generated as follows:(5)B=signF+G2.AsF and G are real-valued features, to decrease the information loss from F and G to B in equation (5), it is necessary to force F and G to be as close as possible to B; thus, we introduce the following quantization loss:(6)Jquantization=B−F22+B−G222nk.Integrating the hash representation loss and the quantization loss together, the whole loss function is as follows:(7)J=Jrepresentation+γJquantization,where γ is a hyperparameter to balance the hash representation loss and the quantization loss.


### 2.4. Feature Extraction Networks

In QDCMH, feature extraction includes two deep neural networks: a classical convolutional neural network is used to extract the features of images and a multiscale fusion model is utilized to learn features from texts. Specifically, for image modality, we deploy AlexNet [11] pretrained on the ImageNet [19] dataset. We then fine-tune the last layer using a new fully connected hash layer which consists of k hidden nodes. Therefore, the learned deep features have been embedded into a k-dimensional Hamming space. For text modality, the TxtNet in SSAH [15] is used, which comprises a three-layer feedforward neural network and a multiscale (MS) fusion model Input⟶MS⟶4096⟶512⟶k.


## 2.1. Notations

Assume that the training data comprisesn image-text pairs, i.e., the original image features V∈Rn×dv and the original text features T∈Rn×dt. Besides, there is a label vector associated with each image-text pair and label vectors for all training instances constitute a label matrix L∈Rn×dl. dv and dt are the corresponding original dimensions of image features and text features, respectively, and dl is the total number of class categories. If image-text pair Vi,Ti attaches to the jth category, then Lij=1, otherwise Lij=0. The quadruplet Vq,Tp,Tn1,Tn2 denotes that Vq is a query instance from the image modality, and Tp,Tn1,Tn2 are three retrieval instances from the text modality, where Vq and Tp have at least one common categories, while Vq and Tn1, Vq and Tn2, and Tn1 and Tn2 are three pairwise instances and the two instances in each pairwise have no common label.With the known quadrupletVq,Tp,Tn1,Tn2, the target of our proposed QDCMH is to learn the corresponding hash codes BVq,BTp,BTn1,BTn2, where BVq,BTp,BTn1,BTn2 are the hash codes of instances Vq,Tp,Tn1,Tn2, respectively. To learn the above hash codes, we first learn the hash representations FVq,GTp,GTn1,GTn2 from the quadruplet Vq,Tp,Tn1,Tn2 with deep neural networks, where FVq=fVq,θV and GTp=gTp,θT are the hash representations of instance Vq and Tp, respectively. f.,θV and g.,θT are the hash representation learning functions for the image modality and the text modality, respectively. θV and θT are the parameters of deep neural networks to extract features for the image modality and for the text modality, respectively. Secondly, we can utilize the following sign function to approximately map the hash representations into the corresponding hash codes, i.e., BVq=signFVq and BTp=signGTp. In the same way, we can learn the hash codes of quadruplet Tq,Vp,Vn1,Vn2. For convenience, we denote the hash codes of all training image-text pairs, the hash representations of all training image instances, and the hash representations of all training text instances as B∈−1,1n×k, F∈Rn×k, and G∈Rn×k, respectively, where k is the length of hash codes:(1)y=1,ifx>=0,x∈R,−1,ifx<0,x∈R.


## 2.2. Quadruplet-Based Cross-Modal Semantic Preserving Module

In cross-modal hashing retrieval, given an image instanceVi and a text instance Tj, it is intractable to preserve the semantic relativity during the hash code learning procedure as the huge semantic gap across modalities. To solve this, DCMH [13] defines pairwise loss to map similar/dissimilar image-text pairs into similar/dissimilar hash codes. TDH [16] utilizes triplet loss to learn similar hash codes for similar cross-modal instances and generate distinct hash codes for semantic irrelevant cross-modal instances. Both pairwise loss and triplet loss can preserve the relevance in the original instance space; however, pairwise loss- and triplet loss-based hashing methods often suffer from a weaker generalization capability from the training set to the testing set [17, 18]. To solve this problem, in this section, a quadruplet-based cross-modal semantic preserving module is proposed to boost the generalization capability and better preserve the semantic relevance for cross-modal hashing.For a quadrupletVq,Tp,Tn1,Tn2, we should keep the semantic relevance unchanged during the hash representation learning, i.e., FVq should be similar to GTp, FVq should be distinct to GTn1 and GTn2, and GTn1 should be dissimilar with GTn2. Thus, we can define the following quadruplet loss for cross-modal hashing:(2)JquadrupletI⟶TFVq,GTp,GTn1,GTn2=∑Vq,Tp,Tn1max0,FVq−GTp22−FVq−GTn122+α1+∑Vq,Tp,Tn1,Tn2max0,FVq−GTp22−GTn1−GTn222+α2,where Vq is a query instance from the image modality, Tp, Tn1, and Tn2 are three retrieval instances from the text modality, and Vq and Tp are semantic similar. While Vq and Tn1, Vq and Tn2, and Tn1 and Tn2 are three pairwise instances, and the two instances in each pairwise have distinct semantics. Equation (2) denotes that the distance of hash representations of similar cross-modal pairwise instances should be smaller than that of dissimilar pairwise instances (both from intermodalities and from intramodalities) with a positive margin α1orα2. This can ensure that similar cross-modal instances have similar hash representations while dissimilar instances have distinct hash representations. By this quadruplet loss, the cross-modal semantic relevance can be preserved during the hash representation learning stage.Similarly, given a quadrupletTq,Vp,Vn1,Vn2, we can have the following cross-modal quadruplet loss:(3)JquadrupletT⟶IGTq,FVp,FVn1,FVn2=∑Tq,Vp,Vn1max0,GTq−FVp22−GTq−FVn122+α3+∑Tq,Vp,Vn1,Vn2max0,GTq−FVp22−FVn1−FVn222+α4,where Tq is a query instance from the text modality, Vp, Vn1, and Vn2 are three retrieval instances from the image modality, GTq, FVp, FVn1, and FVn2 are hash representations for instances Tq, Vp, Vn1, and Vn2, respectively, and α3 and α4 are two positive margins. Equation (3) is distinct to equation (2) as the modality of query instance and the modality of retrieval instances are inverse.


## 2.3. Hash Representation Learning and Hash Code Learning

For each quadruplet from training set, it is easy to learn their hash representations and fully protect the semantic similarity with the above quadruplet-based cross-modal semantic relevance preserving module, so we have the following hash representation learning loss:(4)Jrepresentation=1nI⟶TJquadrupletI⟶TFVq,GTp,GTn1,GTn2+βnT⟶IJquadrupletT⟶IGTq,FVp,FVn1,FVn2,where nI⟶T is the number of quadruplets for utilizing image to retrieve text, nT⟶I is the number of quadruplets for utilizing text to retrieve images, and β is a hyperparameter to balance the two parts.Additionally, to learn high-quality hash codes, we generate hash codes from the learned hash representations with the sign function in equation (1), and the final hash codes matrix for all training image-text pairs are generated as follows:(5)B=signF+G2.AsF and G are real-valued features, to decrease the information loss from F and G to B in equation (5), it is necessary to force F and G to be as close as possible to B; thus, we introduce the following quantization loss:(6)Jquantization=B−F22+B−G222nk.Integrating the hash representation loss and the quantization loss together, the whole loss function is as follows:(7)J=Jrepresentation+γJquantization,where γ is a hyperparameter to balance the hash representation loss and the quantization loss.


## 2.4. Feature Extraction Networks

In QDCMH, feature extraction includes two deep neural networks: a classical convolutional neural network is used to extract the features of images and a multiscale fusion model is utilized to learn features from texts. Specifically, for image modality, we deploy AlexNet [11] pretrained on the ImageNet [19] dataset. We then fine-tune the last layer using a new fully connected hash layer which consists of k hidden nodes. Therefore, the learned deep features have been embedded into a k-dimensional Hamming space. For text modality, the TxtNet in SSAH [15] is used, which comprises a three-layer feedforward neural network and a multiscale (MS) fusion model Input⟶MS⟶4096⟶512⟶k.


## 3. Learning Algorithm of QDCMH

For QDCMH, we utilize alternating strategy to learn parametersθV of deep neural networks for image modality and parameters θT of deep neural networks for text modality and hash codes matrix B for all training image-text pairs. When we learn one of θV, θT, and B, we keep the other two fixed. The specific algorithm for QDCMH is depicted in Algorithm 1.Algorithm 1: QDCMH: quadruplet-based deep cross-modal hashing.
Input:training data set:V,T,L. The maximal number of epoches of the algorithm is max_epoch. Mini-batch size nbatch=128.Output:ParametersθV, θT of the deep neural networks, and corresponding hash codes matrix B.(1)
GeneratingnI⟶TVq,Tp,Tn1,Tn2 quadruplets (named QuadI2T) from training set, generating nT⟶ITq,Vp,Vn1,Vn2 quadruplets (named QuadT2I) from training set.(2) Initialize the deep neural network parametersθV, θT, the whole training image hash representations F, the whole training text hash representations G, the hash codes matrix B, and the epoch numbers batchnumv=batchnumt=⌈nI⟶T+nT⟶I/nbatch⌉.(3)repeat(4)forj=1tobatchnumvdo(5)   Randomly samplenv images from QuadI2T∪QuadT2I to construct a mini-batch of images.(6)   For each instanceVi in the mini-batch, calculate FVi=fVi,θV by forward propagation.(7)   UpdateF.(8)   Calculate the derivative ofθV in equation (7).(9)   Update the network parametersθI by utilizing backpropagation.(10)end for(11)forj=1tobatchnumtdo(12)   Randomly samplent texts from QuadI2T∪QuadT2I to construct a mini-batch of texts.(13)   For each instanceTi in the mini-batch, calculate GTi=gTi,θT by forward propagation.(14)   UpdateG.(15)   Calculate the derivative ofθT in equation (7).(16)   Update the network parametersθT by using backpropagation.(17)end for(18)  UpdateB using equation (5).(19)until the max epoch number max_epoch.


### 3.1. UpdateθV with θT and B Fixed

WhenθT and B are maintained fixed, we utilize stochastic gradient descent and backpropagation to optimize the deep neural network parameters θV.


### 3.2. UpdateθT with θV and B Fixed

When we fix the values ofθV and B, we use stochastic gradient descent and backpropagation to learn the deep neural network parameters θT.


### 3.3. UpdateB with θT and θV Fixed

When the deep neural networks’ parametersθT and θV are kept unchanged, the hash codes matrix B can be optimized with equation (5).


## 3.1. UpdateθV with θT and B Fixed

WhenθT and B are maintained fixed, we utilize stochastic gradient descent and backpropagation to optimize the deep neural network parameters θV.


## 3.2. UpdateθT with θV and B Fixed

When we fix the values ofθV and B, we use stochastic gradient descent and backpropagation to learn the deep neural network parameters θT.


## 3.3. UpdateB with θT and θV Fixed

When the deep neural networks’ parametersθT and θV are kept unchanged, the hash codes matrix B can be optimized with equation (5).


## 4. Experiments

### 4.1. Datasets

To investigate the performance of QDCMH, we conduct experiments on two benchmark cross-modal retrieval datasets: MIRFLICKR-25K [20] and Microsoft COCO2014 [21], and the brief descriptions of the datasets are listed in Table 1.Table 1
Brief description of the experimental datasets.
DatasetUsedTrainQueryRetrieveTag dimensionLabelsMIRFLICKR-25K20,01510,0002,00018,0151,38624MS-COCO2014122,21810,0005,000117,2182,02680


### 4.2. Evaluation Metrics

In our experiments, we utilize mean average precision (MAP), topN-precision curves (top N Curves), and precision-recall curves (PR Curves) as evaluation metrics; for the detailed description of these evaluation metrics, refer to [22, 23].


### 4.3. Baselines and Implementation Details

We compare our proposed QDCMH method with eight state-of-the-art cross-modal hashing methods, including four handcrafted ones, i.e., cross-modal similarity sensitive hashing (CMSSH) method [7], semantics-preserving hashing (SePH) [9] method, semantic correlation maximization (SCM) method [8], and generalized semantic preserving hashing (GSPH) method [10] and four deep feature-based ones, i.e., deep cross-modal hashing (DCMH) method [13], pairwise relationship guided deep hashing (PRDH) method [14], self-supervised adversarial hashing (SSAH) method [15], and triplet-based deep hashing (TDH) method [16]. Most baseline methods are carefully implemented based on the codes provided by the authors. A few baseline methods are implemented by us following the suggestions and descriptions of the original papers.All the experiments are executed by using the open source deep learning framework pytorch and running on an NVIDIA GTX Titan XP GPU server. In our experiments, we setnI⟶T=nT⟶I=10000, max_epoch=500, and λ=10−5 and the learning rate is initialized to 10−1.5 and gradually decreased to 10−6 in 500 epochs. For those handcrafted feature-based baselines, each image in the two datasets is represented by a bag of words (BoW) histogram or feature vector having 512 dimensions. For the whole experiment, we use I⟶T to denote using a querying image while returning text and T⟶I to denote using a querying text while returning an image.


### 4.4. Performance Evaluation and Discussion

Firstly, we investigate the performance of QDCMH with different hyperparametersβ and γ. To this goal, we experiment on MIRFLICKR-25K with the hash code length k=64 and record the corresponding MAPs under different values of β and γ, as shown in Figure 3. We find that high performance can be acquired when β=1 and γ=0.2.Figure 3
A sensitivity analysis of the hyperparameters. (a) Hyperparameterβ on MIRFLICKR-25K dataset. (b) Hyperparameter γ on MIRFLICKR-25K dataset.
(a)(b)Secondly, to validate the performance of QDCMH, we perform the experiment to compare QDCMH with baseline methods in terms of MAP on datasets MIRFLICKR-25K and MS-COCO2014. Table2 presents the MAPs of each method for different hash code lengths, i.e., 16, 32, and 64. DSePH represents the SePH method whose features of the original images are extracted by CNN–F. From Table 2, we can see that the following. (1) The MAPs of our proposed QDCMH are higher than the MAPs of most baseline methods in most cases, which demonstrates the superiority of QDCMH. We can also observe that SSAH outperforms than our proposed QDCMH in most cases, which is partly because SSAH takes self-supervised learning and generative adversarial networks into account during hash representation learning procedure. (2) The MAPs of QDCMH is always higher than the MAPs of TDH, which shows that quadruplet loss can better preserve semantic relevance than triplet loss in cross-modal hashing retrieval. (3) The MAPs of DSePH is always higher than the MAPs of SePH, which demonstrates that deep neural networks have powerful features learning capacity. (4) Our proposed QDCMH can achieve better performance on MS-COCO 2014 dataset than on MIRFlickr-25K dataset, which is partly because the instances in MS-COCO 2014 dataset belong to 80 categories while the instances in MIRFlickr-25K dataset belong to 24 categories, and this makes the quadruplets generated from the MS-COCO 2014 dataset have better generalization ability than the quadruplets generated from the MIRFlickr-25K dataset.Table 2
Comparison to baselines in terms of MAP on two datasets: MIRFLICKR-25K, and Microsoft COCO2014, respectively. The best accuracy is shown in boldface.
TaskMethodsMIRFlickr-25KMS-COCO16bits32bits64bits16bits32bits64bitsI⟶THandcrafted methodsCMSSH [7]0.56000.57090.58360.54390.54500.5410SePH [9]0.67400.68130.68030.42950.43530.4726SCM [8]0.63540.64070.65560.42520.43440.4574GSPH [10]0.60680.61910.62300.44270.47330.4840Deep methodsDCMH [13]0.73160.73430.74460.52280.54380.5419PRDH [14]0.69520.70720.71080.52380.55210.5572SSAH [15]0.77450.78820.79900.51270.52560.5067TDH [16]0.74230.74780.75120.51640.52220.5276DSePH [9]0.71280.72850.74220.46210.49580.5112QDCMH0.76350.76880.77130.52860.53130.5371T⟶IHandcrafted methodsCMSSH [7]0.57260.57760.57530.37930.38760.3899SePH [9]0.71390.72580.72940.43480.46060.5195SCM [8]0.63400.64580.65410.41180.41830.4345GSPH [10]0.62820.64580.65030.54350.60390.6461Deep methodsDCMH [13]0.76070.77370.78050.48830.49420.5145PRDH [14]0.76260.77180.77550.51220.51900.5404SSAH [15]0.78600.79740.79100.48320.48310.4922TDH [16]0.75160.75770.76340.51980.53320.5399DSePH [9]0.74220.75780.77600.46160.48820.5305QDCMH0.77620.77250.78590.52450.53980.5487Thirdly, to further investigate the performance of QDCMH, we plot the precision-recall curves and topN-precision curves of QDCMH and baseline methods with hash code lengths 64 on datasets MIRFLICKR-25K, Microsoft COCO2014, respectively, as presented in Figures 4 and 5. From this figure, we can see that the precision-recall curves and top N-precision curves are nearly consistent with the MAPs in Table 2.Figure 4
Precision-recall curves on datasets MIRFLICKR-25K and Microsoft COCO2014.
(a)(b)(c)(d)Figure 5
TopN-precision curves on datasets MIRFLICKR-25K and Microsoft COCO2014.
(a)(b)(c)(d)


## 4.1. Datasets

To investigate the performance of QDCMH, we conduct experiments on two benchmark cross-modal retrieval datasets: MIRFLICKR-25K [20] and Microsoft COCO2014 [21], and the brief descriptions of the datasets are listed in Table 1.Table 1
Brief description of the experimental datasets.
DatasetUsedTrainQueryRetrieveTag dimensionLabelsMIRFLICKR-25K20,01510,0002,00018,0151,38624MS-COCO2014122,21810,0005,000117,2182,02680


## 4.2. Evaluation Metrics

In our experiments, we utilize mean average precision (MAP), topN-precision curves (top N Curves), and precision-recall curves (PR Curves) as evaluation metrics; for the detailed description of these evaluation metrics, refer to [22, 23].


## 4.3. Baselines and Implementation Details

We compare our proposed QDCMH method with eight state-of-the-art cross-modal hashing methods, including four handcrafted ones, i.e., cross-modal similarity sensitive hashing (CMSSH) method [7], semantics-preserving hashing (SePH) [9] method, semantic correlation maximization (SCM) method [8], and generalized semantic preserving hashing (GSPH) method [10] and four deep feature-based ones, i.e., deep cross-modal hashing (DCMH) method [13], pairwise relationship guided deep hashing (PRDH) method [14], self-supervised adversarial hashing (SSAH) method [15], and triplet-based deep hashing (TDH) method [16]. Most baseline methods are carefully implemented based on the codes provided by the authors. A few baseline methods are implemented by us following the suggestions and descriptions of the original papers.All the experiments are executed by using the open source deep learning framework pytorch and running on an NVIDIA GTX Titan XP GPU server. In our experiments, we setnI⟶T=nT⟶I=10000, max_epoch=500, and λ=10−5 and the learning rate is initialized to 10−1.5 and gradually decreased to 10−6 in 500 epochs. For those handcrafted feature-based baselines, each image in the two datasets is represented by a bag of words (BoW) histogram or feature vector having 512 dimensions. For the whole experiment, we use I⟶T to denote using a querying image while returning text and T⟶I to denote using a querying text while returning an image.


## 4.4. Performance Evaluation and Discussion

Firstly, we investigate the performance of QDCMH with different hyperparametersβ and γ. To this goal, we experiment on MIRFLICKR-25K with the hash code length k=64 and record the corresponding MAPs under different values of β and γ, as shown in Figure 3. We find that high performance can be acquired when β=1 and γ=0.2.Figure 3
A sensitivity analysis of the hyperparameters. (a) Hyperparameterβ on MIRFLICKR-25K dataset. (b) Hyperparameter γ on MIRFLICKR-25K dataset.
(a)(b)Secondly, to validate the performance of QDCMH, we perform the experiment to compare QDCMH with baseline methods in terms of MAP on datasets MIRFLICKR-25K and MS-COCO2014. Table2 presents the MAPs of each method for different hash code lengths, i.e., 16, 32, and 64. DSePH represents the SePH method whose features of the original images are extracted by CNN–F. From Table 2, we can see that the following. (1) The MAPs of our proposed QDCMH are higher than the MAPs of most baseline methods in most cases, which demonstrates the superiority of QDCMH. We can also observe that SSAH outperforms than our proposed QDCMH in most cases, which is partly because SSAH takes self-supervised learning and generative adversarial networks into account during hash representation learning procedure. (2) The MAPs of QDCMH is always higher than the MAPs of TDH, which shows that quadruplet loss can better preserve semantic relevance than triplet loss in cross-modal hashing retrieval. (3) The MAPs of DSePH is always higher than the MAPs of SePH, which demonstrates that deep neural networks have powerful features learning capacity. (4) Our proposed QDCMH can achieve better performance on MS-COCO 2014 dataset than on MIRFlickr-25K dataset, which is partly because the instances in MS-COCO 2014 dataset belong to 80 categories while the instances in MIRFlickr-25K dataset belong to 24 categories, and this makes the quadruplets generated from the MS-COCO 2014 dataset have better generalization ability than the quadruplets generated from the MIRFlickr-25K dataset.Table 2
Comparison to baselines in terms of MAP on two datasets: MIRFLICKR-25K, and Microsoft COCO2014, respectively. The best accuracy is shown in boldface.
TaskMethodsMIRFlickr-25KMS-COCO16bits32bits64bits16bits32bits64bitsI⟶THandcrafted methodsCMSSH [7]0.56000.57090.58360.54390.54500.5410SePH [9]0.67400.68130.68030.42950.43530.4726SCM [8]0.63540.64070.65560.42520.43440.4574GSPH [10]0.60680.61910.62300.44270.47330.4840Deep methodsDCMH [13]0.73160.73430.74460.52280.54380.5419PRDH [14]0.69520.70720.71080.52380.55210.5572SSAH [15]0.77450.78820.79900.51270.52560.5067TDH [16]0.74230.74780.75120.51640.52220.5276DSePH [9]0.71280.72850.74220.46210.49580.5112QDCMH0.76350.76880.77130.52860.53130.5371T⟶IHandcrafted methodsCMSSH [7]0.57260.57760.57530.37930.38760.3899SePH [9]0.71390.72580.72940.43480.46060.5195SCM [8]0.63400.64580.65410.41180.41830.4345GSPH [10]0.62820.64580.65030.54350.60390.6461Deep methodsDCMH [13]0.76070.77370.78050.48830.49420.5145PRDH [14]0.76260.77180.77550.51220.51900.5404SSAH [15]0.78600.79740.79100.48320.48310.4922TDH [16]0.75160.75770.76340.51980.53320.5399DSePH [9]0.74220.75780.77600.46160.48820.5305QDCMH0.77620.77250.78590.52450.53980.5487Thirdly, to further investigate the performance of QDCMH, we plot the precision-recall curves and topN-precision curves of QDCMH and baseline methods with hash code lengths 64 on datasets MIRFLICKR-25K, Microsoft COCO2014, respectively, as presented in Figures 4 and 5. From this figure, we can see that the precision-recall curves and top N-precision curves are nearly consistent with the MAPs in Table 2.Figure 4
Precision-recall curves on datasets MIRFLICKR-25K and Microsoft COCO2014.
(a)(b)(c)(d)Figure 5
TopN-precision curves on datasets MIRFLICKR-25K and Microsoft COCO2014.
(a)(b)(c)(d)


## 5. Conclusions

In this paper, we introduce a quadruplet loss into deep cross-modal hashing to fully preserve semantic relevance of original cross-modal quadruple instances and propose a quadruplet based deep cross-modal hashing method (QDCMH). QDCMH integrates quadruplet-based cross-modal semantic relevance preserving module, hash representation learning, and hash code generation into an end-to-end framework. Experiments on two benchmark cross-modal retrieval datasets demonstrate the efficiency of our proposed QDCMH.



---

*Source: 9968716-2021-07-02.xml*