Distribution Consistency Guided Hashing for Cross-Modal Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-04 07:45:03

Abstract: With the massive emergence of multi-modal data, cross-modal retrieval (CMR) has become one of the hot topics. Thanks to fast retrieval and efficient storage, cross-modal hashing (CMH) provides a feasible solution for large-scale multi-modal data. Previous CMH methods always directly learn common hash codes to fuse different modalities. Although they have obtained some success, there are still some limitations: 1) These approaches often prioritize reducing the heterogeneity in multi-modal data by learning consensus hash codes, yet they may sacrifice specific information unique to each modality. 2) They frequently utilize pairwise similarities to guide hashing learning, neglecting class distribution correlations, which do not notably contribute to reducing differences among various modalities. To overcome these two issues, we propose a novel Distribution Consistency Guided Hashing (DCGH) framework. Specifically, we first learn the modality-specific representation to extract the private discriminative information. Further, we learn consensus hash codes from the private representation by consensus hashing learning, thereby merging the specifics with consistency. Finally, we propose distribution consistency learning to guide hash codes following a similar class distribution principle between multi-modal data, thereby exploring more consistent information. Lots of experimental results on four benchmark datasets demonstrate the effectiveness of our DCGH on both fully paired and partially paired CMR tasks.

Main paper: 
 introduction: 
 WITH the rapid development of social media and networks, the volume of multimedia data has experienced a massive expansion, typically encompassing diverse cross-modal data types like text, images, and videos. Such a swift proliferation of multimedia data highlights the pressing necessity for the efficient storage and retrieval of these extensive volumes of data. Therefore, cross-modal retrieval (CMR) [25, 35, 37, 42] is gaining significant attention as a fundamental task for retrieving semantically correlated instances across various query modalities. Recently, a considerable body of research on CMR has advocated for the acquisition of a unified, real-valued representation that encompasses all modalities, thereby mitigating the heterogeneous disparities in multi-modal data. While these methods have shown effectiveness, they compromise query and training efficiency. Hence, the new challenge lies in how to accurately and efficiently search for semantic similarities across modalities.Owing to the great advantages of fast query and efficient storage for extensive multi-modal data, the hashing techniques [17, 24, 40] provide a feasible solution to apply in large-scale CMR. Therefore, many cross-modal hashing (CMH) methods have been proposed, whose key idea is to project cross-modal data into the Hamming space as binary hash codes while preserving the inherent semantic similarities. Then, the similarities between different instances can be calculated as Hamming distances through the XOR operation [20] , resulting in minimal computational costs. The existing CMH methods can be broadly classified into two main types: supervised and unsupervised. As for supervised ones, some methods directly use label information to guide hash learning [7, 19, 22, 34] , while others firstly use labels to construct similarity and then proceed to learn hash codes [16, 27, 28, 39] . For unsupervised CMH, some methods capture the structural similarity through graph learning [30, 36, 38] , while others use matrix factorization to capture semantic similarity [2, 5, 26] . Generally, supervised CMH outperforms unsupervised ones by leveraging semantic information.Although existing supervised methods have achieved significant progress, they still exhibit certain limitations that need to be addressed: (1) These methods tend to eliminate the heterogeneous gap of multi-modal data by learning the consensus hash codes, which leading to sacrifice modality-specific information from each modality; (2) They often rely on pairwise similarities to learn hash codes and ignore the spatial distribution correlations, which could not be able to fully mine consensus class information.To address the aforementioned issues, we propose a Distribution Consistency Guided Hashing framework (DCGH) for cross-modal retrieval. DCGH considers that each modality-specific representation should obey the unified prior class information. In other words, they should enjoy the same class centers as the original multi-modal data. The basic framework of DCGH is depicted in Fig. 1 . Specifically, our DCGH first learns the modality-specific representation to extract the modality-specific discriminative information from each modality, thereby improving the quality of each private property to some extent. Then, DCGH adopts consensus hashing learning to represent the consistent structural information of all modalities. Finally, based on the principle of class center consistency between multi-modal data, DCGH proposes a distribution consistency learning strategy that enables dual representations (i.e., modality-specific representation and consensus hash codes) to follow a similar class distribution, thereby exploring more consistent information. We summarize the following contributions from this paper:• We propose a novel distribution consistency guided hashing for CMR. To our best knowledge, it is the first to simultaneously investigate modality-specific and distribution consistency learning, which enables a comprehensive exploration of the inherent correlations between all modalities. • In order to promote the modality-specific discriminative representations to more realistically reflect the data distribution of all modalities, we propose distribution consistency learning that makes each specific representation obey the same class centers as the original data.• Extensive experiments verify that our DCGH has better performance than some state-of-the-art comparison methods on four widely used datasets.  
related work 2.1 unsupervised cross-modal hashing: 
 Since unsupervised CMH does not rely on the label semantic information provided by the original labels, it typically learns hash codes by exploring the structure similarities of the raw data. It can typically be categorized into matrix factorization-based methods and graph-based methods. To better explore the inherent structure of modalities, latent structure discrete hashing factorization (LSDHF) [5] extracts hash codes by aligning the eigenvalues of the similarity matrix. Regarding graph-based methods, adaptive structural similarity preserving hashing (ASSPH) [11] proposes an asymmetric semantic learning approach and uses graph-based structure to measure similarity. To better evacuate and preserve the intrinsic cross-modal semantics, correlation-identity reconstruction hashing (CIRH) [41] constructs a cross-modal collaborative graph to model heterogeneous cross-modal correlations. Considering the potent feature extraction and pattern analysis ability of deep neural networks (DNNs), there is a continuous emergence of unsupervised CMH methods based on DNNs. For instance, deep adaptivelyenhanced hashing (DAEH) [18] proposes an adaptively-enhanced optimization strategy and to learn hash functions. To address the performance degradation caused by hash binarization optimization, unsupervised contrastive cross-modal hashing (UCCH) [6] proposes a framework that performs hash optimization in contrastive learning. Although unsupervised CMH generally achieves satisfactory performance, they still exhibit a noticeable gap compared with supervised CMH.Supervised CMH, which uses semantic information provided by labels to guide hashing learning, usually performs better than unsupervised CMH. To model the potential manifold structure between heterogeneous data, label guided discrete hashing (LGDH) [9] simultaneously learns the manifold structure and hash codes with the guidance of labels. In order to bridge the semantic loss in the hashing process, scalable pairwise embedding constraint hashing (SPECH) [31] measures the semantic correlation of cross-modal data by using paired data to calculate the loss of likelihood similarity. More recently, asymmetric learning strategies have also been introduced into the CMH field. For instance, scalable asymmetric discrete cross-modal hashing (BATCH) [29] uses distance minimization to embed semantic information into hash codes, and an asymmetric strategy is proposed to bridge the gaps between the shared space and the hash space. Aimed at investigating the potential relevance of multi-label semantics, adaptive label correlation based asymmetric cross-modal hashing (ALECH) [10] proposes an asymmetric strategy to connect different feature spaces and adaptively learn latent features utilizing higher-order semantic labels. To handle the intrinsic correlations across modalities, asymmetric supervised fusion-oriented hashing (ASFOH) [32] establishes correlations between the shared latent representations and the semantic label matrix. Moreover, deep-based methods such as attention-aware deep cross-modal Hashing (TEACH) [33] , which designs modalityspecific attention modules for learning hash codes, also demonstrate impressive performance.In general, existing CMH methods typically project multi-modal data into shared hash codes. Although these methods achieve promising performance, most of them overlook the fact that features from each modality contain both common semantics and modality-specific semantics. Additionally, they always rely on pairwise similarities to promote hashing learning. More importantly, they have yet to consider the preservation of class distribution consistency. To this end, we propose distribution consistency guided hashing to enhance both the expression and discriminative abilities of hash codes. 
proposed method 3.1 notations: 
 In this paper, we denote the number of instances, classes, and modalities as 𝑛, 𝑐, and 𝑚, respectively. Then, multi-modal data can be represented as𝑶 𝑣 = [𝒐 𝑣 1 , 𝒐 𝑣 2 , • • • , 𝒐 𝑣 𝑛 ] ∈ R ℎ 𝑣 ×𝑛 (𝑣 = 1, 2, • • • , 𝑚), where ℎ 𝑣 is the feature dimensionality of the 𝑣-th modality. The corresponding labels are 𝒀 ∈ {0, 1} 𝑐 ×𝑛 . To be specific, if the 𝑗th instance comes from the 𝑖-th class, its label is 𝒀 𝑖 𝑗 = 1, otherwise 𝒀 𝑖 𝑗 = 0. We adopt ℓ 2 -norm to normalize the i-th column label, i.e., 𝒀 𝑖 = 𝒀 𝑖 ∥𝒀 𝑖 ∥ 2 . To extract the nonlinear features from multimodal data, we use radial basis function (RBF) kernel mapping to generate kernel features. To be specific, we randomly choose 𝑑 samples from each modality as anchors 𝒂 𝑣 𝑖 and use the Gaussian kernel function to obtain the nonlinear features. Therefore, the kernel features of the 𝑣-th modality can be represented as𝑿 𝑣 𝑖 = [𝑒𝑥𝑝 ( ∥𝒐 𝑣 𝑖 -𝒂 𝑣 1 ∥ 2 2 -2𝜎 2 ), 𝑒𝑥𝑝 ( ∥𝒐 𝑣 𝑖 -𝒂 𝑣 2 ∥ 2 2 -2𝜎 2 ), • • • , 𝑒𝑥𝑝 ( ∥ 𝒐 𝑣 𝑖 -𝒂 𝑣 𝑑 ∥ 2 2 -2𝜎 2 )] 𝑇, where 𝜎 is the kernel width.CMH aims to learn compact hash codes while maintaining the inherent similarities of the original features in Hamming space. As is known to us all, learning consensus discriminative hash codes from multi-modal data has a significant challenge. Most CMH methods mainly emphasize leveraging the consistency of multi-modal data to learn consensus hash codes, which largely ignore modality-specific properties. Consequently, these methods cannot comprehensively explore the underlying data distribution among different modalities.To this end, we first employ modality-specific representations 𝑩 𝑣 to capture the private property of each modality by the following formula. minEQUATIONwhere 𝑾 𝑣 is the independent orthogonal projection matrix. In Eq.1, we adopt the bit decorrelation and balance constraints (i.e., (𝑩 𝑣 ) ⊤ 1 = 0, (𝑩 𝑣 ) ⊤ 𝑩 𝑣 = 𝑛𝑰 ) to encourage the compactness of modality-specific representations. Considering that multi-modal data encompasses information from the same instance, there should exist some shared information between different modalities. Further, we propose consensus hashing learning to excavate the common semantic information from modality-specific representations. Specifically, we preserve the similarities between the modality-specific properties and consensus hash codes 𝑯 from different modalities. Mathematically, the above problem can be represented as minEQUATIONwhere 𝛽 is the trade-off parameter. In Eq.2, we can extract the common semantic information from modality-specific representations, thereby obtaining the consensus hash codes. As shown in Fig. 1 , multi-modal data from different modalities have different latent distributions. However, due to the lack of any prior information, orthogonal projection matrices could not be able to extract accurate modality-specific properties from multi-modal data. In other words, the learned specific and consensus representations lack the guidance of consistent information, thereby leading to deviation from the real data distribution. Naturally, we consider that each modality should adhere to a unified class center consistency criterion in order to mitigate disparities within latent distributions. Afterward, we propose distribution consistency learning to ensure that specific and consensus representations adhere to the same class centers as the original data. Leveraging this concealed prior information, consensus hash codes can tap into a richer reservoir of underlying consensus data. Mathematically, we denote the problem as minEQUATIONwhere 𝑭 𝑣 ∈ R 𝑐 ×𝑛 and 𝑬 ∈ R 𝑐 ×𝑛 are the encoding matrices, and 𝑼 ∈ R 𝑙 ×𝑐 is the expected class center. We can observe that these matrices are orthogonal, which means each column of their matrices is independent. In essence, it can identify the most representative points to serve as class centers, thereby guiding hashing learning to aggregate iteratively hash codes around each class center. Theoretically, distribution consistency learning offers the following two advantages: (1) it ensures that the projection matrices effectively map the original data closer to their respective class centers, thus mitigating the differences within latent distributions;(2) specific and consensus representations share the same class centers, which can sufficiently excavate the intrinsic correlation among each modality to improve the quality of hash codes.In general, we can obtain the following unified objective function, i.e., minEQUATIONwhere 𝛼 and 𝛽 are the trade-off parameters.In this section, we adopt the alternating solution strategy to optimize our proposed objection function. Specifically, we optimize one variable in each iteration while fixing other variables, and repeat all steps iteratively. ▶ 𝑩 𝑣 -Step: Fixing the irrelevant variables except 𝑩 𝑣 , we can solve the optimization problem, i.e., minEQUATIONThen, we can transform Eq.5 as the trace form, i.e.,EQUATIONwhere𝑱 𝑣 = 𝑾 𝑣 𝑿 𝑣 + 𝛼𝑼 𝑭 𝑣 + 𝛽𝑟 𝑯 𝒀 ⊤ 𝒀 . We first define Δ = 𝑰 𝑛 - 1 𝑛 1 𝑛 1 ⊤ 𝑛 .Then, we compute the singular value decomposition (SVD) of 𝑱 𝑣 Δ(𝑱 𝑣 ) ⊤ as follows .EQUATIONAfterwards, we define 𝑸 𝑣 = Δ(𝑸 𝑣 ) ⊤ 𝑺 𝑣 (Λ 𝑣 ) -1/2 ∈ R 𝑟 ×𝑟 * and the random matrix Q𝑣 ∈ R 𝑟 × (𝑟 -𝑟 * ) . Finally, according to [35] , we can obtain the optimal solution of 𝑩 𝑣 as followsEQUATIONWhen 𝑟 = 𝑟 * , Ŝ𝑣 , 𝑺 𝑣 , and Q𝑣 are empty.▶ 𝑯 -Step: Dropping the irrelevant variables, the 𝑯 -subproblem can be expressed as minEQUATIONEquivalently, we can transform Eq.9 into the following problem max 𝑯 𝑡𝑟 ((EQUATIONTherefore, the solution can be obtained as followsEQUATIONwhere 𝑠𝑔𝑛(•) represents the element-wise indicator operator. ▶ 𝑾 𝑣 -Step: To update 𝑾 𝑣 , we fix the other variables to solve the subproblem as follows minEQUATIONThen, we adopt the trace operator to replace Eq.12. Thus, we can obtain maxEQUATIONWe compute the singular value decomposition (SVD) on 𝑩 𝑣 (𝑿 𝑣 ) ⊤ . Therefore, 𝑾 𝑣 can be solved by 𝑴 𝑣 𝑤 (𝑵 𝑣 𝑤 ) ⊤ , where 𝑴 𝑣 𝑤 and 𝑵 𝑣 𝑤 are the left and right singular values, respectively. ▶ 𝑼 -Step: To obtain 𝑼 , the subproblem can be rewritten as minEQUATIONSimilar to problem 1, we have maxEQUATIONTherefore, the optimal solution can be computed by SVD, i.e., 𝑴 𝑣 𝑢 (𝑵 𝑣 𝑢 ) ⊤ . ▶ 𝑭 𝑣 -Step: To update 𝑭 𝑣 , the Eq.5 can be formed as minEQUATIONSimilar to problem 1, we have maxEQUATIONThen, we employ SVD to compute the subproblem, i.e., 𝑴 𝑣 𝑓 (𝑵 𝑣 𝑓 ) ⊤ . ▶ 𝑬-Step: We drop irrelevant terms and obtain the following simplified problem minEQUATIONHence, similar to problem 1, we can obtain the solution by SVD, i.e., 𝑴 𝑒 (𝑵 𝑒 ) ⊤ .Once we generate the consensus hash codes, we can further learn hash function to achieve CMR for new queries. Specifically, we can obtain modality-specific hash function by linear regression, i.e., minEQUATION)where 𝜆 is the trade-off parameter that avoids the trivial solution.Afterwards, we can directly optimize hash function 𝑷 𝑣 asEQUATIONThen, to achieve out-of-sample extension, we map all query samples into binary codes as followsEQUATIONwhere 𝑲 𝑣 is the kernelized features of query samples.Our proposed DCGH mainly contains the computational complexity of each subproblem. To be specific, the computational complexity includes O (𝑉 𝑙𝑑𝑛 +𝑉 𝑙𝑑𝑛 +𝑐𝑙𝑛 +𝑐  
experiments 4.1 datasets: 
 Four widely used benchmark datasets are meticulously selected and preprocessed to fulfill the requirements of our experiments, including WIKI [15] , MIRFlickr-25K [8] , IAPR-TC12 [4] , and NUS-WIDE [3] . We randomly choose 693 data pairs from WIKI to serve as the query set, while 1867 pairs from NUS-WIDE. For the other two datasets, we randomly choose 2,000 pairs from each as the query set. The remaining portions of the four datasets constitute the training set. The detailed dataset statistics could be found in the supplementrary materials. To evaluate the performance of our DCGH, we conduct some experiments for 11 state-of-the-art CMH methods, including RFDH [23] , LCMFH [21] , MTFH [12] , FCMH [28] , FDDH [13] , BATCH [29] , EDMH [1] , DAH [37] , ALECH [10] , WASH [35] and AMSH [14] . For the fairness, the parameters of baseline methods are consistent with those in the original literature. For the hyper-parameters (i.e., 𝛼, 𝛽, 𝜆), we set them as {10 1 , 10 1 , 10 -2 }, {10 4 , 10 -1 , 10 -2 }, {10 3 , 10 -2 , 10 -3 }, and {10 4 , 10 -1 , 10 -4 } on the previous mentioned four datasets, respectively. Moreover, the number of anchors in the RBF operation is configured as 1500, while the maximum iteration number is set to 10.For ease of expression, we use the image and text modalities as an illustrative example to perform CMR tasks. To evaluate the performance of our proposed DCGH, we choose two typical CMR tasks: using visual data to search relevant textual data (i.e., Image → Text task) and using textual data to search relevant visual data (i.e., Text → Image task). Besides, we employ mean Average Precision (mAP) and Precision-Recall (PR) curves as evaluation metrics for the two retrieval tasks. In practical retrieval scenarios, cross-modal data is often collected from image-text pairs on the Internet, which inevitably leads to some mismatched pairs, i.e., unpaired data. In our experiments, we randomly shuffle the first 25% of image data and the last 25% of text data to simulate the situation with 50% unpaired data. Therefore, considering the prevalence of unpaired data in practical retrieval scenarios, we evaluate the performance of DCGH with both fully paired and partially paired settings on four datasets. In the scenario where data is fully paired, we adjust the hash lengths from 8 to 64 (i.e., 8, 16, 32, 64 bits) to assess the performance of our DCGH. Table 1 showcases the mAP results of DCGH and other baselines with different hash lengths on four datasets. Additionally, Fig. 2 and Fig. 3 illustrate the PR and Top-k precision curves for various methods with 8-bit hash codes on four datasets. It is imperative to highlight that the MTFH method cannot execute on the NUS-WIDE dataset due to memory overflow. According to these results, we can obtain the following observations: (1) The the quality of hash codes, thereby obtaining higher performance.(2) Generally, the performance of Text → Image tasks is better than that of Image → Text tasks. This could be attributed to the fact that features extracted from textual information contain more semantic information. (3) As the hash length increases, the mAP scores of all comparison methods also improve. This proves that longer hash codes are capable of encoding more semantic information. Furthermore, we find that DCGH achieves more significant performance improvements in cases with shorter hash code lengths. This indicates that the consensus hashing learning strategy used in our DCGH can learn more compact and discriminative hash codes. ( 4) By comparing the PR curves of various methods, we find that the coverage area of DCGH exceeds almost all baselines. Furthermore, the top-k precision curves illustrate that DCGH achieves nearly the highest precision across different k values. These observations underscore the superiority of DCGH in terms of query coverage and accuracy.On three partially paired datasets, we adjust the code lengths from 8 to 64 bits to evaluate the performance of our DCGH. The mAP scores of all methods are illustrated in Table 2 . In addition, to show the performance under different proportions of unpaired data, we draw the corresponding mAP results on MIRFlickr-25K with 8 bits in Fig. 4 . Based on these results, we can obtain the following observations: (1) On three partially paired datasets, our DCGH still exhibits the highest performance, which demonstrates its high robustness against unpaired data. With the hash length increases, the mAP scores of all comparison methods also improve. (2) The performance of all methods decreases as the proportion of unpaired data increases. This could be attributed to the significant impact of the semantic information provided by labels on supervised learning. (3) Under all proportions of unpaired data and different code lengths on MIRFlickr-25K, our DCGH achieves minimal performance degradation. This could be attributed to our proposed distribution consistency learning strategy, which utilizes the latent class centers to alleviate the errors introduced by mismatched pairs. (4) Similar to the fully paired situation, the performance of all methods increases with the growth of code length, as longer code lengths can carry more semantic information.We investigate the sensitivity of our DCGH on four datasets, which involve three hyper-parameters, namely, 𝛼, 𝛽, and 𝜆.Parameter 𝜶 and 𝜷 determine the contribution of distribution consistency learning and consensus hash learning, respectively. To analyze the parameters sensitivity, we adopt grid search and set the range of 𝛼 and 𝛽 as [10 -4 , 10 4 ]. The corresponding mAP scores are depicted in Fig. 5 . It is evident that DCGH consistently achieves superior results across most ranges, demonstrating the stability of our approach.Parameter 𝝀 serves as the regularization coefficient in the hash function learning phase, and Fig. 6 shows the mAP results of DCGH with different 𝜆 values on four datasets. Clearly, within the range of [10 -4 , 10 -1 ], the mAP curves remain almost horizontal, indicating that the values of 𝜆 are feasible within this range. In addition, we further discuss the impact of eliminating the 𝜆 parameter on the performance of DCGH in the ablation analysis section.As previously mentioned, we can iteratively solve each sub-problem to obtain the optimal solution. In order to illustrate the convergence characteristics of DCGH more intuitively, we draw the convergence curves and mAP curves across four datasets in Fig. 7 . From the figure, we can make the following observations: (1) The objective values decline rapidly within the first five steps and manage to converge before 10-th step. (2) Similarly, the mAP values consistently increase with the progression of the iterative steps, attaining stable points around the 10-th step. These results demonstrate that our proposed DCGH boasts rapid and stable convergence advantages.To show the computational complexity of DCGH and its scalability on large datasets, we conduct experiments on NUS-WIDE with varying hash code lengths and data scales. The bit length extends from 8 to 64, and the data scale varies from 10K to 184K. All the results are depicted in Fig. 8 . The observation reveals that the time To further validate the effectiveness and necessity of the individual components in our proposed DCGH, we perform an ablation study on three variants of DCGH across four datasets. Specifically, DCGH-d removes the distribution consistency learning component represented in Eq.3, DCGH-r eliminates the regularization term, i.e., the second term in Eq.19, and DCGH-c discards the consensus hashing learning strategy. It should be noted that discarding the consensus hashing learning strategy implies abandoning the overarching DCGH framework. Hence, we utilize FCMH [28] as the substitute for DCGH-c in our study. Table 3 shows the mAP scores of DCGH and its variants across four datasets. Overall, the original DCGH consistently exhibits superior performance compared to its variant models in the majority of scenarios. However, the performance of DCGH-r drops most significantly because, without the constraint of the regularization term, the method can be susceptible to converge into a trivial solution. In addition, DCGH-d and DCGH-c also experience noticeable performance degradation, which proves the importance of distribution consistency learning and consensus hashing learning to DCGH, respectively.To provide an intuitive understanding for our proposed DCGH, we utilize t-SNE to visualize the original data, the learned modalityspecific representations, and the final consensus hash codes on the WIKI dataset. Fig. 9 (a) and (b) show the visualization results of the original image and text data. We can observe that the original data is chaotic and indistinguishable. Fig. 9 (c) and (d) depict the spatial distribution of the two modality-specific representations. It can be observed that the specific representations from different categories start to exhibit a preliminary class structure. Fig. 9 (e) represents the final consensus hash codes, which indicates our DCGH enjoys more discrimination and a more compact class structure. Moreover, we also adopt t-SNE to visualize the learned consensus hash codes and class centers as shown in Fig. 10 . We can observe that instances from the same category cluster around their respective class centers under the guidance of the class centers. This demonstrates the effectiveness of the proposed distribution consistency learning strategy. 
conclusion: 
 In this paper, we propose a novel distribution consistency guided hashing (DCGH) for cross-modal retrieval, which can handle the cases of paired and unpaired. To overcome the heterogeneity of multi-modal data, we learn consistent hash codes from the extracted modality-specific representation, thereby endowing the learned hash codes more semantics with specific and consistent information. Moreover, we propose distribution consistency learning to project the original data closer to their respective class centers, thus mitigating the differences within latent distributions. Experimental results on four benchmarks demonstrate that our DCGH outperforms 11 state-of-the-art CMH methods. 
 
 Appendix:  


Decision: After rebuttal, the paper finally received four consistent positive recommendations. The reviewers agreed that the paper was innovative, well-experimented and convincing. I agree with the reviewers' suggestions and vote to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel distribution consistency guided hashing (DCGH) for cross-modal retrieval, which can handle the cases of paired and unpaired. The proposed distribution consistency learning aims to project the original data closer to their respective class centers, thus mitigating the differences within latent distributions. Experimental results on four benchmarks demonstrate that the proposed DCGH outperforms 11 state-of-the-art CMH methods. In general, this paper has clear motivation and is interesting.
strength_weakness: strength_weakness: - The structure of the paper is clear and well-written, which can make it easy for the readers to follow the proposed methodology.
- The idea is interesting and the proposed method is reasonable. The distribution consistency learning can make each specific representation to obey the same class centers as the original data, thereby exploring more consistent information.
- The experimental evaluation is impressive and convincing. A series of experimental results on four benchmarks demonstrate that the proposed DCGH outperforms 11 state-of-the-art CMH methods under the cases of paired and unpaired data.
- The proposed distribution consistency learning is novel, which can makes each specific representation obey the same class centers as the original data.
limitations: limitations: - For Figure 4, the authors should uniformly use the IAPR-TC12 dataset to present experimental results with different rates of unpaired data.
- The authors claim that the computational complexity of the proposed method is O(n), so the training time of all methods should be given for comparison.
- For t-SNE visualizations, the image representation has a more cluttered data structure than the text representation. The authors should further analyze it.
- For Figure 7, the legend seems to be wrong. What does Img mAP means?
- Authors should further analyze why different proportions of unpaired data has little impact on retrieval performance.
paper_summary: paper_summary: This paper proposes a novel distribution consistency guided hashing (DCGH) for cross-modal retrieval, which can handle the cases of paired and unpaired. Specifically, DCGH first learns consistent hash codes from the extracted modality-specific representation, thereby endowing the learned hash codes more semantics with specific and consistent information. Then, distribution consistency learning is proposed that projects the original data closer to their respective class centers, thus mitigating the differences within latent distributions. Lots of experimental results show the effectiveness of DCGH.
strength_weakness: strength_weakness: 1.The framework diagram and visualization results help the reader to effectively understand the algorithm motivation and flow.
2.The concepts of distribution consistency learning seem be interesting, and the paper is well-organized.
3. The work seems to be novel, and this paper has clear motivations and reasonable. It is the first to simultaneously investigate modality-specific and distribution consistency learning, which enables a comprehensive exploration of the inherent correlations between all modalities
4. The effectiveness and superiority are well demonstrated by experimental results with state-of-the-art CMH methods on four datasets with both fully paired and partially paired.
limitations: limitations: 1.How the author judged the centers of distribution consistency.
2.The baselines exclude any deep learning hashing methods. The authors should compare some deep hashing methods.
3.There are some typos, such as the legend of Figure 7.
4.Some class center consistency methods have been adopted in some hashing methods. The authors should give some discussions about them. Such as Two-stream deep hashing with class-specific centers for supervised image search, Joint specifics and consistency hash learning for large-scale cross-modal retrieval.
paper_summary: paper_summary: This paper presents a novel Distribution Consistency Guided Hashing (DCGH) for cross-modal retrieval. It adopts the modality-specific representation to extract the private discriminative information. Subsequently, DCGH learns consensus hash codes from the private representation by consensus hashing learning, thereby merging the specifics with consistency. Finally, the authors proposes distribution consistency learning to guide hash codes following a similar class distribution principle between multi-modal data, thereby exploring more consistent information. 
Expensive experiments demonstrate the effectiveness of the proposed DCGH on both fully paired and partially paired cross-modal retrieval tasks.
strength_weakness: strength_weakness: (1) This work is interesting and novel, which proposes distribution consistency learning that makes each specific representation obey the same class centers as the original data. 
(2) The paper is well written and the motivation is clear. The existing methods often rely on pairwise similarities to learn hash codes and ignore the spatial distribution correlations, which could not be able to fully mine consensus class information. 
(3) Extensive experimental results show the DCGH outperforms state-of-the-art methods on the evaluation metrics. The analysis on distribution consistency learning is performed effectively.
limitations: limitations: (1) In Figure 2, authors should analyze the reason why DSFH seems to perform worse than AMSH on image retrieval text tasks.
(2) For unpaired data, the authors should analyze why the performance drop is not obvious under different ratios?
(3) The English presentation of the work needs to be improved. Such as '\lambda variations' in line 692.
(4) For Figure 6, what happens to the retrieval performance when the value of \lambda is small.
paper_summary: paper_summary: The paper propose a Distribution Consistency Guided Hashing (DCGH) framework, which learns
the modality-specific representation to extract the private discriminative information and consensus hash codes from
the private representation by consensus hashing learning, thereby merging the specifics with consistency. Also, a distribution consistency learning  between multi-modal data is proposed. DCGH first extracts modality-specific representations to enhance the quality of private properties in each modality. It then employs consensus hashing learning to capture consistent structural information across all modalities. By ensuring a similar class distribution for both modality-specific representations and consensus hash codes, DCGH explores more consistent information for improved retrieval accuracy. The framework's contributions include simultaneously investigating modality-specific and distribution consistency learning, enabling a comprehensive exploration of correlations between modalities.Overall, DCGH offers a promising approach to CMR by balancing modality-specific details with consensus information, ultimately enhancing retrieval efficiency and performance for large-scale multi-modal datasets.
strength_weakness: strength_weakness: - English writing is easy to follow.
- Experiment is suffient.
- Theory proof is consummate.


In paper novelity, this paper has highlight such as:
- By extracting modality-specific representations, DCGH can capture the unique discriminative features present in each modality. This allows for a more detailed and nuanced representation of the data, improving the quality of the learned features.
- The inclusion of modality-specific representations helps in better capturing the inherent characteristics of each modality, leading to improved retrieval accuracy.
limitations: limitations: - The framework figure should be bigger.
- The citation is not suffient, more categories can be summaried in related work part. 
- more categories can be summaried in related work part. 
-  Some insignificant figure can place in supplementary material, such as Fig.7.