Title: Dynamic Masking and Auxiliary Hash Learning for Enhanced Cross-Modal Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: Cross-modal hash retrieval

Auxiliary hash

Dynamic masking
Abstract: The demand for multimodal data processing drives the development of information technology. Cross-modal hash retrieval has attracted much attention because it can overcome modal differences and achieve efficient retrieval, and has shown great application potential in many practical scenarios. Existing cross-modal hashing methods have difficulties in fully capturing the semantic information of different modal data, which leads to a significant semantic gap between modalities. Moreover, these methods often ignore the importance differences of channels, and due to the limitation of a single goal, the matching effect between hash codes is also affected to a certain extent, thus facing many challenges. To address these issues, we propose a Dynamic Masking and Auxiliary Hash Learning (AHLR) method for enhanced cross-modal retrieval. By jointly leveraging the dynamic masking and auxiliary hash learning mechanisms, our approach effectively resolves the problems of channel information imbalance and insufficient key information capture, thereby significantly improving the retrieval accuracy. Specifically, we introduce a dynamic masking mechanism that automatically screens and weights the key information in images and texts during the training process, enhancing the accuracy of feature matching. We further construct an auxiliary hash layer to adaptively balance the weights of features across each channel, compensating for the deficiencies of traditional methods in key information capture and channel processing. In addition, we design a contrastive loss function to optimize the generation of hash codes and enhance their discriminative power, further improving the performance of cross-modal retrieval. Comprehensive experimental results on NUS-WIDE, MIRFlickr-25K and MS-COCO benchmark datasets show that the proposed AHLR algorithm outperforms several existing algorithms.


Peer reviews:
comments: SUMMARY:
This paper proposed a novel hash learning method for hash learning enhanced cross-modal retrieval. The main challenge in hash learning based cross-model retrieval is the mismatch problem among different modalities, thus leading to wrong retrieval. The proposed method contains two components: Dynamic Masking (DM) and Auxiliary Hash Learning (AHLR). The experimental results show that DM and AHLP could successfully work for hash learning enhanced cross-modal retrieval.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The overall method is sound.

2. The proposed method achieves the SOTA results.

3. The writing is clear and easy to follow.

Weaknesses:

1. Some necessary details about the DM strategy are missing. Based on the ablation study, DM is important for the overall method and even important than AHLR. However, this paper only describes the DM strategy with a few contents in Sec. 2.1. The author should discuss more about the DM such as the rate that should be masked for image features, which may further enhance the contribution of this paper.


To sum up, I think this paper is sound and well-written. The proposed method solves a challenge in hash learning. Meanwhile, i did not find any technical flaws. Thus, i rate it as accept.

QUESTIONS:
1. Regarding the DM policy, can the mask cause a negative influence, such as slow convergence speed?

2. Can the author report the computation cost for the proposed method? The motivation for this question is that hash learning suffers from the convergent problem. This paper introduces an additional loss. Thus, I guess the proposed method needs extra training epochs, hindering it from being implemented in real tasks.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a cross-modal retrieval method called AHLR, which automatically filters and weights key information in images and texts through the dynamic mask mechanism, and combines the auxiliary hash layer to adaptively balance the feature weights of each channel, effectively solving the shortcomings of traditional methods in capturing key information and channel processing, and significantly improving retrieval accuracy. The experimental results show that the AHLR method performs well on multiple benchmark datasets, providing a new solution for cross-modal retrieval.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A novel method of dynamic masking and auxiliary hash learning (AHLR) has been proposed, which effectively solves the problems of channel information imbalance and insufficient capture of key information by jointly utilizing dynamic masking and auxiliary hash learning mechanisms.\n2. By constructing an auxiliary hash learning layer, the problem of traditional hash layers ignoring importance differences when processing different channel features has been solved, enhancing the ability to capture key information and effectively suppressing noise and redundant information.\n3. The experiment was validated on three benchmark datasets, NUS-WIDE, MIRFlickr25K, and MS-COCO, demonstrating the performance of the method.\n\nWeaknesses:\n1. There is a typo in the scale of the NUS-WIDE dataset in Table 3, which needs to be corrected.\n2. The depth of performance experiment analysis needs to be strengthened, and the reasons for obtaining relevant experimental results need to be analyzed.\n3. In practical applications, if there are problems such as noise, missing values, or inaccurate labeling in the input data, it may affect the performance of the model and require additional data cleaning and preprocessing work to ensure data quality.\n\nQUESTIONS:\n1. Has a differentiated masking strategy been used for sequence length differences in different modalities, such as images and text?\n2. Is there a detailed design for the network structure of the cross-modal fusion module, such as the number of layers and attention heads? \n3.How to ensure semantic alignment of different modal features during the fusion process?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces dynamic Masking and Auxiliary Hash-Learning (AHLR) for cross-modal (image–text) retrieval. A dynamic masking block first filters out redundant visual patches and text tokens, keeping only the most informative ones. An auxiliary hash-learning layer then re-weights channel features before hashing, addressing the imbalance between modalities. Trained with a contrastive loss, the two components jointly yield markedly better retrieval accuracy on several widely-used benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n+ The auxiliary hash-learning (AHL) layer assigns data-dependent weights to each channel, generating more discriminative hash codes and improving cross-modal retrieval precision.\n\n+ Dynamic masking, paired with a contrastive loss, automatically highlights key regions/tokens and enforces tighter semantic alignment between images and text.\n\n+ Experiments on three public datasets show consistent, often sizeable, gains over strong baselines. Furthermore, ablations isolate the contribution of each module.\n\n** Weaknesses**\n\n- Dynamic masks plus AHL add parameters and operations, yet no analysis of training/inference latency or memory use is provided.\n\n- While addressing channel information imbalance and insufficient key information capture is a novel perspective in hash learning, there are some literatures adopting similar thoughts. For example, [A, B] model the relevance between visual and textual modalities, and [C] addresses modality imbalance across modalities. It would strengthen the paper to draw clearer connections with these related works and explicitly clarify the differences. [A] Noise-robust Vision-language Pre-training with Positive-negative Learning. [B] Cross-modal Retrieval with Noisy Correspondence via Consistency Refining and Mining. [C] Test-time Adaption against Multi-modal Reliability Bias.\n\n- The paper states that image and text features are fused via a Transformer, but omits the exact token construction, positional encoding, and cross-attention setup—making reproduction difficult.\n\nQUESTIONS:\nQ1. What are the actual training and inference runtimes (or FLOPs/parameter counts) of AHLR versus DCHMT and UCCH on MS-COCO?\n\nQ2. What are the advantages of the channel weight allocation strategy of the auxiliary hash layer compared to classical methods such as SENet?\n\nQ3. Equation (5) imposes a causal-style mask. How does this pattern select “key” tokens? Would a learnable top-k pruning score be superior?\n\nQ4. Could the dynamic mask be reused for other multimodal tasks (classification, generation)? What modifications would be required?\n\nI would consider to give a higher score if the authors could address my concerns in Weaknesses and Questions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on research in the field of cross-modal hash retrieval and proposes a novel AHLR method aimed at effectively enhancing feature extraction and alignment capabilities, generating more detailed hash codes, and improving the accuracy of cross-modal hash retrieval. This topic has important research value and practical application significance, and the proposed method has a certain degree of innovation.\n\nSTRENGTHS AND WEAKNESSES:\nS1:A well-written academic paper offers is given. It provides a rigorous and evidence-based approach (AHLR) to solving the semantic gap between heterogeneous modalities such as images and text. \nS2: AHLR uses Vision Transformer and Transformer-based text encoders for unified cross-modal feature learning.\nS3: The dynamic mask filters irrelevant inputs by sequence length, while the auxiliary module adaptively reweights channels for better hash representations.\n\nW1: Section 2.2 mentions \"Transformer is used to further fuse and process the feature representation of images and texts\", but Section 2.3 treats image and text learning separately.\nW2: The innovation of dynamic masking mechanism lies in automatically filtering key information, but its essential difference from existing attention mechanisms is not clear. \nW3: Authors mention using an Adam optimizer with “dynamically adjusting the learning rate,” but do not specify the initial learning rate, decay schedule, or weight decay.\n\nQUESTIONS:\n1. Please explain how the image and text features are interactively fused at which step. If not, delete the related description in the article.\n2. The dynamic masking mechanism can automatically filter and weight key information in images and text, but how does this mechanism determine which information is key?\n3. The paper introduces a contrastive loss function to optimize the generation of hash codes. May I ask, what is the design principle of this loss function? How does it ensure that the generated hash code has stronger discriminative power?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}