Title: MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-modal

vision and language
Abstract: Semantic retrieval is crucial for modern applications yet remains underexplored in current research. 
Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. 
However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images.
Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products 
in 5 languages, covering 7 distinct product categories.
Extensive experiments on MERIT identify existing models's critical limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries.
Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics.
Experiments demonstrate that Coral achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. 
Collectively, our contributions—a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework—establish a foundation for future research in interleaved multi-condition semantic retrieval. 
**Data & Code**: [MERIT-2025.github.io](https://merit-2025.github.io)


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new benchmark dataset, MERIT, and an improved fine-tuning method, CORAL, to advance the field of semantic retrieval. Existing benchmarks are often limited to single-language, single-image, and single-condition queries, failing to reflect complex, real-world e-commerce scenarios. To address this, they created MERIT, a multilingual dataset for text-image interleaved, multi-condition product retrieval, containing 320,000 queries over 135,000 products in 5 languages. They also propose CORAL to overcome the limitation that existing embedding models lose fine-grained conditional details in queries, that combines contrastive learning with text and image reconstruction. The authors show that CORAL achieves a 45.9% performance improvement on MERIT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles a highly relevant and underserved problem. As retrieval systems are integrated into real-world applications, the limitations of simple, single-condition benchmarks become increasingly apparent. The authors' focus on creating a benchmark for multilingual, multi-image, and multi-condition retrieval is a crucial step to address this gap.\n2. The creation of the MERIT dataset is a massive undertaking and a significant contribution. The data collection process is well-documented and rigorous, involving multiple stages. This dataset will be an invaluable resource for the community.\n3. The paper proposes a novel and effective solution CORAL, motivated by the error analysis performed on existing models. The combination of contrastive learning with reconstruction is an effective approach, with the reported 45.9% performance gain on MERIT and strong generalization to other benchmarks.\n\nWeaknesses:\n1. The product data is sourced from a private, \"internal dataset\". While the authors have anonymized the data, the lack of information about the original source (e.g., a single e-commerce platform vs. multiple) makes it difficult to reason about potential inherent biases in the product distribution, metadata structure, or visual style.\n2. The focused modality (text, image) and scenario (product search) are not accurately reflected in the title (semantic retrieval is too broad).  \n3. Some experiment settings/details are not clear.\n\nQUESTIONS:\n1. In Figure 2, the query condition \"same selling method\" is ambiguous. Could you clarify this example?\n2. How does model performance vary with the number of conditions in a query (e.g., two vs. three vs. four)? The main results do not show this breakdown.\n3. For the out-of-distribution results in Figure 6(b), could you clarify the setting? Was the Qwen2.5-VL trained on a subset of MERIT?\n4. For clarity, consider making the introduction of the CORAL framework (line 241) a new subsection separate from \"Preliminaries\".\n5. In Eq5, the CORAL loss function includes two hyperparameters, λ1​ and λ2​, to balance the reconstruction losses. Could you elaborate on how these weights were selected and their sensitivity?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes three contributions: 1) it introduces a new multilingual, multimodal product retrieval dataset called MERIT, 2) finds that existing retrieval models perform poorly on the MERIT task, and 3) proposes a fine-tuning technique called CORAL that achieves a 46% improvement over baselines on MERIT. The first portion of the paper is devoted to discussing the construction of the MERIT dataset and, in particular, how the data was collected annotated and refined. The authors then apply existing retrieval models to MERIT where they find that these off-the-shelf techniques exhibit poor performance. This finding motivates the authors to introduce CORAL, a multi-modal fine-tuning framework consisting of three loss functions that achieves strong performance on 8 previously introduced retrieval tasks as well as top performance on MERIT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n1. [Originality/Significance] As a multimodal and multilingual product retrieval dataset focused on Southeast Asian languages, I believe that MERIT has the potential to stand out as a novel dataset contribution to the community. The authors motivate the need for a dataset that more tightly integrates different modalities (though they do not precisely define their term \"interleaved\" -- please see the \"Weaknesses\" section below) quite well. \n\n2. [Quality/Significance] The experiments with the CORAL framework are well-executed with results on both MERIT as well as 8 existing retrieval datasets which provides robust evidence that the technique is effective. \n\nWeaknesses\n\n1. [Clarity] The paper lacks sufficient details in several places. First and foremost, as alluded to above, the authors, to my knowledge, do not provide a precise definition of what they mean by \"interleaved\" and, to my knowledge, this is not a standard term in the retrieval literature. It would be very helpful for the authors to clearly define what they mean by interleaved, which would help in understanding how the MERIT dataset stands out from prior work. \n\n2. [Quality] The paper also lacks sufficient details on how the MERIT dataset was constructed. In particular, it was not clear to me from reading the paper 1) how the products were sourced and from where, 2) what specific role GPT-4o played in the dataset construction (line 130 of the paper), and 3) who the annotators are and how they were selected/compensated? While the authors provide examples of what individual items in the dataset look like, I did not see any link to be able to download and inspect the dataset files directly (though I may have missed this). Overall, this paper covers a substantial amount of ground in introducing a new benchmark and proposing a new method and I would encourage the authors to consider whether these contributions might be better appreciated if there were more space to devote to each one as standalone papers. \n\n3. [Quality] There is a rich body of literature on contrastive learning for multimodal models, but the authors do not discuss this related work in much detail in the paper. I would suggest discussing more of this related work to better position the CORAL framework so that readers can understand what is novel about the technique. As it stands, I believe that the triple contrastive loss framework proposed in the paper is limited in novelty since similar ideas have been proposed (though the strength of the empirical results still validates the contribution). A deeper discussion of this related work might help the authors' novel contributions stand out more effectively.\n\nQUESTIONS:\nQ1: What precisely do the authors mean by \"interleaved\" and why do existing benchmarks fail to meet this property. *I would be willing to raise my evaluation score if the authors can provide a clear definition of interleaved and explain why this is a novel property of their dataset.*\n\nQ2: How were the products sourced for the MERIT dataset? What precisely is the role of GPT-4o in the dataset construction process? How do the authors control for potential errors from using GPT-4o in the dataset construction process? *I would be willing to raise my evaluation score if the authors can provide answers to these questions and, in general, provide more details on how the dataset was sourced (including perhaps a file with data examples)*\n\nQ3: Are there any works in the multimodal contrastive learning literature that are related to CORAL? What distinguishes CORAL from these prior works? *I would be willing to raise my evaluation score if the authors can include a deeper discussion of related work around the CORAL model (particularly in contrastive learning). Currently, I feel that this important positioning of CORAL in relation to the literature is lacking in the paper*\n\nQ4: How were the expert annotators recruited for this project and how were they compensated? What is the license of the MERIT dataset? *I would be willing to raise my evaluation score if the authors can provide satisfactory answers to these questions*\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MERIT, the first large-scale benchmark that targets multilingual, multi-image, and multi-condition product retrieval. MERIT contains 135 k products and 320 k query–target pairs across seven e-commerce categories .After showing that nine state-of-the-art (SOTA) retrievers achieve < 12 % R@1 on MERIT , the authors diagnose that current fine-tuning regimes over-focus on a single [EOS] token and thus miss fine-grained attribute cues. They propose CORAL, a fine-tuning framework that couples supervised contrastive learning with masked embedding reconstruction for both vision & text streams. CORAL lifts R@1 by 45.9 % on MERIT and improves eight external benchmarks (MS-COCO, VisDial, etc.) without task-specific tweaks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The benchmark is timely and realistic, which is the first to combine multilingual, multi-image, and compositional attribute retrieval at scale.\n2. CORAL directly tackles the identified issue of missing fine-grained, conceptually simple yet effective.\n3. The experiments are strong and broad, which contain 9 baselines on MERIT plus 8 public datasets, full ablations, and error analysis.\n\nWeakness\n1. The section of “Masked Embedding Reconstruction” is poorly written and needs to be reorganized. For example, Mask_v and Mask_l are sometimes one parameter, sometimes 2; symbols such as q and E do not clearly state what they are; and the meaning of the corner markers under (i) in Equation 4 is unclear.\n2. Lack the analysis of reconstruction mask ratio δ, choice of decoders, and temperature $\\tao$.\n3. The metric relies solely on recall/MRR. User-centric metrics (CTR, dwell time) or ranking quality are absent.\n\nQUESTIONS:\n1. How many GPU-hours did full CORAL training require for the 7B model, and what is the inference latency compared to baseline contrastive tuning?\n2. Have you measured language- or category-specific biases in the retrieval results beyond the small sample in Fig. 7? Any mitigation plans?\n3. How sensitive is CORAL to the 50 % visual/text masking ratio? Could adaptive masking yield further gains?\n\nLIMITATIONS:\n1. Dataset focuses on consumer goods; tasks like scene or document retrieval are out of scope.\n2. CORAL needs access to image embeddings during decoding; models without token-level vision hooks may not benefit.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors made two contributions: \n\nThey first proposed a novel opensource multi-lingual dataset for interleaved multi-condition semantic retrieval (MERIT), highlighting difficult queries that involve multiple conditions that's lacking in previous open datasets.\n\nSecondly, they proposed a contrastive reconstruction for multimodal retrieval (CORAL) as a novel fine-tuning framework to adapt pretrained LLM for retrieval. In essence, this encodes a series of product images and product title descriptions as image/text interleaved embeddings and use an LLM to encode into EOS hidden state, and use this hidden state as query while image/text embeddings as key/value. Three objectives were used to train the model, a contrastive InfoNCE loss upon query and pos/neg product embeddings; a vision reconstruction bert layer that reconstructs the masked visual embeddings from unmasked input embeddings and h_EOS global embedding; a language reconstruction module that decodes masked text embedding similarly.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The authors first identified the lack of sequential multi-condition queries as a limitation in the current mainstream benchmarks and invested considerable effort in constructing a new dataset.The authors meticulously tested multiple mainstream models (InternVL/QwenVL etc) and performed comprehensive analysis and documented their methodologies thoroughly. This should prove valuable for the field. \n\n2. By testing existing models with the new benchmarking system, and proofreading error cases, the authors identified two major limitations in the current semantic retrieval system: a. focusing on global semantic information while failing to extract specific attributes; b. Failing to support sequential image inputs, despite having the ability, due to existing training datasets are predominantly single-image based. These are valuable insights in guiding the community on designing next-gen benchmarks\n\n3. The ablations with vision/language modules convincingly demonstrated the effectiveness of the reconstruction losses design.\n\n\n## Weakness:\n1. While the dataset is multilingual and sizable, it seems to be domain/business specific (e-commerce in Southeast Asia), limiting its generalizability as a standard benchmark for semantic retrieval.\n\n2. The authors demonstrated that existing methods overall have poor performance on MERIT dataset in Table 2. the metrics however might be biased since some models are finetuned on the ecommerce scenario more than others, like the best in the cohort GME-QwenVL2 vs pretrained Qwen2.5-VL. It is therefore not entirely representative of what the prior models would have behaved if they were also finetuned. Undermining the significance of improvements claimed\n\n3. The authors benchmarked and evaluated CORAL/MERIT as a symbiotic duo that worked well with each other, and evaluated CORAL on a few prior tasks according to Ref 33 (Training\nvision-language models for massive multimodal embedding tasks) in Fig 9, but it seems different from what MERIT compared against in Table 1. It would be nice to isolate and demonstrate CORAL's effect on some of these benchmarks as well.\n\nQUESTIONS:\n1. The core diff of CORAL is the introduction of BERT like reconstruction losses, with the vision reconstruction task having visibility to all text(especially the text title associated with the image itself), this in essense is similar to CLIP, I'm wondering if the authors have intuitions on why this new task helped metrics.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}