MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query
Authors: Wei Chow
Yuan Gao
Linfeng Li
Xian Wang
Qi Xu
Hang Song
Lingdong Kong
Ran Zhou
Yi Zeng
Yidong Cai
Botian Jiang
Shilin Xu
Jiajunzhang
Minghui Qiu
Xiangtai Li
Tianshu Yang
Siliang Tang
Juncheng Li
Venue: NeurIPS 2025 poster ()
Keywords: multi-modal
vision and language
Primary area: applications
TL;DR: We introduce MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, revealing current methods' limitations and proposing new method to enhance retrieval performance.

Abstract: Semantic retrieval is crucial for modern applications yet remains underexplored in current research. 
Existing datasets are limited to single languages, single images, or singular retrieval conditions, often failing to fully exploit the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions. 
However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images.
Hence, this paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products 
in 5 languages, covering 7 distinct product categories.
Extensive experiments on MERIT identify existing models's critical limitation: focusing solely on global semantic information while neglecting specific conditional elements in queries.
Consequently, we propose Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction to preserve fine-grained conditional elements and contrastive learning to extract comprehensive global semantics.
Experiments demonstrate that Coral achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. 
Collectively, our contributions—a novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework—establish a foundation for future research in interleaved multi-condition semantic retrieval. 
**Data & Code**: [MERIT-2025.github.io](https://merit-2025.github.io)

MERIT: Multilingual Semantic Retrieval with Interleaved Mulli-oodiition Query
Wei Chow Yuan Gao Linfeng Li Xian Wang', Qi Xu', Hang Song', Lingdong Kong', Ran Zhou', Yi Zeng', Yidong Cai', Botian Jiang', Shilin Jiajun Zhang', Minghui Qiu', Xiangtai Li', Tianshu Yang', Siliang Tang" Juncheng Li
Shili'n Xu
ByteDDance Inc.
7Zhejiang University
"Equal Contributions
Corresponding Author
Data & Code: MERIT- github.io
Clothing
Electronics
Top
brand
material
color
brand
CPU
camera
Phone
HEW
T-Shirt
color
pattern
MERIT
Fruit
brand
type
brand
pattern
Chair
pack
type
weight
Food
Fumituree
Figure 1: Illustrative examples of interleaved multi-condition semantic retrieval. MERIT enables the first multilingual semantic retrieval with composite multi-condition queries that interleave textual descriptions and visual references, reflecting real-world product search scenarios where users specify multiple attributes through both and images.
Abstract
Semantic retrieval is crucial for modern applications yet remains underexplored in current research. Existing datasets are limited to single languages, single images, singular retrieval conditions, often failing fully exploit the expressive capacity visual information, as evidenced by maintained performance when images are replaced with captions. However, practical retrieval scenarios frequently involve interleaved multi-condition queries with multiple images. Hence, this paper in- troduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval, comprising 320,000 queries with 135,000 products in lan- guages, covering distinct product categories. Extensive experiments MERIT identify the existing models' critical limitation: focusing solely global semantic information while neglecting specific conditional elements in queries. Conse- quently, we propose CORAL, novel fine-tuning framework that adapts pre-trained MI.LMs by integrating embedding reconstruction to preserve fine-grained condi- tional elements and contrastive learning to extract comprehensive global semantics.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Experiments demonstrate that CORAL achieves a 45.9% performance improvement over conventional approaches on MERIT, with strong generalization capabilities validated across 8 established retrieval benchmarks. Collectively, our contributions - novel dataset, identification of critical limitations in existing approaches, and an innovative fine-tuning framework establish foundation for future research in interleaved multi-condition semantic retrieval.
1 Introduction
Semantic retrieval is pivotal task that involves sourcing relevant information from vast data collec- tions to meet specific user requirements [79, 62, 26, 45]. This task has become increasingly important with the advent of AI, as it not only enables precise user recall [100, 86, 73] but also mitigates the risk of inaccuracies in the generated content of Multimodal Large Language Models (MLLM) [2, 72].
However, semantic retrieval remains confined to narrow research scopes, which are limited to single languages [102, 84], single images [29, 11, 104], or employing only singular retrieval condition [64, 87], as illustrated in the left part of Fig. Furthermore, many existing works [86, 109, 101] fail to fully exploit the expressive capacity of images, as evidenced by their maintained performance when images replaced with corresponding captions (Vision Unnecessarity in Fig. 2). Moreover, in practical applications, product retrieval tasks frequently involve interleaved multi-condition queries (e.g., specific patterns and particular texture), with many aspects requiring visual representation through images [77, 78, 8), as demonstrated in the right part of Fig. 2.
To further investigate this issue, we pose two fundamental research questions:
How can we comprehensively measure the capability of existing models in the interleaved multi- condition semantic retrieval lask? To address this question and comprehensively assess the perfor- mance gap interleaved multi-condition semantic retrieval tasks, we introduce MERIT, the first multilingual semantic retrieval dataset with composite contiiconniiion queries. Our dataset comprises 135,000 products, forming 320,000 retrieval pairs in 5 languages, covering 7 distinct product retrieval scenarios. Given the challenges in acquiring such data, we employed open-set attribute annotation to increase diversity, closed-sed set product annotation three sampling algorithms to enhance richness and distributional uniformity. After multiple rounds of filtering, we finalized the dataset, investing a total of 10,000 labor hours in the annotation process.
improve precision and recall, and designed
2) What are the important factors that limit their performance, and how can we enhance the retrieval effectiveness for such a challenging task? To address this question, we evaluate 9 existing retrieval models on MERIT and demonstrate that recall rates remain substantially below expectation, despite these methods effectively solving established semantic retrieval tasks [41, 102]. Through in-depth analysis, we identify that these methods neglect specific conditional elements in queries, failing to correctly extract targeted attributes and misinterpreting visual content. This limitation stems primarily
Single Language
Single Image
Multi-Lingual, Multi-Image, Multi-Condition
Query
G Target
Queryy
Query
S SoTarget
Givil ovep Baa
J-Paice Falding
dress
cogg with his puppy running
style pr product
same selling as product
Single Condition
Vision Unnecessarity
Vision Necessarity
Query
Target
Query
Target
Baju Atoson Wanktaanka
Piik T. F.Shii
Won Trogg!
mini deevellos dress
Find other attractions this country
product with same color as product its and same pattern as product
Figure 2: Comparisons among and existing datasets [59, 86, 101]. Left: Previous works are
limited single-condition, single-image, single-language scenarios. Right: Our benchmark enables multilingual semantic retrieval, featuring composite multi-condition queries.

---

from existing retrieval models [5, 38, 41] that typically fine-tune pre-trained MLLMs through contrastive learning with supervision applied exclusively at the [EOS] token [92]. thereby prioritizing global semantic information while inadequately addressing specific conditional elements [46], such as material attributes in product deseriptions or distinctive visual textures images.
To address this limitation, we propose Contrastive-reconssttuiii for multimodal retrieval (CORAL), a novel fine-tuning framework adapt pre-trained MLLMs into multimodal retrieval models. CORAL simuitaneously preserves detailed conditional elements through multi-modal embedding reconstruc- tion while effectively extracting global semantics via contrastive learning. Experimental results demonstrate that our method achieves a 45.9% performance improvement compared to conventional approaches on MERIT, with efficacy further validated across 8 established retrieval benchmarks.
Interestingly, we discover that existing MIL.M-based retrieval models achieve performance approxi- mately 16 times higher in @@ I when multiple images are concatenated into a single input image, compared to an interleaved input of multiple images. This occurs despite the fact that pre-trained MLLMS support interleaved image inputs, which contradicts established MLLM behavior on visual comprehension tasks [90, 47] and zero-shot performance. We hypothesize that this discrepancy may stem from existing retrieval datasets containing at most one image, potentially causing MLLMs to lose their capability to process interleaved inputs effectively. After training on MERIT, sequence input performance improved by 14.3%, further validating our hypothesis. These findings underscore the significance of our dataset as the first interleaved semantic retrieval dataset.
In summary, this paper makes three contributions to the retrieval research community:
We introduce MERIT, the first multilingual dataset for interleaved retrieval, and provide insightful observations based on it.
semantic
We identify critical limitations of existing methods: focusing solely on global semantic information while neglecting conditional query elements, failing to extract specific attributes, and misinterpreting visual content.
We propose CORAL, which combines embedding reconstruction preserve fine- fine-graine grained conditional elements and contrastive learning to extract comprehensive global semantics, demonstrating strong performance across our dataset and eight standard benchmarks.
2
Related Work
Multimodal Large Language Models (MLLMs) are large-scale models that integrate visual modal- ities with language understanding [33, 54, 4, 66, 13, 9]. Real-world multimodal documents [108] often contain interleaved image-text pairs, and recent research [50, 47] has begun to extend MLLMS toward processing such interleaved inputs, resulting in the development of several relevant bench- marks [23, 43, 24). Prior to the emergence of interleaved MILMs, models supporting only single- image inputs [55, 107] typically processed multiple images by concatenating them into single image. However, this approach results in reduced resolution and loss of sequential information, leading to inferior performance compared to sequential input of multiple images [73, 89]. Contrary to these findings, for fine-tuned retrieval models, concatenating image conditions into a single input image significantly outperforms sequential image input. Interestingly, on MERIT, models fine-tuned with sequential inputs demonstrate superior performance. This suggests that fine-tuning with previous datasets, which predominantly contain single-image examples, compromises the model's ability to maintain information across sequential inputs, further emphasizing the uniqueness of our dataset.
Semantic Retrieval is not only crucial application in real-world scenarios, such as product search [86, 24, 47] and webpage retrieval [7], but also facilitates content generation (retrieval- augmented [4 88, 31, 30] and training for reasoning tasks[72]. However, existing semantic retrieval datasets are limited to single languages [102, 84], single images [29, 11, 104], or singular retrieval condition [64, 87], often failing to fully exploit [14] the expressive capacity of visual information as evidenced by maintained performance when images are replaced with captions as shown in Fig. 6. MERIT is the first multilingual dataset for interleaved multi-condition semantic retrieval. Comparison of related works can be seen in Tab. 1.
Multimodal Retrieval Models have primarily focused on cross-modal retrieval [58, 84, 105, 106], typically leveraging models such as CLIP [68, 29] or BLIP [84, 48] for multimodal embeddings. However, these approaches exhibit limited instruction comprehension capabilities. Subsequent
3

---

Table 1: Summary multi-modal query retrieval datasets. We compare existing works from aspects including: 'semantics, 'multilingual data, 'multiple types, "interleaved queries. "multi- attributes queries, and whether manual annotations and filtering are applied. Note that this table does not include datasets [84, 102] that are collated and summarized but not yet newly marked.
Multi Lingual
Multi Type
Inter
Multi
Manual Anno.
Benchmark
Venue
Sem.
#Queries
Leaved
Attri
Fashion200K [27]
ICCV'17
X
X
200,000
CIRR [59]
ICCV'21
X
X
X
36,554
Fashion-IQ [86]
CVPR'21 21
V
X
X
X
X
/
20,090
DTIN [70]
CVPR'23
X
X
X
X
X
10,000
OVEN [29]
ICCV'23
X
X
X
$39,000
InfoSeck [11]
EMNLP'23
X
1,350,000
CIRCO [6]
ICCV'23
V
800
INSTRUCTIR [64]
arXiv'24
X
16,072
SciMMIR [87]
ACT/24
X
530.975
Magiclens [101]
ICML'24
36,700 70,7000000
MIRACLE [63]
CVPR'24 24
X
X
X
26,221
MERIT
Ours
/
320,
Statistic
Number
Clothing
MS
Total Queries
320,000
Two Conditions
319,800
Bags
Three Conditions
Electronics
Four Conditions
100
Category
Language
Unique Altribotes
116
Food
Unique
2,504
Furniture
Products Number
135,000
Maximum Product Title Length
Jawelry
Averuge Product Lengh
95.83
Figure 3: Dataset statistics.
Figure 4: Summary of product categories and language distributions.
research has adapted MILMs to function as embedding models for retrieval tasks, capitalizing on
their robust instruction comprehension capabilities [51, 56, 38, 102, 39, 34]. These methods [5, 38, 41] typically fine-tune existing MLLMs through contrastive learning, utilizing only the [EOS] token [92] for supervision. This approach results in an over-reliance on contrastive learning to supervise global information [92], while neglecting detailed semantic information, which often leads to semantic misunderstandings [78, 74]. To overcome these shortcomings, CORAL preserves original detailed information by integrating embedding reconstruction and contrastive learning fine- tuning framework to adapt pre-trained MILMS into multimodal retrieval models. Our experiments demonstrate strong performance across MERIT and eight standard benchmarks.
3 MERIT: A Multi-Condition Smantie Retrieval Benchmark
To evaluate the effectiveness of existing retrieval models in addressing the interleaved multi-condition semantic retrieval task, we introduce MERIT in Sec. 3.1. Subsequently, we provide a comprehensive description the data collection methodology in Sec. 3.2. Leveraging our data, conduct extensive experiments on 9 state-of-the-ar retrieval models and derive insights regarding visual conditioning necessity, interleaving support, and out-of-distribttion scenarios in Sec. 3.3, Finally, in Sec. 3.4, we present an in-depth analysis of the factors potentially contributing to suboptimal performance.
3.1 Benchmark Overview
In practice, product retrieval tasks frequently encompass multiple simultaneous conditions (e.g., specific patterns, precise colors, and particular styles), with many attributes necessitating visual representation through images [8]. However, existing semantic retrieval datasets are limited to single languages [102, 84], single images [29, 11, 104], or singular retrieval conditions [64, 87], often

---

Product Selection
Product Annotation
Search Query Composition
Filter Refine
S Internal Data
Forsention
Uniffomm Sempling
Automatic Fitering
Product
Unitfo Sampling
Pozulor Products
Ruie Check
Stansoical Method
refor
O
Prior testtonn Sampling
Agent Refinement
M
Aesth
Filtering
Check Corretteess
Attribute
Dalineation
Query Generator
Human Refinement
High Quality
Product
Annotation
Collcction
Amritune
Cold-Starr' Expansion
Producta
Coharance
Relevance
Practical
Figure 5: The data annotation pipeline for MERIT. We ensure data diversity and quality through open-set deduplication and multi-round filtering procedures steps. We first select high-quality products and annotate their attributes, then combine them into query pairs before performing data cleaning to produce MERIT. Details can be found in Appendix A.
failing to fully exploit [14] the expressive capacity of visual information-a limitation evidenced by maintained performance when images are replaced with textual captions.
To bridge this gap, present MERIT, which encompasses 135, 000 products, resulting in 320, 000 retrieval pairs across languages (English, Malay, Indonesian, Vietnamese, Thai), encompassing 7 distinct product retrieval scenarios. Our dataset constitutes structured query dataset, where each fundamental unit is a product comprising an image and its corresponding title generated by GPT-4o [33], illustrated in Fig. 2. Tab. presents the key statistical characteristics of our MERIT, while Fig. 4 displays the category and language distribution of the product pool. Each search query contains at least one positive sample. For convenience, the dataset is partitioned into training and test sets, containing 310, 000 and 10, 000 entries respectively. Additional statistical results and examples provided in Appendix C and Appendix D, respectively.
3.2 Data Collection Process
To ensure data quality, all data underwent manual filtering by annotators proficient in all five languages, complemented by multiple rounds of automated filtering during the collection process. Specifically, our dataset collection pipeline comprises the following four steps:
HighhQQulityy Product Selections. While maintaining diversity, carefully selected popular products from our internal dataset across 6 Southeast Asian countries in 5 languages with each product title is generated by GPT-4o [33]. Each product was further filtered based on popularity and aesthetic scores [71, 95] to form our product inventory used in the following steps.
2) Product Annotations. To accommodate diverse real-world search requirements, we needed to obtain a variety of greegga product attributes for combination. However, attribute information in real-world E-commerce data is often insufficient, resulting in suboptimal retrieval for specific user needs. This gap arises from the limited attribute richness constrained by operational attribute structure versus the need for fineggrained, precise product attribute information in search relevance systems. Consequently, we adopted an open annotation approach followed by statistical analysis for Attribute Delineation, and subsequently tagged products based on these derived attributes.
3) Search Query Compositions. To simultancously enhance dataset quality and diversity, we implemented composite sampling approach for constructing retrieval pairs. This approach integrates three distinct methods: Conventional Uniform Sampling (Appendix A.3.1), Attribute Uniform Sampling (Appendix A.3.2), and High-Similarity Product Prioritization Sampling (Appendix A.3.3). Furthermore, our pipeline supports cold-start expansion, enabling the extension of our dataset to previously unseen product classes, as detailed in Appendix A.3.4.
4) Filtering & Refinement. Finally, we introduce a two-stage filtering process, encompassing automatic filtering and manual curation, respectively. The automatic filtering stage employs rule-
5

---

Table 2: Comparative study of retrieval performance on MERIT. Details about the baselines can be found in Appendix E.I. "Seq", "Cat", and "Avg" denote sequential multi-imap input, concatenated images as a single image input, and averaged embeddings, respectively.
Type
Method
Size
Venue
Type
R@11
R@51
R@
MRR
IB
urXiv'24 24
Cal
0.20
0.98
1.72
0.56
IntemVL22.5MMPO [83]
IB
arXiv' 24
Cat
0.24
1.04
1.81
0.60
Qwen2.5.VL.
3B
arXiv'25
Cat
0.05
0.27
0.40
0.14
Zero-Shot MILM
V1.2m [12]
1B
arXiv'24
Soq
0.27
1.24
1.94
0.69
Inlem\1.2.5-MPO [83]
arXiv'24
Sey
0.87
Qwen2.5-VL
3B
urXiv'25
Sey
0.09
0.39
0.56
0.21
ES-V [38]
8B
arXiv'24
Avg
3.10
7.54
9.90
5.03
LLaVE [41]
0.5B
CVPR'25
Cat
4.89
33.11
41.98
16.95
GME-Qwen2VL (102]
2B
arXiv'24 24
Cat
8.47
47.13
56,
25.02
LLaVE [41]
2B
CVPR'25
Cal
5.80
43.62
53.51
21.78
LamRA-Qwen2.577 [56]
7B
urXiv'24
Cat
12.05
39.13
48.03
23.80
LLaVE [41]
7B
CVPR'25
Cat
45.34
55.32
Embedding MLLMM
BGE-VL [104]
7B
urXiv 25
Cal
11.55
38.01
46.26
23.00
LLLVE [41]
0.5B
CVPR'25
Seq
0.38
1.17
1.79
0.71
GMIE-Qwen2VL
2B
urXiv 24
Sey
5.29
24.18
30.66
13:42
[4]VB[[4]]
2B
CVPR'25
Sey
0.12
1.03
1.67
0.51
VLM2 [39]
4B
arXiv'24
Seq
0.43
1.86
1.04
LamRA LamRA-Qwen2.nn Qwen2.5VL [56]
7B
urXiv 24
Sey
3.26
13.10
19.03
7.57
LLaVE [41]
7B
CVPR'25
Seq
0.39
1.76
2.77
1.03
60
Performance
original
wijo image
w/o title
R@1 (96)
Zero-shot
Mixed
0O0
40
20
FashioniO
CIRR
Ours
Language
Class
Attribute
Figure 6: Comparisons on (a) Visual Necessity Test, and (b) Out-of-Distrriuun Scenarios.
based systems and statistical methods to eliminate obvious inconsistencies and low-quality samples, while the manual filtering stage involves expert annotators who apply nuanced judgment to ensure semantic coherence and practical relevance. This rigorous quality control process results in high-fidelity dataset that meets stringent academic standards.
Due to space limits, we provide a more detailed description our annotation process and the rationale behind our design choices in Appendix A.
3.3 How Far to MERIT
To evaluate the effectiveness of existing retrieval models in addressing the interleaved semantic retrieval task, we conduct experiments on state-of-the-arr retrieval models. The principal results presented in Tab. 2. Detailed information regarding the experimental settings and datasets can be found in Appendix E. MERIT is divided into training and test sets, consisting of 310,000 and 10,000 queries respectively mentioned in Sec. 3.1.
Main Results. Existing retrieval methods struggle to address interleaved multi-condition semantic tasks, with even the best Recall @ being only 12.05%. Additionally, we identify several key insights:
Visual Conditioning Necessity. To verify the necessity of visual information, we conducted experi- ments using BGE-VL [104] on CIRR [59], FashionIQ [86]. and MERIT. We report R @@ for CIRR, @@ for FashionIQ, and our dataset. As shown in Fig. 6(a), when replacing images with their corresponding captions for retrieval, the performance on FashionIQ and CIRR does not significantly deteriorate. In contrast, we exhibit substantial performance degradation when either replacing images with their corresponding captions (w/o image) removing product titles (w/o title), with image removal resulting a particularly severe decline of 73.9%. This demonstrates the effectiveness of dataset, indicating that both images and product titles are indispensable components.

---

Attribute
33
Visual Understaning Erro:
InterVL-225
42
Owen2.5-VL
5wen2.5.VL
InterVL2.5
Category Error
BGE-VL
Detail Error
Annotetion Error
39
Figure 7: (a) Different Language's Performance on MERIT @R@) 1). (b) Distribution of Error Types.
Interleaving Support. As shown in Tab. concatenating multiple images into single image significantly outperforms sequential input such as GME- MMEE-Qwe2VLL [102], with concatenation achiev- ing 119.7% improvement in R@5 over its sequential version. This occurs despite the fact that pre-trained MILMs support interleaved image inputs [5], which contradicts established MLLM behavior visual comprehension tasks [90, 47] and zero-shot performance MERIT IT, where sequential processing typically excels by preserving more image information [40, 89, 44, 1, 25]. We hypothesize that this discrepancy may stem from existing retrieval datasets containing at most one image, potentially causing MLLMs to lose their capability to process interleaved inputs. After training, sequence input performance improved by 14.3% Tab. 3, further validating our hypothesis. This underscores the significance of MERIT as the first interleaved semantic retrieval dataset.
Out-of-Distribution Scenarios. We evaluated Qwen2.5-VL. [5] on three types of OOD scenarios (Class OOD, Language OOD, and Attribute OOD), with results illustrated in Fig. 6(b). Detailed numeric results can be seen in Tab. 7,8,9 in the Appendix E.4. Specifically, performance in the Language OOD) scenario shows a notable gap compared to full training (Mixed); however, it still demonstrates substantial improvement over zero-shot performance due the activation of the MILM's multilingual capabilities. In both Class and Attribute OOD scenarios, the performance gap between OOD and full training relatively small, reflecting the diversity of our dataset.
3.4 Error Analysis
To investigate the poor performance of retrieval models on MERIT, we first analyzed whether success rates correlate with specific languages. As shown in Fig. 7(),, the statistical results reveal minimal variation across different languages, with observable advantage for English despite its predominance in the initial training data of MLLMs.
We then randomly selected 500 queries and obtained explanations from Qwen2.5-Vi and Intern VL 2.5, both which underwent full aall-parrmeter contrastive learning training. Expert annotators classified the root causes of mispredictions into five categories (details can be seen in Appendix E.5).
The distribution these error types, shown in Fig. 7(b), reveals that attribute and visual understanding errors constitute the largest proportion of failures. This analysis reveals these methods neglect conditional query elements, failing to extract specific attributes and misinterpreting visual content. This likely stems from vetrieval-orienter fine-tuning, where MLLMs prioritize global over specific semantic information. Furthermore, since current retrieval datasets are predominantly single-image based, existing methods fail to leverage the image sequence understanding capabilities of interleaved MILMS as analyzed in Sec. 3.3. This limitation likely leads to failures understanding precise semantics, resulting in attribute extraction errors (causing Attribute Errors) and incorrect interpretation of visual features such as patterns (causing Visual Understanding Errors).
4 CORAL: Contrastive Reconstruction for Multimodal Retrieval
Recognizing neglecting specific conditional elements in queries as a primary source of error high- lighted Sec. 3.4, we introduce CORAL Sec. 4.1 to enhance M- retriever performance in addressing interleaved multi-condition semantic retrieval tasks through the integration of visual re- construction during the fine-tunin fine-tuning process of the Ml--M-oo-retrrivall model adaptation. Subsequently, validate the effectiveness of our approach in Sec. 4.2.

---

Contrastive Learning
Masked Positive Embedding Reconstruction
Coral
Postive
Negative
Product Title in TH
L
Emse Lmim
Language
F
Vision
Decoder
K,V
LLM
EOS
YOOO Embidding
VVT
Projecto-VIT VIT
Embescing
[EO]]
Image Portion
Product Title
Product Title
Language Portian
MAAKI,
MASK
Figure Overview for CORAT. The loss function of CORAL consists of three components: Contrastive Learning Loss Lel, Vision Reconstruction Loss mase. and Masked Language Modeling Loss Fmlm. During training, we reconstruct both the query and its corresponding positive sample.
4.1 Preliminaries
Prerained MLLM. For common MLLM [55, 61, 40], it has image and text input Ting and Tixt- We assume d the hidden state dimension of the language model. We first process Fing subject to visual representation backbone [68, 98] Vu that outputs a sequence of features Ping V (Xingg) E IR Nv Xd, Next, we map Ping to sequence of embeddings via learned projector F where Eimg = F (Ping) IR xd Finally, we concatenate the sequence Sing with the text prompt embeddings exxt = e E NR. xd passing the result to the language model.
Generally, have the interleaved image-tex: input Timput by concatenating all the exxt and Cimg- The language model generates output hidden state /tgen LMg (Cing: (xx)) In particular, we denote the hidden layer representation of the [EOS] position as heas. Finally. fggen can be transferred into text output Mgenn The composition LM (P (e(D (xmbed(((xx)) then defines the MILMM. Given a triple (Zing, Tixt, ügen) during training, MILM minimizes that loss L = logp( üggnn [rng Tixt).
Masked Embedding Reconstruction. During training, we apply masking to the attention maps of individual modalities. We define the functions MASKK, (E)) and ( to mask the visual and linguistic portions [91, 80] of the input multi-modal embedding E = [Fimgi éxxo] at a fixed ratio example, for given multi-modal
5 (we set 8 0.5 in experiments). Taking MASKK, input embedding, we retain all textual attention while randomly masking the visual self-attention and crossaatenttinn from the complete text to the image. We set with position embedding [18], which has the same length with e. This process is illustrated in Fig. 8 (a), and the reconstructed multi-modal embedding is then obtained using:
[,, heas] and q [ heas.
Q =
K k
V eW
0, attended,
QJK Vd
(I)
M,
Crec = softmax
+ +MV.
00, masked;
CORAL. We introduce fine-tuning method designed to adapt pretrained MLLMs into multimodal retrieval models. It enhances visual understanding capabilities while preserving the model's original linguistic comprehension. Specifically, for a pretrained MILMM, we perform fine-tuning as follows: Contrastive Learning Loss Let. We employ the InfoNCE Loss [85] for supervised contrastive learning. Given a batch N samples, where denotes the temperature coefficient, represents the query sample, and ki is the encoded vector the positive sample corresponding to query the contrastive loss is computed as:
N
exp
gerket
Lel
log
Σ
exp
Qe-kk
Vision Reconstruction Loss Lusse- We employ a decoder Fýy randomly initialized as BERT layer [91, 17, 67]. Using the full input representation heos the query, compute the MSE loss

---

Table Ablation results of existing methods and CORAL on MERIT using Qwen2.5- VL [5].
Ablation Factor
Method
LoRA
Type
R@11
R@51
R@101
MRR
CL
Seq
48,52
73.11
77.93
59.48
Baseline
X
Seq
47.76
73.97
80.47
59.06
Zero-Shot
Seq
0.09
0.39
0.56
0.21
Input Type
Zero-Shot
Cat
0.05
0.27
0.40
0.14
+CORAL (Ours)
X
Cat
60.94
85.60
90.40
71.70
+Vison
V
Seq
58.18
83.19
88.02
69.13
+Language
V
Seq
58.38
83.01
88.26
69.35
Partial Reconstruct
+Viiion
X
Seq
59.46
85.46
90.81
70.89
+Langugge
X
Seq
59.98
86.01
90.72
71.22
+CORAI. (Ours)
V
Seq
59.40
82.80
87.94
69.
Final Version
+CORAL (Ours)
x
Seq
69.68
89.26
93.08
78.33
90
R@= (96)
Baseline
CL
Coral
70
50
30
VisDial
CIRR
VisualNews (t2i)
VisualNews (i2t)
MSCOCO (t2i)
MSCOCO (i2t)
NIGHTS
WebQA
Figure 9: Comparisons of method with other methods eight established retrieval tasks. We take zero-shot Qwen2-VL [81] as our baseline. CL denotes contrast learning.
between the original unmasked embedding and the reconstructed embedding from F8 follows:
N
1
mmse
E
where = = MAAK, heoel
N
Masked Language Modeling Loss fmtm-- Similar to vision reconstruction, use decoder F/J for reconstruction. To reduce trainable parameters, F/J shares weights with the language modeling head of the MILM. The masked language modeling loss computed as:
mmlm
log P (#: X), where f; = [Fl MASK((F)); heeaall))
(4)
N
The overall training objective CORAL formulated as:
max 0 0,6,.0 .
Let + A1 + ArCrec
(5)
Here, Lreg and Lree represent the reconstructions of the retrieval target using the conditions' [EOS] token and the target's own [EDS] token as attention queries, respectively. For both terms, the attention keys and values referenced in Eq. are derived from the embeddings of the retrieval target. Each reconstruction component encompasses both image reconstruction and language reconstruction.
4.2 Experiments
To validate the effectiveness of CORAL, conducted experiments MERIT and 8 established retrieval tasks. Due to space constraints, the implementation details are placed in the Appendix.
Main Results on MERIT. Results lead to the following conclusions: (i) Embedding reconstruction contributes significantly to retrieval performance. Both partial feature reconstruction (Tab. 3, rows 6-11) enhance model performance, with multimodal reconstruction yielding 45.9% improvement compared to contrastive learning alone. (ūi) Multi-moda reconstruction outperforms partial recon- struction. Comparing Tab. 3, rows 6-9 and 10-11 11 reveal superior performance when reconstructing

---

both modalities simultaneously. (iii) Sequential input surpasses image concatenation. Based on rows 3-5 and 11, we observe that sequential inputs achieve higher performance. We hypothesize that se- quential representation preserves more information than image concatenation [37, 97, 24, 47], which aligns with our findings Sec. 3.3. (iv) Full parameter fine-tuning yields optimal results. Due the substantial divergence between retrieval tasks and pre-training objectives, full parameter fine-tuning generally produces better outcomes, consistent with conclusions from previous work (9].
Results on Eight Retrieval Tasks. To further validate the efficacy of CORAL, we conducted evaluations on retrieval benchmarks with experimental configurations following the methodology described in [39]. The results are illustrated in Fig. 9. Comparative analyses between our approach and other foundational models, such as CLIP [68] and ES-V [38], are presented Appendix E.4. Experimental results demonstrate that our method achieves consistent improvements across these eight retrieval tasks, with particularly notable performance on VisDial [16], where our approach exhibits a 181% % enhancement over the baseline.
5
Conclusion
This paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval. Extensive experiments identify existing critical limitations of existing models: focusing solely on global semantic information while neglecting conditional query elements, failing to extract specific attributes, and misinterpreting visual content. To address this limitation, we propose CORAL, novel fine-tuning framework that adapts pre-trained MLLMs by integrating embedding reconstruction for preserving detailed conditional elements with contrastive learning for extracting global semantics. Ablation study across benchmarks demonstrates the effectiveness of CORAL.
Acknowledgment. This work was supported by the National Natural Science Foundation of China (62436007).
10

---

References
[1] Stanislaw Antol. Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruy Batra, C Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In Proceedings of IEEE international conference on computer vision, pages 2223,,43,,22 2015.
[2] Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval- language models and appli- cations. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41-46, 2023.
[3] Jinbin Bai, Wei Chow, Ling Yang, Xiangtai Li, Juncheng Li, Hanwang Zhang, and Shuicheng Yan. Humanedit: A high-quality human-rowarded dataset for instruction -asioo image editing. arXiv preprint 2Xii:24422.04880 2024.
[4] Jinbin Bai, Tian Ye, Wei Chow, Enxin Song. Qing-Gua Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations. 2024.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijic 2025.
Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint
[6] Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15338-15347 2023.
[7] Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16495-16504, 2022.
[8] Haibin Chen, Kangtao Lv, Chengwei Hu. Yanshi Li, Yujin Yuan, Yancheng He, Xingyao Zhang, Langming Liu, Shilei Liu, Wenbo Su, et al. Chineseecomga: A scalable e-commence concept evaluation benchmark 2025.
for large language models. arXiv preprint arXiv:2502.
[9] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, 7echun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model unified interface for vision-language multi-task learning. arX1v:23100.99778 2023.
[10] Sixiang Chen, Jinbin Bai, Zhuoran Zhao, Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang, et al. An empirical study of gpt-4o image generation capabilities. arXiv preprint arXiv:2504.0597. 2025.
[11] Yang Chen, Hexiang Hu, Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint 1.Xiv:2302.11713. 2023.
[12) Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu. Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 2rXiv:24/2.05271, 2024.
[13] Wei Chow, Juncheng Li, Qifan Yu, Kaihang Pan, Hao Fei, Zhiqi Ge, Shuai Yang. Siliang Tang, Hanwang Zhang, and Qianru Sun. Unified generative and discriminative training for multi-modal large language models. arXiv preprini arXiv:2411.00333 2024.
[14] Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physhench: Bench- marking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501. 1641 1, 2025.
[15] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. GitHub repository, 2023.
[16] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 326-335, 2017.
[17] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: tre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. volume I (long and short papers). pages 41866 2019.
11

---

[18] Li Dong. Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019.
[19] Taoran Fang, Tianhong Gao, Chunping Wang. Yihao Shang, Wei Chow, Lei Chen, and Yang Yang. Kaa: Kolmogorov-art arnold attention for enhancing attentive graph neural networks. arXiv preprint arXiv:2501.13550 2025.
[20] Taoran Fang, Wei Zhou, Yifei Sun, Kaiqiao Han, Lvbin Ma, and Yang Yang. Exploring correlations of self-supervised tasks for graphs. arXiv preprint arXiv:2405. 04245,, 2024.
[21] Hao Fei, Yuan Zhou, Juncheng Li, Xiangtai Li, Qingshan Xu, Bobo Li, Shengqiong Wu, Yaoting Wang. Junbao Zhou, Jiahao Meng, Qingyu Shi, Zhiyuan Zhou, Liangtao Shi, Minghe Gao, Daoan Zhang. Zhiqi Ge, Weiming Wu, Siliang Tang, Kaihang Pan, Yaobo Ye, Haobo Yuan, Tao Zhang, Tianjie Ju, Zixiang Meng. Shilin Xu, Liyu Jia, Wentao Hu, Meng Luo, Jiebo Luo, Tat-Seng Chua, Shuicheng Yan, and Hanwang Zhang. On path to multimodal generalist: General-level and general-bench, 2025.
[22] Stephanic Fu, Netanel Tamir, Shobhita Sundaram, Luey Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint 2.36.v:3306099444 2023.
[23] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148-166. Springer, 2024.
[24] Zhiqi Gc, Juncheng Li, Qifan Yu, Wei Zhou, Siliang Tang, and Yueting Zhuang. Demon24: Acm mm24 demonstrative instruction following challenge. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11426-1144 1442-11428,, 2024.
[25] Yash Goyal, Tejas Khot, Douglas Summers- Stay. Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role image understanding in visual question answering. Proceedings of the I.E.F. conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[26] Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng. Semantic models for the first-stage retrieval: A comprehensive review, ACM Transactions on Information Systems (TOIS), 40(4): 42, 2022.
[27] Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pages 1463-1471, 2017.
[28] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[29] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming- Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12065- 12075, 2023.
[30] Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. Mrag-hench: Vision-centric evaluation for arXiv:2410.08182. 2024.
multimodal models. arXiv preprint
[31] Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval- visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IERF/CVF conference on computer vision and pattern recognition, pages
2023.
[32] Xuanwen Huang, Wei Chow, Yang Wang, Ziwei Chai, Chunping Wang. Lei Chen, and Yang Yang. One graph model for cross-domain dynamic link prediction. arXiv preprint arXiv:2402.02 088 2024.
[33] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint ar 2110:22410. 2024.
[34] Chuong Huynh, Jinyu Yang. Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, and Abhinav Shrivastava. Collm: A large language model for composed image retrieval. arXiv preprint MXXi:2503319910 2025.
12

---

[35] Gabriel Ilharco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori. Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
[36] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-388 2023.
[37] Dongfu Jiang. Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi image instruction tuning. Transactions on Machine Learning Research, 2024, 2024.
[38] Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407. 12580, 2024.
[39] Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingho Zhou, and Wenhu Chen. V1m2vec: Training lasion-language models for massive multimodal embedding tasks. arXiv preprint urXiv:2410.05160. 2024.
[40] Siddharth Karamcheti, Suraj Nair. Ashwin Balakrishna. Percy Liang. Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visnally-conditioned language models. In Forty-first International Conference on Machine Learning, 2024.
[41] Zhibin Lan, Liqiang Niu, Fandong Meng, Zhou, and Jinsong Su. Llave: Large language and vision embedding models with hardness-weighted contrastive learning. arXiv preprint arXiv:2503.0m 048/2, 2025.
[42] Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-auge augmented generation for nlp tasks, Advances in neural information processing systems, 33:9459-994 99444, 2020.
Rerrieval-uggmented
[43] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 33299-13308 2024.
[44] Feng Li, Renrui Zhang. Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. sewva-net-interleavve Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07899. 2024.
[45] Hui Li, Tsz Nam Chan, Man Lung Yiu, and Nikos Mamoulis. Fexipro: fast and exact inner product retrieval in recommender systems. In Proceedings of the 2017 ACM International Conference on Management of Data, pages 835-850, 2017.
[46] Juncheng Li, Xin He, Longhui Wei. Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-languse pre-training. Advances in neural information processing systems, 35:7290-7300 2022.
[47] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal llss to follow zero-shot demonstrative instructions. arXiv preprint Xivx:3008.00152 2023.
[48] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language image pre- pre-training for unified vision- language understanding and generation. International conference on machine learning, pages 12888-12906 12900. PMLR, 2022.
[49] Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang. Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. Embeddingg product retrieval in taobao search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3181-3189, 2021.
[50] Ji Lin, Hongxu Yin. Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao. Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
[51] Sheng-Chier Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal lms. arXiv preprint vrXiv:2411.02511 2024.
[52) Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
13

---

[53] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.03743, 2020.
[54] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Proceediggs of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16296-263366 2024.
[55] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In NeurlPS, 2023.
[56] Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. arXiv preprint arXiv:2412.0172,, 2024.
[57] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference on computer vision, pages 10012-10022, 2021.
[58] Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. Universal vision- languaee dense retrieval: Learning a unified representation space for multi-modal retrieval. arXiv preprint arXiv:2209.00179 2022.
[59] Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Tency, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-lan lannn-uge models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2125-2134, 2021.
[60] Hanqing Lu, Youna Hu, Tong Zhao, Tony Wu, Yiwei Song, and Bing Yin. Graph product retrieval in e-commerce search. arXiv preprint arXiv:2 2021.
Graph-based based multilingual
[61] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong. Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseck-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.0552 205.05555 2024.
[62] Alessandro Magnani, Feng Liu, Min Xie, and Somnath Banerjee. Neural product retricval at walmart. com. In Companion Proceedings of The 2019 World Wide Web Conference, pages 367-30 307- 2019.
[63] Md Messal Monem Miah, Agent Chatterjee, Arindam Mitra, Ruihong Huang, and Man Luo. Miracle: Multimodal image-text retrieval and analysis for contextual long-form evaluation.
[64] Hanseok Oh, Hyunji Lee, Seonghyeon Ye, Haebin Shin, Hansol Jang, Changwook Jun, and Minjoon Seo. Instructir: A benchmark for instruction following of information retrieval models. arXiv preprint arXiv:2402. 14334, 2024.
[65] Maxime Oquab Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 4rXiv:2304.07193 2023.
[66] Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang, Auto-encoding morph-tokens for multimodal 1lm. arXiv preprint PrXiv:2405.01922 2024.
[67] Dan Qiao, Yuan Gao, Zheming Yang, Di Yang, Ziheng Wu, Pengcheng Lu, Minghui Qiu. Juntao Li, and Min Zhang. oncyder-only Imms can be masked auto-encoders. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 713-723, 2025.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PmLR, 2021.
[69] Tim Rädsch, Leon Mayer, Simon Pavicic, A Emre Kavur. Marcel Knopp. Bars Öztürk, Klaus Maier-Hein. Paul F Jaeger, Fabian Isensee, Annika Reinke, et al. Bridging vision language model (vlm) evaluation gaps with framework for scalable and cost-effcctive benchmark generation. arXiv preprint 255. 2025.
[70] Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Plister. Pic2word: Mapping pictures to words for sero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19305- 13305-19314 19314, 2023.
14

---

[71] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. 5aion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278- 25294, 2022.
[72] Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint urXiv:2504.20590 2025.
[73] Qi She, Junwen Pan, Xin Wan, Rui Zhang, Dawei Lu, and Kai Huang. Mammothmoda: Multi modal large language model. arXiv preprint arXiv:2406. 18793, 2024.
[74] Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu, Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412. 15188, 2024.
[75] Manish Sihag. From videos to requirement: data-driven approuch for finding requirements relevant feedbuck from TikTok and YouTube. PhD thesis, 2023.
[76] Gemini Team, Petko Georgiev, Ving Ian Lei. Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555 2024.
[77] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagini IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision vision-centric exploration of multimodal Ilms, Advances in Neural Information Processing Systems, 37:87310-885 1187730--7756... 2024.
[78] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xic, Eyes wide shut? exploring the visual shortcomings of multimodal 1lss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[79] Anthony D Wagner, E Juliana Paré- Blag-B.a Jill Clark, and Russell A Poldrack. Recovering meaning: left prefrontal cortex guides controlled semantic retrieval. Neuron, 31(2):329-333 2001.
[80] Benyou Wang. Lifeng Shang. Christina Lioma. Xin Jiang, Hao Yang. Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. In International conference learning representations, 2020.
[81] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024.
[82] Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang. Hang Song, Xingean Hu, Pengyu Wang, and Li Xiao. Advancing fine-grained visual understanding with multi-scale alignment in multi- modal models. arXiv preprint arXiv:241 1,0691, 2024.
[83] Weiyun Wang, Zhe Chen, Wenhai Wang. Yue Cao. Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu. Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411. 10442, 2024.
[84] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang. Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. European Conference on Computer Vision, pages 387-404. Springer, 2024.
[85] Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. Rethinking infonce: How many negative samples do you need? arXiv preprint 1rXiv:2105.1300 13003, 2021.
[86] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language foedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11307-11317, 2021.
[87] Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, et al. Scimmir: Benchmarking scientific multi-modal information retrieval. arXiv preprint arXiv:2401.1 13478, 2024.
[88] Yin Wu, Quanyu Long, Jing Li, Jianfei Yu, and Wenya Wang. Visual-rag: Benchmarking text-to-imne -moge retrieval augmented generation for visual knowledge intensive queries. arXiv preprint arXiv:2502. 16636. 2025.
15

---

[89] Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang. Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, and Minghui Qiu. Valley2: Exploring multimodal models with scalable vision-language design. arXiv preprint arXiv:2501.08 0059010900110. 2025.
[90] Robert S Wyer Jr, Iris W Hung, and Yuwei Jiang. Visual and verbal processing strategies in comprehension and judgment. Journal of Consumer Psychology, 18(4))244-257, 2008.
[91] Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. Retromae: Pre-training retrieval-oriented language models via masked auto-encoder. arXiv preprint arXiv:2205. /2035, 2022.
[92] Brent Xu, Dhruv Luthra, Zak Cole, and Nate Blakely. Eos: An architectural, performance, and economic analysis. Retrieved June, 11(2019):41, 2018.
[93] Shilin Xu, Yanwei Li, Rui Yang. Tao Zhang, Yueyi Sun, Wei Chow, Linfeng Li, Hang Song. Qi Xu, Yunhai Tong, et al. Mixed-rl: Unified reward perspective for reasoning capability in multimodal large language models. arXiv preprint 2XXiv:555..24144 2025.
[94] Yifan Yan, Shuai Yang, Xiuzhen Guo, Xiangguang Wang, Wei Chow, Yuanchao Shu, and Shibo He., mm- expert: Integrating large language models for comprehensive mmwave data synthesis and understanding. arXiv preprint wxXv:2509.1652) 2025.
[95] Qifan Yu, Wei Chow, Zhongqi Yuc, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang. Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint 1xXx::2411,,57:33 2024.
[96] Qifan Yu, Juncheng Li. Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang, Hallucidoctor: Mitigating hallucinatory toxicity visual instruction data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944-12955 2024.
[97] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 9556-9567, 2024.
[98] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975-11999 11986, 2023.
[99] Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. ProductIm: Towards weakly supervised instance-level product retrieval via cross-modal pretrain- ing. In Proceedings of the IEEE/CVF international conference on computer vision, pages I 1172-117791, 2021.
[100] Haotian Zhang, Mustafa Abualsaud, Nimesh Ghelani, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. Effective user interaction for high-recall retrieval: Less is more. In Proceedings of the 27th ACM international conference on information and knowledge management, pages 187 199-196,22 2018
[101] Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lec, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint MrXiv:2403.19651 2024.
[102] Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Zigi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval hy multimodal llms. arXiv preprint arXiv:2412. 2.... 16855, 2024.
[103] Xiaoyang Zheng, Zilong Wang. Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu, and Xiaoyi Zeng. Make: Vision-language pre-training based product retrieval in taobao search. In Companion Proceedings of the ACM Web Conference 2023, pages 356-360, 2023.
[104] Junjic Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong, Mcgapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint ar1iv:2412. 14445 2024.
[105] Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. Vista: visualized text emhedding for universal multi-modal retrieval. arXiv preprint 240600220, 2024.
[106] Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong. Zhiyuan Liu, Yu Gu, and Ge Yu. Marvel: unlocking the muuli--mdda capability of dense retrieval via visual module plugin. arXiv preprint ar1iv:2310.1 14037, 2023.
16

---

[107] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 1rX1v:2304.1059.. 2023.
[108] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang. Youngjae Yu, Ludwig Schmidt, William Yang Wang. Yejin Choi. Multimodal C4: open, billion-scale corpus of images interleaved with text. arXiv preprini wxXiv:2304009933 2023.
[109] Jian Zou, Tianyu Huang. Guanglei Yang, Zhenhua Guo, Tao Luo, Chun-Mei Feng, and Wangmeng Zuo. Unim 2 ae: Multi-modal masked autoencoders with unified 3d representation for 3d perception in autonomous driving. In European Conference on Computer Vision, pages 293-313. Springer, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Both contributions and scope have been discussed in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The detailed analysis on limitations have been discussed the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This is an empirical study that excludes theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information needed to reproduce the experimental results have been disclosed. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: The detailed implementation procedures have been included the appendix.
To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ htt.///eeuiisiiooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSumias //nip..cooo wttpccccppppc//ccccc/cccc htalllic/iiieeee gomoeeSSSS ssionnii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All training and test details have been discussed in either main body or appendix. To ensure reproducibility, code and data are committed to be publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Sufficient information about experiment settings have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details on computing resources have been discussed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss//naarrss....pbbl. htte//ublcc/tthieeeeeeeeeeeeeeee httpr//s/.cpppoeeetttooooooooae.
Answer: [Yes]
Justification:
This research follows the NeurIPS Code of Ethics properly.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The discussion societal impacts has been included in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The discussion safeguards has been included in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The acknowledgments on licenses have been included in the appendix.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The discussions new assets have been included in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : http://een.../eeeeeeeeeee /ttpe/rips.c/ooofeeeeeeeeee / for what should or should be described.
24

---

Contents
Detailed Dataset Collection Process
26
A.1
Product Source
26
A.2
Attribute Delineation and Annotation
26
A.2.1
Attribute Delineation
................................
26
A.2.2
Attribute Annotation
31
A.3
Retrieve Data Build
................................
32
A.3.1
Conventional Uniform Sampling
................................
32
A.3.2
Attribute Uniform Sampling
33
A.3.3
High-Similarity Product Prioritization Sampling
................................
33
A.3.4
Cold start expansion
35
A.4
Auto and Human Filter
37
A.4.1
Auto Filter
37
A.4.2
Human Filter
37
B Data Annotation Protocol
38
B.1
General Guidelines
38
B.2
Data Format and Structure
38
B.3
Quality Control and Validation Protocol
38
B.4
Handling Ambiguities
39
B.5
Ethical Considerations
39
B.6 Data Contamination Considerations
39
B.7 Annotation Platform
39
B.8 Data Preparation and Release
C More Dataset Analysis
41
C.I Global Statics
C.2 Attribute and Product
43
Dataset Examples
48
E Experiments Details
55
E.1
Baselines
55
E,2 Other Dataset Usage
56
E.3
Main Experiments Settings
57
E,4
More Experiments Results
57
E.5
Error Analysis Details
57
E.6
Human preference...
61
E.7
Condition Count on MERIT
61
Broader Impact
62
25

---

F.1
Impact
62
F2
Limitations
62
A
Detailed Dataset Collection Process
The complete data generation pipeline is illustrated in Fig. 5. Below, we provide a more detailed description of annotation process and the rationale behind design choices than in the main text. We confirm that all data has been anonymized and complies with open-source guidelines. Specifically, in Appendix A.I, describe the sources of our product data, and in Appendix A.2, we elaborate our methodology for labeling products to obtain diverse attributes.
Subsequently, in Appendix A.3, we describe approach for generating rich set of retrieval pairs (which were later manually filtered and curated). Specifically, to enhance dataset quality and diversity, employed composite sampling approach integrating three methods: Conventional Uniform Sampling (Appendix A.3.1), Attribute Uniform Sampling (Appendix A.3.2), and High-similarity Product Prioritization Sampling (Appendix A.3.3). Additionally, pipeline supports cold-start expansion as detailed in Appendix A.3.4.
Finally, we introduce our two-stage filtering process in Appendix 1.4.1 and Appendix A.4.2, encom- passing automatic filtering and manual curation, respectively. The automatic filtering stage employs rule-based systems and statistical methods eliminate obvious inconsistencies and low-quality samples, while the manual filtering stage involves expert annotators who apply nuanced judgment to ensure semantic coherence and practical relevance. This rigorous quality control process results in high-fidelity dataset that meets stringent academic standards.
A.1 Product Source
The MERIT dataset comprises high-quality, anonymized items from our internal dataset. Specifically, we curated popular products across six Southeast Asian countries, covering five languages. The items were sampled from both category- listings (as the primary source) and high-quality open-category offerings to enhance data diversity. Each item underwent aesthetic scoring [71. 95] before inclusion in the final repository. The distribution of categories and their respective quantities detailed in Tab. 4. Note that these categories are simplified for annotation purposes and do not reflect the full hierarchical taxonomy. All sensitive information has been removed to comply with open-source data standards.
However, among these well-performing products, some exhibit suboptimal visual presentations that diminish their search augmentation potential. The imagery associated with these products lacks visual efficacy and fails to fulfill the objective of facilitating retrieval through visual cues. Fig. 10 illustrates several examples of such products.
A.2 Attribute Delineation and Annotation
A.2.1
Attribute Delineation
To support diverse real-world retrieval needs, it essential to obtain thee-graied and varied product attributes for effective combinatorial search. However, existing product retrieval data often suffers from insufficient attribute richness, leading to suboptimal recall for specific user queries. This limitation stems from fundamental gap: operational attribute schemas (typically rigid and sparse) fail to meet the demands of search relevance tasks, which require precise, granular attribute information.
Directly predefining attributes and values for each product category whether through manual annotation or MLLMS [33, 76, -intriruus two challenges: (1) restricted coverage due to finite predefined options, and (2) biases (human LLM-induced) that may overlook critical product features. To address this, we propose an open-ended attribute delineation approach, where attributes and candidate values first generated openly and then refined. As illustrated in Fig. I1, pipeline begins with raw platform properties, employs LLM verification to ensure diversity, and finally derives structured attribute-value pairs through frequency inversion and label refinement.
Specifically, the L.MM-based attribute extraction module takes a product's image and textual information input and generates set of open-ended attributes. Figs. 12, 13, 14, 15, and 16 provide
26

---

Table 4: Product Categories and Countings in MERIT.
Primary Type
Second Type
Number
Fruits
1, 001
Food
Snacks
3, 077
Beverages
3, 481
Tops
32, 100
Clothing
Pants
26, 398
Shoes
24, 267
Phones
11, 671
Headphones
4, 670
Electronics
Laptops
842
Other Screens 351
Backpacks
5, 881
Bags
Handbags
4, 755
Suitcases
5, 540
Gold
405
Jewelry
Silver
608
Diamonds
590
Tables
3, 104
Furniture
Chairs
6, 101
Others
158
Total
135, 000
examples of annotated outputs. However, such open-set annotations frequently generate excessive irrelevant attributes (i.e., attributes that do not align with practical product retrieval needs). For instance, in Fig. 12, attributes such "Isbn/Issn": ("97802262642199
"Language": ["English"], "Version": ["First Edition"], "Cover Type": (""aae"r""Paabaa ["Paperback"], "Number Of Pages": ["464"], "Features": ["27 photographs", "6 maps". "25 ilustrations/diaarrams]] "Publication Date": ["May 15, 2018"J, and "Publisher": ["Basic Books"] ] are rarely used for product retrieval queries and should therefore be excluded. Similarly, in Fig. 14, the attribute "Batteries Included": ["no"] ambiguous and unsuitable for retrieval purposes.
The LLM attribute verification step evaluates whether the extracted attributes reasonable for the given product category, retaining only those that are valid to narrow down the candidate set. During this process, we also standardize the format of attributes and their corresponding values by ensuring consistent formatting, avoiding capitalization of first letters, and removing underscores.
Next, attribute frequency inversion and filtering organizes the verified attributes by product category, as outlined in Tab. This step accounts the distinct characteristics of different products- for instance, clothing may include attributes such pattern, neckline style, zipper type, and sleeve length, whereas electronics like smartphones may feature RAM, storage capacity, and battery size.
Considering these factors, we employed the following methodology to define closed set of attributes:
Rule-based filtering: We initially identified 100 attributes using inverted indexing (employ- ing normalized strings: converting all text to lowercase and removing extrancous spaces
27

---

BLOUSE 250 CHECKOUT
200 FOR CHECKOUT ONLY
Off-white
SP3/CEP
Silk Blouse for Work: Versatile, Comfortable, and Stylish Office Attire
Men's Regular Fit Cotton Blend Denim Jeans with Button Closure
Kerudung Hurberry Off White dengan motif kupu-kupu
Examples of low- now-quality products. The associated images fail to accurately represent the visual characteristics the products, instead containing irrelevant content.
Figure 10:
Wei Zhou 1814
Q
Various product
LLM attribbuee verification
Attribute frequency inversion and
Original properties
b O
attriiute
Wel Zhou 1814
Product- attribute item-Attribute value
Figure 11: An Overview our Attribute Delineation Pipeline. The process begins with extract- ing original platform properties and diverse product attributes, followed by LLM-based attribute verification. We then perform attribute frequency inversion and generate refined labels, resulting in structured product information with open-domate attribute-value pairs.
and punctuation). Selecting frequently occurring instances helps avoid annotating niche attributes that are unlikely be used retrieval tasks.
Human-LLM, collaboration: Based on the 100 identified attributes and our domain exper- tise, we curated distinctive set of 20 attributes for each product category.
The prompt used for attribute selection via LLM as follows:
You are an assistant who communicates only in JSON and is an expert in physics. Do not am working with a dataset for attribute retrieval in clothing categories. Below is list of extracted attributes. Your goal is to help me select the top 20 attributes that best meet the following criteria: Relevance: The attribute should be common across multiple clothing categories (e.g., dresses, shirts, pants).
Searchability: The attribute should be easily combinable with other attributes in queries (e.g., "long sleeve striped").
User Demand: The attribute should reflect features users frequently search for filter by. Visual Detectability: The attribute should be identifiable from product image (e.g., "color", not subjective traits).
Please prioritize attributes that satisfy all four criteria. If needed, suggest rephrasing attributes for clarity consistency (e.g., "sleeve length" instead of "long sleeve").
Exclusions: Exclude overly niche rarely used attributes.
No overlap: Remove synonymous terms
You should return your answer in Python list:
list: [tape here]
28

---

The following is an example of the list of attribute values returned by this operation (products of cloth type):
"color": ["red", "green", "yellow", "orange", "purple"],
"size"; ["small". "medium". "large", "extra large"),
"flavor": ["sweet", "sour", "tart", "bland", "Iangy"].
"fruit type": ["apple", "berry", "citrus", "melon", "tropical"], "organic": ["organic" "non-organie",,
"quantity per pack": ['1 piece". "3 "3-pack", "5 lbs", "10 kg"], "ripeness": (""uripp"" "ripe". "overripe"],
"texture": ["smooth" "bumpy", "fuzzy", "rough"),
"shape": ["round", "oval", "elongated", "lobed"],
"weight": ["100g". "500g", "Ikg". "2kg"].
"pattern": ( ("striped", "spotted", "solid", "motrled"), "seed presence": ["scedless", "with seeds"]. "skin thickness": ("thin-skinned", thick-skinnedd", "juiciness": ["juicy", "dry", "moderate",,
"sweetness": ["low sugar", "medium sweet", "very sweet"), "storage type": l"refrigerate", "room temperature", "cool dry place"], "region of origin": ["Florida", "California", "Chile", "Spain"], "material": ["plastic clamshell", "mesh bag", "paper bag". "loose"], "height": ("short", "medium", "tall"), "allergen information": ("citrus-free", "latex-free", "sulfite-free")
After obtaining the attributes, we expanded each attribute to its corresponding values. Specifically, similar the previous step, we employed the following methodology:
Rule-based approach: We retrieved the top 100 values for each of the 20 attributes through inverted index (ensuring that attributes were clearly specified and excluding cases where attribute values contained multiple values). The returned content was consistent with the previous step.
Human + LLM approach: From the 100 values for each attribute, we identified 20 distinctive values.
The prompt used for attribute selection via LLM is as follows:
am working on attribute retrieval for clothing categories and need select the top to values per attribute for labeling. The chosen values should optimize for:
Relevance Common across multiple clothing categories (e.g., dresses, shirts, pants). Searchability Easily combinable in queries (e.g., "long sleeve + striped").
User Demand Frequently searched or filtered by users.
Visual Detectability Clearly identifiable from product images. For example, the specific numerical value of weight and size cannot be seen and should not appear in your answer). Universality Collectively covers all scenarios without redundancy.
Requirements:
No overlap: Remove synonymous terms (e.g., "round neck" - "crew neck").
Prioritization: Favor attributes that meet all five criteria.
Clarity: Suggest rephrasing for consistency (e.g., "button-front" vs. "buttoned"), Exclusions: Omit niche, ambiguous, rarely used terms (e.g., "peasant sleeve").
Please return the attribute (dict's key) with its choice values (dict's value) JSON. [tape here]
Our approach yields attributes with two key advantages:

---

Title:
Sopiens A Brief Hatory of Humankind Paperbock- - May 15, 2018
Description:
Sapiens
Yuval Nosh Haraci
groundbrecking narrative humenity's creation and evaltion- that explores the ways in which biology and bieery have dieined and enhanced our understanding of what it meore to be "human" One hurdred thaveand ago, at least eis different species of humam Inhabited Forth. Yet today there sopiens,
From renownedl histrrian
Sapiens
happeried the otherst And what may hoppon to us? books about the sitory himonity purive etter a Niporical or biological approach, but Dr. Yoval Nooh Harori brooks the nold with this highly originel book that begins about 70,000 role evolving humars played the global ccosystem charting the rise emplnes, Sepless integrates History and science to recomider accepted norratives connect past developments with contemporary concerns, and examine specific events Haradi compals
only
Brief History of Humankind
ogo with
of modem cogrition. From examiiing the
A Brief History of Humankind
lorger
look chand,
over few decades hurrem have begun hand laws of selection that have govemnd life for the past four billion years We are acquiring the absiity to design not curseives Where this loading and what de photogrohh, 6 maps,
only the world around
Ilustrotions (dlagrams, provocetive and insighiful work is sure to spork debots and essertial reading for efidenedos Jared Diamond, Jomes Gleik,, Matt Ridley, Robert Wright, and Shoron Moolem.
want become? Teoturiig
Sapiens
Yuval Noah Harari
Attribute
(SSpecial Solling ("Gvoundbrrcoking narrative, "Explores humonn evolution", "integranes Nistryy and science)) "bdbn//onr: ('97802262442199 19), "Language":
["tngfish]] "Verdon": ("Fist telttion)] "Cover Type": ("hoperback), "Number Of Pages" ( "464"), "Fecoures' ['27 photographs"; ; "6 mapd", "25
[lustration ilurratiomddaarams /diagrams", "Gift For': ("Hatory enhhsiest"), "Yoer" 1 ('2018"] "Author': ("Ywall Noch Harorf]) "Publicarion Dere': : (MMay 15, 2018"), "Publisher':
["Basic Bools"))
Figure 12: Open-ended Attributes Annotated Product Case.
Tille:
Even Dorkness Stickers Samwite Gongee Lord Engs Quote, Imspire, Waterproof Vinyl
Decal
Description:
Feataring icanic quote Semwise Gamgee, this stiker a perfect reminder hape and perseverence Design Inspired by J.R.P. Talkien's Lord of the Sings. Features the powerful quote: "Even darkeess must pas." Matertal & features: Made from high- quality PVC/PP
Embrace
Midcle-eart with Darkness
Waterpreof and stain-esistent for durability Eesy-renve adhesive for hassle-free opplication. Plat slicker type with smooth finish, Perfect for: loptops, water bottles, notebooks, and more. Decorating walls, windows, doon, and wardrobes. Ideal for any fan of Lord of the Rings and impiring lbe
Dimensiom: Height inch surfoce. Avoid excessive benciing or scratching to mointain qualty. Corry the spirt of Samwise with you, wherever go. Get your Even Darkness Must Pass sticker today!
Length inches Weight
Apply dlaan, dry
Attribute:
("Special Selling Paint: (Toppired by L.R. Tolkien's Lord of the Rings", "any fan of Lord of the Rings") "Langth" ('8 inches") ") "Size" ("Height 1 Inch", "Widlhu B Inches', "Length inches"). ) "Bomertes Included) ['No"] "Matertol", (PVC,/PP), "Product Type": nicker'), "Occosion", ['Decoratingg, 1, "Fecttuees" ("Waterproof,, "Stais-resstem", "eoove odhesive", "fler stickr type"). "Gilt For': ('Foms of Lord of the Rings") "Pattem/Prinl' ("Quore"), "Color' ["Black"), "Banafin': ('Durability,, "Henele-fres opplication)] "Style"l ("Iaapreed IRR.R Tolkien"), 'Room/Lecation" Location' i ("Waile', "Windows", "Doors", "Wardraber",
"uunttty Pack' [T1"
Figure 13: Open-ended Attributes Annotated Product Case.
Title:
Wonchful Mark iPhone Tough Cose
Description:
Moterials Dedinn < 1(MMade from duroble polycorbonate jouter shell) and TPU (lmer linng) for added shock absorption, "1r2-plece dedin wrhonces impact resiztacce disperses shock effectively. "Uteduudes interfor rubber liter extro protection (mey very besed on phone model), +iyGoey firish for sleek and polished lock. WWWirele charging compatible (Magaele not inducied). Technology Inku DDDeiigedd using dye sublimation process involves printing on heat-recktom transfer paper and embedding the image into the case with heot press. Production Time: vlyCoses are produced withi 1:2 business days, ensuring quick turnoround
vibrant, lang-lasting prim. "ifTe
Shipping: "(Akfer production, orders shioped within 3.5 business deys, depending on destination. Quality Control: '/yfach case undergoes individual inspection for print quality, design elignment, and functionaiity before shipping. Pockaging Control s/PPookaged peel-and-seal securely protected during
mailing emelaper
Attribute:
Batterias Included' ("re"), 'Applicoble Mobile Brand": ('Phone"),, "Material' ("pelycarbonnte"" "TPU', "robber'), "Cave Type": i ("oough case") "I "Features": Chock obsorption", "iieeees charging comporible", 1, "Pottem/hrin'' ('anine prin') "Color" ("ree"))
Figure 14: Open-ended Attributes Annotated Product Case.
30

---

Title:
Make Chritmos Green Agoin, Punny Trunp Dancing Ugly Sweeter, Trump 2024 Chrismos All-Over-PPinne -Prined Sweetdirr P161110
Description:
Make Chritmes Great Again, Funny Trump Dancing Ugll Swecner, Trump 2024 Christess Al-Over-Priited Sirectdirt P1611T0 Brianloephhds Product Detail Weight: Crafted with substantial weight of 200gun, our uply sweeter esceptional
TO MAKE CHRISTMAS
chilly days. Saft Cory, Featuring bruahed texturo, this sweater offers a losurioulyy soft and plush feel ogainst the skin, ersuring you stay comfortoblee oll day long. Duroble Condruction Dedgned to wihhtand wear and tear, the thick and sturyy fobrie makes on ideal choice colder dinates, provicing losting quelity... Molstrreeeeee Absorbent: Our ogly sweater effectivcly wicks away matoure, keeping you dry comfortoble whether you're Indoors or oundoors quality and longevity of your sweeter, please follow these care instructions_ Weshing: Mechine wash cold or the fabric and colors Drying You tumble dry on law heat opt for softeess lroiing needed, iran preaiing datn avoid direct contact with the fobrie. Stain Trentmern einins, treat avoid bleoch prevent domage the fabric. Storage: Store your sweeter a cool, dry place owey from direct vibrant colors and integilly. Avoldi Please do not dry dean sook sweeter, this
comfort, moking perfect
instructions to
ware water gartle sydle to maintaics in
drying
heat using detergent and
GREAT AGAIN!
sunlight to load to unwanted danage.
Attribute:
("Designn: ("Trump 2024 Christoos, "Funyy Trump Donding), "Cooure Type': ["Pulover"), "Sutable Scasons" ("Winee"], 'Neckline: ('Crew (Nok"] "Stze's
["1161110) "Fi Type': ("Regslor fir' "Sleeve Length" ('Leng Sleeve'l "Core Instruction; ('Machine colef, "uubble low", "hon low hear", "Store cool, dry place", "Avold dry cleen") "Material" ("300gen fobric") "Ocendon", ["Chiimma"), "Faatome" ("Sch Cary", "Durable Conerrctto", "Moleture Absorbent") "Gif Far', ('Those who enjoy political sotire), [) "Patte /Print. ('Ali-Oon-Priined] "Coler" ('Block), "Styla", ("Ugly Sweater") "Composiion"' ["300gmn], "Quantty Per Pock) (*I
Figure 15: Open- Attributes Annotated Product Case.
Title:
Lipatick Silk Finish, Wama Bibir yong Kaya dan Mudah Ditumpuk dengen Kandungan Palembap, Diformulesikan dengan Vitamin A, Minyak Macadamia antuk Hidrasi Maksimal
Description:
LPSTICC SILK FINISH Warna bibbir kami yang sangst pigmen difermdlsskkn dengan Vitamin A & E, Lidch Buoya, don minyck kocong mocadamio koya antooksidnn, sehinggel bibir terasa lembop dan halus dengan warna yong menzalak.
RAWAT BIBRR ANDA: Lpprick Silk Finish membertkan wana tchon loma yang bold dengon kandungon dan bahan clami, sehingga Ando menciptekan tompilan sempumo. Gunakan etou lapli dengan lip liner don lp gloss komi
TAMPIL SEMPURNA, Nulal dori bronzer, blush primer, highlighter, bedok natural, hinggo concealer-kant menyediokan semuai yang
bututkan unluk tampil flawless slop menghadopi momen penting: Kini, Anda telch menemukan posengon makeup terbaik. BBBA KEKERASAN HEWAN 100% Produk kami bebas uji coba pode hewan (cruelly-rre) sejek awal, Kani monyedicken produk vegon dan alat makeup sepertl foundation, kutek, polet kontur, liostk, kuos makeup, dan lamnyo, STAY WILD: Tujuan kecontikan terpercaya bogi pecint makeup dari segals usia, suku, warna kulit, dan mana pun beroda, kani punya produk yang cocok
status ckonomi. Sicpp pun Andi.
Attribute:
("Special Selling Point: ("Vitomins A & E", "Aloe Vera", "Macodomia nut oif). "Makeup Finish Type": ['Sik flnish"), "Sutable Seasond": ('All seasons"]
'Application Areo": ["Lip="], "Product Form', ("Stick), "Skin Types" types", "Gender': ["Unlsex",, "), "Bomertes Incuuddd ['No"), " "Produer Type";
[Upatick" "Occesion": ("Face forward") "Teotures": ["Hydraiing", "Long-losting "Bold color", "Cruelty free") "I. "Gifi For": ('Boouty lovers") "Color:: ("Rieh")
'Beneftt'' ["Molsuutinng,, "Long-lore lostin bold color') "Pock Type's ("Single]] "Quantiy Per Pack' (")"))
Figure 16: Open-ended Attributes Annotated Product Case.
1. Diversity and Reduced Bias: The extracted attributes are both comprehensive and varied, demonstrating greater resistance to biases from either the LLM or human annotators [3, 32], while simultaneously improving recall of long-tail tail attributes.
2. Product-Centtee Representation: Since attributes are first annotated at the product level before being aggregated, they more accurately reflect real-world product characteristics.
A.2.2 Attribute Annotation
Through the methodology described in Appendix 1.2.1, we obtain a rich set of attributes and their corresponding values for each product category. This attribute set forms a closed set that is for retrieval tasks. Utilizing these closed attribute tables, we can annotate each product with its corresponding attributes. Specifically, we employ the following prompt:
suitable
31

---

Objective:
Annotate clothing product attributes based on the provided inputs. Your annotations must align with the given attribute options and labeling rules.
Inputs:
List of Attributes and Options: A predefined list of attributes (e.g., "color," " "sleeve length") and their valid options.
Product Image: The primary reference for annotation.
Image Attribute Table (Optional): Existing attribute descriptions for reference (may be incomplete empty).
Product Description (Optional): Textual description of the product (may be inaccurate verify against the image).
Labeling Rules:
Select the most accurate option for each attribute based on visible/verifiable details. Skip an attribute if cannot be determined from the inputs. Do not add new attributes. Leverage the Image Attribute Table (if available):
If attribute a semantically similar one) exists in the table, summarize its content as one of the predefined options.
If your summarized value isn't t in the options, ensure adheres Exclusions: Omit niche/rare attributes.
No Overlap: Remove synonymous terms.
Your Output Format:
Please output your answer in JSON format. The answer is a dict, where the key is the attribute and the value value of the attribute
The list of Attributes and Options is [paste here] Product Image is [paste here] Image Attribute Table is [paste here] Product Description is [paste here]
Following annotation completion, automatic filtering is performed to: eliminate newly constructed attributes; remove attributes with values designated as "none" or "skip"; exclude the most frequent value within each attribute category avoid generic descriptors such as "all ages"; and discard terms that appear only once in the dataset.
A.3 Retrieve Data Build
To enhance dataset quality and diversity, we employed a composite sampling approach integrating three methods: Conventional Uniform Sampling (Appendix A.3.1), Autribute Uniform Sampling (Ap- pendix A.3.2), and High-similarity Product Prioritization Sampling (Appendix A.3.3). Additionally, our pipeline supports cold-start expansion as detailed Appendix A.3.4. worth emphasizing that all data generated by our automatic combination algorithm subsequently underwent manual review and refinement. Furthermore, due the inherent limitations of closed-set attributes, many matches lacked precision (e.g., products labeled "Bohemian style" might not exhibit particularly similar stylistic elements, thus being filtered out). Consequently. a significant proportion of data was eliminated during post-prose post-processing, reflecting the high quality standards of our final dataset.
A.3.1 Conventional Uniform Sampling
Conventional Uniform Sampling is our most frequently utilized sampling algorithm. Specifically, we randomly select two products and randomly choose product attributes. then retrieve all products from the product pool that simultaneously satisfy these two conditions. We apply certain constraints to these recalled products:
The value of attribute for product a and product b cannot be identical, as this would render the use of two products for retrieval unnecessary
32

---

in an suult wwwuklooollsssssssn) valan th treenautta
stylish, streechy knitwnee strined pattem, featuring neck, sleases, comfortable daily
Condition Product
Condition
Leher Pandek Cataloan Jaker Bira und ik lebiki dan Perempuan, Saiz hingps Musim
Komeja Wanitta Korwa V-back Katur Salur denpan Kanzing clsn Gelatan Delakang,
Ag than minn sants size länn SKC,, csi coltoe riiiin usieen va tham má song trong tol riglt
Nakumi Makabi Simm
Longon Panjang Blouse Cur Line dangen Pals Switip,, Terhoat danl Colton Cambed yang Myoman Wenyera
ge ou wen 2h-a0 Evirns 11..
dn
Hordhrade dangen Dehon Katun Premium Grede yang Koriiggr Aan Rusut.
Kell
High Similarity
Low Similarity
Figure 17: The CLIP Similarity [35, 98] textbfRanking of Recalled Products Case 1. The first row the two condition products, and the second row is the recalled products. The CLIP similarity with the condition products decreases from left to right.
When randomly selecting values, we employ a probability distribution that suppresses frequently occurring keys, such color
Our dataset contains 2-4 conditions per query. When forming combinations, probability distribution as follows: 75 for conditions, 10 for 3 conditions, and 15 for 4 conditions (the higher proportion for conditions accounts for potential filtering during subsequent annotation)
the
At this stage, numerous products still remain. enhance the efficiency of subsequent manual annotation, we use CLIP Similarity to rank all qualifying samples according to their average similarity with the two products in descending order. This arrangement prioritizes similar products, facilitating the annotation process. The CLIP Similarity Ranking of Recalled Products can be seen in Figs. 17, 18, and 19. These figures demonstrate that CLIP similarity ranking serves a meaningful purpose.
However, this sampling method suffers from long-tail distribution of attributes (uncommon attributes rarely appear in the combined retrieval entries). Furthermore, due the extensive size of the product repository, we do not process all qualifying products in each iteration but instead conduct selective sampling, which may result in the absence of products that genuinely match the conditions in the recalled set. To address these issues, two novel sampling algorithms are introduced in Appendix A.3.3 and Appendix A.3.4.
A.3.2 Attribute Uniform Sampling
Relying solely Conventional Uniform Sampling strategies can be limited by the long-tail distribu- tion of attributes, as shown in Fig. 20. For instance, the "color" attribute appears with significantly higher frequency, while other attributes we introduce the Attribute Uniform Sampling algorithm. Specifically, we first uniformly select a particular value of given attribute, and then identify corresponding items to form combinations. This approach effectively increases the occurrence of less common keys and introduces long-tail scenarios, thereby enhancing data diversity. As illustrated in Fig. 20, after implementing this sampling method, the attribute distribution becomes more balanced, and a wider range of attributes emerges.
much less frequently [20]. To address this issue,
A.3.3 High-Similarity Product Prioritization Sampling
Due to the vast size of the product catalog, we do not process all qualifying products at once, but instead employ sampling, which may result in the retrieved products not truly matching the specified conditions. To address this issue, we introduce the High-similarity Product Prioritization Sampling algorithm, which identifies product and one of its similar products (manually pre-annotated).
33

---

Chān vay dāi träng thanh lich voi chift ligu mēm miti va thiet les se tá binh
Assyyiih pair of up canvas sneakers upper material, perfect
Condition Product
Condition Product
Neaut Kaaual Wan Karmes Premiam
Astylihh par of laceup carees commort and mibiiiy
Adlyfhhaal pai of whise carnees uporoakers
Torahal Turn whlshto
Be i8 canson toding this dindung
High Similarity
Similarity
Figure 18: The CLIP Similarity Ranking of Recalled Products Case 2. The first row is the two-condition products, and the second row the recalled products. The CLIP similarity with the condition products decreases from left to right.
gioy spc, liču ting
Smartphone with Dual Bear Carnerus While Finish
lsing
Condition 1roduct
Condition Product
V29e
ViGa Anane
sundillas Who I0M bots, POM wad D4118, 1T8, LCD sut aominolgriee Andruid 14
Vioo murrilltuo Furch tide atnnn 20008 siiitguul
with 468 RVN 6460 Storago.
with KAR 12tel LCB Digily. Stapdrapon SOMP Rebeased 2825.
with Punch Hoi Digpily Co Engineere ZEiS5
Snapshage pogplagge firm 1208
Buthery, and
RAM Storape
Battery, FSS Water and bust
High Similarity
Low Similarrty
Figure 19: The CLIP Similarity Ranking of Recalled Products Case 3. The first row is the two-condition products, and the second row the recalled products. The CLIP similarity with the condition products decreases from left to right.
Figure 20: Frequency distribution of conditional attributes before and after applying the Attribute Uniform Sampling strategy. The left panel shows the distribution before applying the Attribute Uniform Sampling algorithm, while the right panel demonstrates the distribution after implementing 10 Attribute Uniform Sampling. Notably, the attribute distribution becomes more balanced after applying the Attribute Uniform Sampling algorithm.
The algorithm then derives several conditions that would transform product into product 2, and
34

---

subsequently retrieves candidate products from the catalog based on these conditions. Finally, human annotators perform additional filtering to construct the retrieval query.
Additionally, the High-similarity Product Prioritization Sampling algorithm accounts for long-tail distributions by probabilistically reducing the frequency of the most common attributes (such as color-based differences between product pairs).
A.3.4 Cold start expansion
Furthermore, our pipeline demonstrates extensibility; upon establishing a comprehensive attribute list scale, we can perform cold-start initialization for previously unseen categories of data. Specifically, we leverage LLMs for automatic expansion through the following systematic methodology.
First, for any given product, we employ similarity matching approaches [49, 65, 68, 19] to retrieve analogous products as query conditions and corresponding target images. Subsequently, we utilize the following LLM prompt to characterize the distinctions between the two products. Where all attribute represents the distinguishing attributes previously identified from our retrieval dataset, therehy enabling newly integrated data to seamlessly adapt to the existing retrieval paradigm.
Task: Identify the Most Significant Attribute Difference Between Two Products
Instructions:
1. Examine the two product images carefully.
2. From the provided attribute list, select ONLY ONE attribute that represents the most significant difference between these two products. By modifying this attribute, you can convert picture to picture
3. Even though multiple differences may exist, focus on identifying the single most visually distinct and important attribute difference. Try not answer vague attributes like style. 4. Your response should contain ONLY the attribute name no explanations, no quotes, no additional text.
Image 1: <image> Image 2: <image>
Available attributes to choose from: { (all_attre attribbute }
Remember:
Your response must be exactly one attribute from the provided list
Do not add quotation marks or any additional text
Select the most visually distinctive difference
- IIf multiple differences exist, choose the most significant one
Response (attribute name only):
Upon obtaining the differential product attributes, analogous statistical analysis of the original retrieval data enables the extraction of the various differential values for each attribute. Given that our preceding data generation methodology prioritizes comprehensive diversity, the differential value options are nearly certain to encompass the most salient distinguishing characteristies within product pairs. Specifically, the annotation process utilizes the following prompt:
35

---

Task: Identify the Specific Value of a Differing Attribute Between Two Products
Instructions:
1. Carefully examine the two product images.
2. Focus ONLY on the following differing attribute: {diff_ [
3. From the provided list of possible values, select EX ONE value that best describes the
{diff_ _ that can make the first product (Image 1) into the product (Image 2).
4. Your response must contain ONLY the selected value explanations, quotes, additional
text.
5. If none of the values in the provided list accurately describe the difference, respond with ONLY the word: no Image 1: <image> Image 2: <image> Differing attribute: [diff_ attribute)
Available values to choose from: (all_values]
Remember:
Your response must be exactly one value from the provided list
Do not add quotation marks any additional text
Focus exclusively on the [diff attribute } difference between the products
Response (value only):
Task: Generate Enhanced Product Search Queries Based on Visual Comparison
Instructions:
1. Examine both the reference product image (Product 1) and the target product image carefully.
2. Create a natural, concise search query that helps retrieve the target product by:
Maintaining the original statement structure
Adding ONE new distinctive attribute unique to the target image IIclludin the target product's category/type in your search query
Key Requirements:
Focus only on visually detectable attributes of the specific product in both images Highlight meaningful differences similarities between products
Use natural, conversational language
Incorporate the new attribute seamlessly without explicitly stating "this product has attribute"
Preserve all Product tag formatting exactly as shown: <Product 1> limagel </Product 1> and
<Product 2>product description</Produe 2>
Note: Product will be represented by an image, while Product will be represented by text attributes.
Example: Product Image: [IMAGE]
Target Image: [IMAGE]
Search Statement: Find product with the same color as in <Product limagel </Product 1> and
the same brand as in <Product 2> product with brand Nike</Product 2>, Your Return is: Find a T-shirt with the same color as in <Product 1> limagel </Product and the same brand as <Product 2> product with brand: Nike</Product 2> with small logo.
Product Image: <image> Target Image:
<image>
Search Statement: Find product that has the material <Product 1> limagel </Product 1> and the same (diff_attr) as <Product 2> a product with (diff_attr): (diff_ value)</Producc 2>,
Your Return is:
Upon obtaining the differentiating attributes and their corresponding values, we can retrieve additional products through iitribute-based lookup and assemble them together with the initial conditions to
36

---

form a complete search query. It should be noted that the example presented above illustrates cases with a single differentiating factor, resulting in query formations with two conditions retrieving one product. We can further prompt the model to identify additional existing differentiating factors (utilizing the aforementioned prompts), thereby expanding the number of conditions and enhancing the precision of both search statements and retrieved samples. Once we have acquired the conditions and target positive sample products, we can employ the following prompt to generate the retrieval instruction through the LLM. As shown in the black block above the text.
Finally, this augmented data, like the data generated by the original pipeline, is fed into the filtering pipeline described in Appendix A.4.
A.4
Auto and Human Filter
A.4.1 Auto Filter
To enhance efficiency, implement initial LLM filtering stage prior to manual annotation. Specifically, when employing machine-based annotation, we require adherence to the following three fundamental principles:
Non-omission requirement: Essential information must be present in the text. For instance, material attributes must be explicitly mentioned within the textual content when required for retrieval.
Vision-centric approach: Attributes that can be effectively communicated through visual representation should rely on visual information rather than textual description. For example, product color attributes should be primarily identified through visual features rather than written descriptions.
Accuracy criterion:
The characteristies of positive samples must correspond precisely with the specified retrieval features. For instance, when color consistency is required as search criterion, the positive samples must exhibit color attributes that exactly match those of corresponding conditional products.
This multi-modal evaluation framework ensures that the automated filtering process maintains semantic consistency across both textual and visual dimensions while maximizing retrieval relevance. The LLM prompt employed in our implementation follows structured guidelines that prioritize these three principles, resulting in higher-quality candidate samples for subsequent manual verification. For the LLM implementation, the specific prompt utilized is:
A.4.2 Human Filter
Following the automated filtering process, data undergoes rigorous manual refinement to en- sure quality and comprehensiveness. This human verification stage consists of the following key components:
1. Data Curation: Annotators critically review and eliminate data entries deemed inappro- priate for the dataset. Furthermore, they refine imprecise search queries to enhance their accuracy and relevance to the target products. This process ensures that the search statements precisely capture the intended product characteristics and filtering criteria.
2. False Negative Correction and Positive Sample Expansion: Recognizing that search queries may retrieve multiple valid products beyond those initially annotated, annotators systematically identify and include all products satisfying the established search criteria as positive samples. This step addresses potential false negatives from the automated process and creates a more comprehensive and representative set of positive examples, thereby improving the overall quality and completeness of the dataset.
The detailed annotation protocol, platform specifications, and comprehensive documentation for this manual verification process are provided in Appendix B. These resources offer complete guidance on annotation standards, quality control measures, and platform-specific instructions for the human curation phase.
37

---

B Data Annotation Protocol
B.1 General Guidelines
As previously discussed, there is a significant gap in existing benchmarks [86, 63], which primarily focus on homogeneous retrieval scenarios with predefined formats, limiting themselves to single domains, single languages, employing only singular conditions. To bridge this gap, our benchmark, MERIT, is designed to provide comprehensive dataset for interleaved multi-condition semantic retrieval, integrating multilingual understanding with the assessment of composite multi-condition queries across diverse product categories. Our dataset follows the guidelines outlined below for data collection:
General Principles:
Annotations must be accurate, consistent, and adhere to high standard of academic rigor.
It covers multiple product categories and languages to mirror real-world applications across Southeast Asia.
incorporates diverse visual contexts and attribute combinations to foster a well- rounded evaluation.
provides robust evaluation settings for deterministic assessments.
Specifie Instructions:
All retrieval pairs must contain one or more images expressing specific properties.
All queries should available in five languages spanning six Southeast Asian countries.
All queries should meet real-world e-commerce search complexity.
Queries should not be ambiguous and must be answerable with the products in the dataset.
Clearly categorize each query across seven distinct product retrieval scenarios. Annotate all fields, including attribute tags, visual descriptors, and other elements that follow the format requirement.
Review Process: Ensure that every annotation undergoes a peer review to maintain high standards and minimize errors or inaccuracy.
Annotations such product attributes, language, country, category also collected, providing detailed examples that demonstrate the semantic retrieval capabilities of the models for further analysis and usage.
B.2 Data Format and Structure
Detailed examples of annotated retrieval pairs are provided in the guidance to serve as a reference for the annotators.
JSON File Format:
The structured JSON format will include fields for product identifiers, attribute types, attribute values, query conditions, target products, language identifiers, and region information.
Naming Conventions:
Each collected sample will be stored in separate JSON file following standard naming rule: product cettuet__cteorr___e catteaorreNNumbaaaa [Number].json
Image Files: image_ (ProduclD)
B.3 Quality Control and Validation Protocol
The integrity and reliability of the MERIT dataset are maintained through a comprehensive quality assurance framework that encompasses multilingual expertise and systematic validation procedures. The following protocol has been established and rigorously implemented throughout the data collec- tion process:
38

---

(1) Product Selection and Authentication Protocol. All candidate products must (i) look authentic. (ii) demonstrate substantial commercial viability as evidenced by popularity. and (iii) meet or exceed the predetermined aesthetic quality threshold quantified by established metrics [71, 95]. Each product undergoes verification by authorized annotators before inclusion in the product inventory.
(2) Attribute Standardization and Verification Protocol. Given the inherent insufficiency attribute richness in commercial e-commerce data, all products must undergo attribute enrichment via our prescribed open annotation methodology. Annotators shall (i) document all observable attributes using a standardized lexicon, (ii) participate in statistical analysis for Attribute Delineation, and (iii) verify attribute consistency across similar products. All attribute assignments must receive secondary validation before entering the final database.
To ensure statistical robustness and eco- logical validity, every search query must be generated through
(3) Query Formulation Compliance Protocol.
tri-modal sampling methodology: (i) Conventional Uniform Sampling, (ii) Attribute Uniform Sampling, and (iii) High-similarity Product Prioritization Sampling, as detailed in Appendices A.3.1-A.3.3 Cold-start expansion procedures (Appendix A.3.4) shall be employed only after primary sampling methods have been exhausted. Each query composition undergoes verification against pre-established diversity metrics.
(4) Multi-stage Validation Protocol. All dataset entries are subject to mandatory dual- phase validation: (i) Automated Validation Phase-employing deterministic rule-based systems and statistical outlier detection methods to identify and eliminate non non-conforming samples; followed by (ii) Expert Validation Phase Sease-rrquiring assessment by annotators with demonstrated proficiency in the relevant languages and product domains. Samples must achieve unanimous approval through both validation phases MERIT inclusion in the final dataset. Non-compliant samples shall be documented for methodological refinement purposes.
B.4 Handling Ambiguities
Instances of ambiguity unclear data are flagged for detailed review. These instances collabora- tively examined during team meetings to establish a standardized approach for annotation. Particular attention is paid to visual attribute interpretation across different cultural and linguistic contexts.
B.5 Ethical Considerations
Copyright and Licensing: Adherence to copyright and licensing regulations is strictly enforced. Data from sources that prohibit copying or redistribution is explicitly avoided.
Data Privacy: Compliance with privacy laws and ethical standards in data handling is paramount. Annotators must avoid collecting product information that contains any private information.
Ethical Data Usage: All data collection and usage must respect ethical guidelines. This includes avoiding biased or harmful content and ensuring that the datasets promote fairness and inclusivity across diverse cultural contexts.
B.6 Data Contamination Considerations
The risk of data contamination is mitigated by assigning annotators carefully select products and attributes that extend beyond straightforward queries with easily accessible answers. It is essential that retrieval tasks rely on provided images for attribute identification rather than the common knowledge of large language models. This approach is beneficial for creating benchmarks that genuinely test the model's 's ability to comprehend and synthesize information from diverse visual sources across multiple languages and cultural contexts.
B.7 Annotation Platform
We developed a GUT-based annotation platform [3, 14, 93, 94], as illustrated in Fig. 22, specifically engincered to facilitate the data annotation process for human experts. This system enables specialists
39

---

Check the rast
tellee cansiilacc
type
Tick "NO"
Figure 21: Flowchart manual review labeled data.
to efficiently visualize images, while performing annotations and modifications directly within an intuitive interface. The platform's streamlined layout significantly enhances user experience, enabling experts to execute annotation tasks with increased precision and efficiency, thus elevating both the quality and productivity of the annotation process. The primary objective of this tool is to optimize the complex annotation workflow, minimize manual intervention, and substantially improve the overall efficiency of data annotation procedures.
Ever
Fin prodect back tyoc -Fredect to -inago- Experornc Onay Mish Adjustatie tacknot voler choodact cimugas Hand: Guy Aczent Bock, Suliowary Support ofhoddst with hums.
for Untinate Confort seme
Upprosae servee seree 1ww.
Ohe Soctey Dan equulalle tae Preaaryy insilall
bucking bethiigge iddes
Figure 22: Annotation Platform.
40

---

B.8 Data Preparation and Release
For evaluation purposes, we selected 10, 000 queries from total of 320, 000 queries in MERIT as the test set. To ensure equitable representation of each source dataset within the test split and maintain distributional consisteney of language and product categories between the test set and the complete dataset, we implemented stratified sampling methodology:
1. Random sampling of queries to achieve proportional representation of languages and product categories in alignment with the full dataset distribution.
2. Supplementary random sampling of remaining queries from individual source datasets according to their respective volumetric contributions to the complete corpus.
After dividing the dataset into training and test sets, the test set underwent two additional rounds of manual annotation to further enhance quality. Our initial pre-annotation test set consisted of approximately 20, 000 queries, which was subsequently refined 10, 000 queries through careful curation.
C
More Dataset Analysis
C.1
Global Statics
Tab. 4 delineates the taxonomic structure and distribution of product categories within the MERIT dataset. The hierarchical organization comprises seven primary product types (Food, Clothing, Electronics, Bags, Jewelry, Furniture, and Others), further subdivided into specific secondary types enable fine-grained analysis. This comprehensive categorization encompasses essential consumer products ranging from perishables such as fruits and beverages durable items, including electronics and furniture. The diverse product spectrum reflects real-world product retrieval inventory diversity, providing robust foundation for evaluating retrieval models across various product domains. With approximately 160,000 total products, MERIT represents one of the most extensive multimodal semantic retrieval benchmarks in the literature, facilitating meaningful performance assessments across different product categories and their associated attributes.
Tab. 23 and Tab. 24 present the geographical distribution of samples within the MERIT dataset across six Southeast Asian countries. Indonesia contributes the largest portion with X samples, reflecting its significant market presence in the region. The Philippines follows with 569 samples, while Thailand, Malaysia, and Vietnam contribute X, and X samples respectively. Singapore, with its smaller market size but strategic importance, accounts for X samples. This diverse geographical representation ensures that the benchmark captures the linguistic and cultural nuances essential for evaluating retrieval systems in a multilingual Southeast Asian product retrieval context.
Country
Number
Language Number
Indonesia 2281
ID
2281
Philippincs
569
EN
597
Thailand
366
TH
366
Malaysia 284
MS
284
Vietnam 268
VN
268
Singapore 28
Figure 23: Language distribution.
Figure 24: Country distribution.
Fig. 25 illustrates the cross-linguistic patterns in the retrieval process within the MERIT dataset. This Sankey diagram visualizes the flow between source query languages and target product languages, demonstrating the prevalence of both intra-linguistic searches (where source and target languages match) and cross-linguistic retrieval scenarios. The diagram reveals that Indonesian (id) serves as the predominant source language, with significant flows to other Southeast Asian languages, including Thai (th), Vietnamese (vi), and Malay (ms), as well as English (en). The thickness of each connection represents the frequency of language transitions, highlighting the multilingual nature of product retrieval interactions in the region. This visualization underscores the importance of robust cross-lingual retrieval capabilities in real-world applications, particularly in linguistically diverse markets.
41

---

Figure 25: Language Flow Diagram. The flow chart shows the flow conditional products target products.
Fig. 26 presents the geographic distribution of retrieval patterns across six Southeast Asian countries within the MERIT benchmark. The Sankey visualization maps the flow of queries originating from one country products associated with potentially different countries. Indonesia (ID) emerges as the primary source of queries, with substantial connections other regional markets including the Philippines (PH), Thailand (TH), Malaysia (MY), and Vietnam (VN). The diagram illustrates both intra-national retrieval (where source and target countries match) and cross-border retrieval scenarios that reflect the interconnected nature product retrieval in Southeast Asia. The varying widths of the flow connections quantify the frequency of these cross- markettrket interactions, providing valuable insights into regional commerce patterns and highlighting the necessity for retricval systems to effectively operate across national boundaries.
1D
Figure 26: Country Flow Diagram. The flow chart shows the flow of conditional products to target products.
Fig. 27 delineates the inter-category retrieval dynamics within the MERIT dataset through a compre- hensive Sankey diagram. This visualization maps the relationships between query source product classes and retrieved target product classes, revealing both intra-category retrieval (where queries and results belong to the same product category) and cross-category retrieval scenarios. The diagram
42

---

demonstrates how users searching within one product domain (e.g., clothing) may retrieve items not only within that same category but also from related categories (e.g., accessories or footwear). The varying thickness of connecting flows represents the frequency of these category transitions, providing quantitative insights into product category relationships. This visualization is particularly valuable for understanding the complex cross cross-categorice nature of product retrieval search behavior and for developing retrieval systems capable of accommodating diverse user intentions that span multiple product domains.
table
laptop
cloth
fruit
Shoes
pants
gold
phone
backpaek Chair
snack
handbag- headphanrdig
Figure 27: Product Class Type Flow Diagram. The flow chart shows the flow of conditional products target products.
Fig. 28 presents a comprehensive visualization of product distribution across linguistic and geograph- ical dimensions in the MERIT dataset. The left sunburst chart illustrates the distribution by language, with the inner ring representing five languages (id, en, th, ms, vi) and the outer ring depicting the corresponding product categories. Indonesian (id) dominates the linguistic landscape, constituting the largest portion, followed by English (en), Thai (th), Malay (ms), and Vietnamese (vi). The right sun- burst chart demonstrates the geographical distribution, with the inner ring representing six Southeast. Asian countries (ID, PH, TH, MY, VN) and the outer ring showing product categorics within each country. Indonesia (ID) accounts for the most substantial proportion, followed by the Philippines (PH), Thailand (TH), Malaysia (MY), and Vietnam (VN). Across both dimensions, clothing items (particularly pants and clothes) and electronies (predominantly phones) emerge as the most prevalent product categories. This dual visualization effectively captures the multilingual and multicultural nature of the dataset, highlighting the proportional distribution of product types across different languages and markets in Southeast Asia.
C.2 Attribute and Product
Attribute Distribution. Fig. 29 presents a bar chart of the top 30 most frequent attributes within the MERIT dataset. The chart, ordered by descending frequency, highlights the predominant attribute categories such as 'color' and "matrrial, providing a quantitative overview of attribute distribution. This visualization aids in understanding the key characteristics emphasized in the dataset.
Attribute Word Cloud. Fig. 30 displays a word cloud of the top 30 attributes from the MERIT dataset. Larger font sizes indicate higher frequency, with 'color' and 'material' standing out, offering a visual summary of the most common attributes. This representation enhances the perception of attribute prominence within the benchmark.
Value Distribution. Fig. 31 illustrates a bar chart of the top 30 most frequent values associated with attributes in the MERIT dataset. Ordered by frequency, it showcases dominant values like 'gray' and "pink', providing a detailed view of value distribution. This chart supports the analysis of specific attribute values prevalent in the datasct.
43

---

2004
2h04s
DATEA
7H
50
parto
zenb
coth
Figure 28: Sunburst visualization of product distribution in MERIT. The left chart shows the hierarchical relationship between languages (inner ring) and product categories (outer ring), while the right chart illustrates the distribution between countries (inner ring) and product categories (outer ring).
Frequency
materia
styie cloure the spec slac type tes
nattern Toint eccautorr
dosurs prrte wellght
target danigg
Hemtoree number garean
pleevs
releass Por
reactani Neo
auta irsoleelli el
eror fiorapo thtee
contin battery Capa
inigio of oetions
hem shipe
prodice
Cat Burh 1T
grap
upreed aet UPP sorooe
porini Dr openna platform lucation
display
wates
Attributes
Figure 29: A bar chart the top 30 most frequent attributes in MERIT, ordered by frequency, highlighting key attribute categories.
Value Word Cloud. Fig. 32 presents word cloud of the top 30 values from the MERIT dataset. I Larger font sizes reflect higher frequency, with 'gray" and "pink' being prominent, offering a visual insight into the
common attribute values. This visualization facilitates the exploration value diversity within the benchmark.
Product Word Cloud Fig. 33 presents word cloud visualization of the 'query instruction' strings from the MERIT dataset. The word cloud highlights the most frequent terms, with larger font sizes indicating higher frequency, providing clear overview of the key themes and vocabulary used in user queries. This visualization underscores the diversity and focus of query instructions within the benchmark, offering insights into user search behavior.
Title Fig. 34 displays a word cloud representation of the product titles within the MERIT dataset. The visualization emphasizes frequently occurring words with larger font sizes, revealing common
44

---

Attribute Word Cloud
brand target
demographic
style
special features
storage copaesty
S
waist line heel height
pat pattenn/print
mater
closure
display technology
mishelismeet
year
stretch
insole
soveent tpp
1
fit
a
persaieer type
s sho Leve length
type
resolution
closure
connectrvity optiom
type
olor
pr processor
occasion
collar
type
planform beight
pockets capicity
lacation
toe
style
betton type
screen $ize
Sleeve type
outsolo nater ial
A word cloud of the top 30 attributes in MERIT, with larger fonts indicating higher frequency.
Figure 30:
350
300
Frequency
250
200
150
100
de gray
Porty
taless
sid saat der
mid heal C2. 1005
lave heal
Values
Figure 31: A bar chart of the top 30 most frequent values in MERIT, ordered by frequency. highlighting dominant attribute values.
descriptors and product characteristics in product retrieval titles. This word cloud provides a snapshot of the linguistic patterns and descriptive focus of product titles, shedding light on the nature of product representations in the benchmark.
Product Length Fig. 35 illustrates the distribution of character counts in product titles within MERIT. The visualization reveals significant variation in title lengths across the dataset, reflecting the diverse nature of e-commerce product descriptions. For visualization simplicity, titles exceeding 190 characters in length have been consolidated into a single category. This distribution provides insight into the complexity and descriptive depth of product representations within the benchmark.
Query Length Fig. 36 presents the distribution of character counts in query instructions across MERIT. The distribution demonstrates considerable variation in query complexity, reflecting the multi-condition and multilingual nature of the benchmark. For visualization clarity, query instructions exceeding 500 characters have been aggregated into a single category. This analysis provides valuable context regarding the linguistic complexity of retrieval instructions that models must process to accurately identify relevant products.
45

---

Cloud
elegant
waist
open tow
polyester
cotton blend
Z1
ope
teens
up
navy
sports
pr
purple denie
non- slip
formal lace-
sneakers
waise
febric
polo
low
on
r een
n
g
low heel (1-2 inches)
lightweight ght
a
white
creans
striped
EVA
carg go pockets
sporty
bur
gundyndy
b
rra
own
inches)
straight
slim
bluebe
red
ebeer floral
neck
buckle
trendy
l
g
e
elastic
pointed
Figure 32: A word cloud of the top 30 values in MERIT, with larger fonts indicating higher frequency.
image Kemejapants sho
fayanage
erue
of
mater
drgan
deelaal
clssure
tye "phone
image Celana
color
untuk
Keny anwaamaaannccmmm mudi %
featuring
Ukur
Elegant
S Premium um
the
Par
pattern print
wit
Polyes
shoe
uni
que
type
Kaos
Find
Top
fiit
iimge g
dompr
style
fir
Size
Pr
Product
featur
Clöth
knt
Ford
dan
desla
luba
res1gn
1gg
Amage
Supenu
Find
iroo
Women
en
C
chât
liêu
Mu
wut
Limage
ge
0%
Makestnal
Sindal
Fabric
Geal
Pirt
e
product style
Celana Pangind
dengan Bahan
relaxed fit Tera
s stylish
Neccal Teturn S Stretch
collar type
Figure 33: word cloud of the "query instruction' strings in MERIT. Larger font sizes represent higher frequency of terms, illustrating the key themes in user queries.
ME Ber batten
Ukür
Celanu Jeans
an
Fabri
I
Available
Cottoo
Polyes
leeesteer
Casual
Taopiian
deng
Shirt
ann
Sneaker
Men
SLip
Ba
m
Elegant
Ant
Lips
stylee
Cotton
Lace
Usia Tahun
B
Blend
derpaa
untuu
mude prafted
un
cukk
annyaman
gink
lavy
dian
Wyaman
teate
Design
Confort Cocat
n
untut
featur
celana Panjjang
Made!
Fukuran
Forme
ja
pato
tyrish
P
meme
dengan Kerah
ulmsiiman
F
featr
Fitt
Q
Si
Size
bukk Usig
Hyanon Ban Fo
Sandal
Un1 S e ex
Wive
stile
t-
Blouse
PEemAUmm dengan
Figure 34: word cloud of the product titles in MERIT. Larger font sizes indicate higher frequency of terms, highlighting common descriptors in product titles.
46

---

2.0%
270 280 290 310 330 340 350 370 380 390 400 420 430 440 450 470 480 490 500 Moan 100.42
Median 96
1.5%
Percentage (%)
1.0%
0.5%
80 95 Title Length (Characters)
Figure 35: The distribution of the string length of product title in MERIT. Titles with length greater than 190 are categorized as 190 for visualization simplicity.
368.97
Median 385.00 365.00
Percentage (%)
Query Instruction Length (Characters)
Figure 36: The distribution of the string length of query instruction in MERIT. Query instructions with length greater than 500 are categorized as 500 for visualization simplicity.
47

---

D Dataset Examples
Table Table index of case study figures by meta-task with associated emror categories. there are more than 2 conditions, only show the first 2.
Case Figure
Attribute
Attribute
Target Product Type
Target Product Language
Figure 37
Style
Color
Pants
Vietnamese
Figure
Toe Style
Color
Shoes
Thai
Figure 39
Heel Height
Color
Shoes
Thai
Figure 40
Design Style
Size
Handbag
Thai
Figure 41
Brand
Fashion Style
Backpack
English
Figure
Insole Material
Color
Shoes
Indonesian
Figure 43
Material
Color
Pants
Vietnamese
Figure 44
Tightness
Pattern
Pants
Thai
Figure 45
Movement Type
Color
Chair
English
Figure
Color
Pants
Thai
Figure 47
Closure
Color
Shoes
Thai
Figure 48
Color
Material
Cloth
English
Figure 49
Color
Material
Pants
English
Figure
Brand
Color
Hcadphone
English
Figure
Wheel Type
Color
Suitcase
Thai
Figure 52
Camera
Color
Phone
English
Figure 53
Neck shape
Color
Cloth
Indonesian
Figure 54
Material
Patter
Pants
Thai
Figure
Collar type
Color
Cluth
Thai
Figure
Embellishment
Sleeve lype
Cloth
Thai
Figure 57
Brand
Storage capacity
Phone
Thai
Figure
Material
Closure type
Shoes
English
Figure
Occasion
Material
Pants
English
Figure
Lace style
Color
Shoes
Thai
Figure 61
Brand
Color
Headphones
Thai
Figure 62
Pockets
Color
Pants
English
Figure
Capacity
Brand
Suitcase
English
Figure
Color
Shoe type
Shoes
English
Figure 65
Shoe type
Closure type
Shoes
English
Query Text:
Find product of that have the product style with Product and the color with Product with visible brand logo.
Quinn Carga mau Unisex thit bitu fhryng tá bling cottinn
Quinn läng Corgo Unisex cá théu bišu
Outdoor Jacket with Febric and Mal Martiple Pockets
furong viil cottonn
khoon chun mēu
Condition
Condition
Ground Truth
Figure 37: A sample case. Back to List of Figures.
48

---

Query Find a product of shoes that have the some toe style with Product and the some color with Product 2with a glossy finish.
POSÉE Lightweight Open Slippers for Couples, Suitable for All Seasons
Condition
Conditiion
Ground Truth
Figure 38: sample case. Back to List Figures.
Query
Find of height Product
Product metallic finish.
Apple iPhone Silver 6468, 4G8 RAM, 5.8" Super Retino OLED Display, Duel 12MP iOS, Lightning Connectivity.
recohdugeenisu Wwszualnnfyurr
Condition
Ground Truth
Figure 39: A sample case. Back to List Figures.
Query Text:
Find product of handbag that the design style with Product and the color with Product with denim texture.
EELATO WITH DENIN
meedhhhhmmmne
iPad 13 inci, dilengkkppi dengan chip A3
mulul minu
iPad
Condition
Condition 2
Ground
Figure 40: sample case. Back to List of Figures.
Query
Find product of backpack that have the some brand with and the some fashion style with quitted texture and choin strop.
M
3d
Elegent
Polyester dengen
Ransel MOS3DOOM
Elegont Pirk Flutt with
Handibg
Piek Lesther
Beser; Ukuran 30x12x3B
Kampuher dan Penyimpuran
Clesure for Stylish
Strap and Lange
0.32kg
Party
Capectty,
Size 18v9 5x35 em
Condition
Ground Truth
Figure 41: A sample case. Back to List of Figures.
Query Text:
Find product shoes that the
with Product and
with Product minimolist design.
Sondal Woriit
Premium French Temy
Flots MINNIK
dengon Material Kulf Sintetts don 1PR, Insoie EVA, Tieggi 4 cm.
with Deep Shourder Design, from 80% Cotton Blend for Rich
dengen Ujung
Runcing, Matariu PU
Alami dan Insole
Condition
Condition 2
Figure 42: A sample case. Back to List of Figures.
49

---

Query
Find a product of parts that have the some material with Product and the some color with Product 2 with a subtie logo on the side.
deerrwyrrnhhh unmoaaauu Turget
Elegant Pink Syethetic Party flltts with Buckie closure and Stylish Hoel
Quin short ni köng möm moj fleace, of a kich or S/ML chitu cae càn nong kide
"umumnu rinls
Condition
Condition
Ground Truth
Figure 43: A sample case. Back to List Figures.
Query
Find product of ponts that have the some tightress with Product and the some pattern with Product 2.
arrch
Qjáin ling ring than gān
coo cáp vil hoa hite hiws
maunamnnu Argaamaurr uncensflavllne
muunnednga иeиырее aтмbiиe Dh
phù hop to 40kg din 70kg. click tre firing và lich cha phói dep.
Condition
Ground Truth
Figure 44: A sample case. Back to List of Figures.
Query Text:
Find product of choir
Type Product the
WProduct a plush Texture.
y if Tr halueeuu ln puchh sura 1xx110 mpi. daedn Thauehlllrrggh tuursh (u tustun
Rainstyle Shirt Entoreideeed Lien and Fode Resistort Material
luarious Japanese velvet adgustable
metel frome, ideal for support.
Condition
Condition 2
Figure 45: A sample case. Back to List Figures.
Query Text:
Find a product of pants that have the some fit with Product and the some color with Product 2, featuring distinct brand lobel.
maunsfuuull -uwaa gansosmuiww.wani Waznre al al wrnu urogn 25 il unoylodo 38 Dn ih mauua 10.5 On
neee meefullaamaaa wnulla Aulginen @uängumin 379 lu wazi Iela dnwii
muunssurhh Sinee fowsenfuuddwaarit Manwemong 25 ynoo monuerssitéi Dn
5-3XL
Condition
Condition 2
Ground Truth
Figure 46: sample case. Back to List Figures.
Query
Find a product of shoes that have the some closure type with Product and the some color with Product with controsting sole.
roauThunsshudle oe Tnunze Fenffuiii
maunssrfrunelll andnfouresddment in
maashunesaan Mussuu/aggrs woousaee/idlee
Condition
Condition 2
Ground Truth
Figure 47: A sample case. Back to List of Figures.
50

---

Query Text:
Find product of cloth that the color Product and material Product with button-up design.
stylish Reakkculture
Backpack with Brecthable, le.
Ae thun unisex hinh déo litu 100% mžm mul.
crofted from vintage high coller design.
Wear Resistont Design and 20-35L Cypocity.
Condition
Condition
Ground Truth
Figure 48: sample case. Back to List Figures.
Query Text:
Find a product of pants that have the some color with Product and the some material with Product 2, featuring tailored fit and sleek design.
naz nerfulagneeuenn and an alaeouuns unaga duuuwaaaee Aau se inn uu uunerllee
Chân viy xoé dii 93mm polyester wii dai hes, thii ki hijn phù tile.
APTIN
elugent
polyester anide langth
ponts
zipper closure for men.
Condition
Condition
Ground Truth
Figure 49: A sample case. Back to List Figures.
Query Text:
Find a product of headphone that have some brond with Product the some color with Product with matching cose.
2501 Blaetsoth
Anker R501 TWS Earphones Blustooth: 22 EQ Options, Battery Life, Fost USB Changing, Microphones, Guriigg Mods, IPX5 Waterprsof, Control Pirk.
Earphones with Gamiin Mode, IPX5 Wiseeppooff and 22 Customizoble Equilizer Options, Festuring USB-C Changing and bual Microphanesl Clear Calls
Promium Moccosin
Lauther Open Sandals with Soft
Non-Slip Grip Outsole
Condition
Condittion 2
Ground Truth
Figure 50: A sample case. Back to List of Figures.
Query Text:
Find a product of suitcase that has the some wheel type in Product and the same color as in Product 2 with larger size and
sleek handle design.
masthhuuumnnn for Riw Fan simunn un weeieerin
nruuthuuurn [anesihamu hiju
Ling Cas Khod Sai Vii Polyester 6ün
11/9 29 #e19 weeudeedyaa 360
360 94 garnuacAudnn
eq equiMennluureiiu
norhuazWuuin eqiidaann
Condition
Condition 2
Ground Truth
Figure 51: A sample case. Back to List of Figures.
Query Text:
Find product of phone that has the
Product the color Product with sleeker design.
6 I-inch Super Retina XDR Disproy. Al6 Blonic Chio, buel Comera, and Offiad Apple Warranty.
Dép nom nguyān ledi siiu chiing mčn, dding the trung năng dongg chift cip,
iPhone A16 Bronic Chip. Dual Comaro, and Officia
Apple Wancenty
Condition
Condition 2
Ground
Figure 52: A sample case. Back to List of Figures.
51

---

Query Text:
Find product of cloth that neck shope Product the color Product detail.
Coneol
Komejo Rubai By- ió Katun Poplin Slin-Fit desger Destin Serut Alsen Rito Kerch, Sizee Lb 82-92 om.
Komeja hagoo Kanun Poplin dengon Deson
Tól xöch do binng niu whi city dee xich sang trong.
Korah Berhios, Ukurrn All Size Lb 82-92
loyi
Condition
Condition 2
Ground Truth
Figure 53: A sample case, Back to List of Figures.
Query
Find a product of ponts that have the some material with Product and the same pattern/print with Product 2.
yn unuuunn mensäry Sossrron javggusi/uunans drwi anlainlaanee
desunannnama
Condition
Ground Truth
Figure 54: A sample case. Back to List of Figures.
Query Find product of cloth that has the some collor type as in Product 1 and the some color as in Product with intricate sleeve detailing..
Ào Than Botry Form Vas 23.3 cotton chitu 230gon bén, giōn, thier tre trung. ding, phii
douuthiuulll Yi
(amis andan sueei horiuoooiiiiii owFepegn gelegngnnl mnunia
Condittin
Condition 2
Ground Truth
Figure 55: A sample case. Back to List Figures.
Query Text:
Find a product cleth that some embellishment as Product the some sleeve type as Product with a light color and loyered design.
@ewwnullin unnu rouneeeech Anusig Surfsinn
Diarro Gardigan Raju!
dengon Hizzon Fite, Ukuran
stylish red bank with ruffled sleaves. designed to butt aizes
Porjjong Lingkar
Dade 110cm, Cocok untuk Buckan 40-65ing Tingg 140-160cm.
tim aalssas "alFingu
Condition
Ground Truth
Figure 56: A sample case. Back to List of Figures.
Query
Find product of phone that has the same brand as in Product 1 and the some storage capacity as in Product 2 with a vibrant color finish.
sundhiiwu Teomo Spork #tmo,, 1868, #rruya 51268, nans sadia 108Me 3iidin
Tranfuní Tecno Sprak
10:00
with 868 RAM, 25668 Gespdragon. 685, 5.77 AMOLEb 120Hz Display, and Triple
DO
1208. #71314
An1n. RAM
256GB néee
108MP. 7u
Smapdroggn 8 Sen 3, U1.A 6000mAh
P5,699
Snapdraagn 8
Condition
Ground Truth
Figure 57: sample case. Back to List of Figures.
52

---

Query
Find product of shoes that have the some material with Product I ond the some closure with Product 2 featuring prominent logo and reinforced toe design.
Unisex Our Work Microf Leatter, Zipper Clesure, Sip Rubber Sale
gouthdnluaiodee wamamm war wwmmwweuean u
Mid-Cut Microfiber
Lesther Work Bocts
and Kevlor Midsole for
Condition
Condition 2
Ground Truth
Figure 58: A sample case. Back to List of Figures.
Query Text:
Find product of pants that have the some occasion with Product and the material with Product neutral color
Rek Duyinn Fornal Slin derrgan Knit Premium dan Karet
Dong
Elegont
Sufing Soe Phong Clch Chilt Polyester Cos Cip to Thuong SARA.
Marnaid skirt by Rolizshaphoune, crafted from premium stretch polyester
Pinggong Belakang Relizshophouse
Condition
Condition
Ground Truth
Figure 59: A sample case. Back to List Figures.
Query Text:
Find product of shoes that have the some lace style with Product 1, the some color with Product and same heel color with Product 3.
Gidy Mary Jone màu tuoi LilyWei luon sóng, gidy don môt quai cip ngot igão vå
wenaudenanun
Zeke Polo Shirt Embroidered Logo Brown
Drawstrigg anviu
gneuuse Awa
Condition
Condition
Condition 3
Ground Truth
Figure 60: A sample case. Back to List of Figures.
Query Text:
Find product of headphones that have
brand Product and the color Product with transparent casing design.
Ear ug Yo 5.3 Anu suncu Tavi dia 'H' mnggagg 40 yu. fuunhffuuuu 1P54 Huudeesaiu....
A transparant silicone with cartoon design, offering duroble protection and eaaly acceds headphones
NNA TaL ugns dirueeerrun whes/luf #o 18
emagano 40 nui Fuudirfuuu 1954
Condition
Condition
Ground Truth
Figure 61: A sample case. Back to List of Figures.
53

---

Query
Find product of parts that have the some pockets with Product I and the some color with Product with dlightly darker shode.
High-Woist Khoki Cargo Pants and Coffor: SCUBA Blend
Kons Unisex dergon Sablon DT Tayan dan Bahan Cotton Combed 30s Premium
LF RAINA High Woist Bierd
Ponts with Elastic
Closure Straight Design
BUACI
Condition
Ground Truth
Figure 62: A sample case. Back to List Figures.
Query Text:
Find product of suitcase that has the same capacity as Product1, same brand as Product 2 and the same color as Product with sleek, minimalist design.
PIAN
Metaverse Koper Original dengon Material Polycarbenate don Aluminium Frame, Trally Aluminium Stainless, TSA Combination Lock, serto Garansi Pecah Tahun dan Sporepart Tahun
Vali 8812 Ban Thuvnng 24
Koper Metaverse
Original dengon Material
Inch Chät Liëu Nhua ABS+PC, Khán S6 Bão Mät, Bánh Coo Su Non 360 bi.
85cm ABS-PC Checked Luggage 24 inch Lightweight Suitcase
Polycarbonate
Aluminium Frame, inci.
Tay Kéo Hop Nhôm, Ngän Chira Rōng Räi.
Condition
Condition
Condition 3
Ground Truth
Figure 63: A sample case. Back to List of Figures.
Query Text:
Find product of shoes that have the color with Product I the shoe with Product featuring decorative elements.
A charming adonned with
lesther baots featuring
RAANMARR X5-6XL
zioper clezure for effortless
rhinestones fluffy box accent.
Condition
Condition
Figure 64: sample case. Back to List of Figures.
Query Text:
Find product
type
type
design and low heel.
resuiranis & dodionio nseuuu gurnnnnz e Bnwuunngema
IMBI
Giby oso gff LoviEoermer thei vuing, mbn, da PVC chinn tron trest, 9on, phong eách iitn del wii kehia odi.
Clossic Leather
Lesfers with Square
Buckle
Clssaee Festuring
Hedl (3:2
Condition
Condition 2
Ground Truth
Figure 65: A sample case. Back to List of Figures.
54

---

E Experiments Details
E.1 Baselines
For most MLLM-based models, we adhered to the standard evaluation protocol delineated in their respective original configurations. In instances where such protocols were not specified, we followed the established conventions outlined in VLMEvalKit [15, 69], consistently setting the temperature parameter to O. Specifically, we categorized the evaluated models into two distinct types: Zero- Shot MILMs, which have undergone training on dedicated retrieval datasets, and Embedding MLLMs, which have been specifically trained on existing retrieval datasets through particular methodologies for retrieval purposes.
Zero-Shot MLLM:
InternVL2.5-VL. [12] is an advanced multimodal large language model (MLLM) series that builds upon Intern 2.0, maintaining its core model architecture while introducing significant enhance- ments in training and testing strategies as well as data quality. Through extensive evaluations on wide range of benchmarks, including multi-discipline reasoning, document understanding, interleaved multi-image understanding, real-world comprehension. multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL. 2.5 exhibits competitive performance, rivaling leading commercial models such GPT-4o and Sluude3.5--onnge Notably, Intern 2.5 is the first open-source MLLM to achieve over 70% on the MMMU benchmark. For experiments, we utilized the 1B variant, as well as the version enhanced through MPO training. Both models were evaluated with the maximum number of tiles constrained to one.
Qwen2.5-VL. [5]. Qwen2.5-VV LL represents the latest flagship model in the Qwen vision-language series, achieving significant advancements in foundational multimodal capabilities including enhanced visual recognition, precise object localization via bounding boxes, robust document parsing, and long-video comprehension with second-level event localization. The model's innovative architecture incorporates dynamic resolution processing and absolute time encoding, enabling it to natively perceive spatial scales and temporal dynamics without traditional normalization techniques, while its native dynamic-resolution Vision Transformer with Window Attention [57] maintains resolution integrity with reduced computational overhead. For our experiments, we employed the 3B parameter variant of this model, configuring the max pox_li parameter to 576 x 28 28.
Embedding MLLM:
E5-V [38]. ES-V adapts Multimodal Large Language Models (MLLMs) to generate universal multi- modal embeddings, effectively bridging the modality gap between different input types. Employing an innovative single-modality training approach using exclusively text pairs, ES-V demonstrates superior performance compared to traditional multimodal training methods while reducing training costs by approximately 95% and eliminating the need for expensive multimodal data collection. Following the original paper, we obtain the global token through the prompt 'Summary above image in one word:
LLaVE [41]. LLaVE represents groundbreaking framework for universal multimodal embeddings that effectively addresses the challenge of distinguishing hard negative pairs in image-text retrieval tasks through dynamic representation learning based on discriminative difficulty. We evaluated three model sizes: 0.5B, 2B, and 7B parameter versions.
GME-Qwen2VL. [102]. General Multimodal Embedder (GME) functions as an ML..M-based dense retriever capable of processing queries and candidates across text, images, or multimodal combinations. Developed using novel training data synthesis pipeline that addresses modality imbalance issues, GME overcomes the limitations of previous approaches that relied solely on text data for training. In our experiments, we utilized the GME implementation based on Qwen2VL-2B.
LamRA-Qwen2.5VL. [56]. LamRA is a versatile framework that repurposes MLLMs for comprehen- sive retrieval tasks, eliminating the need for task-specific fine-tuning, Employing a two-stage training pre-training followed by multimodal instruction tuning-ann joint training for both pointwise and listwise reranking, LamRA A demonstrates exceptional performance across more than ten retrieval tasks in both supervised and zero-shot settings. Our evaluation utilized the LamRA implementation based on Qwen2.5VL-7B.
mettoodolggy-tnnuugee-o-
55

---

BGE-VL [104]. We evaluated the BGE-VL-MLLM-SI model, which was obtained from https: //huggingface Co/BAAI/BGE. vase BGE-VL-MLLLE is trained exclusively on the Mega- Pairs dataset, achieving outstanding performance in composed image retrieval tasks.
VLM2Vec [39]. VeM2 is a novel multimodal embedding framework designed encode se- quences of images and text into a unified representation space for diverse downstream applications. Unlike conventional CLIP or BLIP embeddings that operate under constraints of fixed image res- olutions and text lengths, VeM2 accommodates inputs of arbitrary dimensions and sequence lengths, significantly enhancing its versatility across multimodal tasks. Our experiments utilized the 4B parameter version of VLM2
E.2 Other Dataset Usage
In this section, we provide a detailed introduction to the additional datasets utilized in our paper. To validate the effectiveness of our proposed method, CORAL, we conducted comprehensive evaluations on several widely-used retrieval benchmarks.
This dataset presents interactive visual dialogues generated through a controlled collaboration between two annotators on Amazon Mechanical Turk. In this conversational framework, one participant assumes the role of the "questioner," with access only to a textual caption of an image, while the other serves as the "answerer," with complete visual access to the image itself. They engage in structured 10-round question-and and-ssswe exchanges regarding image content. We repurpose this dialogically rich dataset as cross-modal retrieval task, where the objective to identify and retrieve the precise image that corresponds to the given conversational dialogue, thereby testing models' abilities to construct visual representations from textual discourse.
VisDial [16]:
CIRR [59]: This dataset is meticulously designed for composed image retrieval tasks, focusing on natural language modifications of visual content. It comprises pairs of real-world reference and target images, accompanied by linguistically nuanced modification descriptions that articulate the transformative differences between the source and target images. This configuration presents particularly challenging evaluation scenario that requires models to understand both visual foundations and linguistic modifications in a compositional manner.
VisualNews [53]: This corpus encompasses a substantial collection of publicly available news images paired with professionally written captions. The dataset features content from major news organizations and represents a diverse range of visual journalism across various domains, providing a real-world test of multimodal understanding in the context of current events and factual reporting. The caption-image pairs exhibit complex relationships that often require world knowledge and contextual understanding of news content.
MSCOCO [52]: This comprehensive benchmark dataset features over 330,000 images, each meticu- lously annotated with multiple human- generated captions. Originally designed for object detection, segmentation, and captioning tasks, MSCOCO has become a fundamental standard for evaluating multimodal capabilities. Its diversity spans 91 object categories captured in everyday contexts with multiple objects per image. The dataset's 'ss rich annotations facilitate cross-modal retrieval in both text- to-image and image-to-ter directions, providing robust assessment of bidirectional understanding between visual and linguistic modalities.
This dataset introduces perceptually calibrated human similarity judgments on image pairs that exhibit diverse forms of visual correspondences. The corpus consists of carefully constructed triplets: reference image and two systematically perturbed variations, accompanied by human perceptual judgments indicating which variation maintains greater similarity to the reference. Following the methodology established in M-BEIR [84], reconfigure this dataset into retrieval framework for image-to-imet to-ge-tt-image matching, where the reference image functions as the query, and the perturbed version that aligns with human perceptual judgment serves as the target. This transformation provides a rigorous test of a model's ability to replicate human perceptual similarity assessments.
NIGHTS [22]:
WebQA [7]:
This dataset presents a multihop, multimodal necessitates the retrieval and integration of information from Wikipedia pages to formulate responses to given queries. The dataset's complexity emerges from its requirement for models to navigate both textual and visual information across multiple reasoning steps. In our experimental context, we utilize the Wikipedia page's images and accompanying textual descriptions as candidate elements
framework that
56

---

Table 6: Comparision with other method across retrieval tasks.
EN
ID
TH
VN
MS
Qwen2.5-VL [5]
48.73
56.23
55.34
55.13
47.98
InternVL2.5-V4 [12]
55.38
60.75
58.97
62.82
52.47
BGE-VL VL [104]
13.76
14.02
14.83
20.51
14.08
for retrieval, thereby evaluating models' capacities identify relevant multimodal content based on query specifications.
E.3 Main Experiments Settings
All experiments were conducted on computing node equipped with 8xH100 GPUs.
Experiments were conducted for a single epoch with the following training configuration: A per- employed with gradient accumulation steps to 2, resulting in effective
device batch size of global batch size 64. The InfoNCE contrastive loss temperature parameter (r) was fixed at 0.02. For negative sampling, implemented in-batch negatives combined with cross-device negative 63.
sample gathering, achieving a final positive- Desitive-to-negatie to-negative ratio
For full- parameeere fine-tuning, adopted learning rate of le 5 with weight decay of 0.0005 and linear warmup ratio 0.01. The LoRA [28] configuration employed the following parameters: learning rate of le 4 (10 times higher than full fine-tuning). identical weight decay (0.0005) and warmup ratio (0.01), with LoRA-specific hyperparameters set to r = 8, a = 16, no bias terms, and a dropout rate of 0.05 between LoRA layers.
The CORAL framework was configured with the following hyperparameters: the loss weighting coefficients X andA2 were both set to 0.1 to balance the objective components, while maintaining uniform masking probabilities of 0.5 for both visual and linguistic modalities. This symmetric configuration ensures equal contribution from both vision and language streams during the masked reconstruction tasks.
Across all training regimes, we kept the vision tower completely frozen to preserve its pretrained representations. For LoRA-based adaptation. we specifically applied low-rank adaptation only the LLM backbone components, while maintaining standard full-paramete training for all BERT decoder layers. This hybrid approach allowed us efficiently adapt the language model while preserving the decoder's s complete expressive capacity.
E.4 More Experiments Results
Different Languages' Performance In Section 3.4, we present the accuracy distribution across different languages, with specific values shown in Table 6. We define a query as belonging to a particular language only when all products both the query statement and the positive samples are in that language.
Out of Distribution Scenarios. As mentioned in Sec. 3.3, tested several -ut-of-distributtin (OOD) scenarios. Tables 8, 7, and 9 correspond to the Zero-shot, OOD, and Mixed results depicted in Figure 6(b), respectively. Zero-shot refers direct inference using Qwen2.5-VL's [EOS] token. OOD indicates that for each row in Table 7, we excluded the specified OOD data (e.g., for the first row concerning Language ID OOD, we removed all queries containing the ID language from the training set, while for the test set, we only selected data where all languages were ID). Mixed refers training conducted the complete training dataset.
Results on 8 Retrieval Tasks, To further validate the efficacy of CORAL, we conducted evaluations on several retrieval datasets. The experimental results presented in Tab. 10, with experimental configurations following the methodology described in [39].
F.5 Error Analysis Details
In this section, we present a case study analysis of the error types made by Qwen2.5-VL--3 [5] and GMFF-Qwen2VL-228 [102] across various tasks. The errors are classified into the following five categories. Other less frequent error types are not included in this analysis. For the analysis, as
57

---

Table 7: Out-of-distribution test results for Qwen2.5-VL trained on in-domain data. AVG represents the average value, and indicates the total number of entries in the test set.
MRR
R@5
R 10
513
44.26
15.98
79.34
87.91
MS
434
42.82
17.97
74.65
84.33
Language
TH
1020
45.00
19.80
75.29
84.22
AVG
1967
44.03
17.92
76.43
85.49
Brand
889
60.72
48.03
78.07
85.60
Pattern
1002
54.93
37.23
77.74
84.53
Attribute
Region
89
44.29
19.10
78.65
86.52
AVG
1980
53.31
34.79
78.15
85.55
Drink
475
51.14
33.47
73.47
81.05
Phone
1958
43.47
31.77
58.63
67.72
Class
Table
422
48.57
35.07
64.69
74.64
AVG
2855
47.73
33.44
65.60
74.47
Table Out-of-distribution test results for zero-shot Qwen2.5-VL. AVG represents the average value, and # indicates the total number of entries in the test set.
#
MRR
R @
R@5
R@ 10
ID
513
4.28
2.34
7.02
9.55
MS
434
6.47
3.92
9.22
12.44
Language
TH
1020
2.67
0.69
5.20
7.94
AVG
1967
4.47
2.32
7.15
9.98
Brand
889
5.46
2.81
8.44
13.05
Pattern
1002
0.95
0.50
1.50
2.40
Attribute
Region
89
7.46
4.49
12.36
14.61
AVG
1980
4.62
2.60
7.43
10.02
Drink
475
6.68
3.58
10.95
13.89
Phone
1958
2.18
1.02
3.47
5.72
Class
Table
422
2.82
1.18
4.98
7.35
AVG
2855
3.89
1.93
6.47
8.99
mentioned Sec. 3.4, we selected 500 samples for each model, but due to space limitations, we present only some of them here, as shown in the following Figures.
Attribute Error: Retrieval models often misidentify incorrectly select attributes in product retrieval tasks, resulting in recommendations that fail to meet the specified attribute criteria. As illustrated in Fig. 66, while the recalled product matches the pattern Product 1, its color does not align with the requirement of Product 2 (which shares the same condition). This discrepancy highlights the model's misinterpretation of attribute- based constraints.
Visual Understanding Error: Retrieval models accurately identify the requested attributes but fail generate corresponding and consistent visual outputs. In such cases, while the model correctly interprets the textual instructions, struggles to align them with appropriate visual representations. As illustrated in Fig. 67, although the language command is comprehended correctly, the retrieved product image is incorrect [82].
Category Error: Retrieval models retrieve products from incorrect categories, indicating failure to accurately classify items within the prescribed product taxonomy. As illustrated in Fig. 68, the model is tasked with identifying mobile phone bag but instead retrieves generic bag. demonstrating misclassification.
Detail Error: Retrieval models accurately identify the primary product conditions but often overlook specific secondary requirements or finer details. Consequently, their recommendation satisfy the main criteria but fail to capture critical nuances specified in the query. As illustrated in Fig. 69,
58

---

Table 9: Out-of-distribution test results for Qwen2.5-VL. trained on the whole train queries. AVG represents the average value, and # indicates the total number of entries in the test set.
MRR
R@5
R @@ 10
ID
513
71.34
57.31
88.89
94.93
MS
434
67.51
53.23
85.48
91.94
Language
TH
1020
69.79
55.29
87.65
91.57
AVG
1967
69.55
55.28
87.34
92.81
Brand
889
71.87
59.62
88.98
93.59
Pattern
1002
63.50
50.30
80.44
87.03
Attribute
Region
89
57.81
38.20
83.15
93,26
AVG
1980
64.39
49.37
84.19
91.29
Drink
475
53.54
33.26
78.53
86.74
Phone
1958
63.64
51.58
79.78
86.57
Class
Table
422
52.40
38.63
69.19
77.96
AVG
2855
56.53
41.16
75.83
83.76
Table 10: Comparision with other method across retrieval tasks.
Model
VisDial
CIRR
VisualNewsrar
VisualNews,07
COCOrs:
pm
NIGHTS
CLJIP
30.70 70
12.60
78.90
79.60
59.50
57.70
60.40
67.50
OpenCLIP
25.40
15.40
74.00
78.00
63.60
62.10
66.10
62.10
SigLIP
21.50
15.10
51.00
52.40
58.30
55.00
62.90
58.10
BLIP2
18.00
9.80
48.10
13.50
53.70
20.30
56.50
55.40
MagicLens
24.80
39.10
50.70
21.10
54.10
40.00
58.10
43,00
E5-V
9.20
6.10
13.50
8.10
20.70
14.00
17.70
WMFF-Qee2-VIi V1-222
26.00
38.00
66.00
71.00
62.00
56.00
64.00
$3.00
Qwen2
Twen2-VL-2B--o
51.00
39.00
$6.00
$2.00
$6.00
45.00
$8.00
67.00
Qwen2 CO-RAAL
72.00
Query Text:
Find T-shirt that has the same text pattern as Product and the same color as Product 2.
EATERAL
viconiiie
ei cilfe
RAKERAL adi-tii
La vou Riveeeeneeiee a dunuiiuaiaa Roinday wauch saadhhhhinnnunnn wa
Baseball- Themed Burgundy T-Shirt "Baseball is My Favorite Season"
Beseball-Themed Novy T-
Baseball Favorite Season Grophic T-Shirt
Shirt with 'Baseball My Favorite Season' Graphic
Condition
Condition 2
Error Recall
Ground Truth
Figure 66: A case for Attribute Error. In this example, the same condition as product is color. The recalled product meets the pattern of product 1, but the color does meet the requirements of product 2. This is misunderstanding of the attribute content.
although the retrieved product fulfills the two primary conditions, it does not meet the detailed requirement of "having visibly displayed brand logo."
Annotation Error: Inaccuracies in dataset annotations may result in cases where the model's response appears incorrect when evaluated against imprecise ground truth. Such errors are rare, as our data undergoes multiple rounds of manual review. However, given the vast scale of the candidate set, we cannot entirely rule out the possibility of a small number of positive samples remaining unlabeled.
59

---

Query Text:
Find T-shirt that has the same material Product and Product
Men's 's Black Graphic Crew Neck Short Sleeve Polyester
Åo thun trón thuöng ngdy dônh cho båing vái Polyester ching nhōn
Casual Hooded Pullover with Geometric Wing Grophic, 100% Polyester, Long Sleeve, Slight Stretch
Aadaduug
Tee Wrinkle Resistont, Fast Drying, Ultre-Durable Casual Pullover
Condition
Condition 2
Error Recall
Ground Truth
Figure 67: case for Visual Understanding Error. In this example, the language instruction was understood correctly, but the visual image of the recalled product wrong.
Query Text:
Find phone bag that has the same material as Product and the same color as Product
J
unnud drudi iPhone
Brown Polo T-Shirt with Classic Collar,
Business PU iPhone
X/XS ifforraattoiii RFID
Backpack with Water- Resistont Polyester Cotton Lining, 28x40x12 for Work and Events
X/XS 5.8 t wherruaaeer'u
de-lidk arlee noddhr un sestrerddifffemmmma.....
Available Sizes XS ZXL
RFID de/ldlir mesiivl
Lacdvendherrrnun onald!
Condition
Condition
Error Recall
Ground Truth
Figure 68: A case for Category Error. this example, the goal is to find mobile phone bag, but bag is recalled, so it is wrong.
Query Text:
Find product pants that have same product style with Product the same color with Product with visible brand logo.
Quán ling Cargo máu xanh Unisex oó thêu biéu turing vá túi sou bäng väi cotton Whoan
Credifox Olive Hoaded Outdoor Jacket with Taslon Zn Waterproof Fabric and Multiple Pockets
Quân lirng Cargo xanh cñy Unisex eá théu bilf turyng và tüi sau bång väi cotton khoan
Quán long Cargo Unisex thēu biéu tuyin, väi cotton khoan và cap chun màu 8 liu
Condition
Condition 2
Error Recall
Ground Truth
Figure 69; A case for Detail Error. In this example, although the recalled product meets both conditions, the detail condition "with a visible brand logo" not met.
60

---

E.6 Human preference
We have supplemented our evaluation with human preference experiments. Our human preference study is conducted in multiple-chocee comparison format. Specifically, we randomly sampled 100 items from MERIT, with each model providing their top-1 (Choice@) 1) or top-3 (Choice 3) most similar items as options. Five annotators were asked to select the best option (when the best item was retrieved by multiple models, we credited the model with the highest similarity score during recall). We report the average selection rate. As shown in Table 11, CORAL demonstrates superior performance the human preference study, particularly achieving 58. 9% selection rate in the Choice @ 3 setting.
Table 11: Human preference study on MERIT
GME-
LamRA-
BGE-
CORAL-
Qwen2VL-2B
Qwen2.5VL-7B
VL-7B
Qwen2.5VL-3B
Choice @
15.0
18.0
13.4
53.6
11.7
14.4
14.9
58.9
E.7 Condition Count MERIT
As demonstrated Table 12, we observe that the performance nearly all models degrades with increasing condition count. Notably, under the most complex scenario with conditions, even the best-performing model achieves only 36.84 R@5, highlighting the inherent difficulty posed by the combinatorial complexity of multi-condition queries.
Table 12: Model Performance Comparison
Model
#Condition
R@ 11
R@5t
R@ 10
MRR
2
0.42
1.39
2.30
0.88
Intem
0.00
0.00
0.78
0.10
4
0.00
0.00
0.00
0.00
2
0.09
0.39
0.55
0.20
Qwen2.5-VL-3B-Ser
3
0.00
0.78
1.56
0.37
0.00
0.00
0.00
0.00
2
8.45
53.12
61.24
27.73
GME-Qwen2VL-2B-CGr
3
10.24
55.91
59.06
28.51
5.26
36.84
47.37
15.89
2
12.05
39.20
48.13
23,83
LamRA-QQwen25V1--7B-cca
3
13.39
35.43
42,52
22.58
4
5.56
27.78
33.33
15.28
2
0.44
1.87
2.98
1.04
VLMSVVc-4B-SSq
0.00
1.57
2.36
0.70
4
0.00
0.00
0.00
0.00
61

---

Broader Impact
F.1
Impact
The broader impact of MERIT carries both potential benefits and risks upon deployment and release. Some considerations are unique due the multimodal nature our dataset, while others reflect challenges common to retrieval systems in product retrieval environments. Built upon multilingual semantic understanding across Southeast Asian markets, MERIT inherits issues associated with cross-cultural product retrieval and multi-condition query interpretation. Below, we outline risks and mitigation strategies for its release.
Hallucination. On one hand, since the product titles in our dataset were generated by GPT-4o [10], there exists potential for hallucination issues [36, 96]. On the other hand, similar to other retrieval datasets [49, 103, 75], models trained on MERIT may generate outputs that are disconnected from user intentions or input conditions. This raises concerns, particularly product retrieval applications where purchase decisions depend on accurate results, as user requirements and their modes of expression are inherently variable.
Biases. Bias in training data can propagate to models employing MERIT, arising from both visual feature extraction and linguistic interpretation. This may result in biased retrieval outcomes unfair representations across diverse cultural contexts. Additionally, multilingual processing can introduce further biases in language alignment, noted by [60].
Ethical Impacts. This research does not present substantial ethical concerns. Furthermore, we affirm that open-source data and model distribution comply with all corporate guidelines and industry regulations governing intellectual property and data sharing practices.
Expected Societal Implications. A significant societal benefit lies in enhancing cross-cultural product retrieval experiences through improved interleaved multi-condition semantic retrieval. However, challenges remain in ensuring fairness across linguistic and cultural boundaries. Strong ethical standards and ongoing evaluation are essential for maximizing positive impact. These issues aren't unique to our method but are prevalent across different techniques for multi-condition retrieval. Despite the challenges, we believe the benefits significantly outweigh the potential limitations, allowing ongoing investigation and improvement of retrieval models while engaging the community in developing better approaches [102, 84, 62, 99]. Moreover, the release of MERIT can foster new applications and research directions, contributing to the progress and responsible deployment of retrieval systems in multilingual product retrieval environments.
F.2 Limitations
(i) Our dataset, while comprehensive, may inherit limitations from real-world product retrieval data, such as imbalances product categories and potential biases in attribute distributions across different Southeast Asian markets. (ii) Despite rigorous filtering, the dataset might inevitably contain some inconsistencies between visual attributes and textual descriptions, which could adversely affect model training and evaluation.
62


Reviews:
[{"review_id": "ZwQWD0Le3h", "reviewer": "Reviewer_YCpi", "summary": "The authors made two contributions: \n\nThey first proposed a novel opensource multi-lingual dataset for interleaved multi-condition semantic retrieval (MERIT), highlighting difficult queries that involve multiple conditions that's lacking in previous open datasets.\n\nSecondly, they proposed a contrastive reconstruction for multimodal retrieval (CORAL) as a novel fine-tuning framework to adapt pretrained LLM for retrieval. In essence, this encodes a series of product images and product title descriptions as image/text interleaved embeddings and use an LLM to encode into EOS hidden state, and use this hidden state as query while image/text embeddings as key/value. Three objectives were used to train the model, a contrastive InfoNCE loss upon query and pos/neg product embeddings; a vision reconstruction bert layer that reconstructs the masked visual embeddings from unmasked input embeddings and h_EOS global embedding; a language reconstruction module that decodes masked text embedding similarly.", "questions": "1. The core diff of CORAL is the introduction of BERT like reconstruction losses, with the vision reconstruction task having visibility to all text(especially the text title associated with the image itself), this in essense is similar to CLIP, I'm wondering if the authors have intuitions on why this new task helped metrics.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The authors first identified the lack of sequential multi-condition queries as a limitation in the current mainstream benchmarks and invested considerable effort in constructing a new dataset.The authors meticulously tested multiple mainstream models (InternVL/QwenVL etc) and performed comprehensive analysis and documented their methodologies thoroughly. This should prove valuable for the field. \n\n2. By testing existing models with the new benchmarking system, and proofreading error cases, the authors identified two major limitations in the current semantic retrieval system: a. focusing on global semantic information while failing to extract specific attributes; b. Failing to support sequential image inputs, despite having the ability, due to existing training datasets are predominantly single-image based. These are valuable insights in guiding the community on designing next-gen benchmarks\n\n3. The ablations with vision/language modules convincingly demonstrated the effectiveness of the reconstruction losses design.\n\n\n## Weakness:\n1. While the dataset is multilingual and sizable, it seems to be domain/business specific (e-commerce in Southeast Asia), limiting its generalizability as a standard benchmark for semantic retrieval.\n\n2. The authors demonstrated that existing methods overall have poor performance on MERIT dataset in Table 2. the metrics however might be biased since some models are finetuned on the ecommerce scenario more than others, like the best in the cohort GME-QwenVL2 vs pretrained Qwen2.5-VL. It is therefore not entirely representative of what the prior models would have behaved if they were also finetuned. Undermining the significance of improvements claimed\n\n3. The authors benchmarked and evaluated CORAL/MERIT as a symbiotic duo that worked well with each other, and evaluated CORAL on a few prior tasks according to Ref 33 (Training\nvision-language models for massive multimodal embedding tasks) in Fig 9, but it seems different from what MERIT compared against in Table 1. It would be nice to isolate and demonstrate CORAL's effect on some of these benchmarks as well.", "was_revised": true, "final_justification": "The rating didnt include any shortcoming to be addressed by the authors but I thank the authors for responding to my questions", "rebuttal": "We sincerely thank Reviewer `YCpi` for the thoughtful and constructive feedback. We appreciate your recognition of our novel MERIT dataset and the CORAL framework for improving multi-condition semantic retrieval. Below, we address your concerns point by point.\n\n---\n> **`Q1`:** *\"While the dataset is multilingual and sizable, it seems to be domain/business specific (e-commerce in Southeast Asia), limiting its generalizability as a standard benchmark for semantic retrieval.\"*\n\n**A1:** Thank you for this important point. We respond as follows:\n\n- **(1.1)** Interleaved multi-condition retrieval represents **a novel and inherently challenging task** that has not been previously explored in literature due to the complexity of data collection. Our proposed MERIT constitutes the first dataset specifically designed for this task formulation. Product retrieval represents a canonical multi-condition retrieval scenario; thus, we chose to approach this problem from the product retrieval domain.\n\n- **(1.2)** Southeast Asia constitutes **an active multilingual and culturally diverse market**, whose products exhibit **representative** characteristics.\n\n- **(1.3)** We acknowledge that extending interleaved multi-condition retrieval to additional domains beyond product search represents an important direction for future work, which we defer to subsequent investigations. Additionally, based on your suggestion, we have further refined the elaboration in the manuscript.\n\n---\n> **`Q2`:** *\"The authors demonstrated that existing methods overall have poor performance on MERIT dataset in Table 2. The metrics however, might be biased since some models are finetuned on the e-commerce scenario more than others, like GME-QwenVL2 vs pretrained Qwen2.5-VL...\"*\n\n**A2:** Thank you for your observation. We clarify as follows:\n\n- **(2.1)** The purpose of Table `2` is to measure the **zero-shot capability of existing models** on the interleaved multi-condition semantic retrieval task. Most existing models are trained on proprietary datasets with inherent distributional differences, making it impossible to guarantee complete consistency across training data.\n\n- **(2.2)** For our method, based on Qwen2.5-VL, we conducted comprehensive ablation studies as shown in Table `3`, where we ensure that the fine-tuning data is completely consistent. As shown in Table `3`, our method achieves a **45.9%** improvement in R@1 over traditional contrastive learning methods, which we believe **provides a fair and convincing validation of our approach’s effectiveness**.\n\nIf you have any remaining concerns about our experimental setup, please let us know, and we would be happy to discuss them further.\n\n---\n> **`Q3`:** *\"The authors benchmarked and evaluated CORAL/MERIT as a symbiotic duo… it would be nice to isolate and demonstrate CORAL’s effect on other benchmarks as well.\"*\n\n**A3:** We appreciate this suggestion. We have now updated results on **4 new standard benchmarks** and provided more comprehensive comparisons on the 9 standard benchmarks shown in Figure `9`, following the same experimental setting as depicted in Figure `9`, as presented in the *Table A* below:\n\n***Table A.** Comparisons with other methods across 12 standard retrieval tasks*\n| Model | VisDial | CIRR | VisualNews_T2I | VisualNews_I2T | COCO_T2I | COCO_I2T | NIGHTS | WebQA | FashionIQ | Wiki-SS-NQ | OVEN | EDIS |\n|-|-|-|-|--|-|-|-|-|-|-|-|-|\n| CLIP | 30.7 | 12.6 | 78.9 | 79.6 | 59.5 | 57.7 | 60.4 | 67.5 | 11.4 | 55.0 | 41.1 | 81.0 |\n| OpenCLIP | 25.4 | 15.4 | 74.0 | 78.0 | 63.6 | 62.1 | 66.1 | 62.1 | 13.8 | 44.6 | 45.0 | 77.5 |\n| SigLIP | 21.5 | 15.1 | 51.0 | 52.4 | 58.3 | 55.0 | 62.9 | 58.1 | 20.1 | 55.1 | 56.0 | 23.6 |\n| BLIP2 | 18.0 | 9.8 | 48.1 | 13.5 | 53.7 | 20.3 | 56.5 | 55.4 | 9.3 | 28.7 | 39.5 | 54.4 |\n| MagicLens | 24.8 | 39.1 | 50.7 | 21.1 | 54.1 | 40.0 | 58.1 | 43.0 | 11.2 | 18.7 | 1.6 | 62.6 |\n| E5-V | 9.2 | 6.1 | 13.5 | 8.1 | 20.7 | 14.0 | 4.2 | 17.7 | 2.8 | 8.6 | 5.9 | 26.8 |\n| Qwen2-VL-2B | 13.0 | 20.0 | 40.0 | 43.0 | 49.0 | 39.0 | 59.0 | 20.0 | 3.0 | 34.0 | 57.0 | 44.0 |\n| Qwen2-VL-2B+CL | 51.0 | 39.0 | 56.0 | 52.0 | 56.0 | 45.0 | 58.0 | 67.0 | 3.0 | 24.0 | 57.0 | 56.0 |\n| Qwen2-VL-2B+CORAL | 73.0 | 50.0 | 67.0 | 72.0 | 68.0 | 64.0 | 65.0 | 84.0 | 9.0 | 39.0 | 67.0 | 75.0 |\n\nExperimental results demonstrate that our method (Qwen2-VL-2B+CORAL) achieves consistent improvements across these 12 standard retrieval tasks, with particularly notable performance on VisDial, where our approach exhibits a **181% enhancement over the contrastive learning baseline**. Additionally, our method shows **advantages compared to other foundational models**, such as CLIP and E5-V, with especially significant improvements on CIRR, where we achieve a 275% enhancement over OpenCLIP.\n\nWe commit to updating our revised manuscript with the new experimental results and detailed experimental configurations. We thank you once again for your valuable feedback.\n\n---\n> **`Q4`:** *\"The core diff of CORAL is the introduction of BERT-like reconstruction losses... This seems similar to CLIP. I'm wondering why this new task helped.\"*\n\n**A4:** Thank you for your insightful question. We provide the following detailed responses:\n\n- **(4.1)** Recognizing that neglecting specific conditional elements in queries constitutes a primary source of error as highlighted in Section `3.4`, it **naturally follows that reconstructing these elements could prove beneficial**. Moreover, reconstruction can enhance task complexity to improve **data utilization efficiency** *[Xiao et al., 2020]*. Therefore, we propose CORAL to enhance MLLM-based retriever performance in addressing interleaved multi-condition semantic retrieval tasks through the integration of visual reconstruction during the fine-tuning process of the MLLM-to-retrieval model adaptation.\n\n- **(4.2)** Simultaneously, we emphasize that our reconstruction approach is symmetric. When performing visual reconstruction, we maintain text visibility with the intention that textual information can assist in visual recovery, thereby enhancing cross-modal understanding capabilities. As demonstrated in the **ablation study** in Table `3`, reconstructing only vision or language modalities in isolation yields suboptimal performance.\n\n---\n**Reference:**\n- *[Xiao et al., 2020]* \"RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder.\" EMNLP, 2022.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `YCpi` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "RkUbudHO0c", "reviewer": "Reviewer_MHMi", "summary": "The paper introduces MERIT, the first large-scale benchmark that targets multilingual, multi-image, and multi-condition product retrieval. MERIT contains 135 k products and 320 k query–target pairs across seven e-commerce categories .After showing that nine state-of-the-art (SOTA) retrievers achieve < 12 % R@1 on MERIT , the authors diagnose that current fine-tuning regimes over-focus on a single [EOS] token and thus miss fine-grained attribute cues. They propose CORAL, a fine-tuning framework that couples supervised contrastive learning with masked embedding reconstruction for both vision & text streams. CORAL lifts R@1 by 45.9 % on MERIT and improves eight external benchmarks (MS-COCO, VisDial, etc.) without task-specific tweaks.", "questions": "1. How many GPU-hours did full CORAL training require for the 7B model, and what is the inference latency compared to baseline contrastive tuning?\n2. Have you measured language- or category-specific biases in the retrieval results beyond the small sample in Fig. 7? Any mitigation plans?\n3. How sensitive is CORAL to the 50 % visual/text masking ratio? Could adaptive masking yield further gains?", "limitations": "1. Dataset focuses on consumer goods; tasks like scene or document retrieval are out of scope.\n2. CORAL needs access to image embeddings during decoding; models without token-level vision hooks may not benefit.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The benchmark is timely and realistic, which is the first to combine multilingual, multi-image, and compositional attribute retrieval at scale.\n2. CORAL directly tackles the identified issue of missing fine-grained, conceptually simple yet effective.\n3. The experiments are strong and broad, which contain 9 baselines on MERIT plus 8 public datasets, full ablations, and error analysis.\n\nWeakness\n1. The section of “Masked Embedding Reconstruction” is poorly written and needs to be reorganized. For example, Mask_v and Mask_l are sometimes one parameter, sometimes 2; symbols such as q and E do not clearly state what they are; and the meaning of the corner markers under (i) in Equation 4 is unclear.\n2. Lack the analysis of reconstruction mask ratio δ, choice of decoders, and temperature $\\tao$.\n3. The metric relies solely on recall/MRR. User-centric metrics (CTR, dwell time) or ranking quality are absent.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer `MHMi` for the thoughtful and constructive feedback. We are glad that you found our benchmark timely and realistic, CORAL conceptually simple yet effective, and our experiments strong and comprehensive. Below, we address your questions and concerns in detail.\n\n---\n> **`Q1`:** *\"The section of 'Masked Embedding Reconstruction' is poorly written and needs to be reorganized...\"*\n\n**A1:** Thanks a lot for your suggestion. We have addressed the mentioned typos and improved the elaboration in the manuscript. We commit to making detailed revisions to these unclear aspects to improve readability. Additionally, we commit to open-sourcing the code and our trained models for public use.\n\n---\n> **`Q2`:** *\"Lack the analysis of reconstruction mask ratio δ, choice of decoders, and temperature.\"*\n>\n> **`Q3`:** *\"How sensitive is CORAL to the 50 % visual/text masking ratio? Could adaptive masking yield further gains?\"*\n\n**A2** **&** **A3:** Thank you for your comment. We conduct ablation studies on the mask ratio, temperature (τ), loss weights (λ), and the decoder, as shown in Table `D` below:\n\n***Table D.** Further ablation study for CORAL on MERIT using Qwen2.5-VL-3B*\n|     |     | R@1↑  | R@5↑  | R@10↑  | MRR↑  |\n| - | - | - | - | - | - |\n| Mask Ratio           |  language 0.25 vision 0.25    | 69.98 | 88.88 | 92.83  | 78.22 |\n|                      |  language 0.5 vision 0.5      | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      |  language 0.75 vision 0.75    | 60.97 | 82.31 | 87.47  | 70.38 |\n| Temperature ($τ$)  | 0.001                         | 67.48 | 87.51 | 91.77  | 76.24 |\n|                      | 0.02                          | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | 0.1                           | 67.34 | 87.1  | 91.07  | 76.00    |\n| Loss Weights ($λ$) | $λ_l$=0.05, $λ_v$=0.05 (default)          | 69.6  | 89.08 | 93.21  | 78.22 |\n|                      |  $λ_l$=0.1, $λ_v$=0.1  | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | $λ_l$=1.0, $λ_v$=1.0            | 58.41 | 80.31 | 86.25  | 68.18 |\n| Decoder Head         | 4                             | 67.89 | 88.26 | 92.28  | 76.85 |\n|                      | 8                             | 68.68 | 88.62 | 92.76  | 77.46 |\n|                      | 16                            | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | 32                            | 70.22 | 89.6  | 93.36  | 78.68 |\n\nKey findings:\n- Overall, CORAL demonstrates **robust training** dynamics and exhibits low sensitivity to variations in temperature and the number of decoder heads.\n- However, we observe that excessively high mask ratios can be detrimental to performance. This phenomenon likely stems from the model's diminished capacity to leverage existing knowledge for embedding reconstruction *[Xiao et al., 2020]*. Specifically, when the mask ratio is increased to **0.75**, we observe a substantial **10.1%** decrease in MRR compared to the 0.5 setting.\n- Furthermore, overly high loss weights can lead the model to overemphasize the reconstruction objective, consequently degrading retrieval performance.\n\n**Reference:**\n- *[Xiao et al., 2020]* \"RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder.\" EMNLP, 2022.\n\n---\n> **`Q4`:** *\"The metric relies solely on recall/MRR. User-centric metrics (CTR, dwell time) or ranking quality are absent.\"*\n\n**A4:** Thank you for your question. We address this concern from the following aspects:\n\n- **(4.1)** First, it is important to clarify that our task falls **within the domain of semantic retrieval** based on **multi-modal content comprehension** rather than **recommendation systems**. Therefore, the standard evaluation metrics are recall and MRR, and our related work also does not involve user-centric metrics such as CTR.\n- **(4.2)** Nevertheless, we have supplemented our evaluation with **human preference experiments**. Our human preference study is conducted in a multiple-choice comparison format. Specifically, we randomly sampled 100 items from MERIT, with each model providing their top-1 (Choice@1) or top-3 (Choice@3) most similar items as options. Five annotators were asked to select the best option (when the best item was retrieved by multiple models, we credited the model with the highest similarity score during recall). We report the average selection rate. As shown in Table `B`, CORAL demonstrates **superior performance** in the human preference study, particularly achieving **58.9%** selection rate in the Choice@3 setting.\n\n***Table B.** Human preference study on MERIT*\n| Model | GME-Qwen2VL-2B | LamRA-Qwen2.5VL-7B | BGE-VL-7B | CORAL-Qwen2.5VL-3B |\n|-------|:-:|:-:|:-:|:-:|\n| Choice@1 | 15.0 | 18.0 | 13.4 | 53.6 |\n| Choice@3 | 11.7 | 14.4 | 14.9 | 58.9 |\n\n---\n> **`Q5`:** *\"How many GPU-hours did full CORAL training require for the 7B model, and what is the inference latency compared to baseline contrastive tuning?\"*\n\n**A5:** Thank you for asking.\n\n- **(5.1)** We first clarify that our paper presents results for the 3B model rather than the 7B model. Under full-parameter training conditions using 8×H100 GPUs with batch size 64 for the LLM backbone, **one epoch requires approximately 5 hours for the 3B model and 6.5 hours for the 7B model.**\n- **(5.2)** Since our approach only introduces an additional decoder during training compared to the baseline, our method **incurs no additional inference latency** relative to the baseline model.\n\n---\n> **`Q6`:** *\"Have you measured language- or category-specific biases in the retrieval results beyond the small sample in Fig. 7? Any mitigation plans?\"*\n\n**A6:** Thank you for your insightful question. We measure the biases as follows:\n\n- **(6.1)** In Figure `5`(a), we evaluate language bias and observe that models trained with multilingual data **do not exhibit significant language bias**.\n- **(6.2)** As demonstrated in Table `7` of the supplemental material, Qwen2.5-VL exhibits biased zero-shot performance across different languages and classes, though the magnitude of bias remains relatively modest. This bias likely stems from the distributional characteristics of the pretraining data.\n- **(6.3)** Comparing Table `7` and Table `9` reveals that **jointly training** across multiple classes and languages **improves performance** on individual classes and languages. This suggests that increasing data diversity can effectively mitigate such biases.\n\n***Table 7.** Out-of-distribution test results for Qwen2.5-VL trained on in-domain data*\n|               |         | MRR   | R@1   | R@5   | R@10  |\n| ------------- | ------- | ----- | ----- | ----- | ----- |\n| **Language**  | ID      | 44.26 | 15.98 | 79.34 | 87.91 |\n|               | MS      | 42.82 | 17.97 | 74.65 | 84.33 |\n|               | TH      | 45.00 | 19.80 | 75.29 | 84.22 |\n| **Attribute** | Brand   | 60.72 | 48.03 | 78.07 | 85.60 |\n|               | Pattern | 54.93 | 37.23 | 77.74 | 84.53 |\n|               | Region  | 44.29 | 19.10 | 78.65 | 86.52 |\n| **Class**     | Drink   | 51.14 | 33.47 | 73.47 | 81.05 |\n|               | Phone   | 43.47 | 31.77 | 58.63 | 67.72 |\n|               | Table   | 48.57 | 35.07 | 64.69 | 74.64 |\n\n***Table 9.** Out-of-distribution test results for Qwen2.5-VL trained on the whole train set*\n|               |         | MRR   | R@1   | R@5   | R@10  |\n| ------------- | ------- | ----- | ----- | ----- | ----- |\n| **Language**  | ID      | 71.34 | 57.31 | 88.89 | 94.93 |\n|               | MS      | 67.51 | 53.23 | 85.48 | 91.94 |\n|               | TH      | 69.79 | 55.29 | 87.65 | 91.57 |\n| **Attribute** | Brand   | 71.87 | 59.62 | 88.98 | 93.59 |\n|               | Pattern | 63.50 | 50.30 | 80.44 | 87.03 |\n|               | Region  | 57.81 | 38.20 | 83.15 | 93.26 |\n| **Class**     | Drink   | 53.54 | 33.26 | 78.53 | 86.74 |\n|               | Phone   | 63.64 | 51.58 | 79.78 | 86.57 |\n|               | Table   | 52.40 | 38.63 | 69.19 | 77.96 |\n\n---\n> **`Q7`:** *\"The dataset focuses on consumer goods; tasks like scene or document retrieval are out of scope.\"*\n\n**A7:** Thanks for your comment.\n\n- **(7.1)** Interleaved multi-condition retrieval represents a novel and inherently **challenging task that has not been previously explored** in the literature due to the **complexity of data collection.** Our proposed MERIT constitutes the **first** dataset specifically designed for this task formulation, as collecting multi-condition retrieval data for document retrieval presents significant practical difficulties.\n- **(7.2)** Meanwhile, product retrieval serves a **representative and tractable domain** for multi-condition retrieval scenarios and **holds substantial real-world significance**, which motivated our domain selection.\n- **(7.3)** We acknowledge that extending interleaved multi-condition retrieval to additional domains beyond product search represents an important direction for future work, which we defer to subsequent investigations.\n\n---\n> **`Q8`:** *\"CORAL needs access to image embeddings during decoding; models without token-level vision hooks may not benefit.\"*\n\n**A8:** We appreciate your insightful question. \n\n- **(8.1)** MLLM-based models demonstrate superior performance in addressing semantic retrieval challenges *[Jiang et al., 2024]*, and CORAL is specifically **designed as a methodology for MLLM-based retrievers.**\n- **(8.2)** Current **mainstream MLLMs are equipped with token-level vision hooks**, including but not limited to GPT, InternVL, Qwen, Phi-V, LLaVa, and other prominent models in this domain. Thus, our method can **benefit most MLLM-based retrievers**.\n\n**Reference:**\n- *[Jiang et al., 2024]* \"VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks.\" ICLR, 2025.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `MHMi` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "WuEiNSD1UJ", "reviewer": "Reviewer_WJjZ", "summary": "This paper makes three contributions: 1) it introduces a new multilingual, multimodal product retrieval dataset called MERIT, 2) finds that existing retrieval models perform poorly on the MERIT task, and 3) proposes a fine-tuning technique called CORAL that achieves a 46% improvement over baselines on MERIT. The first portion of the paper is devoted to discussing the construction of the MERIT dataset and, in particular, how the data was collected annotated and refined. The authors then apply existing retrieval models to MERIT where they find that these off-the-shelf techniques exhibit poor performance. This finding motivates the authors to introduce CORAL, a multi-modal fine-tuning framework consisting of three loss functions that achieves strong performance on 8 previously introduced retrieval tasks as well as top performance on MERIT.", "questions": "Q1: What precisely do the authors mean by \"interleaved\" and why do existing benchmarks fail to meet this property. *I would be willing to raise my evaluation score if the authors can provide a clear definition of interleaved and explain why this is a novel property of their dataset.*\n\nQ2: How were the products sourced for the MERIT dataset? What precisely is the role of GPT-4o in the dataset construction process? How do the authors control for potential errors from using GPT-4o in the dataset construction process? *I would be willing to raise my evaluation score if the authors can provide answers to these questions and, in general, provide more details on how the dataset was sourced (including perhaps a file with data examples)*\n\nQ3: Are there any works in the multimodal contrastive learning literature that are related to CORAL? What distinguishes CORAL from these prior works? *I would be willing to raise my evaluation score if the authors can include a deeper discussion of related work around the CORAL model (particularly in contrastive learning). Currently, I feel that this important positioning of CORAL in relation to the literature is lacking in the paper*\n\nQ4: How were the expert annotators recruited for this project and how were they compensated? What is the license of the MERIT dataset? *I would be willing to raise my evaluation score if the authors can provide satisfactory answers to these questions*", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n1. [Originality/Significance] As a multimodal and multilingual product retrieval dataset focused on Southeast Asian languages, I believe that MERIT has the potential to stand out as a novel dataset contribution to the community. The authors motivate the need for a dataset that more tightly integrates different modalities (though they do not precisely define their term \"interleaved\" -- please see the \"Weaknesses\" section below) quite well. \n\n2. [Quality/Significance] The experiments with the CORAL framework are well-executed with results on both MERIT as well as 8 existing retrieval datasets which provides robust evidence that the technique is effective. \n\nWeaknesses\n\n1. [Clarity] The paper lacks sufficient details in several places. First and foremost, as alluded to above, the authors, to my knowledge, do not provide a precise definition of what they mean by \"interleaved\" and, to my knowledge, this is not a standard term in the retrieval literature. It would be very helpful for the authors to clearly define what they mean by interleaved, which would help in understanding how the MERIT dataset stands out from prior work. \n\n2. [Quality] The paper also lacks sufficient details on how the MERIT dataset was constructed. In particular, it was not clear to me from reading the paper 1) how the products were sourced and from where, 2) what specific role GPT-4o played in the dataset construction (line 130 of the paper), and 3) who the annotators are and how they were selected/compensated? While the authors provide examples of what individual items in the dataset look like, I did not see any link to be able to download and inspect the dataset files directly (though I may have missed this). Overall, this paper covers a substantial amount of ground in introducing a new benchmark and proposing a new method and I would encourage the authors to consider whether these contributions might be better appreciated if there were more space to devote to each one as standalone papers. \n\n3. [Quality] There is a rich body of literature on contrastive learning for multimodal models, but the authors do not discuss this related work in much detail in the paper. I would suggest discussing more of this related work to better position the CORAL framework so that readers can understand what is novel about the technique. As it stands, I believe that the triple contrastive loss framework proposed in the paper is limited in novelty since similar ideas have been proposed (though the strength of the empirical results still validates the contribution). A deeper discussion of this related work might help the authors' novel contributions stand out more effectively.", "was_revised": true, "final_justification": "During the rebuttal period, the authors addressed my concerns around providing a formal definition of interleaving and providing important details on the dataset license and human annotation process. Thus, I have decided to raise my score from a 3 to 4. One unresolved issue (which motivated me not to give a higher rating) is that the authors were not able to provide access to a slice of the data for review. This is a requirement in the Datasets and Benchmarks track, for example, and is an important component of evaluating a new dataset contribution.", "rebuttal": "We sincerely thank Reviewer `WJjZ` for the thoughtful and constructive feedback. We are encouraged that you found our benchmark MERIT to be a novel and significant contribution, especially in highlighting multilingual and multimodal product retrieval, and appreciated the effectiveness and breadth of CORAL’s empirical evaluations. Below, we address each concern point by point.\n\n---\n> **`Q1`:** *\"What precisely do the authors mean by 'interleaved' and why do existing benchmarks fail to meet this property. I would be willing to raise my evaluation score if the authors can provide a clear definition of interleaved and explain why this is a novel property of their dataset.\"*\n\n**A1:** We provide a point-by-point clarification of what constitutes interleaved:\n\n- **(1.1) Definition:** In MLLMs, \"interleaved\" refers to the model's capability to simultaneously **process alternating sequences** of images and text. \n  - This capability enables more flexible understanding and generation of multimodal content, analogous to human cognitive processes *\\[Dong et al., 2024]*.\n  - The typical sequence structure follows: `[Text₁] → [Image₁] → [Text₂] → [Image₂] → ...`\n  - Post-2024 MLLMs have predominantly adopted **interleaved sequence processing**, as this better reflects real-world usage patterns where users frequently present queries containing multiple distributed images.\n\n- **(1.2) Our Contribution:** Existing retrieval datasets exclusively consider single images or singular retrieval conditions, namely using an image to retrieve text, or using an image-text pair to retrieve an image-text pair. Our work presents the **first interleaved retrieval dataset**. As demonstrated in Figure `2` in the manuscript, our query format follows: `[Text₁] → [Product Image₁] → [Product Title₁] → [Product Image₂] → [Product Title₂] → ...`, representing a fundamental departure from previous dataset formulations.\n\n- **(1.3)** **Practical Significance**: Interleaved data enables the construction of multi-condition, cross-modal retrieval tasks, allowing for **more precise interpretation of user intent**. Moreover, the interleaved format aligns with the input structure of MLLMs, which uses interleaved input dat&#x61;*\\[Alayrac et al., 2022]*, enhancing their **semantic representation capabilities** and thereby advancing progress in multimodal learning *\\[Zhu et al., 2024]* .\n\nWe have clarified the definition in the revised manuscript and better emphasized its novelty.\n\n**References:**\n- *\\[Dong et al., 2024]* \"Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.\" arXiv:2401.16420, 2024.\n- *\\[Zhu et al., 2024]* \"Multimodal c4: An open, billion-scale corpus of images interleaved with text.\" NeurIPS, 2023.\n- *\\[Alayrac et al., 2022]* \"Flamingo: A visual language model for few-shot learning.\" NeurIPS, 2022.\n\n---\n> **`Q2`:** *\"How were the products sourced for the MERIT dataset? What precisely is the role of GPT-4o in the dataset construction process? How do the authors control for potential errors from using GPT-4o in the dataset construction process? I would be willing to raise my evaluation score if the authors can provide answers to these questions and, in general, provide more details on how the dataset was sourced (including perhaps a file with data examples).\"*\n\n**A2:** We apologize for any unclear aspects in our paper and will address your concerns point by point.\n\n- **(2.1)** As described in Appendix Section `A.1`, our data is sourced from our internal dataset. Specifically, we curated popular products across six Southeast Asian countries. Due to the double-blind review process, we cannot provide further details at this time. However, we commit to releasing more comprehensive details once the paper is accepted.\n\n- **(2.2)** Overall, GPT was involved in attribute selection, product annotation, query generation, and filtering processes, as indicated by the robot icons in Figure `2` in the manuscript. We take **annotation accuracy very seriously**. Following each GPT annotation step, we implemented GPT self-verification procedures. For critical steps (attribute identification and query generation), we employed manual filtering (10,000 labor hours in total) to ensure quality. We provide an 11-page detailed description of our annotation process in Appendix Section `A`, which thoroughly explains the usage of GPT in our pipeline.\n\n- **(2.3)** Regarding detailed data information, we provide distribution statistics in Appendix Section `D` and data examples in Appendix Section `E`. Due to rebuttal limitations preventing file uploads and the rule stating *\"All the texts you post (rebuttal, discussion) should not contain any links to external pages\"*, we cannot upload the requested folders containing data examples and sources. However, we commit to fully open-sourcing our data and code (both annotation and training) following the completion of the review process.\n\nWe acknowledge these unclear aspects you have identified and commit to addressing them comprehensively in the revised version of our paper.\n\n---\n> **`Q3`:** *\"How were the expert annotators recruited for this project, and how were they compensated? What is the license of the MERIT dataset?\"*\n\n**A3:** Thanks for your question.\n\n- **(3.1)** Our annotators are professional in-house annotation staff with **extensive labeling experience** and bachelor's degrees or higher. The specific compensation is determined by our institution's policy, with an estimated total exceeding $10,000 USD. We ensure that annotators receive fair compensation, maintain appropriate workloads, and prioritize their physical and mental well-being.\n\n- **(3.2)** Quality control was strictly enforced during annotation. All annotators **underwent training and a qualification phase** (including trial annotations) before formal participation. Additionally, we employed dedicated quality-check specialists to conduct periodic random inspections.\n\n- **(3.3)** Our licensing follows **CC BY-NC 4.0**, which we have explicitly specified in the revised version of the manuscript.\n\n---\n> **`Q4`:** *\"Are there any works in the multimodal contrastive learning literature that are related to CORAL? What distinguishes CORAL from these prior works? I would be willing to raise my evaluation score if the authors could include a deeper discussion of related work around the CORAL model (particularly in contrastive learning). Currently, I feel that this important positioning of CORAL in relation to the literature is lacking in the paper.\"*\n\n**A4:** We thank you for your insightful question and will address each point systematically:\n\n- **(4.1)** While we acknowledge that the literature on multimodal contrastive learning is indeed extensive, our method specifically targets **MLLM-based retrieval approaches**, focusing on training algorithms for transitioning from MLLMs to multimodal retrieval models. Therefore, our related work section concentrates on discussing MLLMs and Multimodal Retrieval Models as most relevant to our contribution.\n\n- **(4.2)** Through our experimental analysis, we identified key limitations in existing MLLM-based retrieval methods: over-reliance on global semantic information while neglecting conditional query elements, failing to extract specific attributes, and misinterpreting visual content. To address these challenges, we propose CORAL, which introduces embedding reconstruction on top of conventional contrastive learning to preserve **fine-grained conditional elements rather than solely supervising the \\[EOS] token,&#x20;**&#x61;s detaile&#x64;**&#x20;in lines 100-110&#x20;**&#x6F;f our manuscript.\n\n- **(4.3)** Moreover, to the best of our knowledge, we are the **first to introduce multimodal embedding reconstruction in MLLM-based retrieval**. The results on MERIT and 12 standard benchmarks also demonstrate the effectiveness of our approach.\n\nWe greatly appreciate your valuable feedback and have committed to incorporating more comprehensive discussions of contrastive learning-related work in the revised version of our manuscript.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `WJjZ` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "JpOqvAOTDh", "reviewer": "Reviewer_6FnV", "summary": "This paper introduces a new benchmark dataset, MERIT, and an improved fine-tuning method, CORAL, to advance the field of semantic retrieval. Existing benchmarks are often limited to single-language, single-image, and single-condition queries, failing to reflect complex, real-world e-commerce scenarios. To address this, they created MERIT, a multilingual dataset for text-image interleaved, multi-condition product retrieval, containing 320,000 queries over 135,000 products in 5 languages. They also propose CORAL to overcome the limitation that existing embedding models lose fine-grained conditional details in queries, that combines contrastive learning with text and image reconstruction. The authors show that CORAL achieves a 45.9% performance improvement on MERIT.", "questions": "1. In Figure 2, the query condition \"same selling method\" is ambiguous. Could you clarify this example?\n2. How does model performance vary with the number of conditions in a query (e.g., two vs. three vs. four)? The main results do not show this breakdown.\n3. For the out-of-distribution results in Figure 6(b), could you clarify the setting? Was the Qwen2.5-VL trained on a subset of MERIT?\n4. For clarity, consider making the introduction of the CORAL framework (line 241) a new subsection separate from \"Preliminaries\".\n5. In Eq5, the CORAL loss function includes two hyperparameters, λ1​ and λ2​, to balance the reconstruction losses. Could you elaborate on how these weights were selected and their sensitivity?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles a highly relevant and underserved problem. As retrieval systems are integrated into real-world applications, the limitations of simple, single-condition benchmarks become increasingly apparent. The authors' focus on creating a benchmark for multilingual, multi-image, and multi-condition retrieval is a crucial step to address this gap.\n2. The creation of the MERIT dataset is a massive undertaking and a significant contribution. The data collection process is well-documented and rigorous, involving multiple stages. This dataset will be an invaluable resource for the community.\n3. The paper proposes a novel and effective solution CORAL, motivated by the error analysis performed on existing models. The combination of contrastive learning with reconstruction is an effective approach, with the reported 45.9% performance gain on MERIT and strong generalization to other benchmarks.\n\nWeaknesses:\n1. The product data is sourced from a private, \"internal dataset\". While the authors have anonymized the data, the lack of information about the original source (e.g., a single e-commerce platform vs. multiple) makes it difficult to reason about potential inherent biases in the product distribution, metadata structure, or visual style.\n2. The focused modality (text, image) and scenario (product search) are not accurately reflected in the title (semantic retrieval is too broad).  \n3. Some experiment settings/details are not clear.", "was_revised": true, "final_justification": "The authors addressed my concerns on content clarity and ablation results. I keep my original positive rating since I think the dataset can be valuable to the community.", "rebuttal": "We sincerely thank Reviewer `6FnV` for the thoughtful and constructive feedback. We are especially encouraged by your recognition of the real-world relevance of our problem setting, the rigor and scale of the MERIT dataset, and the effectiveness of our CORAL framework. Below, we address your comments point by point:\n\n---\n> **`Q1`:** *\"The product data is sourced from a private, 'internal dataset'. While the authors have anonymized the data, the lack of information about the original source (e.g., a single e-commerce platform vs. multiple) makes it difficult to reason about potential inherent biases in the product distribution, metadata structure, or visual style.\"*\n\n**A1:** Thank you for your question. Our original data is sourced from **a single e-commerce platform**. We have provided **distribution statistics** in Appendix Section `D` and **data examples** in Appendix Section `E`. Due to the double-blind reviewing process, we cannot disclose additional data information at this time. However, we have committed to providing more comprehensive data details and fully open-sourcing our data and code (both annotation and training components).\n\n---\n> **`Q2`:** *\"The focused modality (text, image) and scenario (product search) are not accurately reflected in the title (semantic retrieval is too broad).\"*\n\n**A2:** Thank you for your suggestion. We will consider revising the title of our paper in future iterations, as you know, modifications are not permitted during the rebuttal phase. We will consider incorporating keywords related to modality and scenario.\n\n---\n> **`Q3`:** *\"Some experiment settings/details are not clear.\"*\n>\n> * **`Q3.1`**: *\"In Figure 2, the query condition 'same selling method' is ambiguous. Could you clarify this example?\"*\n>\n> * **`Q3.2`**: *\"How does model performance vary with the number of conditions in a query (e.g., two vs. three vs. four)? The main results do not show this breakdown.\"*\n>\n> * **`Q3.3`**: *\"For the out-of-distribution results in Figure 6(b), could you clarify the setting? Was the Qwen2.5-VL trained on a subset of MERIT?&#x20;\"*\n>\n> * **`Q3.4`**: *\"For clarity, consider making the introduction of the CORAL framework (Line 241) a new subsection separate from 'Preliminaries'.\"*\n>\n> * **`Q3.5`**: *\"In Eq. 5, the CORAL loss function includes two hyperparameters, λ1 and λ2, to balance the reconstruction losses. Could you elaborate on how these weights were selected and their sensitivity?\"*\n\n**A3:** Thank you for your question! We apologize for any unclear details in the manuscript. Here, we will explain them point by point.&#x20;\n\n- **A 3.1:** We inadvertently reversed the plotting of product1 and product2 in the figure. \"Same selling method\" refers to the bundled selling approach where 3 items are sold together as a package. We have revised this mirror in the revised manuscript.\n\n- **A 3.2**: We thank you for this valuable suggestion. Due to time constraints, we have supplemented Table `C` in the supplementary material with results showing the performance of several key models as a function of the number of query conditions. As demonstrated in Table `C` below, we observe tha&#x74;**&#x20;the performance of nearly all models degrades with increasing condition count**. Notably, under the most complex scenario with 4 conditions, even the best-performing model achieves only 36.84 R@5, highlighting the inherent difficulty posed by the combinatorial complexity of multi-condition queries.\n\n***Table C.** Impact of query condition count on model performance on MERIT*\n|  | #condition | R@1↑  | R@5↑  | R@10↑  | MRR↑  |\n| ---------------------- | ---------- | ----- | ----- | ------ | ----- |\n| InternVL2.5-MPO-1B-Seq | 2          | 0.42  | 1.39  | 2.30   | 0.88  |\n|                        | 3          | 0.00  | 0.00  | 0.78   | 0.10  |\n|                        | 4          | 0.00  | 0.00  | 0.00   | 0.00  |\n|  Qwen2.5-VL-3B-Seq     | 2          | 0.09  | 0.39  | 0.55   | 0.20  |\n|                        | 3          | 0.00  | 0.78  | 1.56   | 0.37  |\n|                        | 4          | 0.00  | 0.00  | 0.00   | 0.00  |\n| GME-Qwen2VL-2B-Cat     | 2          | 8.45  | 53.12 | 61.24  | 27.73 |\n|                        | 3          | 10.24 | 55.91 | 59.06  | 28.51 |\n|                        | 4          | 5.26  | 36.84 | 47.37  | 15.89 |\n| LamRA-Qwen2.5VL-7B-Cat | 2          | 12.05 | 39.20 | 48.13  | 23.83 |\n|                        | 3          | 13.39 | 35.43 | 42.52  | 22.58 |\n|                        | 4          | 5.56  | 27.78 | 33.33  | 15.28 |\n| VLM2Vec-4B-Seq         | 2          | 0.44  | 1.87  | 2.98   | 1.04  |\n|                        | 3          | 0.00  | 1.57  | 2.36   | 0.70  |\n|                        | 4          | 0.00  | 0.00  | 0.00   | 0.00  |\n\n\n- **A 3.3:** In Figure `6`(b), we evaluate three types of OOD scenarios. You can find the specific experimental details and more comprehensive result tables (please refer to Tables `7`, `8`, and `9`) in Appendix Section `E.4`. In brief, *\"OOD\"* refers to removing the same language, class, or attribute from the training set during testing, while *\"mixed\"* refers to training on the complete dataset.\n\n- **A 3.4:** Thank you for your feedback. We have committed to incorporating your suggestions in the revised version of the manuscript.\n\n- **A 3.5**:  We conduct ablation studies on the mask ratio, temperature (τ), loss weights (λ), and the decoder, as shown in Table `D`. Overall, CORAL demonstrates **robust training dynamics&#x20;**&#x61;nd exhibits low sensitivity to variations in temperature and the number of decoder heads.\n\nRegarding the selection of loss weights, we first establish through ablation experiments in Table `3` tha&#x74;**&#x20;joint reconstruction of language and vision modalities** is more effective than reconstructing individual modalities. Therefore, in the ablation experiments presented in Table `D` below, we maintain equal weights for both modalities. Our findings indicate that smaller values of λ have minimal impact on the results. However, overly high loss weights can lead the model to overemphasize the reconstruction objective, consequently degrading retrieval performance.\n\n***Table D.** Further ablation study for CORAL on MERIT using Qwen2.5-VL-3B*\n|                      |                               | R@1↑  | R@5↑  | R@10↑  | MRR↑  |\n| -------------------- | ----------------------------- | ----- | ----- | ------ | ----- |\n| Mask Ratio           |  language 0.25 vision 0.25    | 69.98 | 88.88 | 92.83  | 78.22 |\n|                      |  language 0.5 vision 0.5      | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      |  language 0.75 vision 0.75    | 60.97 | 82.31 | 87.47  | 70.38 |\n| Temperature ($τ$)  | 0.001                         | 67.48 | 87.51 | 91.77  | 76.24 |\n|                      | 0.02                          | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | 0.1                           | 67.34 | 87.10  | 91.07  | 76.00    |\n| Loss Weights ($λ$) | $λ_l$=0.05, $λ_v$=0.05 (default)          | 69.60  | 89.08 | 93.21  | 78.22 |\n|                      |  $λ_l$=0.1, $λ_v$=0.1  | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | $λ_l$=1.0, $λ_v$=1.0            | 58.41 | 80.31 | 86.25  | 68.18 |\n| Decoder Head         | 4                             | 67.89 | 88.26 | 92.28  | 76.85 |\n|                      | 8                             | 68.68 | 88.62 | 92.76  | 77.46 |\n|                      | 16                            | 69.68 | 89.26 | 93.08  | 78.33 |\n|                      | 32                            | 70.22 | 89.60  | 93.36  | 78.68 |\n\n---\nLast but not least, we would like to sincerely thank Reviewer `6FnV` again for the valuable time and constructive feedback provided during this review."}]

Author rebuttal:

Meta-review: There are two primary contributions of this work: (1) the authors propose MERIT, claimed to be the first multi-lingual dataset for interleaved (i.e., text, images) multi-condition and multi-attribute semantic (product) retrieval and (2) observing limitations on existing models with MERIT, the authors proposed CORAL, a fine-tuning method for multimodal retrieval that integrates embedding reconstruction and contrastive learning to derive global semantics. MERIT contains 135k products and 320k query–target pairs across five languages and seven product retrieval scenarios. The process for generating data is well-documented (Figure 5 -- although reviewer WJjZ made some sensible recommendations for further improvements), sensible, and includes human validation. Several state-of-the-art baseline methods are shown to perform poorly on MERIT, motivating the authors to introduce CORAL to prevent over-focusing on a single [EOS] token that misses fine-grained attribute cues. CORAL is shown to outperform existing baselines on MERIT and several widely-used retrieval benchmarks. Additionally, ablation studies are performed to better understand relative component contributions and dynamics.

Strengths identified by the reviewers include:
- Sequential multi-condition, multi-modal queries is missing in widely-used benchmarks and is common in many practical deployed applications (which likely develop internal datasets that may not explicitly capture the sequential dependencies). The dataset appears carefully constructed, is sizable, and likely useful to other researchers -- and increasingly so given deployed AI products being increasingly multi-modal.
- The authors experimented with several strong benchmark models and identified consistent root cause error types, developing a new method that is shown to perform well on MERIT and other widely used retrieval benchmarks (VisDial, CIRR, VisualNews, MSCOCO, NIGHTS, WebQA) relative to strong baselines (with additional baselines included in the rebuttal -- that still need to be integrated into the main paper).
- The rebuttal was particularly effective in addressing reviewer concerns around sensitivity analyses, human evaluation, dataset biases, query condition cardinality, hyperparameters, etc. -- all of which can easily be integrated into a manuscript revision (even if some in appendices).

Weaknesses identified by the reviewers include:
- While product retrieval is an excellent multi-condition retrieval scenario, it may limit adoption in not being multi-domain and introduce some biases in the types of methods that will tend to perform well. CORAL also performs well on other tasks, so this isn't a significant concern for this specific paper, but will likely introduce some biases if not combined with other benchmarks in future papers.
- There were several clarification questions and technical details (including some fundamental aspects), but these were well-addressed during rebuttal (and can be easily added to the next revision) such that the reviewers generally changed their score. 
- The dataset is likely the more significant contribution and as this isn't the datasets track, the reviewers were not able to actually look at the dataset to evaluate the quality 'first hand'. I am confident it is fine based on the paper, but it is a valid concern. 
- [my concern] Given the paper is introducing a dataset and method, I perceived the discussion of experimental results to be somewhat short -- which is why I believe there were multiple questions regarding additional experiments. Thus, I strongly encourage the authors to integrate some of these results in the main text and some in the appendices -- maintaining thorough discussion in both cases as this may point the way to future work. 

As the strongest contribution is likely the dataset (even though there are other solid contributions), I believe it meets the core criteria for a good resources paper: (1) the provided benchmark is unique, useful, and sufficiently large/diverse, (2) the benchmark lowers the barrier for other researchers to work on this problem, and (3) the baselines provided isn't the simplest baselines. I think MERIT/CORAL stands out in terms of (1) and (3). Regarding (2), the authors will release the dataset as CC BY-NC 4.0, which isn't the most permissive, but shouldn't restrict wide adoption if researchers find the data useful. They also claim a public release plan for code, etc. -- but this hasn't been verified. Overall, I believe this is a solid and useful research & dataset contribution.
Decision: Accept (poster)