MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition
  Query
Authors: name: Wei Chow, name: Yuan Gao, name: Linfeng Li, name: Xian Wang, name: Qi Xu, name: Hang Song, name: Lingdong Kong, name: Ran Zhou, name: Yi Zeng, name: Yidong Cai, name: Botian Jiang, name: Shilin Xu, name: Jiajun Zhang, name: Minghui Qiu, name: Xiangtai Li, name: Tianshu Yang, name: Siliang Tang, name: Juncheng Li
Organization: ByteDance

Semantic retrieval is crucial for modern applications yet remains
underexplored in current research. Existing datasets are limited to single
languages, single images, or singular retrieval conditions, often failing to
fully exploit the expressive capacity of visual information as evidenced by
maintained performance when images are replaced with captions. However,
practical retrieval scenarios frequently involve interleaved multi-condition
queries with multiple images. Hence, this paper introduces MERIT, the first
multilingual dataset for interleaved multi-condition semantic retrieval,
comprising 320,000 queries with 135,000 products in 5 languages, covering 7
distinct product categories. Extensive experiments on MERIT identify existing
models's limitation: focusing solely on global semantic information while
neglecting specific conditional elements in queries. Consequently, we propose
Coral, a novel fine-tuning framework that adapts pre-trained MLLMs by
integrating embedding reconstruction to preserve fine-grained conditional
elements and contrastive learning to extract comprehensive global semantics.
Experiments demonstrate that Coral achieves a 45.9% performance improvement
over conventional approaches on MERIT, with strong generalization capabilities
validated across 8 established retrieval benchmarks. Collectively, our
contributions - a novel dataset, identification of critical limitations in
existing approaches, and an innovative fine-tuning framework - establish a
foundation for future research in interleaved multi-condition semantic
retrieval.