MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Authors: name: Junjie Zhou; name: Zheng Liu; name: Ze Liu; name: Shitao Xiao; name: Yueze Wang; name: Bo Zhao; name: Chen Jason Zhang; name: Defu Lian; name: Yongping Xiong
Published: 2024-12-19 02:49:55

Abstract: Despite the rapidly growing demand for multimodal retrieval, progress in this
field remains severely constrained by a lack of training data. In this paper,
we introduce MegaPairs, a novel data synthesis method that leverages vision
language models (VLMs) and open-domain images, together with a massive
synthetic dataset generated from this method. Our empirical analysis shows that
MegaPairs generates high-quality data, enabling the multimodal retriever to
significantly outperform the baseline model trained on 70times more data
from existing datasets. Moreover, since MegaPairs solely relies on general
image corpora and open-source VLMs, it can be easily scaled up, enabling
continuous improvements in retrieval performance. In this stage, we produced
more than 26 million training instances and trained several models of varying
sizes using this data. These new models achieve state-of-the-art zero-shot
performance across 4 popular composed image retrieval (CIR) benchmarks and the
highest overall performance on the 36 datasets provided by MMEB. They also
demonstrate notable performance improvements with additional downstream
fine-tuning. Our produced dataset, well-trained models, and data synthesis
pipeline will be made publicly available to facilitate the future development
of this field.