Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
  Models
Authors: name: Qirui Jiao; name: Daoyuan Chen; name: Yilun Huang; name: Yaliang Li; name: Ying Shen
Published: 2024-08-08 17:10:16

Abstract: High-performance Multimodal Large Language Models (MLLMs) rely heavily on
data quality. This study introduces a novel dataset named Img-Diff, designed to
enhance fine-grained image recognition in MLLMs by leveraging insights from
contrastive learning and image difference captioning. By analyzing object
differences between similar images, we challenge models to identify both
matching and distinct components. We utilize the Stable-Diffusion-XL model and
advanced image editing techniques to create pairs of similar images that
highlight object replacements. Our methodology includes a Difference Area
Generator for object differences identifying, followed by a Difference Captions
Generator for detailed difference descriptions. The result is a relatively
small but high-quality dataset of "object replacement" samples. We use the the
proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
yielding comprehensive improvements of performance scores over SOTA models that
trained with larger-scale datasets, in numerous image difference and Visual
Question Answering tasks. For instance, our trained models notably surpass the
SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
alternative methods for generating image difference data through "object
removal" and conduct thorough evaluation to confirm the dataset's diversity,
quality, and robustness, presenting several insights on synthesis of such
contrastive dataset. To encourage further research and advance the field of
multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
for image understanding, we release our codes and dataset at
https://github.com/modelscope/data-juicer/tree/ImgDiff.