MMIU: Multimodal Multi-image Understanding for Evaluating Large
  Vision-Language Models
Authors: name: Fanqing Meng; name: Jin Wang; name: Chuanhao Li; name: Quanfeng Lu; name: Hao Tian; name: Jiaqi Liao; name: Xizhou Zhu; name: Jifeng Dai; name: Yu Qiao; name: Ping Luo; name: Kaipeng Zhang; name: Wenqi Shao
Published: 2024-08-05 17:56:41

Abstract: The capability to process multiple images is crucial for Large
Vision-Language Models (LVLMs) to develop a more thorough and nuanced
understanding of a scene. Recent multi-image LVLMs have begun to address this
need. However, their evaluation has not kept pace with their development. To
fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
relationships, 52 tasks, 77K images, and 11K meticulously curated
multiple-choice questions, making it the most extensive benchmark of its kind.
Our evaluation of 24 popular LVLMs, including both open-source and proprietary
models, reveals significant challenges in multi-image comprehension,
particularly in tasks involving spatial understanding. Even the most advanced
models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
multi-faceted analytical experiments, we identify key performance gaps and
limitations, providing valuable insights for future model and data
improvements. We aim for MMIU to advance the frontier of LVLM research and
development, moving us toward achieving sophisticated multimodal multi-image
user interactions.