MuirBench: A Comprehensive Benchmark for Robust Multi-image
  Understanding
Authors: name: Fei Wang; name: Xingyu Fu; name: James Y. Huang; name: Zekun Li; name: Qin Liu; name: Xiaogeng Liu; name: Mingyu Derek Ma; name: Nan Xu; name: Wenxuan Zhou; name: Kai Zhang; name: Tianyi Lorena Yan; name: Wenjie Jacky Mo; name: Hsiang-Hui Liu; name: Pan Lu; name: Chunyuan Li; name: Chaowei Xiao; name: Kai-Wei Chang; name: Dan Roth; name: Sheng Zhang; name: Hoifung Poon; name: Muhao Chen
Published: 2024-06-13 17:59:52

Abstract: We introduce MuirBench, a comprehensive benchmark that focuses on robust
multi-image understanding capabilities of multimodal LLMs. MuirBench consists
of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
involve 10 categories of multi-image relations (e.g., multiview, temporal
relations). Comprising 11,264 images and 2,600 multiple-choice questions,
MuirBench is created in a pairwise manner, where each standard instance is
paired with an unanswerable variant that has minimal semantic differences, in
order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
results reveal that even the best-performing models like GPT-4o and Gemini Pro
find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
Open-source multimodal LLMs trained on single images can hardly generalize to
multi-image questions, hovering below 33.3% in accuracy. These results
highlight the importance of MuirBench in encouraging the community to develop
multimodal LLMs that can look beyond a single image, suggesting potential
pathways for future improvements.