MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

Authors: Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Tianshuo Yang, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, Wenqi Shao
Primary area: datasets and benchmarks
Keywords: Multi-image Understanding, Benchmark, LVLM, Evaluation

Abstract:
The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of nearly 30 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7\% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development. We release the data and code at https://github.com/MMIUBenchmark/MMIU.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729952069775

thread_type:
official_review

<thread_object>
creator: reviewer_g41s

<content>
summary:
The article introduces the MMIU benchmark for evaluating large vision-language models on multi-image tasks. It includes various tasks and a large dataset with over 77K images and 11K questions. The study finds that even advanced models struggle with complex multi-image understanding

strengths:
1、Comprehensive Evaluation: The MMIU benchmark provides a thorough assessment of LVLMs by covering a wide range of multi-image tasks and relationships, making it one of the most extensive benchmarks in this domain.

2、Diverse Task Coverage: It includes a variety of tasks that require different cognitive processes, from low-level visual perception to high-level reasoning, offering a holistic view of a model's capabilities.

3、Insightful Analysis: The article presents detailed analytical experiments that identify key performance gaps and limitations in current models, providing valuable insights for future research and development in the field of multimodal AI.

weaknesses:
1、Typo: 173 line multimodal large language models (LVLMs), abbreviation error.

2、The details of writing can be further polished, such as using Fig. in some places, Some places use Figure

questions:
1、Are there any duplicates between the samples in your dataset and the current multi graph dataset?

2、Why do models without multi-imgae SFT perform better than those with multi-imgae SFT on your benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730001324897

thread_type:
official_review

<thread_object>
creator: reviewer_jjzh

<content>
summary:
The paper introduces the Multimodal Multi-Image Understanding (MMIU) benchmark, which evaluates large vision-language models (LVLMs) on complex multi-image tasks, including spatial, semantic, and temporal relationships. Covering 52 task types with 77,000 images and 11,000 curated questions, MMIU is a comprehensive benchmark that reveals significant challenges, particularly in spatial and temporal tasks, for leading models such as GPT-4o.

strengths:
1. Multi-image understanding is indeed a meaningful and impactful problem within multimodal research, this paper effectively addresses this need by establishing MMIU, a benchmark specifically designed to evaluate large vision-language models (LVLMs) across diverse multi-image tasks.

2. The presentation is clear and well-structured, figures and tables are effectively used to enhance understanding.

weaknesses:
1. The benchmark evaluates numerous LVLMs but could benefit from deeper comparisons across models with distinct visual encoding strategies. This would provide more specific insights into how different architectures impact performance on complex spatial and temporal tasks, which is particularly valuable for the multi-image context.

2. While MMIU successfully identifies gaps in LVLM performance, the paper does not sufficiently explore or propose alternative methods to address these gaps, such as novel training techniques or architectural adjustments. This limits the benchmark’s practical utility for guiding future model development.

questions:
This paper is overall good, it would be better to present deeper analysis on model performance variations across specific multi-image relationships, particularly spatial, temporal, and high-level semantic tasks. Currently, the analysis highlights general performance trends but could benefit from a more granular examination of how and why certain models excel or struggle with specific relationship types.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730107630061

thread_type:
official_review

<thread_object>
creator: reviewer_4ca4

<content>
summary:
This paper presents MMIU, a comprehensive benchmark designed to evaluate large vision-language models (LVLMs) on multi-image tasks, addressing limitations in existing single-image-focused evaluations. MMIU includes 52 tasks spanning seven image relationship types, such as semantic, spatial, and temporal, with a total of 77,659 images and 11,698 multi-choice questions.

strengths:
- Originality: The paper introduces a novel benchmark, MMIU, specifically designed for evaluating large vision-language models (LVLMs) on multi-image tasks. This fills a critical gap, as existing benchmarks predominantly focus on single-image understanding. The inclusion of diverse image relationships—semantic, spatial, and temporal—is a creative approach that highlights areas where LVLMs struggle, especially in tasks requiring complex multi-image reasoning.

- Quality: The experimental methodology is robust and thorough. The authors test 24 popular LVLMs, both open-source and closed-source, providing a comprehensive view of current capabilities. Their use of multi-dimensional analysis tools, including supervised fine-tuning (SFT) and task maps, provides depth to the evaluation, highlighting model strengths and weaknesses across various task types. This approach demonstrates rigorous scientific methodology and enhances the reliability of the conclusions drawn.

- Significance: MMIU has high potential impact for the vision-language research community, particularly given the increasing relevance of multi-image tasks in real-world applications (e.g., video understanding, 3D reasoning). By providing a benchmark with a broad set of tasks and image relationships, MMIU allows for a more nuanced understanding of LVLMs’ strengths and limitations. The insights from this benchmark can guide future LVLM improvements, pushing the field towards more sophisticated models that handle complex visual contexts and multi-image data effectively.

weaknesses:
- Reliability of GPT-4o-Generated Data: Although MMIU’s large dataset is a significant contribution, its reliance on GPT-4o for generating questions and responses raises concerns about potential bias. Given that GPT-4o itself may have inherent preferences or limitations in multi-image reasoning, this could inadvertently affect the construction of questions and answer choices. Additionally, if GPT-4o has limitations in multi-image reasoning (as noted by the authors), this brings into question whether the generated tasks and responses are fully reliable. While human verification can ensure the accuracy of correct answers, it may not catch subtle model biases embedded in the answer choices, potentially skewing results. A clearer discussion or additional safeguards against GPT-4o’s influence on task design would improve the benchmark’s objectivity and robustness.

- Limited Discussion on Practical Applicability: While MMIU’s tasks are inspired by real-world applications, the paper could strengthen its relevance by addressing practical implications more explicitly. For instance, the authors could highlight specific MMIU tasks that align with high-stakes applications—such as high-level reasoning and long-term memory tasks relevant to autonomous driving, surveillance, and medical imaging. Connecting benchmark tasks to these domains would make the work more relatable and impactful for applied researchers. Additionally, discussing how MMIU might be adapted or extended to cover emerging application areas would further enhance the benchmark’s future-facing value and adaptability.

questions:
- Clarification on GPT-4o’s Role in Data Generation: Could you elaborate on how GPT-4o’s limitations in multi-image reasoning are managed during data generation? Specifically:

- How do you ensure that GPT-4o’s generated questions and answer choices are unbiased and accurately reflect the task complexity, given that it might have inherent reasoning limitations?

- To what extent was human verification involved, especially in preventing GPT-4o's possible biases from influencing the multiple-choice options? Further clarification on this process would strengthen confidence in the benchmark’s objectivity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730578430842

thread_type:
official_review

<thread_object>
creator: reviewer_avps

<content>
summary:
This work introduces MMIU, a benchmark to evaluate a LVLM's capability of understanding information when given multiple images simultaneously. MMIU covers a wide range of established cognitive tasks on semantics and compositionality in fine categories.

strengths:
This work offers meticulous details to the design of the benchmark. The authors have shown great effort in the data curation process while following a logical main plot line. I particularly find the results that show the LVLMs pre-trained with only single-image content may outperform their multi-image counterparts of great research insight.

weaknesses:
Please find my two major concerns down below:

1. My bigger issue about this paper is regarding the presentation of experimental results. **I find the benchmarking results are presented in a very cumbersome way.** Table 3 is the best example to illustrate this major weakness. The authors put in all the fine-category results by every single subtask. Although the comprehensiveness is appreciated, such overkill-style of presentation makes it hard to follow, so that we readers may not obtain a clear intuition about where and how well each baseline model thrives or lags against one another. This 'way-too-much-details' problem is also present in Figure 4 of the task maps. I highly recommend the authors show more summarized performance on the coarse categories, such the 3 big ones of Semantic/Spatial/Temporal in the main text, while leaving the finer categories in the Appendix.

2. **Potential position bias in the unanswerable set is unaddressed.** . MMIU's QA test cases are all multiple-choices. However, according to Zheng et al, 2023, the position bias is a significant issue when constructing benchmarks for evaluating LLMs (esp. on non-GPT methods). Basically speaking, the position bias will be prominent if the 'None of the above' option for the unanswerable set is always Option D, while known LLMs tend to favor the first option. This means models intend to give away deflated results shown in Table 12 for the unanswerable tasks. Have the authors tried shuffling the option orders? I am not seeing particular evidence for addressing the position bias in Section 3.2 as supposed to be.

Zheng et al, 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

questions:
Please find my concerns and suggestions in the Weakness section above.

Typo: Line 1389, Table reference is missing.

Update (Nov 23): Raising my ratings according to the authors' response
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702012647

thread_type:
official_review

<thread_object>
creator: reviewer_bvmz

<content>
summary:
This paper introduces the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess Large Vision-Language Models (LVLMs) across a wide range of multi-image tasks. MMIU encompasses seven types of multi-image relationships, including semantic, temporal, and spatial aspects, and consists of 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions. The authors evaluate 24 popular LVLMs, including both open-source and proprietary models, on the MMIU benchmark and identify significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Through multi-faceted analytical experiments, they provide valuable insights for future model and data improvements.

strengths:
1. The paper is well-organized and clearly presented, with a detailed background section, methodological descriptions, and experimental results. 
2. MMIU is the first benchmark to comprehensively evaluate the performance of LVLMs on multi-image tasks. It covers a wide range of multi-image relations and tasks, including semantic, temporal and spatial relations, as well as various image modalities.
3. The design of the MMIU allows it to be easily extended to new multi-image tasks and image modalities. This makes it a useful tool for future research.

weaknesses:
1. MMIU is large in scale, including 77,659 images and 11,698 multiple-choice questions. This may require a lot of computing resources to run experiments and perform analysis.
2. The paper shows that multi-graph understanding requires the capabilities of semantic, temporal and spatial relations. However, most current models do not model the latter two capabilities in the pre-training phase, which can lead to a lack of practical significance in the comparison of various models

questions:
1. The authors mention how they collected and standardized the data to create the MMIU. However, they do not provide detailed information about the data collection process, such as which data sources and data processing techniques they used. 
2. The strong capability in single-image understanding is the foundation of multi-image comprehension. As we know, the ability of single-image understanding comes from powerful language models, such as MMMU benchmark, can you give some performance of pure language models.
3. I hope to provide some model performance that add interleave data or video data in the pre-training or post-training stage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732188324341

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Update

comment:
We deeply appreciate the time and effort invested by all reviewers in providing valuable feedback on our manuscript. Their insightful comments and suggestions have guided substantial improvements to our work. In the revised manuscript, we have employed a color-coding system (**blue, red, green, orange, and purple**) to highlight modifications addressing the specific comments from **Reviewers bVMZ, avPS, 4Ca4, JjzH, and g41s**, respectively. The comprehensive revisions have significantly enhanced the paper's clarity, depth, and overall quality.

The key changes included:

- Introduction of our proposed testmini test set for efficient evaluation in **Section 3**
- Addition of LLM experimental results in **Section 4.1**
- Clarification of MMIU's shuffle mechanism during construction to avoid position bias in **Section 4.1**
- Replacement of **Table 3** with a focused performance analysis across 7 types of image relationships, with the comprehensive task performance table moved to the appendix
- Enhancement of **Figure 4** by separating the TaskMap and model performance heatmap visualizations, with TaskMap relocated to the appendix for better focus
- Documentation of the MMIU data collection process in **Appendix A.4**, including metadata formats and distractor generation techniques
- Analysis of MMIU's independence through comparison with other benchmarks in **Appendix A.4**
- Expansion of experimental analyses in **Appendix B.3**, covering:
  - Performance analysis of testmini
  - Visual signal requirements in MMIU questions
  - Architectural impact on multi-image performance
  - Training stage effects of multi-image data introduction
  - Discussion of MMIU's practical applications
- Detailed error analysis across models and image relationships in **Appendix B.4**
- Addition of task cluster descriptions in **Appendix C Table 20**
- Analysis of the revised task map in **Appendix C.2**
- Inclusion of MMIU test overhead tables and analysis in **Appendix D**



These revisions have addressed the reviewers' concerns while strengthening the paper's contributions and methodological rigor. We believe these changes have markedly improved the manuscript's quality and clarity. We sincerely thank the reviewers for their constructive feedback that has helped shape this improved version of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a benchmark designed to evaluate VLMs across a wide range of multi-image tasks, including 7 types of multi-image relationships, 52 tasks, and 11K meticulously curated multiple-choice questions. With the proposed benchmark, the paper evaluates more than 30 existing VLM models, and analyzes their strengths and weaknesses in multi-image tasks.

MMIU provides a useful and comprehensive benchmark for multi-image understanding. Multi-image understanding is a very important capability for modern VLMs, but the corresponding comprehensive benchmarks are limited, especially the evaluation across diverse tasks and scenarios, e.g., semantic, temporal and spatial relations.

The reviews pointed out weaknesses in insufficient analysis, comparison, and reliability.
The authors responded to each point in details.
The final reviewers are all positive about the paper, with final scores of 6, 6, 6, 6, 6.

I recommend accepting this paper because the carefully designed benchmark for multi-image understanding meets current demands and the extensive analysis on a variety of VLM models is insightful and offers valuable insights into future research.