MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

Authors: Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, Muhao Chen
Primary area: datasets and benchmarks
Keywords: multimodal llm, multi-image understanding, benchmark, robustness

Abstract:
We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729403020111

thread_type:
official_review

<thread_object>
creator: reviewer_1gpk

<content>
summary:
The paper introduces MuirBench mainly focusing on robust multi-image understanding capabilities of VLMs. The benchmark covers 12 multi-image understanding tasks and introduces unanswerable data. 20 recent VLMs are tested on the benchmark and several related analyses are provided.

strengths:
1. The MuirBench covers 12 multi-image understanding tasks, involving diverse scenarios. It is a comprehensive benchmark containing 11,264 images and 2,600 multiple-choice questions.
2. The paper conducts detailed testing on 20 recent VLMs, revealing the limitations of these models in multi-image understanding and the performance gap compared to single-image input.
3. The analyses of unanswerable-type questions, image positioning, and error analysis of GPT-4o are impressive.

weaknesses:
1. The paper only contributes data without offering any insightful analysis of model limitations and lacks novelty and contribution. It does not give people any new knowledge, and people do not feel any insight after reading it.
2. Authors overlooked some of the newer VLMs, such as Gemini 1.5 Pro (Feb. 2024), InternVL1.5-Chat (Apr. 2024) and Claude 3.5 Sonnet (Jun. 2024)
3. Although some limitations of the models in multi-image understanding tasks are shown, there is no discussion on the architectural limitations of these models, why these VLMs underperformed, why certain models perform poorly on different tasks or how models’ architectures could be adapted for better multi-image understanding. Furthermore, the suggestions for future improvements are rather general and lack detailed explanations of specific technical approaches for new VLMs. To sum up, the analysis part in this paper is superficial, more insight is required. 
4. The difficulty of question among different tasks is imbalance.
5. It seems that unanswerable questions have limited practical applications in real-world scenarios.
6. Some of the questions in the benchmark do not need to be presented in multiple images, such as question “Counting” and “Image-Text Matching” in Figure 6. As a questioner, I would consider inputting images one by one to ensure accuracy. A benchmark needs to focus more on practical application value rather than just creating a question that is difficult for VLMs.

questions:
1. What specific shortcomings of the VLMs’ capabilities do your benchmark actually reveal?
2. Authors are encouraged to provide results on some of the latest models, such as InternLM-XComposer2.5-7B (2024/07/03), the InternVL2 series (2024/07/04), and the Qwen2-VL series (2024/08/30). While the results of these models are not mandatory under the guidelines, considering the super-fast advancements in VLMs this year. Could you please include results from some of the aforementioned models to highlight the performance of the latest generation of VLMs in multi-image understanding?
3. The pairwise design of answerable and unanswerable questions seems to create an artificial scenario. Unanswerable questions are more ambiguous and complex. How does your dataset account for the nuance of real-world multi-image understanding?
4. Line 311 mentions "concatenate the images to constitute one input." How exactly are the images concatenated? Could it be that the concatenation method affects the model's performance? What are the results of using different concatenation methods?
5. Please refer to Weaknesses for more potential questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730302654951

thread_type:
official_review

<thread_object>
creator: reviewer_qrsh

<content>
summary:
The paper proposes MuirBench, a benchmark that comprehensively measures LVLM's multi-image understanding capabilities, including multiple image relationships and multiple task types, and conducts a detailed evaluation and analysis on MuirBench.

strengths:
1. The author proposed MuirBench, which is a comprehensive enough benchmark to measure the multi-graph understanding ability of the model
2. The author conducted a relatively detailed evaluation and discussion, which can provide some insights for the community

weaknesses:
1. The first picture of Figure5 seems to be ambiguous: the correct order can be washing dishes and then opening the cabinet, but the reordered order can be interpreted as washing dishes, opening the cabinet and then closing it, depending on the frequency and method of frame extraction. When the number of pictures is small, reordering does not necessarily make the problem unanswerable. (Just like when looking for patterns, if there is not enough prior information, there are many ways to explain the pattern)

2. Unanswerable setting experiment may be ambiguous: Imagine such a scene: when a poor student faces a difficult question, he does not understand the question at all and cannot make a choice. At this time, the student may choose randomly or refuse to answer because he thinks he lacks the necessary information (the large model also has the phenomenon of complete question information, but refuses to answer due to insufficient ability [1]). If we change the question information to make the question unanswerable, and add unanswerable options. At this time, facing the unanswerable option, the poor student may directly choose the unanswerable option, resulting in "cheating". However, we cannot assume that the student understands the question. Similarly, the author points out that most models have poor multi-image understanding performance, which can be compared to the poor students mentioned above, so I have doubts about the settings and results of unanswerable.

3. There are problems with the settings of Multi-Image Input Models: I don’t understand the sentense: “the models that do not support multiple images as input” in 310. For the “SIngle-Image Input Models" defined by the author here, why can’t we concat the image tokens and then input them to the subsequent modules? Like InternVL1.5-chat [2], GLM-4V-9b. It has not been trained with multi-image data, but it can still perform multi-image QA through the above operations. Or should the author’s definition of “Multi-Image Input Models” here be models trained with multi-image data?

4. Although the author used the method of directly concat input images to test "single-image input models" here, this will result in too low image resolution, which may affect the performance of these models. The author should compare the effect of testing the "single-image input models" defined here using concat visual tokens. For specific methods, please refer to the test code provided by InternVL-Chat-V1-5 on HuggingFace (`pixel_values ​​= torch.stack(pixel_values)`). I think it is a more comprehensive way to test the "single-image input models" using both concat image and concat image token.

5. The author lacks some more advanced open-sourced models for evaluation, such as LLava—interleave [3], GLM-4V-9b [4], etc.


[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
[2] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
[3] LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
[4] CogVLM: Visual Expert for Pretrained Language Models

questions:
Please refer weaknesses for details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681340707

thread_type:
official_review

<thread_object>
creator: reviewer_fmcn

<content>
summary:
This paper proposed a new benchmark aims at evaluating the performance of multi-modal language models for handling multiple image inputs.
The benchmark is designed following two key points, comprehensive coverage of tasks and robust evaluation.
For the comprehensive coverage of tasks, the paper seeks to leverage existings datasets from multiple sources as well as sourcing new data.
For the robust evaluation, the benchmark create unanswerable questions paired with each question.

Evaluation on the proposed benchmark demonstrates a gap of current multi-modal language models in understand multiple image inputs compared with human.

strengths:
1. I think the paper is tackling an interesting and relativly novel setting, that is in evaluating the reasoning abilities of models across multiple inputs.
2. The experimental results demonstrate that there are indeed a gap for current models, and a possible direction for future research to be done.
3. The paper tested a range of state-of-the-art models.

weaknesses:
1. The description of how the performance on unanswerable questions are measure is not very clear to me. Is "this question is unanswerable" part of the option or the evaluation just expect the model to give this?
2. Seems like the benchmark is unevenly distributed across different questions types and image types.
3. The benchmark consists of 10 different relations for multi-image, however, it is unclear to me how these 10 relations are chosen.

questions:
1. I would like to know how the 10 relations are chosen, and how they can represent the use cases of multi-image understanding.
2. How is the performance on unanswerable questions measured.
3. The benchmark is imbalanced in terms of the question types and image types, how can one obtain one single score that can reflect the overall ability of the model to understand multiple images?
4. In L146, it is claimed that MIRB is released later than this paper. I'm wondering how is this claim supported? Given that this paper is still underreview.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700377738

thread_type:
official_review

<thread_object>
creator: reviewer_yzea

<content>
summary:
This paper presents a new evaluation benchmark, dubbed MuirBench, which focuses on multi-image understanding capabilities of multimodal LLMs. Specifically, MuirBench contains 11,264 images and 2,600 multiple-choice questions, covering diverse multi-image relations. In experiments, the authors have evaluated 20 recent multi-modal LLMs, revealing that the best-performing models still struggled in multi-image understanding. Besides, the authors have also conducted detailed analyses, demonstrating possible reasons for the unsatisfying performance of multimodal LLMs.

strengths:
- The evaluation aspect about multi-image understanding is novel and important for multimodal LLMs, which present significant contributions to the field.
- The proposed benchmark covers many types of multi-image relations, such as temporal, ordered-pages, or narrative relations, which are comprehensive to support solid evaluations for multimodal LLMs.
- The unanswerable counterpart of the benchmark presents a complementary perspective to assess multimodal LLMs fairly.

weaknesses:
- It would be better if the authors could provide detailed comparison with recent multi-image evaluations benchmarks, such as Milebench, in a summarized table. This can help better capture the unique characteristics of MuirBench.
- The evaluation on MuirBench seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources.

questions:
- Do the authors have plans to expand or adapt MuirBench to include new tasks or modalities as LVLM capabilities evolve?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731123393144

thread_type:
official_review

<thread_object>
creator: reviewer_iba5

<content>
summary:
This paper introduces MUIRBENCH, a new benchmark designed to evaluate the capabilities of multimodal large language models (LLMs) in multi-image understanding. MUIRBENCH comprises 11,264 images and 2,600 multiple-choice questions spanning 12 multi-image tasks and 10 types of multi-image relations. Each task is paired with unanswerable counterparts to test robustness. Evaluations show that prominent models, like GPT-4 and Gemini Pro, perform far below human accuracy, emphasizing the need for improved models capable of holistic multi-image comprehension.

strengths:
Let me first clarify that I am not an expert in evaluation dataset construction.
1) MUIRBENCH covers 12 different multi-image tasks and 10 image relationship categories to ensure the evaluation as comprehensive as possible.
2) Including answerable and unanswerable question variants can evaluate the robustness of models.
3) The image sources are expanded. In addition to existing multi-image data, medical, geographical, etc. are considered.
4) Human quality control and evaluation results are introduced to provide the evidence of faithfulness.

weaknesses:
Let me first clarify that I am not an expert in evaluation dataset construction.

1) Lack of data scale comparison: I am not clear about the data scale of related multimodal evaluation benchmarks. It is recommended to provide a table to compare existing single-image/multi-image evaluation benchmarks, including the number of samples, the number of tasks, the number of relations, the number of image domain, etc. At least it is necessary to explain clearly how much improvement the proposed dataset in this paper has compared with previous work. I am not sure whether 1,300 answerable questions, 12 tasks and 10 relations are a relatively large improvement.

2) Specific explanation of comprehensiveness: I know that this paper considers 12 tasks and 10 relations, which sounds like a number that is neither too many nor too few. It is not clear whether there is evidence or analysis to state that this dataset is comprehensive enough at least within a certain range.

3) Are the ground-truth answers reliable? The human quality control does not seem to mention the analysis of the reliability of the answer. It seems that there is only human quality control of questions and images. I am not sure whether humans will find it difficult to judge or have no correct answers when answering these questions, and whether there will be subjective differences that lead to inconsistent evaluations by multiple people, which will add additional uncertainty to model evaluation.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732857363060

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of strengths recognized by the reviewers

comment:
We appreciate the reviewers' insightful feedback. Below is a brief summary of strengths listed in their reviews:


- **Novelty and significance**: "novel and important, significant contributions to the field" (Yzea), "novel setting" (fMCn).


- **Comprehensive evaluation and insightful analysis**: "ensure the evaluation as comprehensive as possible" (iBA5), "comprehensive to support solid evaluations" (Yzea), "demonstrate gap and direction for future research" (fMCn), "comprehensive enough" (QRSh), "comprehensive benchmark, impressive analyses" (1gpk)


- **Robustness and Quality**: "provide the evidence of faithfulness" (iBA5), "assess multimodal LLMs fairly" (Yzea).
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings

The paper introduces MuirBench, a benchmark designed to evaluate the multi-image understanding capabilities of multimodal large language models (LLMs). MuirBench includes 11,264 images and 2,600 multiple-choice questions across 12 tasks and 10 types of image relations, with unanswerable counterparts to test robustness. Evaluations reveal that current models, such as GPT-4 and Gemini Pro, perform significantly below human accuracy, highlighting the need for improved models. All reviewers note the benchmark's comprehensive coverage and its potential to reveal gaps in current model capabilities.

(b) Strengths

Reviewer iBA5 appreciates the comprehensive coverage of tasks and relations, as well as the inclusion of unanswerable questions to test robustness. Yzea highlights the novel focus on multi-image understanding and the benchmark's comprehensive nature. fMCn commends the benchmark's ability to reveal gaps in current models and its potential to guide future research. QRSh notes the detailed evaluation and discussion provided, while 1gpk emphasizes the benchmark's extensive coverage and the insights gained from testing recent VLMs.

(c) Weaknesses

The reviewers identify several weaknesses. iBA5 suggests providing a data scale comparison with existing benchmarks and questions the reliability of ground-truth answers. Yzea notes the lack of detailed comparison with recent benchmarks and the high computational resources required. fMCn finds the description of unanswerable question evaluation unclear and questions the choice of image relations. QRSh raises concerns about the ambiguity of unanswerable settings and the limitations of multi-image input models. 1gpk criticizes the lack of insightful analysis and novelty, the imbalance in question difficulty, and the limited practical application of unanswerable questions.

(d) Decision Reasons

On balance, AC agrees with positive points raised by reviewers which outweigh the negative points raised the two reviewers who favor rejection. The decision to accept the paper is based on its novel approach to evaluating multi-image understanding in multimodal LLMs and the comprehensive nature of the benchmark. The benchmark's ability to highlight gaps in current models and guide future research is a significant contribution, as noted by reviewers iBA5, Yzea, and fMCn. While there are concerns about the lack of detailed analysis and the practical application of unanswerable questions, the overall strengths in innovation, coverage, and potential impact on the field outweigh these weaknesses. The paper's contributions to advancing the evaluation of multimodal LLMs make it a valuable addition to the conference.