MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Authors: Fei Wang
Xingyu Fu
James Y. Huang
Zekun Li
Qin Liu
Xiaogeng Liu
Mingyu Derek Ma
Nan Xu
Wenxuan Zhou
Kai Zhang
Tianyi Lorena Yan
Wenjie Jacky Mo
Hsiang-Hui Liu
Pan Lu
Chunyuan Li
Chaowei Xiao
Kai-Wei Chang
Dan Roth
Sheng Zhang
Hoifung Poon
Muhao Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal LLM
Multi-image Understanding
Benchmark
Robustness
Primary area: datasets and benchmarks
TL;DR: We introduce MUIRBENCH, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs.

Abstract: We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.

Published as a conference paper at ICLR 2025
MUIRBENCH: A COMPREHENSIVE BENCHMARK FOR ROBUST MULTI-IMAGE UNDERSTANDING
Fei Wangi* Xingyu Fu
James Y. Huang Zekun Li Qin Liu
Xiaogeng Liu
Mingyu Derek Ma
Nan Xu
Wenxuan Zhou Kai Zhang" Tianyi Yan
Wenjie Mo
Hsiang-Hui Liu
Pan Lu
Chunyuan Li Chaowei Xiao Kai-Wei Chang
Dan Roth
Sheng Zhang² Hoifung Poon
Muhao Chen
USC
UPenn
UMN
*UC Davis UU-MMadioon UCLA
TOSU Microsoff Research
A
Project page:
githu.. io/
Aitibutte Simbactt
Counting
Prape-Teot Matchigs
Which imgis connairs same aiject with Neme attributtr handness?
How luards with glanes une
What can be the tille this 2reventationn
Ganarachix Understandins
MUIRBENCH
Geeprehentive Mult-irage
Which exhibits the building?
Which the
Sccoe Undoertrodde
Dagran Undersandiis
Cartoon Understardns
object
What &s heppering ont
Figure The MUIRBENCH Benchmark. MUIRBENCI contains 11,264 images and 2,600 multiple- choice questions, providing robust evaluation 12 multi-image understanding tasks. Each example comes from one task in MUIRBENCH, presenting diverse multi-ing multi-image relations.
ABSTRACT
We introduce MUIRBENCH, comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMMs. MUIRBENCE con- sists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MUIR ENCHI is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve BUICHENCH, achieving 68.0% and 49.3% accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the im- portance MUIRBENCH in encouraging the community develop multimodal LLMs that can look beyond single image, suggesting potential pathways for future improvements.
Equal leadership. Correspondance <fwang598 Srrang598@usc.eedd @usc.edu; xingyuf2@seas.upom
1 Equal contribution; alphahetic order.

---

Published as a conference paper at ICLR 2025
Single-Image Task Majority
Limited Image Relations
Answerable Instances Only
Previous Benchmarks
Q. How many apples?
Q: mood?
Sinilar Objects Only
Q:1s the number hftits equai to the number of bugs?
Diverse Multi-Image Tasks
Various Image Relations fe Tempora//Miitslcwl
Robust Evaiuation with Unanswerable Instances
Ies. Georaapii /hoaaaaammm Understanding)
Ours
Q: Which mape Cover the same reg on?
Q: correct slidd
Reor der park car.
Q Which shows the effect of sad-07
Which two show the some building?
Q: Reonder to park the red
Figure 2: Compared with previous benchmarks, MUIRBENCH has several novel features: (I) evaluates on a comprehensive range of 12 multi-image understanding abilities, e.g. geographic understanding and diagram understanding as introduced in $3 while prior benchmarks generally contain single-image questions. (2) It contains 10 diverse multi-image relations, e.g. narrative and
complementary as discussed in $3 (3) It provides a robust evaluation on models by introducing
unanswerable instance variants. The samples of previous benchmarks are from
(Liu et al. 2023c
Suhr et al. 2019; Jiang et al. 2024a).
INTRODUCTION
The proverb "a picture is worth a thousand words" is often cited to emphasize the richness of visual information hidden in one image (Gropper] 1963 Hibbing & Rankin- Erickson 2003). However, an image is only a single projection of the real world captured from a specific angle at a specific moment in time (Hays & Efros 2008). In contrast, humans naturally observe multiple images multiple pieces of such projections from discrete moments under various scenes - to perceive and holistic part. Humans excel synthesizing information from multiple image sources, whether it involves telling stories from a series of cartoon images (Cohn et al. 2017] Li al., 2023), drawing comparisons among multiple charts and diagrams to infer holistic new insights Massy et al., 2022), learning from diverse visual experiences such as online lesson slides to adopt new skills (Nouri & Shahid, 2005), predicting future event actions from past screenshots (Oh) et al., 2015: Finn et al. 2016)). or conducting temporal reasoning based on nuanced differences between photographs (Fu et al. 2022). Moreover, multi-image multi-image input has the advantage of conveying visuospatial ideas directly combining multiple images of the same scene can reveal spatial relations or other more abstract relations in the world (Faugeras et al. 200T). Multi-image input also overcomes the limitations of resolution that single-iman -mmge input faces, allowing for better visual perception and understanding (Kawulok et al. 2019).
understand the world
As multimodal large language models (LLMs) OpenAl 2023; Team al. 2023 Liu et al. 2023b
2024a Alaarac et al. 2022 Bai et al. 2023 Wang et al. 2023 Dai et al. 2023 Lu et al. 2023b
Zhang et al. 2024b; Team 2024 Chen et 2023c Chaves al. 2024) have begun to show
superior performance across various single-image tasks, we now expect them to solve hard tasks that require an holistic understanding of multiple images. This work aims highlighting crucial aspects of multi-image understanding that have been overlooked when evaluating multimodal LLMs, and providing a comprehensive benchmark for robust Millt--image reasoning. As shown in Figure 2 current evaluations (Goyal et al.) [217] et al. 2023c Li et al. (023) Yue et al., 2023 Lu et al.
2024 Liu al. 2023a d) generally focus on single-image understanding. thereby neglecting the richer, more complex tasks of integrating and reasoning across multiple images. While many of these benchmarks have been popularized as the de facto evaluation measures for influential models like GPT-4-Turbo (OpenAl 2023) and Gemini-Pro (Team et al.) 2023). this oversight limits the potential of these models to conduct advanced-lever multimodal comprehension. Though some recent benchmarks start to include multi-image questions in evaluation (e.g., Mantis-ER Eval (Jiang (et al. (2024a) and BLINK (Fu et al.; 20244), they are far from being comprehensive in multi-image evaluation that involve multi-persectiver multi-relations and robustness concerns.
2

---

Published as a conference paper at ICLR 2025
In this paper, we introduce MUIRBENCH (MULTI-IMAGE UNDERSTANDING BENCHMARK), comprehensive benchmark designed to rigorously assess and evaluate -malt understanding by multimodal LLMs. MUIRBENCH encompasses 11,264 images and 2.600 multiple-choice ques- tions spanning across 12 distinctive multi-image understanding tasks, e.g. g.viuulal retrieval, cartoon understanding, and attribute similarity, etc.. As illustrated in Figure 1] there can be multiple images interleaved in the contexts or questions, or presented as choices in our benchmark. Instances in MUIRBENCH also contain diverse kinds of multi-image relations, e.g. temporal, ordered-pages, or narrative relations, etc. as shown in Figure 4 The questions and choices are either derived from the datasets, or manually written by experts. Additionally, MUIR adopts a pairwise design approach, where each question-answer -nswering instance is paired with expert-annotated unanswerable counterpart (Rajpurkar et al. 2018) featuring minimal differences following Figuree 5| This design ensures reliable assessment of multimodal LLMs,, mitigating the risk of achieving correct answers through vision or language shortcuts. We also include various fine- grained expert annotated labels such image positions and image types in MUICBENCH, to facilitate detailed model analysis.
We conduct a comprehensive evaluation on MUIRBENCE using 20 multimodal LLMs of various sizes, including models that accept multi-image inputs and those originally designed for single- image inputs. Experimental results underscore the current limitations of even the most influential multimodal LLMs, e.g. GPT-4o and Gemini Pro, in handling multi-image scenarios. For instance, GPT-4o and Gemini Pro achieve mere 68.0% and 49.3% of accuracy respectively, which are 25.1 and 43.8% 43.8% lower than human performance. We also show that multimodal LLMs perform much worse on unanswerable questions than their answerable counterparts, with GPT-4o and Gemini Pro exhibiting accuracy gaps of 26.8% and 21.5%. Furthermore, multimodal LLMs trained solely on single images demonstrate impaired generalization to multi-image contexts. These findings highlight the significance of MUIRBENCH in driving the development of multimodal LLMs transcending single-image limitations. We believe MUIRBENCH can serve an effective testbed for holistic multi- image understanding, encouraging the community to cultivate models with a more comprehensive and integrated understanding of the visual world.
2
RELATED WORK
2.1 MULTIMODAL UNDERSTANDING BENCIIARRSS
A number of recent benchmarks have been developed to comprehensively assess the multimodal understanding and reasoning capabilities of multimodal language models (LLMs) (Lu et al. 202 la
Li et al. 2023; Liu et al. 2023c Lu et al. 2024 Yue et al., 2023 Zhang et al.) 2024c Ying ct al. 2024; Wu & Xie 2023). However, most of these benchmarks primarily focus on single image
2022
scenarios. While some benchmarks include multi-image examples (Lu et al. 2024 Yue et al. 2023
Fuu et al. 2024 Ying et al. 2024) Wang et al., 2024b) Zhao et al.) 2024). they typically require limited aspects of capacities (e.g.. image comparison for Vista) and do not provide a comprehensive assessment of multimodal LLMs in multi-image scenarios. While some benchmarks feature video understanding (Grauman et al. 2022; Maaz et al. 2023 in-context learning (Shukor et al.) 2023 Jiang et al. 2024b), the assessed capabilities are fundamentally different from multi-image understanding Video understanding focuses continuous streams of frames capturing dynamic changes over time, while in-context learning focuses on task adaptation using few-shot examples. In contrast, multi-image understanding challenges models to integrate and analyze spatial and contextual cues from varied perspectives, settings, and moments, thereby simulating the way humans process information from multiple visual sources. Recently, there have been dedicated efforts to assess multimodal LLMs in multi-image scenarios. For example, MANTIS-ER a annotaaed benchmark comprising 207 examples for multi-image reasoning, such as size perceptions and weight comparisons. DEMON (Li et al. (024c) evaluates whether multimodal L.MSs can follow zero-shot demonstrative instructions. MileBench (Song et al,, 2024) assesses multimodal LLMs' performance under long-context scenarios. However, these benchmarks either still focus on limited multi-image relations and reasoning processes or lack of controlled and robust evaluation. In contrast, MUIRBENCE provides comprchensive assessment of multimodal LLMs, covering broader range of multi-image capacities. We provide a detailed comparison between MUIRBENCE and related benchmarks in Appendix {B]
(Jiang et al.) (024a) is
3

---

Published as a conference paper at ICLR 2025
464
Mrree Tou Metching
Temporal
Complneentaly
196
Ordeing
Visual
Groundigg
Understanting
Scone
Understanding
Action
164
Cropoed/Zoomelt
Bimilartty
340
Differnnce Spotting
Counting
1456
Understandingg
Cartoon
Visual Netrieval
Indopendent
9%
Dieeram
Understersii eedeenteenninn
Geng Congraztie
Inderttndiig
Attrouute
78
292
Overall Siit
11%
Oboject Muttite-MutiiWeew
11%
338
Ordered Pages
Nonathve
100
13%
3%
Data distribution by tasks in MUIRBENCH. More details are in $
Figure 3:
Figure 4: Data distribution by multi relation categories. More details are in $
2.2 MULTIMODAL LARGE LANGUAGE MODELS
Inspired by the remarkable achievements in recent LLMs (Brown et al. 2020 OpenAl 2023 Touvron
ct al. 2023 Zheng et al., 2024 Team 2023b), a series of studies have begun exploring multimodal
LLLMs that concurrently interpret visual and linguistic information. However, most of early
multimodal LLMs are trained on single-image datasets and overlook the complicated tasks of multi-
image understanding (Liu et al., 2023b; Dai et al. 2023 Chen et al.
2023b Wang et 2023)
although some of them process each image as multiple tiles (Liu et al.
2024a Chen et al.
2024a).
Recent work starts training multimodal LLMs on interleaved image-text corpus such as MMC4 (Zhu
et al. 2024) and OBELICS (Laurençon et al. 2024) for pretraining as well as
(Jingg
al.
2024a) for instruction tuning, which enables models generate texts given multiple images.
While some of these models, like Flamingo (Alayrac et al.) 2022), Idefics (Laurençon al. 2024),
Emu (Sun et al. 2023a), and VILA (Lin et al.) (023), have demonstrated in-context learning ca- pabilities, there still lack of evidence regarding their capabilities in understanding multiple images within independent instances. Although instruction tuned models such as Mantis (Jiang et al. 2024a) and Gur-o (OpenAI 2023) have shown to possess counting and comparison skills over multi-image inputs, their ability in understanding and reasoning over multiple images with different relations across diverse tasks, though critical, remain unexplored. Therefore, propose MUIRBENCH to conduct comprehensive evaluation and provide insights to further improve their capabilities in handling realistic multi-image tasks.
MUIRBENCH
Our benchmark is meticulously curated for comprehensively assessing multimodal LLMs' capa- bilities in holistic multi-image understanding. We introduce the overall design and key features of MUIRBENCH in $.1 and delve deep into the data curation process in $3.
3.1 BENCHMARK OVERVIEW
Focusing on multi-image understanding, MUIRBENCH consists of 11,264 images and 2,600 multiple- choice questions, with an average of 4.3 images per instance. In general, MUIRBENCE adheres to two key design principles, First, seeks to provide a comprehensive and holistic evaluation on multimodal LLMs multi-image understanding capabilities, by containing 12 diverse multi-image tasks covering distinctive litti--iage relation categories. Additional fine- tne-ggained labels such as input image positions and image types also included to support comprehensive analysis of models. Second, seeks to provide robust evaluation, following a pairwise design where each answerable instance is paired with unanswerable counterpart featuring minimal differences.
4

---

Published as a conference paper at ICLR 2025
Replace/Reorder /Reorder Images
Modify Questions
Change Options
Answerable
0
4.4
5.5
Which image shows the colony of Virginia?
Q: How many birds in total?
Unanswerable
A
B.2
C.
D.5
Q: What's the action next?
image colony California?
many birds total?
Figure 5: Three major strategies are used in MUIRBENCH create unanswerable instances from their answerable counterparts with minimal changes ($3.2). In the above examples, blue marks denote the original input in the answerable case, and red marks highlight the input in the unanswerable case.
Comprehensive Multi-Image Evaluation. MUIRBENCH provides an comprehensive assessment through 12 distinctive multi-image understanding tasks, with selected examples of each task shown in Figure 6 As illustrated in Figure 3 each task represents 2.5% 17.8% of the whole benchmark.
[ACTION UNDERSTANDINGL aims to evaluate the ability of models to understand continuous images chronological order and match with an action. [ATTRIBUTE SIMILARITY] aims to evaluate the ability of models to identify specific given attribute among multiple images. [CARTOON INDDRRSTANDING aims to evaluate the ability of models to understand stories conveyed in cartoon images. [COUNTING] aims to evaluate the ability of models to count the number of specific objects across multiple images. [DIAGRAM understand information conveyed in diagram images. [DIFFERENCE SPOTTING] aims evaluate the ability of models to identify differences across multiple images. [GEOGRAPHIC UNDERSTANDING) aims to evaluate the ability of models understand maps and reason upon geographic features. [IMAGE-TEX: MATCHING] aims to evaluate the ability of models to understand the meaning of text snippet and match with the corresponding visual content or vice versa. [ORDERING] aims to evaluate the ability of models to order series of images based on the textual description. [SCENE UNDERSTA NDING] aims to evaluate the ability of models understand scene comprised of multiple views from multiple surveillance images. [VISUAL GROUNDING] aims to evaluate the ability of models to ground a specific object and seek information about it within multiple images. [VISUAL RETRIEVAL] aims to evaluate the ability of models to retrieval images that contain the same building.
INDERSTANDING aims to evaluate the ability of models to
Additionally, MUIRBENCH includes images covering various categories multi-image relations, such narrative images conveying stories ideas, ordered pages of documents and slides providing collective insights, images forming a temporal sequence presenting events, and multiple views of objects 3D scenes offering complete vision, with the complete distribution shown in FFigure These relations are summarized based on prior studies and the data collected, which reflect the common focus of the research community. each instance ranges from two to nine, while the input positions of images can be the beginning of question, middle question, end of question, options, and mix of these positions. MUIRBENCE also exhibits various image types, including but not limited to slides, maps, medical images, drone/satellite images, animations, memes, graphics, and 3D views. The data diversity from these perspectives enhances the comprehensiveness of our benchmark. More details can be found in Appendix A
terms of image presentation, the number of images in
Robust Evaluation. Existing datasets primarily assess models' capabilities in solving answerable questions but overlook their ability to recognize what they do not know (Rajpurkar et al., 2018 Miyai et al. 2024)). In real-world scenarios, there is no guarantee that user queries are answerable. A reliable multimodal LLM should directly indicate when a query is unanswerable rather than providing an answer that is most likely to be correct. In light of this, we pair each answerable instance with an unanswerable counterpart, featuring minimal differences, to provide a more robust evaluation, simulating real-world scenarios. We adopt multiple strategies to manually design the unanswerable instances, with major strategies of image replacing or reordering, question modification, and option modification introduced in Figure More details can be found in Appendix A
5

---

Published as a conference paper at ICLR 2025
3.2 DATA COLLECTION
Answerable Data Collection. We invest our efforts in collecting multi-image multiple-choice question answering (MCQA) data covering various tasks and multi-image relations. Diverse data attributes enable fine-grained and diagnostic evaluation, while the multiple-choice format ensures deterministic results. To achieve this goal, we consider three sources of data, including existing datasets, dataset derivations, as well as newly collected data. Existing data (40.8%) come from GeneCIS (Vaze et 2023). SeedBench (Li et al.] [202)), and IconQA (Lu al. [02Tb). Derived data (21.7%) reformat data into MCQA format, using multiple strategies including question generation, option rewriting. and single-image QA combination, etc. upon instances from NLVR2 (Suhr et al. 2019) HallusionBench (Guan et al. 2023), ISVQA (Bansal et al. 2020), and MMBench (Liu et al. 2023c). New data (37.5%) address certain tasks (e.g. geographic understanding and visual retrieval) that are underrepresented in the aforementioned collection to fulfill a more comprehensive evaluation. We manually create the question and choices for these data based images from the National Geologic Map Datahase [1 University-1622 (Zheng et al. 2020 2023), PubMed papers" and SciDuet slides (Sun al. 2021). Details about curation process and data sources can be found Appendix A
Unanswerable Data Collection. As shown in Figure 5| consider three strategies for modifying an answerable instance to its unanswerable counterpart with minimal changes. We first replace or reorder some images to disrupt the question-image and image-image relations (24.2%). We also modify the question to make it incompatible with the images and options (35.3%). In addition, we replace options to create a scenario with no correct answer (40.5%). For each answerable instance, we apply one of these three strategies. More details can be found in Appendix A
Quality Control. We employ two types of quality control throughout the annotation process: automatic check with predefined rules, and a manual examination of each instance to filter out any low-quality data. The automatic check verifies valid instance format, answers, metadata values, and the coreference between image placeholders and images (ensuring no redundant image), as well as the accessibility of images. The manual examination is conducted by four experts working in this field, and filters out ambiguous queries, unclear images, and confusing instances.
4 EXPERIMENTS
In this section, we first describe the experimental setup and the baselines ($4.1). 'hen we present comprehensive evaluation of 20 recent multimodal L.MS ($4.2). We demonstrate that while humans can answer the questions with high accuracy, MUIRBENCH is challenging for existing models.
Finally, we conduct various analyses on multiple experiment settings, including sensitivity to various resolution and error analysis 44.3
4.1 EXPERIMENTAL SETUP
Multimodal LLMs:
We evaluate MUIRBENCH on 20 recent multimodal LLMs, including models designed for considering multi-image inputs and those originally designed for single-image inputs.
For multi-image input multimodal LLMs, we evaluate on GPT-4o, GPT-4-Turbo (OpenAl 2023).
Gemini Pro (Team et al. (202)). Mantis (Idefics2, clip-Ilama3, and Siglip-llmat, versions; 8B) (Jiang
al.
2024a). VILA (vl. I.v.I.-133 13BB (Lin al. 2023), Idefics (9B-Instruct and v2-8B) (Laurençon et al.
2024
Laurençon et al. 2024). Emu2 (Chat) (Sun et al. 2023a) and OpenFlamingo (v2-9B) ( (Awadalla
jet al.
2023). For single-image input multimodal LLMs, we evaluate on LLaVA (v1.5, NeXT,
intern
and xtuner versions, model size 7B, 13B, and 34B) (Liu et al. 2022bb 2023b 2024a Team
2023a
Contributors 2023b), Yi-VL-6B MiniGPT-4-v2 (Chen et al. 2023b), and Cog VLM Wang
et al. 2023). We refer the readers to Appendix C for more details.
Evaluation setup: We follow the standard setup as it is in VLMEvalKit (Contributors, 2023a),
where the temperature is set to 0 and retry is set to 10. For the models that do not support multiple
https::/ngmdb... usgs htwpo//soooo///mm////mmttttt
html
https: //ppmmeddpp/d....o/mm ncbi n min. htw.gggvv//// qcv/
More details are the official wehsite htthhhcps//// /wwp//ww.kkk 01 /

---

Published as a conference paper at ICLR 2025
Overall (2, 600)
Counting (231)
Action. (161)
Grounding. (84)
Matching. (164)
Onlering
Scene.
(64)
(186)
Choice
20.98
Human
93.15
94.
97.56
85.71
94.83
87.50
Mudti--magge-rren MLLMs
GPT-4o OOeenA
2023
68,00
49.15
44,51
36,90
86.85
23.44
71.51
GPT-4-Turb OpenAl
2023
62.31
42.31
39.63
53.57
80.39
35.94
59.14
Gemini Pro Team ct
2025
49.35
28
35,98
28.57
66.59
12.50
59.14
Mantis M.ntts--BB.ae liehuss ling et
2014g
44.50
38 46
33.54
26.19
53.88
18.75
56.99
Mantis-8B Baati--A--lii-llnaaa
Jane ct al
2024a
37.38
29.06
36.59
21.43
43.32
18.75
$6.99
Jiang ml.
2024g
36.12
27.35
37.20
22.62
43.75
7.81
54,30
Idefics 9B Instruct (Laurençon et
2024
33.43
29.91
28.05
13.10
35.99
12.50
27.41
Emu2-Chat (37B) Sun et 20259
33.62
31.20
27.44
26.19
37.28
15.63
48.39
VII.AI.5-3BB.
10.94
Mdefic2-SBB
Laurençon
2024
Open/Tammngo- (Awadalla
2023
23.73
21.79
26.83
30.95
24.14
21.88
22.58
Single-Image-- Traimed MLLMs
HJVAA-eeT--488 (1.i 2024n
33.31
36.32
26.22
33.33
37,93
21.88
54,30
LLAVA-v1.5-7B-. 5-77-xxueee Contributors 2023b
33.23
26.92
25.61
23.81
22.84
4.69
39.78
Yi-VL-GBB
28. 69
28.21
27.44
28.57
25.00
7.81
38.71
Team 2023a
28.15
34.19
26.22
32.14
25.65
7.81
42.47
LLAVA-v1 Lia 20236
24.38
25.21
29.27
14.29
20.26
20.31
36.56
(Lu L1.VA-v1.5.7B
20236
23,46
23.08
27.44
14.29
23.49
23.44
34.95
LLAVA-vI 11144-11.--133-100 5-138-xtuner Contributors
2023b
21.69
23.08
23.17
16.67
21.98
14.06
47.85
CogVI.M Wang et
MiniGPT
18.75
Difference.
Cartoon. (78)
Diagram. (398)
Geographic, (100)
Attribute. (196)
Retrieval.
(340)
(292)
20.00
Multi-Image- Trained
GPT-4o OpenA1 2025
60.29
88.69
56.00
56.12
80.14
Gar---Turho OpenAA
2023
60.59
52.56
79.15
57.00
50.51
64.04
Gemini Pro Team
2023
45.29
47.44
64.82
48.00
41.33
43.84
Mantis-8B. Id-eeess Diang ot 20244
28.82
38.46
67.59
26.00
48.47
35.62
Mantis clip Ilama. Jing et all
2014s
24.12
43.59
$4.27
16.00
33.67
31.85
Mantis-8B Bi---BBBii---lamss-- Diang
20224
27.35
46.15
47.99
22,00
31.63
28.08
Idefics-9B- In-ssretttt [Larregonn et
2024
34.41
48.72
46.98
35.00
32.65
43.49
Emu2 Chat (37B) Sun st 2033
32.65
43.59
37.69
34,00
31.63
23.97
VILAI.5-13B (Lán al. 20233
24.71
30.77
42.71
31.00
24.49
30.14
Idetica--BB Laurengon
17.86
Open
2023
21,76
25.64
25.00
18.88
15.41
Single-Image- Trained MLLMs
LLaA--eeT:3-4B Liu ctal, 2024a
22,06
12,00
38.27
25.00
L.VAAvv..--B--xttt ( (Comribatoes 20236
33.53
29.49
44.72
26.00
38.78
47.60
Yi-VL-6B
25.59
50.00
35.68
17.00
34.18
22.60
M2- Team 20230
19.12
39,74
35.43
12.00
23.96
28.42
LLaVA-v1.5-13B Liu 20236
20.00
25.64
31.66
20.00
22.96
20.89
I.AVA.vl.557
al 2124h
20.00
24.36
25.13
20.00
22.96
19.86
LL.VA.v1.5-33B xttueerr Conitibulors 2025b
12.94
30.77
20.10
11.00
18.37
21.58
CogVLM Wang al
19.71
41.03
19.60
13.00
16.33
15.75
MiniGPT Chen
Table 1: Experiment results on MUIRBENCH. The first row shows task names and number of test
data. We see that most models perform similarly to random choice, and are far from humans ($4.3)
images as input, we concatenate the images to constitute one input. Following Lu et al. (2024), our prompt consists of four parts, the question, options, the hint indicating the answer format, and prefix indicating the beginning the answer. For images, we insert them into the text to form a coherent prompt. Following Yue al. (023). We rule-based automatic tool to extract the exact answer. We refer the readers to Appendix fDr more details on multi-image concatenation, visual prompting, answer extraction, and the human evaluation protocol.
7

---

Published as a conference paper at ICLR 2025
Action Understancing
Attribute Similarity
ayed video?
Which followng imaget object potted
same scere singts
[e] None
Flickin something
ethingg
66) Spi ling something onto something
(C)
(d) bescend something ento nomething
(e) None the choices provided
Cartoon Understanding
Counting
Whatis the main consent of thi cemicc sarp?
many painted design mages?
(e] Nicne of the choicss provided
(bs) duck drinking
Or) A
(b) Three
[c] Four
duck the sola while
dericing enjoying nightlie
(d) Two
3
(e) None of the cholces
Diagram Understanding
Difference Spotting
BC
diflorent
Whish
[a]
(b)
Armelt
You
(di None
Geographic Understancing
Image-Text Matching
Ammong these which decicts
ovedlpping geographic regione liee
image bananse!
[b]
(c)
(b)
Ordering
Scene Understanding
What's the color the car parked behind the black van the given images?
clothes Whet the of according the given consext?
(a) 3-1-2
(02113
choices
[a] (b) the choi oas provided
Red Wfrise
Visual Grounding
Visual Retrieval
McDenaldss my bought cing'
migito you please my McDonuld's
(d
(b) dolar
[<] None of options
(4) dolllar
$
olces
A
Figure 6: Qualitative results on BENCH. For each task, we show the ground-truth answer in blue, and choice of GPT-4o (OpenAI 2023 Gemini Pro (Team et al.) (023)., and Mantis-88P -BB Idefics2 (Jiang et al. 2024a).. Notice that the example cases slightly modified with change of word and image reduction for better illustration.
4.2
MAIN RESULTS
Overall performance: As shown Table 1] the average accuracies of the most advanced multimodal LLMs on MUIRBENCH are no better than 68%, which are still far from enabling satisfactory utility. The mean accuracies of open-source multimodal LLMs that have considered multi-images hover between 23.73% and 44.50% which fall behind from advanced proprietary LLMs. Notably, there obvious correlation between model sizes and performances, indicating the importance of training data and training processes in developing multimodal LLMs with multi-image understanding capabilities. For certain models and tasks, some results only par even below random
8

---

Published as a conference paper at ICLR 2025
Matding
Ordenig
Mousl Grounding
Answerable
Unanewerable
Understand
Undersanding
Acion
50
Counting
25
Candon Underzard Ing
Vissi Rosievel
GPT-40
GP & Turbo
Gemin Pro
Manti 48 ai-deeiaz
LLaVA-terT-MB
Bur dom
Hummnn
Undentsedding
Goographic Undenstanding
Sinlarty
GPT-4o
GPI-4-Turto
Gemini Pro
Figure 8: Model performance answerable and unanswerable instances. An obvious per- formance gap be observed between the two sets on all best-performing models.
8B-idefics2
Figure 7: Model performance by tasks.
guessing. We provide more in-depth model analyses in the following and in Appendix H| Moreover, the model performance by multi-image relation is presented in Table in appendix. We also report the task-level and relation-level macro accuracy in Appendix SG
In which multi-image tasks do multimodal LLMs show relative strengths and weaknesses? Figure 7| visualizes the accuracies of the best-performing models on MUIRBENCH. We observe that multimodal LLMs perform relatively better on image-text matching, visual retrieval, and diagram understanding. In contrast, multi-image ordering and visual grounding appear to be more challenging for these models, because these tasks require understanding the whole multi-image context and conducting more complicated reasoning processes across images and modalities afterwards.
Can models designed for single-image inputs perform multi-image tasks? In general, models accepting multi-image inputs(e.g.. g.. Mantis-8B), even with fewer parameters, perform better than single- image input multimodal LLMs (e.g.. -LAVA-NeXT-34BB This observation shows that generalizing from single-image training to multi-image inference is non-trivial. Reasonably, models benefit from understanding capabilities.
multi-image training data and learning processes to develop
4.3 ANALYSIS
the unanswerable set? Figure 8 compares performances on answerable and unanswerable sets for some best-performing models. All the studied models have severe performance drop when changing answerable instances unanswerable counterparts. A closer look of the eиrоr cases reveals that models often avoid abstention when facing unanswerable questions. These observations not only highlight the importance of assessing model behavior under more realistic setting, but also show that the pairwise design improves the reliability of MUIR
Do multimodal LLMs perform
Do image positions correlate with error rates? We analyze the error rates of varying input positions of images and report the performance GPT-4o, GeminiProVisioe shown in Figure [ the highest accuracy is achieved when images are positioned in options, while the highest error rate can be observed when images are in the middle of questions. This consistent trend across different models suggests that the position of images within question correlates with the error rate. The cause of higher error rates might that images in the middle or end a question may interrupt the flow of context processing, increasing complexity and thus reducing model performance. may also be attributed to the training process. These models may have seen less data with images in the middle during training.
and Mantis-8B- B--iis8B---deiis2.. As
Do unanswerable types correlate with error rates? We further analyze the error rates of varying unanswerable types and report the performance of the same three models in Figure 10 Results show that the error rate also correlates with the type of unanswerable instances. All the three models perform relatively better when we only change the questions to make it incompatible with original images and options. However, all models confused when the correct option is removed and fail

---

Published as a conference paper at ICLR 2025
0.9
beginiing of quession middle queelion and of quaaliion option 1
#chenge image
cchnnge question
@change
0.4
0.3
0.2
02
0.1
0.1
GPT-40
GeminnProVltion
Mant's Sannss-D-iiddee 0D-idefics2
GPT-40
Gemini Pro
Manti-688-deeiss2
Figure 9: Model performance by image positions. Figure 10: Performance by unanswerable types.
to choose "none of the other options" in this scenario. The performance unanswerable instances created by reordering or replacing images is divergent. GPT-4o performs much better than the other models in these cases.
Error analysis of GPT-40: We randomly sampled 100 error instances made by GPT-4o MUIRBENCH and meticulously examined them. The most common error category (26% of er- cases) is the failure of capturing details in images. The rest 20% of eπrrr due to inaccurate object counting or reasoning, followed by errors in logical reasoning (18%), identification of the same object in different scenes (14%), and inferring the intents implied by image sequences (12%).
Qualitative Results: Figure 6 presents some qualitative results, per task. A notable phenomenon that multimodal LLMs may hallucinate by attempting to find an erroneous option that appears be likely correct for an unanswerable question rather than abstaining (see examples for cartoon understanding, diagram understanding, visual grounding, and visual retrieval). This illustrates the obvious performance gap between answerable and unanswerable instances in Figure g
OPPORTUNITIES FOR MODEL IMPROVEMENT
Our findings highlight several opportunities for improving multimodal LLMs multi-image scenarios. Multimodal LLMs struggle with tasks like multi-image ordering and visual grounding, which require complex reasoning across images and modalities, suggesting the need for more sophisticated training processes and model architectures that better integrate structured information of multi-image input (Wang et al. 2022). Additionally, models show weaknesses understanding specific relations, such temporal relation, which could be addressed by training on more temporally annotated data. Our results also reveal that models benefit from multi-image training. Thus, expanding multi-image datasets and training on diverse image types, tasks, and relations could improve generalization (Li) et al. 22244 b). Similarly, the model's performance drop on certain image positions suggests that training data should include a broader range of image positions. Furthermore, multimodal L.LMs often fail to identify unanswerable questions, even though such questions are inevitable and common in the real world, pointing the need for better training recognizing insufficient context (Wang al., 2024a). Lastly, the challenge of inputting multiple images often requires compression or concatenation, which can lead to information loss or long-context issues. This highlights the need for new architectures that can process multiple images more effectively, preserving context and minimizing coreference challenges.
CONCLUSION
In this work, we introduced MUIRBENCH, comprehensive benehmark designed to provide a robust evaluation on the multi-image understanding capabilities of multimodal LLMMs. Experimental results of 20 multimodal LLMs, including the prominent models like GPT-4 and Gemini Pro, revealed substantial limitations their ability to handle multi-image scenarios. These models showed significant performance deficits compared to human accuracy and struggled more with unanswerable questions in MUIRBENCH. Our findings underscore the need for multimodal LLMs to transcend single-image limitations and achieve more holistic visual comprehension. MUIRBENCE provides rigorous framework for such assessments, encouraging the community to develop models that can effectively synthesize and reason across multiple visual sources.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
Fei Wang was supported by an Amazon ML PhD Fellowship. James Y. Huang was supported by gift fund from the USC Center Secure & Trusted ML. Qin Liu, Wenjio Mo and Muhao Chen were supported by the DARPA FoundSci Grant HR001 12490370, the NSF of the United States Grant ITE 2333736. Mingyu Derek Ma was supported by the DARPA FoundSci Grant HR001 12490370. Wenxuan Zhou was supported by the NSF of the United States Grant IIS 2105329. Tianyi Yan was supported by CURVE Fellowship. Xingyu Fu and Dan Roth were funded by ONR Contract N00014-23-1-2366 Special thanks to BLINK (Fu et al. 2024 authors, especially Wei-Chiu Ma, for providing the figure templates used in this paper.
ETHICS STATEMENT
Our work proposes MUIRBENCH, providing robust evaluation multi-image tasks using mul- timodal LLMs. While includes a comprehensive list of 12 tasks, all of them are in English and could induce bias on multilingual research settings. Also, if misused, the multimodal LLMs may be used to generate harmful vision and text artifacts. Nevertheless, this is not directly related to our research, and the data curate do not contain personally identifiable information offensive content. However, more researchers should be encouraged to get involved in research on the safety issues in a multimodal context.
REFERENCES
Jean-Baptias Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Korbllihh, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.013900 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
Ankan Bansal, Yuting Zhang, and Rama Chellappa. Visual question answering on image sets. In European Conference on Computer Vision, pp. 51-67, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 13:187711901, 2020.
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, et al. Training small multimodal models to bridge biomedical competency gap: A case study radiology imaging. arXiv preprint arXiv:2403.0800 08002, 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning, arXiv preprint arXiv:2310.09478. 10.09788. 2023a.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lingustiin tasks. arXiv preprint arXiv:2312. /4238, 2023c.
11

---

Published as a conference paper at ICLR 2025
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint arXiv:2404.16683
2024a.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. Internv12: Better than the best-expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024b.
Neil Cohn, Ryan Taylor, and Kaitlin Pederson. A picture is worth more words over time: Multi- modality and narrative structure across eight decades of american superhero comics. Multimodal Communication, 6(1):19-37, 2017.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. 2023a.
XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1lm. https:// com/
InternLM/ (attrnmMM/utuerrr 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning, 2023.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Olivier Faugeras, Quang-Tuee Tuan Luong, and Theo Papadopoulo. The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications. MIT press, 2001.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, and Dan Roth. There's s a time and place for reasoning beyond the image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404. /2390, 2024.
Yash Goyal, Tejas Khot, Douglas Summers- Stamerr--Sta, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
George L Gropper. Why is picture worth thousand words? Audio Visual Communication Review, pp. 75-95, 1963.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision- lisiun-language models. arXiv preprint arXiv:2310. 14566, 2023.
Jarvis Guo, Tuney Zheng, Yuelin Bai. Bo Li, Yubo Wang. King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint urXiv:2412.05237, 2024.
12

---

Published as a conference paper at ICLR 2025
James Hays and Alexei A Efros. Im2gps: estimating geographic information from single image. In 2008 ieee conference on computer vision and pattern recognition, pp. 1-8. IEEE, 2008
Anne Nielsen Hibbing and Joan L Rankin-Erickssnn Erickson. A picture is worth thousand words: Using visual images to improve comprehension for middle school struggling readers. The reading teacher, 56(8):758-2) 2003.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01488, 2024a.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. shot in-context learning in multimodal foundation models. arXiv preprini arXiv:2405.09798, 2024b.
Michal Kawulok, Pawel Benecki, Szymon Piechaczek, Krzysztof Hrynczenko, Daniel Kostrzewa, and Jakub Nalepa. Deep learning for multiple-image super-resolution. IEEE Geoscience and Remote Sensing Letters, 1766::066-10666 2019.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. Llava-onevision: Easy visual task transfer. arXiv preprint PrXiv:2408.033200 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.1709) 2023.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. seina-nextiinteleeavee Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024b.
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal lims to follow zero-shot demonstrative instructions. The Twelfth International Conference on Learning Representations, 2024c.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre- for visual language models, 2023.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-63 651, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:
//llava-vl. V/ http:/iooooooo///o///000
30- 1et--aaaaanneet..
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arX 2023c.
13

---

Published as a conference paper at ICLR 2025
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chumyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07889, 2023d.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint urXiv:23/2.17112 2023a.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17117 2023b.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan I Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021a.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang. Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Pan Lu. Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. The Twelfth International Conference on Learning Representations, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint NKKv:23366004224 2023.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/w1/2022 hwt.://.i.22......n---22. acl. URL https: http:///ccaanttooooooooo wrg. 2022. www.innnnnnnnn...nnnnn.
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint arXiv:2403. 20331, 2024.
Hossein Nouri and Abdus Shahid. The effect of powerpoint presentations student learning and attitudes. Global perspectives on accounting education, 2:53, 2005.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
OpenAl. Gpt-4 technical report, 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthicu Cord. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. In The Twelffh International Conference on Learning Representations, 2023.
Dingjie Song, Shumian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milmss in long context. arXiv preprint 4r0iv:2404. 38532, 2024.
14

---

Published as a conference paper at ICLR 2025
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual 2019.
Meeting of the Association for Computational Linguistics, pp.
Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, and Nancy XR Wang. D2s: Document-to- slide generation via query- based text summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1405-1418, 2021.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:23122 13286, 2023a.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023b.
Chameleon Team. Chameleon: Mixed-modaa early fusioo foundation models. 2024. URL https
//api. h
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Cean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
InterLMM Team. Internlm: A multillnguul language model with progressively enhanced capabilities. http//gtttbbcmmmmm//e www./mommmII.e htt//Inteennnn InternIneen 2023a.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023b. URL www. www.mcmsmmi.....m.. wttcco//llog /mpt 7l Accessed: 2023-05-05.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6862-6872, 2023.
Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. Robust (controlled) table-to-text genera- tion with structure-aware equivariance learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5037-5048, 2022.
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2024a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu. Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2023.
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jachong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multi- modal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529, 2024b.
Penghao Wu and Saining Xie. v*; Guided visual search core mechanism in multimodal llms. arXiv preprint 13 /4135, 2023.
Kaining Ying, Fanqing Meng, Jin Wang. Zhiqian Li, Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt--benhh comprehensive multimodal benchmark for evaluating large vasion-language models towards multitask agi. arXiv preprint 4404 16006, 2024.
15

---

Published as a conference paper at ICLR 2025
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplime multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang. Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mml. 5: Methods, analysis & insights from multimodal Ilm fine-tuning. arXiv preprint arXiv:2409.20560 2024a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In Internationa Conference on Learning Representations (ICLR), 2024b.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 1e.e22443.444000 14624, 2024c.
Bingehen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. Benchmarking multi-imag understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint arXiv:2406. 12742, 2024.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zhedong Zheng, Yunchao Wei, and Yi Yang. University-1652 A multi-view multi-source benchmark for drone-based geo-localization. ACM Multimedia, 2020.
Zhedong Zheng, Yujiao Shi, Tingyu Wang, Jun Liu, Jianwu Fang, Yunchao Wei, and Tat-seng Chua. Uavm' 2:: 2023 workshop on uavs multimedia: Capturing world from a new perspective. In Proceedings of the 3 st ACM International Conference on Multimedia, pp. 9715-9717, 2023.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2025
Appendices
Table of Contents
MUIRBENCH Details
17
A. Dataset Statistics
17
A.2 Dataset Curation Details
17
A.3 Mult--image Relations
19
A4
Human Evaluation Protoco]
20
B Comparison with Related Benchmarks
20
Baseline Models
21
D Experimental Setting Details
21
D.1 Model Prompts
21
D.2 Evaluation Tool
22
E Performance by Image Relation
23
F Performance Recent Models
23
G Levk- and Relation-Level Average Performance
23
H Additional Analysis
23
Limitation and Future Work
24
License
25
K
Accessibility of BENCH
25
K.
Dataset Documentation and Formal
25
K
2
Links and Maintenance Plan
26
K
3
Author Statement
26
4
Intended Uses
26
A MUIRBENCH DETAILS
A.1 DATASET STATISTICS
Figure II presents the overall statistics of MUIRBENCH. Figure 12 shows the data distribution by the type of images. MUIRBENCII covers wide range of image types, ranging from common types like photography to specific areas such as medical images, slides, and drone and satellite imagery. Figure 13 demonstrates the data distribution by the number of images. MUIRBENCH contains instances ranging from two images to nine images. Figure presents the data distribution by the position of images, including the beginning/middll/endd of question, options, and mix of these positions.
A.2 DATASET CURATION DETAILS
Answerable Data Collection. We invest efforts in collecting multi-image multiple-choice question answering (MCQA) data covering various tasks and multi-image relations. Diverse data attributes enable fine-grained and diagnostic evaluation, while the multiple-choice format ensures
17

---

Published as a conference paper at ICLR 2025
Video
View Animation 156
Total Instances
2600
Total Images
11264
Slides 13%
Visualization 1%
Total Tasks
12
Total Image Relations
10
11%
Answerable Instances
1300
existing data
531 (40.8%)
derived data
282 (21.7%)
new data
487 (37.5%)
Unanswerable Instances
1300
change image
315 (24.2%)
Graphics 16%
change question
459 (35.3%)
change option
526 (40.5%)
Photography 30%
Average image number
4,3
Average question length
21.6
Average option length 3.7
4%
Average option number
Other
Meme
Medical Image 9%
Figure 11: Statistics of MUIR BENCH.
Figure 12: Data distribution by type of images.
begia 14%
2%
middle 6%
mix 42%
3
16%
$
end 24%
24%
option 14%
Figure 13: Data distribution by number
Figure 14: Data distribution by position images.
of images.
deterministic results. To achieve this goal, we consider three sources of data, including existing datasets, dataset derivations, as well as newly collected data. Existing data come from datasets that focus on single aspect of multi-image reasoning, such GeneCIS (Vaze et al.) (023); and from datasets not specifically designed for the multi-image setting but containing a portion of multi-image data, such SeedBench (Li al. 2023) and IconQA (Lu al. 202Tb). For fair representation of each task, we sample up to 200 test examples from each dataset. This part contributes 40.80 of the data the final benchmark. Derived data reformat binary QA. such as NLVR2 (Suhr al.) 2019) and HallusionBench (Guan et al. 2023), into MCQA by modifying questions and options; or rewriting open QA, such as ISVQA (Bansal et 2020), into MCQA by adding options; and reconstructing single-image MCQA, such as MMBench (Liu et al., 2023c), into multi-image MCQA by replacing text options with corresponding images. Similar to those from existing datasets, we sample up to 200 test examples from each dataset. This part contributes 21.7% of the data in the final benchmark.
New data address certain tasks (e.g. geographic understanding), image relations (e.g. multiview), and types (e.g. medical images) remaining absent underrepresented in the aforementioned collection to fulfil a more comprehensive evaluation. We present four new datasets: HistoricalMap. UnivBuilding. PubMedMQA, and SciSlides. HistoricalMap requires identifying map patches covering the same regions collected from the National Geologic Map Database UnivBuilding requires identifying
html
18

---

Published as a conference paper at ICLR 2025
different views of the same building, or buildings from the same universities. The image data from University- (Zheng et al. 20200] 2023). PubMedMQA contains questions regarding the
subfigures from medical papers on PubMed SciSlides consists of questions regarding the slides for paper presentation collected from SciDuet (Sun et al. 2021). This part contributes 37.5% of the data
in the final benchmark.
Unanswerable Data Collection. As shown in Figure 5, consider three strategies for modifying an answerable instance to its unanswerable counterpart with minimal changes. We first replace or reorder some images disrupt the question-image and image-image relations. We also modify the question to make it incompatible with the images and options. In addition, we replace options to create scenario with no correct answer. For each answerable instance, we apply one of these three strategies. Among all the instances, 24.2% of the unanswerable instances are created by replacing or reordering the images in their answerable counterparts, 35.3% by modifying the questions, and 40.5% by changing the options. This step doubles the size of data, leading to a balanced distribution of answerable and unanswerable instances.
Metadata Annotation. Fine-grained metadata enable a diagnostic analysis of multimodal LLMs' weaknesses across various aspects. We annotate image relations, tasks, image types, number of images, and image positions for all instances. Among of these attributes, image relations are crucial factor that influences the model's capability for multi-image reasoning, yet they are rarely annotated in existing data. Therefore, we manually annotate them. Tasks and image types are partially annotated in existing data. We match the existing categories with our taxonomy and manually fill in any missing ones. Number of images and image positions are automatically detectable, so we conduct automatic annotation. The annotation interface is shown in Figure 16.
Quality Control. We employ two types of quality control throughout the annotation process: automatic check with predefined rules, and a manual examination of each instance to filter out any low-quality data. The automatic check verifies valid instance format, answers, metadata values, and the coreference between image placeholders and images (ensuring no redundant image), as well as the accessibility of images. The manual examination last filters out ambiguous queries, unclear images, and instances with other errors, resulting in the retention of 86.3% of instances.
A.3 MULTI-IMAGE RELATIONS
MUIRBENCH consists of 10 multi- iinlti-imagereee relations:
Temporal Relation: Images are related by time, showing progression or change over period, Examples include time-lapse photography or sequential frames from a video.
Ordered Pages: Images are part of a sequence, such pages in a book or slides in presentation, where the order conveys meaning.
Complementary Relation: Images that, when viewed together, provide additional information context that enhances the understanding of the subject. They complement each other by filling in gaps or providing different perspectives.
Cropped/Zoomed Images: One image is zoomed-in or cropped version of another, focusing a specific part of the original image to highlight details.
Narrative: A series of images that together tell a story or convey a sequence of events, much like comic strip storyboard.
Meeei--ulriview: Multiple images of the same scene taken from different angles or perspec- tives, providing a more comprehensive view of the scene.
Object-Mulriview: Images of the same object captured from various angles or perspectives, useful for understanding the object's 's's three-dimensional shape.
Overall Similarity: Images that are generally similar in content, style, or subject matter, but not necessarily identical. They might share common themes visual elements.
Partial Similarity: Images that share some, but not all, elements. They might have overlap- ping features or subjects but also contain distinct differences.
https::/pubbndd ncbi m1m. gcv/
19

---

Published as a conference paper at ICLR 2025
Independent Images: Images that do not have clear relation to each other. They not connected by time, sequence, context, or content.
A.4 HUMAN EVALUATION PROTOCOL
Two experts in domain conduct the human evaluation. Each answerable instance and its unanswerable counterparts are randomly assigned different experts ensuring a fair evaluation. The interface for human evaluation is shown in Figure 15
Counting
many other garments complete mitten pair image?
[IMAGE [IMAGE_ 2]
Options:
(B) Three
(C) One
(D) None the choices provided
Answer:
Images:
[IMAGE_]]
(IMAGE_2)
Figure 15: Human evaluation interface.
B COMPARISON WITH RELATED BENCHMARKS
In Table 2
compare MUTR BENCH with series of related benchmarks.
20

---

Published as a conference paper at ICLR 2025
Benchmark
#Instances #lmage #Relation #Task #Domain Precise Metric Unanswerable Cases
NLVR2
6967
N/
Yes
No
Blink
1902
144
N/A
Yes
No
BeleBBecch-Raaiitie
5197
2-109
2
10
N/A
Partial
No
Mementos
699
2-25
1
1
3
No
No
Mantis-Eval
217
2-5
N/A
N/A
N/A
Yes
No
MuirBench
2600
2-9
10
12
12
Yes
Yes
Table Comparison with related benchmarks.
C BASELINE MODELS
We evaluate MUIRBENCH on 20 recent multimodal LLMss including models designed for considering multi-image inputs and those originally designed for single-image inputs. For most model families, we use the latest and best-performing available checkpoint to date. The list of baseline models are as follows:
(i-ii) GPT-4 (OpenAl 2023) is known to be one of the best multimodal models to date. We test with two most up-to-date checkpoints: gpt-4-turbo and gpt-4o. Notice that the GPT-4 performance would change if this specific checkpoint gets updated. (iii) Gemini Pro (Team et al. 2023) is one of the most powerful multimodal models, and we use the Gemini 1.0 Pro Vision version of it. (iv-
vi) Mantis (Idefics2, clip-Ilama3, and lillip-llama3 versions; 8B) (Jiang et al. 2024a) is recent
strong model specifically finetuned multi-image related tasks. (vii) VILA (v1.5- 13-1BB) (Lin et al.)
2023), (viii-ix) Idefics (9B-Instruct and v2-8B) (Laurençon al. 2024 Laurençon et al. 22024) (x) Emu2 (Chat) (Sun et al., 2023a) and (xi) Open OpenFlamingo (v2-9B) (Awadalla et al. (023) are four
recent multimodal models that can take multiple images input. (xii-xvii) LLa VA (v1.5, NeXT.
internLM, and xtuner versions, model size 7B, 13B, and 34B) (Liu et al. 2022bb 2023b 2024a Team
2023a Contributors 2023b) included well. While they designed for single-image input, concatenate all the images in order. (xvii)) Yi- T-VLL-6B has shown great performance recently. (xix) MiniGPT-4-v2 (Chen al. 2023a) adapts EVA (Fang al. 2023) as visual backbone, LLaMA2-chat
(7B) Touvron et al.) 2023) as language model backbone, and designs a linear projection layer for visual understanding abilities. (xx) Cog VLM (Wang et al., 2023) adds a trainable visual expert module in the attention and FFN layers to bridge different modalities better. It uses EVA-CLIP (Sun
al. 2023b) vision encoder and Vicuna (Zheng et al.) 2024) as language backbone.
D EXPERIMENTAL SETTING DETAILS
D.1 MODEL PROMPTS
Following Lu et al. 2024 our prompt consists of four parts, the question, options, the hint indicating the answer format, and a prefix indicating the beginning of the answer. For images, we insert them into the text to form coherent prompt. The complete prompt is as follows:
Model Prompts
Question: (QUESTION)
Choices:
(A) [OPTION_A) _
(B) [OPTION B)
(C) [OPTION C)
(D) [OPTION
Hint: Please provide the correct option letter, such as A, B, C, D, directly.
Answer:
More details are at the official website at wtt. https:
eva ination/uuiiiii 1 http://ttaatitaaoooooooooe/oaaae 1e31 faa 7530800 2a
21

---

Published as a conference paper at ICLR 2025
Partial Sim.
Temporal
Complementary
Scene-Multiview Overall Sim.
gp14o
59.14
38.43
63.51
71.51
52.54
gpt4turbo
49.43
37.50
69.37
59.14
54.71
GeminiProVision
37.14
29.63
50.00
59.14
44.93
Contis-BB-Idefiess
48.00
29.63
39.19
56.99
39,49
Mantis-SB-siglippl-
32.86
30.09
32.43
54.30
30.07
Maniis-8B-Clip-llama3
31.14
31.94
29.28
56.99
27.17
Idefics2
16.57
23.61
24.77
56,45
24.64
Emu2-Chat
30.29
23.61
30.63
48.39
33.33
VILA-I.5-13b
27.43
23.61
29.28
56.45
28.26
ideficsl
31.43
23.61
30.63
27.42
31,88
DocnFlamingo-BB-vit-mmee
22.86
25.00
24.77
23.12
23.55
Ilavav11.-34b
36.57
25.00
21.62
54.30
34.06
llva-v1.5-70-xttne
30.86
19.91
27.03
39.78
35.14
Yi_ Yi_VL_6 VL_ 6B
30.86
22.22
27.48
38.71
31,52
llva-intermlm2-7h
32,29
21.76
22.97
42,47
22.10
llva_vl..__b
26.00
25.93
16.22
34.95
19.20
Haav_vv1.5_133
25.14
27.31
17.12
36.56
19.57
Ilva-v1.5-133-xtunee
24.86
20.83
13.51
47,85
15.94
cogylm-chat
18.86
23.15
18.47
41.40
19.20
Narrative Cropped/Zoomed Independent Ordered Pages Object-Multiview
gpt4o
51.28
88.69
82.05
68.97
80.14
gpt4turbo
52.56
79.15
77.35
65.23
64.04
GeminiPro Vision
47.44
64.82
58.12
53.16
43.84
Mantis-8B-Idefics2
38.46
67.59
46.58
31.90
35.62
Mantis-8B-sigee
Baanis-8--igglip-llaa siglip-llama3
46.15
47.99
37.18
30.75
28.08
Mantis-8B-clip-lle
43.59
54.27
33.76
36.21
31.85
Idefics2
39.74
25.38
18.38
33.33
17.12
Emu2-Chat
43.59
39,95
27.78
43.10
27.40
VIIA--155-3b
30.77
42.71
37.18
27.87
30.14
ideficsl
48.72
47.24
38.03
32.18
43.49
BeenFlaaingo----vil-nnnpp
26.92
31.91
21.79
22.99
15.75
llava-v 1.634bbbbbbb
42.31
38.19
38,46
27.30
25.00
1ava-vl.5-7b-xtuner
29.49
44.72
25.64
23.56
47.60
YL_VL_6B
50.00
35.68
24.36
18.97
22.60
llva-intermlm2-76
39.74
35.43
24.79
19.54
28.42
Ilava_v1.55 v1.5_7b
24.36
25.13
20.51
24.14
19.86
Ilava_v1.5 v1.5_131
25.64
31.66
20.09
22.13
19.52
Ilava-v1.5. 1.t.vl...-3bb-tueeee
32.05
20.10
22.22
14.66
21.58
cogvim-chat
41.03
19.60
21.37
13.51
15.07
Table
Performance by image relation.
D.2 EVALUATION TOOL
Following Yue et al. 2023 We use rule-based automatic tool to extract the exact answer. First, the tool detects if a valid option index appears in the model output. tool matches the output to the content of each option. If there still match, it will randomly select
no direct answer found, the
com http://th.....ooooooooooeeeeaaaa
7a 3ac of0 9955dbaa0bb
eva
22

---

Published as a conference paper at ICLR 2025
option the answer. When more than valid answer is detected, the tool will use the first one that appears as the final answer.
E PERFORMANCE BY IMAGE RELATION
In Table we report model performances by image relation.
F PERFORMANCE OF RECENT MODELS
In Table 4
further report the results of recent multimodal LLMs released after MUIRBENCH,
including LLaVA- Anee Vision (Li et al. 2024a), InternVL2 (Chen et al. 2024b), MM1.5 (Zhang et al.) 2024a), MAmmoTH-VL (Guo et al. 2024). and Gemini 1.5 Pro. Some of these works have already
used MUIRBENCH as one of the primary benchmarks. The results highlight how MUIRBENCE contributes to advancing the development of multimodal LLMs.
Model
Overall Score
VaaaVA-neeeee Vneiinnn-O
MaaVA-OneVision
41.8
LLaVA-One SANNV--OeVisiin221
54.8
VL2e44
26.8
LLtemBL2-8B
36.8
MM1.5-1 1BB
34.7
MM1.5-IB IB (MoE)
40,9
MM1.5-3B
44.3
MAmmoTH-VVL-SB
55.1
Gemini Pro (001)
64.9
Gemini 1.5 Pro (002)
62.0
Table 4: Results of recent multimodal LLMs. Partial results reported by corresponding papers.
G TASK-LEVEL AND RELATION-LEVEL AVERAGE PERFORMANCE
To obtain rebalanced results, one can also report the task-balanced performance (i.e., the macro average). We provide the reweighted performance Table 5] While the numbers differ than the overall scores (i.e., the micro average), the trends remain roughly the same.
model
H ADDITIONAL ANALYSIS
Can multi-image MILMS understand single-image input? We evaluated GPT-4o using both multiple images and a concatenated image as input for each instance. As shown in Table 6 GPT-4o performs worse when using concatenated image as input. This result is intuitive, is inherently designed to process multiple images separately. Concatenating images can lead to information loss and introduce coreference challenges between subimages and their textual mentions in the prompt.
GPT-4o
Can single-image MILMss understand multi-image input? We further evaluated LLaVA-NeXT- 34B under the same setting. I.LVV--NeTT-5 exhibits similar trend. Although primarily trained on single-image data, has been trained using the AnyRes technique-wwih into several smaller ones melowingg to transfer this capability to handle multiple images effectively. These results suggest that explicitly processing multiple images as input is beneficial, as reduces information loss and minimizes challenges in referencing images.
one large image is split
Can the benchmark solved by dense captioning? Following BLINK (Fu et al. 2024), we employ GPT-4o to caption each image, subsequently replacing the images with their captions inputs. The results, presented in Table 7 show a similar trend to our prior work MMMU, which is also a benchmark for image understanding. This study specifically illustrates the extent which details
23

---

Published as a conference paper at ICLR 2025
Task-Macro Avg
Relation- -aaooo Avg
GPT-4o
58.74
65.63
GPT-4-Turbo
56.24
60.85
Gemini Pro
43.51
48.82
Mantis-8B-Ideficss
39.40
43.35
Mantis-8B-clip clip-1lama
34.14
36,99
33.02
37.62
(defics-9B-Img
32.35
28.00
Emu2-Chat (37B)
32.47
34.81
VII.A1.513BB
30.46
33.37
Idefiex2-8B
26.65
35.46
OpenFlamingo-v2--9B
23,90
23.87
LLaVA-NeXT-34B
32.21
34.28
LLaVA-vl.5-7B-xtuer
30.31
32.37
Yi-VL-6B
28.40
30.24
LLaVA-intemLLM2-7B
27.26
28.95
LLaVA-v1.5-13B
23.92
23.63
LLaVA-vI.5-7B
23.25
24.47
LLaVA-vl.S-1BB-xuunee
21.80
23.36
CogVLM
21.47
23.17
Table 5: Macro-average performance by task and relation.
Model
Input Format
Overall Score
GPT-4o
multi-image 68.00
single-image
60.69
LLaVA-NeXT-34B
multi-image single-image
36.80
33.31
Table 6: Effect of input format.
beyond the provided captions are necessary to answer the questions. does not, however, reflect the quality of the benchmark.
Benchmark
Type
Input Modality
Score
MMBench
Multimodal Perception and Reasoning
text-only input (image caption)
80.8
multimodal input
75.1
BLINK
Visual Perception
text-only input (image caption)
36.0
multimodal input
51.1
MMMU
Multi-discipline Multimodal Understanding
text-only input (image caption)
47.2
multimodal input
56.8
MuirBench
Robust Multi- Multi-image Understanding
text-only input (image caption)
63.4
multimodal input
68.0
Table 7: Effect of input modality.
I LIMITATION AND FUTURE WORK
There are several limitations to this work. We focus our scope on 2D images, and future research can further extend the idea of work to 3D problems, and include more multi-image tasks and relation categories. We focus multiple-choice questions answering following widely used previous benchmark (Fu et al. 2024 Yue et al. (003). as this format ensures structured evaluation and clear criteria for correctness. Nevertheless, other question formats, such open-ended questions, also valuable to explore. We hope our work can guide future efforts in providing robust and faithful evaluation in multimodal benchmarks. In addition, our strategies of creating unanswerable instances,
24

---

Published as a conference paper at ICLR 2025
as in Figure 5 do not cover all strategies that be used create such instances. Also, focus our evaluations on multimodal LLMs. Future work could include more vision-language foundation models such Unified-IO (Lu et al. 2023a) and Chameleon (Team 2024).
LICENSE
We release our data under CC-BY 4.0 license. For specific instances follow their original licenses. The datasets we used and their licenses are as follows:
GeneCIS is released under the CC-BY-NC 4.0 license
SEED-Bench is released under the CC-BY-NC 4.0 license
IconQA is released under the CC BY-NC-SA license
NLVR2 is released under the CC-BY-4.0 license
12
HallusionBench is released under the BSD 3-Clause license
13
ISVQA annotation is released under the CC BY-NC-SA 2.0 license
14 We only use the
images from nuScenes, which released under the CC BY-NC-SA 4.0 license
MMBench is released under the Apache-2.0 license
16
National Geologic Map Database is free in the public domain
University-1652 is released under the MIT license
PubMed is free and public database, with open access articles under Creative Commons or similar license
SciDuet is released under the Apache 2.0 license with paper slides from ACL, ICML, and NeurlPS 20
K
ACCESSIBILITY OF MUIRBENCH
K. DATASET DOCUMENTATION AND FORMAT
The full documentation of MUIRBENCH is the project page athttpss//huggiggffaa co/ datasets Possess/MUn /KUIBENCC//OIRRENCH For each data entry in MUIRBENCII, it includes metadata of index (idx), task, question, options, answer, image relation, image type, images, and counterpart instance idx.
licensing
license
1
readme
25

---

Published as a conference paper at ICLR 2025
K.2 LINKS AND MAINTENANCE PLAN
22 MUIRBENCH is hosted on s where license and metadata are also available. We maintain our benchmark this page and will continually update it. The evaluation code and outputs will be provided to facilitate easy reproduction and analyses of the results in the paper.
K.3 AUTHOR STATEMENT
We confirm that we bear all responsibility in case of violation of rights during the collection of data on MUIRBENCH, ensuring accountability and commitment to maintaining ethical standards. We will take appropriate action when needed.
K.4 INTENDED USES
The dataset is for academic purposes only and not for commercial usage.
htttss/s/haugnnnfee..ce///ssssss http://aaaaatte///////////c////a R RB KNCH
htcpps///ooggnnnneeeece///////ap RHENCH MM ReNNCH/croissantt
26

---

Published as a conference paper at ICLR 2025
Question: Are you able to identify images that showcase the matching structure seen in [IMAGE_] 1/?
Options:
(A) [IMAGE_2 2]
(B) [IMAGE_3]
(G) [IMAGE_] 4]
(D) IN_AANSWER
Answer:
Edit Query/Opions/Answerrrngggg Paths
Images:
[MMGGE 1]
[IMAGE 2]
[IMAGE 3]
[IMAGE_4] 4]
Image Relation: Objeci-Muttvvew
Image Type: Photography
Task: Image Retrieval
Figure 16: Annotation interface.
27


Reviews:
[{"review_id": "wSPhsFRJL8", "reviewer": "Reviewer_iBA5", "summary": "This paper introduces MUIRBENCH, a new benchmark designed to evaluate the capabilities of multimodal large language models (LLMs) in multi-image understanding. MUIRBENCH comprises 11,264 images and 2,600 multiple-choice questions spanning 12 multi-image tasks and 10 types of multi-image relations. Each task is paired with unanswerable counterparts to test robustness. Evaluations show that prominent models, like GPT-4 and Gemini Pro, perform far below human accuracy, emphasizing the need for improved models capable of holistic multi-image comprehension.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Let me first clarify that I am not an expert in evaluation dataset construction.\n1) MUIRBENCH covers 12 different multi-image tasks and 10 image relationship categories to ensure the evaluation as comprehensive as possible.\n2) Including answerable and unanswerable question variants can evaluate the robustness of models.\n3) The image sources are expanded. In addition to existing multi-image data, medical, geographical, etc. are considered.\n4) Human quality control and evaluation results are introduced to provide the evidence of faithfulness.", "weaknesses": "Let me first clarify that I am not an expert in evaluation dataset construction.\n\n1) Lack of data scale comparison: I am not clear about the data scale of related multimodal evaluation benchmarks. It is recommended to provide a table to compare existing single-image/multi-image evaluation benchmarks, including the number of samples, the number of tasks, the number of relations, the number of image domain, etc. At least it is necessary to explain clearly how much improvement the proposed dataset in this paper has compared with previous work. I am not sure whether 1,300 answerable questions, 12 tasks and 10 relations are a relatively large improvement.\n\n2) Specific explanation of comprehensiveness: I know that this paper considers 12 tasks and 10 relations, which sounds like a number that is neither too many nor too few. It is not clear whether there is evidence or analysis to state that this dataset is comprehensive enough at least within a certain range.\n\n3) Are the ground-truth answers reliable? The human quality control does not seem to mention the analysis of the reliability of the answer. It seems that there is only human quality control of questions and images. I am not sure whether humans will find it difficult to judge or have no correct answers when answering these questions, and whether there will be subjective differences that lead to inconsistent evaluations by multiple people, which will add additional uncertainty to model evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpe1aMGnRz", "reviewer": "Reviewer_Yzea", "summary": "This paper presents a new evaluation benchmark, dubbed MuirBench, which focuses on multi-image understanding capabilities of multimodal LLMs. Specifically, MuirBench contains 11,264 images and 2,600 multiple-choice questions, covering diverse multi-image relations. In experiments, the authors have evaluated 20 recent multi-modal LLMs, revealing that the best-performing models still struggled in multi-image understanding. Besides, the authors have also conducted detailed analyses, demonstrating possible reasons for the unsatisfying performance of multimodal LLMs.", "questions": "- Do the authors have plans to expand or adapt MuirBench to include new tasks or modalities as LVLM capabilities evolve?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The evaluation aspect about multi-image understanding is novel and important for multimodal LLMs, which present significant contributions to the field.\n- The proposed benchmark covers many types of multi-image relations, such as temporal, ordered-pages, or narrative relations, which are comprehensive to support solid evaluations for multimodal LLMs.\n- The unanswerable counterpart of the benchmark presents a complementary perspective to assess multimodal LLMs fairly.", "weaknesses": "- It would be better if the authors could provide detailed comparison with recent multi-image evaluations benchmarks, such as Milebench, in a summarized table. This can help better capture the unique characteristics of MuirBench.\n- The evaluation on MuirBench seems to require lots of computation resources, which may limit the accessibility for researchers with fewer resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sT1aXWZfz", "reviewer": "Reviewer_fMCn", "summary": "This paper proposed a new benchmark aims at evaluating the performance of multi-modal language models for handling multiple image inputs.\nThe benchmark is designed following two key points, comprehensive coverage of tasks and robust evaluation.\nFor the comprehensive coverage of tasks, the paper seeks to leverage existings datasets from multiple sources as well as sourcing new data.\nFor the robust evaluation, the benchmark create unanswerable questions paired with each question.\n\nEvaluation on the proposed benchmark demonstrates a gap of current multi-modal language models in understand multiple image inputs compared with human.", "questions": "1. I would like to know how the 10 relations are chosen, and how they can represent the use cases of multi-image understanding.\n2. How is the performance on unanswerable questions measured.\n3. The benchmark is imbalanced in terms of the question types and image types, how can one obtain one single score that can reflect the overall ability of the model to understand multiple images?\n4. In L146, it is claimed that MIRB is released later than this paper. I'm wondering how is this claim supported? Given that this paper is still underreview.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think the paper is tackling an interesting and relativly novel setting, that is in evaluating the reasoning abilities of models across multiple inputs.\n2. The experimental results demonstrate that there are indeed a gap for current models, and a possible direction for future research to be done.\n3. The paper tested a range of state-of-the-art models.", "weaknesses": "1. The description of how the performance on unanswerable questions are measure is not very clear to me. Is \"this question is unanswerable\" part of the option or the evaluation just expect the model to give this?\n2. Seems like the benchmark is unevenly distributed across different questions types and image types.\n3. The benchmark consists of 10 different relations for multi-image, however, it is unclear to me how these 10 relations are chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fPaVhEEcF7", "reviewer": "Reviewer_QRSh", "summary": "The paper proposes MuirBench, a benchmark that comprehensively measures LVLM's multi-image understanding capabilities, including multiple image relationships and multiple task types, and conducts a detailed evaluation and analysis on MuirBench.", "questions": "Please refer weaknesses for details.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The author proposed MuirBench, which is a comprehensive enough benchmark to measure the multi-graph understanding ability of the model\n2. The author conducted a relatively detailed evaluation and discussion, which can provide some insights for the community", "weaknesses": "1. The first picture of Figure5 seems to be ambiguous: the correct order can be washing dishes and then opening the cabinet, but the reordered order can be interpreted as washing dishes, opening the cabinet and then closing it, depending on the frequency and method of frame extraction. When the number of pictures is small, reordering does not necessarily make the problem unanswerable. (Just like when looking for patterns, if there is not enough prior information, there are many ways to explain the pattern)\n\n2. Unanswerable setting experiment may be ambiguous: Imagine such a scene: when a poor student faces a difficult question, he does not understand the question at all and cannot make a choice. At this time, the student may choose randomly or refuse to answer because he thinks he lacks the necessary information (the large model also has the phenomenon of complete question information, but refuses to answer due to insufficient ability [1]). If we change the question information to make the question unanswerable, and add unanswerable options. At this time, facing the unanswerable option, the poor student may directly choose the unanswerable option, resulting in \"cheating\". However, we cannot assume that the student understands the question. Similarly, the author points out that most models have poor multi-image understanding performance, which can be compared to the poor students mentioned above, so I have doubts about the settings and results of unanswerable.\n\n3. There are problems with the settings of Multi-Image Input Models: I don’t understand the sentense: “the models that do not support multiple images as input” in 310. For the “SIngle-Image Input Models\" defined by the author here, why can’t we concat the image tokens and then input them to the subsequent modules? Like InternVL1.5-chat [2], GLM-4V-9b. It has not been trained with multi-image data, but it can still perform multi-image QA through the above operations. Or should the author’s definition of “Multi-Image Input Models” here be models trained with multi-image data?\n\n4. Although the author used the method of directly concat input images to test \"single-image input models\" here, this will result in too low image resolution, which may affect the performance of these models. The author should compare the effect of testing the \"single-image input models\" defined here using concat visual tokens. For specific methods, please refer to the test code provided by InternVL-Chat-V1-5 on HuggingFace (`pixel_values ​​= torch.stack(pixel_values)`). I think it is a more comprehensive way to test the \"single-image input models\" using both concat image and concat image token.\n\n5. The author lacks some more advanced open-sourced models for evaluation, such as LLava—interleave [3], GLM-4V-9b [4], etc.\n\n\n[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs\n[2] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites\n[3] LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models\n[4] CogVLM: Visual Expert for Pretrained Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zXpdZrRKLy", "reviewer": "Reviewer_1gpk", "summary": "The paper introduces MuirBench mainly focusing on robust multi-image understanding capabilities of VLMs. The benchmark covers 12 multi-image understanding tasks and introduces unanswerable data. 20 recent VLMs are tested on the benchmark and several related analyses are provided.", "questions": "1. What specific shortcomings of the VLMs’ capabilities do your benchmark actually reveal?\n2. Authors are encouraged to provide results on some of the latest models, such as InternLM-XComposer2.5-7B (2024/07/03), the InternVL2 series (2024/07/04), and the Qwen2-VL series (2024/08/30). While the results of these models are not mandatory under the guidelines, considering the super-fast advancements in VLMs this year. Could you please include results from some of the aforementioned models to highlight the performance of the latest generation of VLMs in multi-image understanding?\n3. The pairwise design of answerable and unanswerable questions seems to create an artificial scenario. Unanswerable questions are more ambiguous and complex. How does your dataset account for the nuance of real-world multi-image understanding?\n4. Line 311 mentions \"concatenate the images to constitute one input.\" How exactly are the images concatenated? Could it be that the concatenation method affects the model's performance? What are the results of using different concatenation methods?\n5. Please refer to Weaknesses for more potential questions.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The MuirBench covers 12 multi-image understanding tasks, involving diverse scenarios. It is a comprehensive benchmark containing 11,264 images and 2,600 multiple-choice questions.\n2. The paper conducts detailed testing on 20 recent VLMs, revealing the limitations of these models in multi-image understanding and the performance gap compared to single-image input.\n3. The analyses of unanswerable-type questions, image positioning, and error analysis of GPT-4o are impressive.", "weaknesses": "1. The paper only contributes data without offering any insightful analysis of model limitations and lacks novelty and contribution. It does not give people any new knowledge, and people do not feel any insight after reading it.\n2. Authors overlooked some of the newer VLMs, such as Gemini 1.5 Pro (Feb. 2024), InternVL1.5-Chat (Apr. 2024) and Claude 3.5 Sonnet (Jun. 2024)\n3. Although some limitations of the models in multi-image understanding tasks are shown, there is no discussion on the architectural limitations of these models, why these VLMs underperformed, why certain models perform poorly on different tasks or how models’ architectures could be adapted for better multi-image understanding. Furthermore, the suggestions for future improvements are rather general and lack detailed explanations of specific technical approaches for new VLMs. To sum up, the analysis part in this paper is superficial, more insight is required. \n4. The difficulty of question among different tasks is imbalance.\n5. It seems that unanswerable questions have limited practical applications in real-world scenarios.\n6. Some of the questions in the benchmark do not need to be presented in multiple images, such as question “Counting” and “Image-Text Matching” in Figure 6. As a questioner, I would consider inputting images one by one to ensure accuracy. A benchmark needs to focus more on practical application value rather than just creating a question that is difficult for VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)