MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
Authors: Fanqing Meng
Jin Wang
Chuanhao Li
Quanfeng Lu
Hao Tian
Tianshuo Yang
Jiaqi Liao
Xizhou Zhu
Jifeng Dai
Yu Qiao
Ping Luo
Kaipeng Zhang
Wenqi Shao
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-image Understanding
Benchmark
LVLM
Evaluation
Primary area: datasets and benchmarks

Abstract: The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of nearly 30 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7\% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development. We release the data and code at https://github.com/MMIUBenchmark/MMIU.

Published as a conference paper at ICLR 2025
MMIU: MULTIMODAL MULTI-IMAGE UNDERSTAND- ING FOR EVALUATING LARGE VISION-LANGUAGE MODELS
Fanqing Meng".1.2, Jin Wang-.54 Chuanhao Li Quanfeng Lu Hao Tian Jiaqi Liao?, Xizhou Zh, Jifeng Da Yu Qiao", Ping Luo Kaipeng Zhangat Wenqi Shao
Shanghai Jiao Tong University Shanghai Al Laboratory "The University of Hong Kong TSenseTime Research TTinghua University
Project Page: https://mmiu-bench http:///muuubbbeecchhuu....bb github.io
ABSTRACT
The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. How- ever, their evaluation has not kept pace with their development. To fill this gap. we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMMs across wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and IIK meticulously curated multiple-choice questions, making the most extensive benchmark of its kind. Our evaluation of nearly 30 popular LVLMs, including both open-source and proprietary models, reveals sig- nificant challenges in multi--mage comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7% accuracy MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU advance the frontier of LVLM research and development. We release the data and code at https://gi thub. comm/OppenGVLab/MMi
INTRODUCTION
The capability to process multiple images is crucial for multimodal large models, as single image captures information from a specific angle and moment, limiting the model's 's ability to understand and reason about the entire scene (Song et al.] 2024, Wang et al., [024]. Multiple images, on the other hand, provide rich information from different perspectives and time points, enabling the model to synthesize this data and achieve a more comprehensive understanding, such as analyzing consecutive images for action prediction (Lu et al. 2024b) or utilizing multi-view images in 3D navigation (Dai et al. 2017). The ability to process multiple images allows Large Vision-Language Models (LVLMs) to understand and handle complex visual tasks, thereby facilitating real-world applications.
Due the great importance of multi-image understanding, recent LVL.Ms have improved such pability by pre-training on various image-text interleaved data such as M4-Instruct (Li al., (024a)) -antrss-nstruct (Jiang et al. 2024b), and OmniCorpus (Li et al.) 2024b). However, the evaluation of multi-image LVLMs significantly lags behind their development. A good multi-image evaluation benchmark can help identify tasks that lead to poor performance and guide future model design
data collection. Prior datasets such as LVLM-eHub (Xu et al. 2023) and MMBench (Liu et al.
Corresponding Authors: shaowengi pjlab.org.cm.... zhangkaipeng @ @pjlab.org.cn Equal contribution

---

Published as a conference paper at ICLR 2025
High-Lerel (9b3)
Are both of these realictic?
conblnattonn
List used ravigetion
Three-d
High-ioval
Low-twet
Migi-
Discrate
Somantic ie Malti-image
Plaase Leantify the action person performs
Spetial No
Terqural Bel Lange
Rtacrote
TieeD
the Snagus
Cant
Tee-D
Coms Lownns
Fleane
order Sedeons baned the glaen patcles
abject changed status uSen actios?
Figure 1: Visualization of MMIU. Our MMIU contains 77,659 images, 7 types of image relation- ships, and image modalities, along with 11,698 multiple-choice questions, providing a compre- hensive evaluation for 52 multi-image understanding tasks. Each example comes from task from each multi-image relationship. We construct MMIU by adopting top-down hierarchy where image relationships of interest enumerated and multiple tasks associated with each relationship.
2023) focus on single-image image tasks (Xu et al. 2023), which cannot capture the complexity in multi- image scenarios. Although several recent benchmarks have attempted to evaluate the multi-image performance of LVLMs, they have limited coverage of multi-iman multi-image tasks while capturing a few relationships between multiple images shown in Table T For example, Video- 2024a) focuses solely on temporal relationships and MUIRBENCH (Wang et al. 2024) does not consider spatial relationships between objects in multiple images, which is crucial multi-image applications such as 3D navigation. Other works such as SlideQA (Tanaka et al. 2023) and MMMU (Yue al.] 2024) focus on understanding and reasoning within specific input types or disciplines, preventing them from providing a general evaluation for multi-image capabilities.
(Fu al.,
To build comprehensive multi-image evaluation benchmark, we connect multi-image comprehen- sion with manipulating information in working memory in cognitive psychology (Baddeley (2000) As pointed out by Multiple Trace Theory (MTT) (Moscovitch et al., 2006), working memories are categorized into episodic memory which captures sequential information and can arrange events in the order they occur, semantic memory enabling concept comprehension, and spatial memory which helps understand spatial environments. Multiple images can be deemed as visual memory. Under- standing such a visual memory requires models to handle the semantic content, understand spatial relationships, and track temporal sequences of multiple images, closely mirroring human memory mechanisms. This inspires us to construct the evaluation benchmark to measure how well LVLMs tackle multi-image tasks from temporal, semantic and spatial perspectives.
This work introduces the Multimodal Multi-image Understanding (MMIU) benchmark, designed to comprehensively evaluate large visual language models (LVLMs) in multi-image task understand- ing. As shown in Table [ we collect evaluation data through a top-down hierarchy, starting with the enumeration of image relationships spanning temporal, semantic, and spatial correspondences, and subsequently assigning multiple multi- imate tasks to each relationship. The comprehensiveness of MMIU is twofold. First, has the widest coverage of multi- multi--mage evaluation data to date, en- compassing 7 types of multi-image relationships, 52 tasks (e.g. multi-view action recognition). 77k images, and 1 1..k carefully curated multi-choice questions, which 1.81 times larger than Miles- Bench (Song al. 2024). Second, MMIU involves more diverse multi-image analysis tools than previous benchmarks, including performance comparison over image relationships, in- and out-of- domain task discovery by task map, and task learning difficulty by supervised fine-tuning (SFT). The multi-faceted analyses provide useful insights for model and data improvement.
MMIU, including closed-source models such GPT-4o (Ope- nAI (204)) and Geminil.5 (Reid et al. 2024), and open-source models such as GLM4V (GLM et al.) 2024 and Intern VL--Chat (Chen et 2024b). These LVLMS contain both multi-image (support
We test 24 popular LVLMs
2

---

Published as a conference paper at ICLR 2025
Table 1: The comparison between MMIU and existing multi-image evaluation benchmarks includ- ing Video-MME (Fu et al. 2024a), MIRB O, MUIRBENCH (Wang et al. 2024), and MileBench (Song al., 2024). We summarize the image relationships in previous benchmarks according to seven categories defined in Figure [ 'Y&N' indicates that our MMIU comprises both answerable and unanswerable questions. I, T, V, and represent image, text, video, depth map and point cloud, respectively. Compared with prior datasets, MMIU involves massive test samples spanning 52 multimodal tasks and 5 modalities, and comprchensive multi-image analyses by image relation- ships, task map and supervised fine-tuning (SFT).
Statisties
Multi image Analysis
Benchmark
# Sample
Ings.
Relation
Task
Modality
Answerable?
Relation
Task Map
SFT
Video-MME
2.7K
1
30
T.V
Y
X
x
MIRB
0.9K
3.5k
LLV
x
MUIRBENCH
2.6K
11k
4
12
LLV
Y&N
X
MileBench
6.4K
97k
4
28
LTV
Y
x
MMIU
11.6K
7
I.V.DDD Y&N
multi-image input) and single-image (support only single-image input) models. For single-image models, we employ image concatenation to obtain the evaluation performance. The experimental results show that even the most advanced model, GPT-4o (OpenAl 2024), achieves only 55.7% ac- curacy on MMIU, highlighting the inherent difficulty of these tasks. Other than the diverse analytical tools Table T we conduct ablation studies to investigate the impact of unanswerable questions and multi-image concatenation methods on model performance. We summarize our findings as follows:
The best-performing model for multi-image tasks GPT-4o, with Intern V1.2-pro (Chen et al. 20024) being the strongest among open-source models. The best closed-source model GPT-4o leads the best open-source model InterVLL2 by a large margin, (i.e. 5.4% accu- racy). However, GPT-4o achieves only 55.7% accuracy MMIU, indicating substantial challenge our benchmark.
Some powerful LVLMs like InternVL1.5 (Chen et 2024b) and GLM4V (GMM al., (024) whose pre-training data do not contain multi- image content even outperform many multi-image models which undergo multi- image supervised fine-tuning (SFT), indicating the strong capacity in single-image understanding is the foundation of multi-image com- prehension.
By comparing performance the level of image relationships, we conclude that LVLM excels at understanding semantic content multi-image scenarios but has weaker perfor- mance comprehending temporal and spatial relationships in multi-image contexts.
The analysis based on the task map reveals that models perform better on high-level under- standing tasks such as video captioning which are in-domain tasks, but struggle with 3D perception tasks such 3D detection and temporal reasoning tasks such as image ordering which are out-oi- tasks.
By task learning difficulty analysis, tasks involving ordering, retrieval and massive images cannot be overfitted by simple SFT, suggesting that additional pre-training data or training techniques should incorporated for improvement.
In summary, this paper makes three key contributions. First, we introduce and open-source the Mul- timodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite that ad- dresses various complex multi-image tasks, thereby filling critical gap in multi-image comprehen- sion. Second, our evaluation results demonstrate that current large visual language models (LVLMss, including proprietary models like GPT-4o, encounter significant challenges in solving multi-image tasks, particularly those involving spatial understanding. Third, we conduct multi-facetee analytical experiments, shedding light the limitations and performance gaps of current models from various perspectives. We hope that MMIU will push the boundaries LVLM research and development, bringing closer the realization of advanced multimodal multi-image user interactions.
3

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 LARGE VISION-LANGUAGE MODELS
With the advancements in large language models (LLMs) (Touvron et al. 2023 Jiang et al. 2024a). a series of studies have begun exploring multimodal LLMs capable of simultaneously interpreting visual and linguistic information. Through visual pre-training and instruction fine-tuning, LVLMs have demonstrated outstanding performance in understanding multimodal image-text inputs (Li et al. 2024a Lu et al., 2024a Bai et al., 2023). However, most LVLM training data consist pr- marily of single image- pairs or pure text data, limiting their ability to comprehend mulli-image inputs. Therefore. researchers have considered using harge-scale interleaved image-text corpora, such MMC4 (Zhu et al. 2024) and Omnicorpus (Li et al.. 2024b), during the pre-training phase of LVLMs. This approach has led to the development of models like DeepSeek- VI. (Lu et al.) 2024a) and Idefics (Laurençon et al.] (024).. which exhibit notable performance in multi-image tasks. Building on this foundation, recent studies have applied instruction tuning with extensive multi-image data, resulting in models that handle multi-image tasks more effectively while utilizing fewer resources. Notable examples of these advancements include Mantis QJiang et al., (024b)) and LLa VaaV-Neet-iittrleaae (Li et al. 2024a).. Nonetheless, the evaluation of these models' capabilities in handling multiple images has mainly been qualitative, and quantitative assessments of different models' performance across a broad range of multi-image tasks remain insufficiently explored.
2.2 LARGE VISION-LANGUAGE MODELS BENCHMARKS
Benchmarking large vision- Dasion-language models (LVLMs) is crucial for identifying model limitations and guiding their development (Xu et al. [023]] Ying et al. 20244 Liu et al. (203)). Despite the exis- tence of numerous benchmarks aimed at evaluating the perception or reasoning abilities LVLMs, most of these benchmarks focus solely on single-image scenarios. Although some benchmarks in- clude multi-image examples (Jiang et al., 2024b Fu et al. (024a), they usually address limited capabilities. For instance, MANTIS-Eval (Jiang et al.) 2024b) focuses on assessing model's ability to perceive size, while Video-MME (Fu et al. 2024a) emphasizes image sequences and their tempo- ral relationships. Recently, researchers have been dedicated to developing more holistic multi- imige evaluation benchmarks, such as MileBench (Song et al. (2024) and MUIRBench (Wang et al., 2024), to provide more thorough assessment of multi-image cognition. However, these benchmarks fall short in terms of task depth and breadth. For instance, MILEBENCH (Wang et al.) (024) provides relatively comprehensive multi-image evaluation but lacks important multi-image tasks such as 3D spatial understanding and low-level semantics, which are essential for drawing complete conclu- sions. In contrast, MMIU offers a benchmark that combines both task depth and breadth, covering wider range of image relationships, task types, and image categories. This enables more compre- hensive assessment of model capabilities.
MMIU
This section presents the proposed MMIU benchmark. MMIU is a comprehensive evaluation dataset encompass- ing 11K multi-choice questions for multi-image compre- hension. We first give a brief overview of MMIU in Section 3.T Then, we describe the construction process of MMIU in Section 3.2 We provide two versions of MMIU: test and testmini, with the latter being 1/10th the size of the former for quick testing, which has 1040 sam- ples. In this paper, we primarily conduct experiments and analyses on the test set, while results on testmini are recorded in the Table 15 in the appendix for comparison.
Table 2: Key statistics for the MMIU
Statitic
Number
Tital samples
17606
Tixal Images
73660
Trxal oolutices
Average images
6.60
Avcraze qucdion woods
27.9
Range images
Image Num Level
Number
-iin (3..5)
3416
Many (16-32)
Unamoveralle set
Percentage
mnser
411G
Replbz other imagie
595
3.1 BENCHMARK OVERVIEW
Stattle dll inages
556
Ipolevest quesioolnnggeee
MMIU is designed to measure multi-image understand- ing for LVLMs. It has two advantages compared with pre-
4

---

Published as a conference paper at ICLR 2025
Sanam
Spalia Retationship (19)
lingr
Google a arxiv kaggle
Sourtin
...
Semarica
collel
Brattondiis
leuge
Tempend
Vigal ondeiing
Question (1Tck samples)
Mall chol-chllle Ovesdioe and Anorer
The lonol H& site salo gren hannen 3C thes ar peopleme the rrupes,
canena
Saooder
Image Typea
Druak letee
Velro brag Mendlti
Mahheater Mid Mihrowtogg Gu Inage-Uit irags
KiPercoeed Imap-
Figure 2: An illustration of our data collection process. First, we refine multi-image tasks and collect task data based on cognitive psychology. Then, standardize these datasets into uniform mormat-metadata. Next, we generate multiple-choice samples with answerable and unanswerable questions from the metadata using either manually designed rules or GPT-4o. Our benchmarks include capability evaluations across various image types.
vious multi-image evaluation benchmarks as illustrated in Table I First, MMIU provides a comprehensive evaluation by encompassing massive test samples spanning various multi-image tasks and image relationships. Specifically, MMIU consists of 77 77,659 images and 11,698 multi-choice questions (1.81 times more than MileBench (Song et al. 2024) which previously had the most multi-image test samples) with an average of 6.64 images per in- stance. It tests 7 distinctive multi-image relationships covering 52 diverse multi-image tasks, 1.73 times more than VideoMME (Fu et al.) 2024a) which previously contained the most multi-image tasks. In addition, we also create unanswerable set comprising 19 tasks with each task containing 40 questions, considering that LVL.Ms cannot answer all questions in real scenarios. More detailed statisties of MMIU be found in Table [
The diverse evaluation data requires the model to be capable enough to deeply understand semantical, temporal, and spatial clues multi-images with various input types (Figure
Second, MMIU offers thorough analyses in multi-image understanding by utilizing multi-faceted analytical tools. 1) Thanks to the top-down hierarchy collecting data, MMIU can compare per- formance across image relationships. 2) The extensive coverage of multi-image tasks enables eval- uating on task map, facilitating the discovery of in- and out-of-domain tasks. 3) The evaluation samples can be adapted to multi-image instruction tuning data. By SFT, the task learning difficulty can be acquired, which is crucial for the practitioner improve the model and data.
3.2 DATA CURATION PROCESS
Multi-image understanding is essential for Large Vision-Language Models (LVLMs) due to its prevalence real-world applications. We treat a sequence of images as visual memories whose semantic, temporal, and spatial segments are crucial in retrieving information (Moscovicch et al,, 2006). As shown in Figure 2 our Multi-Modal Image Understanding (MMIU) benchmark con- structed using top-down approach. We first categorize multi-image relationships into semantic, spatial, and temporal types, further divided into seven subtypes. We then collect data for each rela- tionship type and standardize its format. Finally, we create multiple-choice questions based these relationships to evaluate LVLMs' multi-image understanding capabilities.
Relationships Tasks. We categorize multi-image relationships into semantic, spatial, and tem- poral aspects. Semantic relationships are further divided into: 1) Low-level semantic relationships comparing visual features like illumination, quality, and saturation; 2) High-level (objective) rela- tionships involving objects, attributes, and interactions (e.g., a person hitting ball, a person catching ball); and 3) High-level (subjective) relationships such thematic, cultural, and emotional asso-
5

---

Published as a conference paper at ICLR 2025
ciations (e.g., the emotions expressed in these images). Temporal relationships are refined into: 4) Continuous temporal relationships for video frame sequence tasks, and 5) Discrete event sequence relationships for understanding multi-step tutorials. Spatial relationships spatial relationships including rotation, translation, and symmetry; and 7) 3D spatial relationships involving different camera perspectives and depth variations. Detailed information on each image relationship is provided in Appendix A with corresponding multi-image tasks presented in Table 4 of the Appendix.
categorized as: 6) 2D
Tasks + Data. We conduct extensive searches for relevant datasets using resources like Google, Paper With Code, and Kaggle, guided by the proposed tasks. After downloading, we thoroughly evaluate each dataset's ss appropriateness for the specific task, ensuring usability and relevance. We then organize the data into a standardized metadata format, which includes task description, ques- tion, answer, input context, and images for each sample. This format facilitates the creation of visual questions and answers. We manually verify the accuracy of this information and its convertibility into a multiple-choice question format. For efficient evaluation, we limit each task to maximum of 200 randomly selected samples, except for tasks with insufficient data. The detailed description of this metadata format is provided in Appendix A.4
Question and Answer Generation. For each subtask, we create multiple-choice visual questions (with a maximum of eight options, depending the task), with the choices and answers derived from their metadata. Specifically, depending on the task, we either manually design rules or use GPT-4o (T 2024) with carefully crafted prompts to ensure efficient and high-quality gener- tasks, we instruct GPT-4o to generate plausible but
ation. For example, in 3D question-answering incorrect options based on the question and the correct answer. For image retrieval tasks, we ran- domly select incorrect images from the metadata as the wrong options. Additionally, we select 19 tasks and create 40 unanswerable samples for each task to construct an unanswerable set for robust evaluation. More details in unanswerable question generation are provided in Appendix AA.4.
Challenges. Designing accurate question templates: Questions must provide all necessary informa- tion for LVLMs to derive correct answers. For example, in 3D object detection, questions should include detailed camera pose information and specify the coordinate system for detected objects. Obtaining and verifying correct answers: This is particularly challenging for 3D spatial relation- ships. In 3D pose estimation, relative camera pose between images is not inherently provided in datasets like ScanNet (Dai al., 2017). We address this by: a) Transforming original camera poses to relative poses through matrix multiplication. b) Carefully examining the correctness by applying relative poses to image pairs and ensuring proper correspondence matching. These challenges high- light the significant effort required to establish MMIU as a comprehensive multi-image evaluation benchmark.
4 EXPERIMENT
This section first introduces the experimental setup in Section 4.1 models used, Following this, we present the main results and multi-faceter analyses in Section 4.2 and Section 4.3 respectively. Ablation studies are included in Section 4.5 We put more detailed information and analysis in the Appendix B
including the testing methods and
4.1 EXPERIMENT SETUP
LVLM Models.
Specifically, we select 9 closed-source models: GPT-4o (OpenAl 2024 and
the other 3 models. Additionally, we evaluate 19 open-source models that support muluple image inputs: Mantis (Jiang et al.) 2024b), InternVL.2 (Chen et al. 2024b). al., (024a)) and other 16 models. Furthermore, we include models that only support single image input including A.AV--V1.5-7P (Liu et al.] 2024a), Monkey-Chat (Li et al.] (024c) and other 5 models. Finally, we also evaluate the performance of I.LMs, including two approaches: direct questioning without images and questioning after generating image captions using GPT-4o. The detailed description of all models can be found in Appendix B.I
LLi
Evaluation Method. With OpenCompass (Contributors) 2023), we first match the model's re- sponse to the corresponding options. If a match cannot be made, we mark it as Z Yue et al. 2023). The accuracy is used as the metric. Specifically: 1) For cases where the input token is too long for the tested model, we randomly sample images until it can be tested. 2) For closed-source models, if the model refuses respond due copyright issues with the images, discard those samples.

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative results across 7 image relationships on test are summarized. Accuracy is the metric, and the Overall score is computed across all tasks. The maximum value of each task is bolded. The caption used for LLMs is generated by GPT-4o.
Model
Osenall (icetiin)))
Ovurll
Diserde
Canimuns
Low-lnsel
Highloonhooh
Hightloschall
Two-
ThreeD
Prequency
30,9
31.5
29.5
29.5
36,1
29.6
36.7
27.31
30.2
Rundom
27.1
27.4
22.1
25.4
33.7
20.7
32.8
26.3
25.4
C Clotad-souree E.LLMs
GPT-4o
55.6
55.5
58.2
53.7
8AU
69.2
57.5
41.7
55.4
Claude3,5
$4.3
53.4
47.9
77.2
64.8
64.5
41.9
45.1
Geminil.5
$4.5
53.4
54.2
50.1
76.1
63.9
64.9
43.3
43.0
Geminil.
41.2
40.2
45.8
49.3
46.7
57.9
36.7
29.7
36,7
Adequate Multi-Tmage SET
Murta-Redio2--BB
SiCCLIP-A
52.9
KGen MM Inserleveed
29,2
Muuti Impur LVE.Ms
46.3
50.5
38.1
Insemll
38.7
46.4
59.1
26.0
33.6
In 2-88
34.0
14.3
$4.2
45.4
96.7
47.9
10.1
30.0
92.2
Mint-bicemVLLce 1-48
32.5
12.1
90.6
42.3
35.4
47.3
79.2
27.3
90.5
Semi-biienLLeheae 1.528
31.8
30.5
31.1
38.6
30.9
37.6
28.7
27.4
25.7
alcfics--88
27.2
27.5
22.9
19.3
42.4
45.2
26.6
33.4
25.7
DespSeck-VV VL--T VL-7B
24.5
24.6
16.4
10.3
39.1
3233
34.2
32.9
16.7
xGen MM Single 4B
25.4
24.5
25.8
25.7
23.9
28.8
24.4
26.5
22.1
XComppser2 7B
23.4
23.5
31.9
31.6
23.4
34.3
20,0
18.7
15.0
DeepSeet VL 1,30
23.8
23.2
14.6
9.2
33.3
24.9
30,8
32.7
19.0
fumingow2
22.7
22.3
20,8
19.5
29.6
24.6
26.9
17.2
21.7
XCampaser2: 188
22.0
21.9
29.4
32.9
22.5
36.2
15.1
20.9
14.6
Qwen-chat
18.0
15.9
14.7
19.5
22.3
21.
14.8
10.5
17.1
dediza-l--immruc
132
12.8
23.
7-3
11.6
220
12.3
12.3
6.7
Qwen Base
3.2
13.2
2.6
5.3
10.1
4.5
3.8
Single- Image impul LYLMS
GLMM44y9h
26.6
21.0
10.9
29.8
20.9
20.1
LLaVA-neat-icuua.
30.7
Liun13
15.7
sharegpoty
chanecapconerr
monkery-hor
12.9
L.MSs
21.5
24.4
10.4
20.0
Caption
48.9
19.1
58.9
32.8
Qwen2.5
30.1
51.7
26.2
37.5
35.4
30.6
25.8
Capiion
41.1
50.0
45.6
51.7
42.7
27.8
31.8
L.me's
30.1
19.5
53.5
76.9
36.7
30.0
10.2
24.4
31.4
Capiion
38.3
19.3
40.9
45.4
$1.7
50.3
18.9
31.2
35.4
Interl M2.5
32.1
31.5
90.2
28.3
36.7
42.2
28.7
27.8
35.9
Cippiin
39.8
38.2
36.7
45.0
45.0
32.5
41.3
30.0
31.8
The detailed setup be found in Appendix [B.] 3) To prevent the model consistently selecting same option (e.g. single-image input models), we shuffle the original options and retest. A result is considered correct only if both tests yield the correct answer. 4) For both the answerable and the unanswerable set, we randomly shuffle the options to ensure that the correct answer's position does introduce any position bias.
4.2
MAIN RESULTS
As shown in Table 3 we report the average accuracy of all models across 7 image relationships alongside Random Choice and Frequent Choice baselines, with "overall" representing the average accuracy all tasks. We record the model's performance across all tasks in Table 14 in the Ap- pendix. Specifically,, we have the following findings.
Multi-image tasks remain challenging. GPT-4o leads with 55.7% accuracy, followed by Geminil.5-Flase and at 50.3%, outperforming Geminil.0 1.0 Pro Vision. A 5.4% accuracy gap exists between closed-source and opencsource models in iillti-imageeee comprehension. Notably, in single-image tasks, open-source models like Intern VI.2-Pro match exceed closed-source models such GPT-4o (Yue al. 2023
at 53.4%. Among open-source models, Intern VL2-Pro tops
Liu et al. 2023 Ying et al. (2044).
The strong capability in single-image understanding the foundation of multi-image compre- hension. Several advanced models such as Intern 1LI.5-Chahhh which have been trained with only single-image data can achieve good performance MMIU. For instance, GLM4V reaches 37.4%
Notice that although multi-image data.
supports multiple image inputs, its training phase did not incorpo-

---

Published as a conference paper at ICLR 2025
Semardti
Teepond
Spwtwl
Ext (iffeced perforennece Affioret reaks on Lomanto, Imagura and spetted beage rebilneakgg
(M) The varsing Pegrocetatii reakih leegg relatieesigg
Figure (a): The average performance comparison of representative models on 3 main image relationships. (b): The average performance comparison of representative models on 7 specific image relationships.
accuracy, surpassing multi-image models from its powerful capability in single-image multimodal understanding. Besides, GLM-4V also out- performs many aulti-image models such as DeepSeck- VL. This is because GLM-4V supports an ultra-high resolution of 120 120, allowing it to understand concatenated images and to reason. For instance, in the video-captioning task, its accuracy reaches 76%.
and Idefics2. Such success stems
Adequate multi-image supervised fine-tuning (SFT) improve the performance of models on multi-image tasks. Notably, we have observed that many models trained extensively with multi- image data during the pre-training phase did not achieve satisfactory results, such as idefics2 and DeepSeek-VL. However, Mantis and common feature is extensive multi-image instruction fine-tuning during the SFT phase. For instance, although idefics2 is trained with a large amount of multi-image data during the pre-training phase, is trained by few multi-image data during the SFT phase. Mantis, after performing multi-image SFT on the basis of idefics2, achieved 17.8% accuracy improvement.
stand out among all models. Their
4.3 MULTITASK ANAYSTS
4.3.1 PERFORMANCE ACROSS IMAGE RELATIONSHIPS
As shown in 3 models exhibit varying capabilities across different image relationships. More detailed visualizations can be found in Figure g in the Appendix. In general, LVLMs excel at understanding semantic content multi-image scenarios, perform moderately in temporal tasks, and obtain the worst performance incomprchending spatial relationships in multi-image contexts. 1) In semantic relationships, models generally perform well on multi-image semantic tasks involy- ing low-level relationships. However, they struggle with high-level tasks, for subjective tasks such as Causality Reasoning and Emotion Recognition, which require the identification and reasoning of implicit visual information, highlighting a gap between model performance and human visual cognition. As for objective tasks such retrieval tasks, most models fail tackle them. 2) tempo- ral relationships, models can handle discrete and continuous temporal relationships relatively well but show mediocre performance on -intuinnnnnittiiveeeeeee multi-image tasks. For instance, in sorting tasks, GPT-4o achieves only 28% and 21.5% accuracy in temporal ordering and visual ordering tasks, respectively. 3) In spatial relationships, we find that models struggle with understanding both 2D and 3D positional relations. This is consistent with the observation in the previous single- image evaluation benchmark Ying et al. (2024) where they find that LVLMs fall short in localization and detection tasks requiring spatial reasoning. The tasks involving spatial relationships in MMIU become more challenging because models need to gather spatial information in multiple images and to reason.
4.3.2 ANALYSIS ON THE TASK MAP
Task map an effective tool for multi-task analysis Ying et al. (0024): Ilharco et al. (202)). Thanks to extensive coverage of multi-image tasks in MMIU, we build a task map analyze the relation- ships between different tasks, allowing us to identify in- and oft-of-domain tasks for current LVLMs. Following MMT-Bench Ying et al. (2024). use QwenVL-chal construct task map where the
8

---

Published as a conference paper at ICLR 2025
Gate
Ouewr
Oatir
Chew
Ouew
datee7
Ohew
Gotad madel
Adagams
Mall mere
Siyle-inge
Opeii
Figure 4: Visualization of categories of LVLMs performance across various clusters.
distance between two tasks is given and cluster all tasks into 8 categories. Detailed construction process, the description of the task map, and more analysis can be found in Appendix
Tasks involving recognition or captioning are in-domain tasks which can be handled by most current multimodal large models. For multi-image tasks, models generally struggle to achieve sat- isfactory results, obtaining good performance on limited number of tasks. Specifically, for tasks in clusters 7, 8, and some tasks cluster 2, which involve recognition or captioning (e.g., video captioning, action recognition), models perform relatively well. This is because these multi-image tasks focus on overall image perception, requiring less comparison and reasoning between images.
Tasks involving temporal ordering and 3D spatial reasoning are out-of-domain Tasks where most models perform poorly. Specif- ically, models struggle with tasks in clusters 4, 5, and 6. Clusters 4 and involve mod- elling semantic relationships or sequential or- der among multiple images, requiring memo- rizing detailed long-context content and strong reasoning skills. Most LVLMs underperform on these tasks such as temporal ordering tasks). Tasks in cluster pertain to 3D visual tasks such 3D detection and restruction. This may be due to the lack of 3D vision- language data in training LVLMs.
Figure 5: The performance AcC and AceSFT different tasks, sorted by ode in descending order.
4.3.3 TASK LEARNING DIFFICULTY
We analyze task learning difficulty by SFT with all evaluation samples in MMIU being instruction tuning data. In this way, we can identify tasks which cannot be improved by simple SFT. To this end, we fine-tune QwenaL-chat on each task for 20 epochs and obtain the accuracy of QwenVL- chat on each task, denoted as AceSFT. The lower accuracy reflects the larger fitting difficulty of the task. Meanwhile, we also obtain the average accuracy of all tested models on each task, denoted as AccMadel This accuracy reflects the difficulty current models face handling these tasks.
As shown in Figure 5 we find that the Spearman correlation coefficient between AccsFT and AcCMadel is 0.66, indicating high correlation. This suggests that both measures can reflect task difficulty some extent. More importantly, we need to focus tasks where both AccsFT and AccMode! are low. A low Aces indicates that the task difficult to overfit even with SFT, suggest- ing that additional pre-training data training techniques might be necessary. These tasks include 1) Ordering and retrieval tasks, which require strong memory and reasoning abilities abilitie--capabilttie that are generally weak in large multimodal models. 2) Tasks involving large number of images, such as EVQA. MEV, and GNAP, require models to support longer context lengths and possess strong memory capabilities. This indicates that future model designs should consider the ability to handle long contexts and emphasize the inclusion of multi-image data during the pre-training.
4.4 ERROR ANALYSIS
We analyzed eтrоr patterns of some Large Vision-Language Models (LVLMs): GPT-4o, Claude 3.5 Sonnet, and VLeemPLL-PP on MMIU. Our approach involved systematically sampling up to incorrect answers per subtask for each model. Task-specific experts then examined these errors

---

Published as a conference paper at ICLR 2025
skoropptie
Eavort Farformanc
Figure 7: Comparison of the performance of different ingle-image models on various tasks in the MMIU when tested with image stitching or visual token stitching methods.
identify their root causes. The error distribution is shown in Figure For more detailed analysis and description the four error categories, refer to Appendix B.4
Specifically, GPT-4o's errors are primarily per- ception (39%) and reasoning (42%), indicat- ing room for improvement in visual capabili- ties and logical reasoning based on visual cues. However, it excels in following instructions due to strong semantic understanding and long con- text handling. GPT-4o shows 17% lack of ca- pacity errors, mainly in 3D and 2D visual tasks, suggesting its visual capabilities are still insuf- ficient in these areas. Similarly, as a represen- tative closed-source model, Claude 3.5 Sonnet exhibits an error distribution similar to GPT-4o, but with more "Lack of capacity" emror.. We find this is mainly due to Claude's weaker visual ca- pabilities compared to GPT-4o. InternVI--Pro as an open-source model, achieves excellent results on MMIU. However, it shows more perception errors, indicating that open-source models may be weaker in multi-image and text alignment compared to closed-source models.
deuk Sweet
Insewtt
Figure 6: Distribution of error types for GPT-40, Claude3.5 Sonnet and InternVL2-Pro.
4.5 ABLATION STUDY
Here have conduct detailed ablation study for different test methods. For more results, please refer to Appendix B.5
Impact of Different Testing Methods Model Performance. For single-image input models handling multi-image tasks, one approach is to concatenate the images into a single image and feed the models. Besides, we explore an alternative method: concatenating all output visual em- beddings before feeding them into LLMs. As shown in Figure 7 we observe that for these models, testing using concatenated visual tokens does not perform better than directly concatenating images. This is especially true for the LLaVA series, where concatenating images significantly outperform concatenating visual tokens. In contrast, GL.M-VV exhibits relatively consistent performance under both testing methods.
5 CONCLUSION
In this paper, we present MMIU, benchmark dedicated to comprehensively evaluating the perfor- mance of LVLMs on multi-image tasks. MMIU includes seven types of image relationships, such as 3D spatial relations, 52 tasks, and various image modalities, filling gap in this field. We test 24 popular LVLMs on MMIU and analyzed the results using various analytical tools, including task maps. The experimental results indicate that current models, including GPT-4, struggle handle complex multi-image tasks. We hope that MMIU will promote the development of more generalized capabilities in future models within the multi-image domain.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This paper is partially supported by the National Key R&D Program of China (NO.20222D01601002 NO. (0.20022ID0111000) and the General Research Fund of Hong Kong No.17200622 and 17209324. This work was done during his internship at Shanghai Artificial Intelligence Laboratory.
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1692-1700, 2018.
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 1:X1::2444414219, 2024.
Anthropic. Claude, 2023. URL http://///www..thhppp....... com.
Accessed: 2023-04-18.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive dision-language models. arXiv preprint arXiv:2308.00 01390, 2023.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe, Scanqa: 3d question an- swering for spatial scene understanding. In proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19191 2022.
Alan Baddeley. The episodic buffer: new component of working memory? Trends cognitive sciences, 4(11):417 2000.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajezyk. Hpatches: A bench- mark and evaluation of handcrafted and learned local descriptors. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5173-5182, 2017.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631, 2020.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jaang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv. Kai Lv, Ma. Runyuan Ma. Zerun Ma. Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internim2 technical report, 2024.
Angel X Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Jianxiong Xiao, Manolis Savva, Shuran Song, Andy Zeng. Yinda Zhang, and Matthias Nießner. Matterpori3d: Learning from rgb-d data in indoor environments. In Proceedings of the International Conference on 3D Vision, pp. 667-676, 2017.
11

---

Published as a conference paper at ICLR 2025
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-200. 2011.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi multi-modal models with better captions. arXiv preprint arXiv:2311.12791, 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far mercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024a.
gpt-4v? closing the gap com-
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In P Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 24185-24199 2024b.
Hsu-kuang Chiu, Ehsan Adeli, Borui Wang, De-An Huang, and Juan Carlos Niebles. Action- agnostic human pose forecasting. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1423-1432. IEEE, 2019.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https:///itthb. wwt.cooooooo//c http:///pppnncccaaaa/ppeeooooooo
2023.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias
Nießnner Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2694-2703, 2023.
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems, 35:13610-13626, 2022.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. intemmmmxccoppose// Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024.
Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al. The 2021 image similarity dataset and challenge. arXiv preprint arXiv:2106.0966 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims in video analysis. arXiv preprint arXiv:2405.21075. 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li. Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404. 12390, 2024b.
Andreas Geiger, Philipp Lenz, and Raquel Urtasun. Vision meets robotics: The kitti dataset. In International Journal Robotics Research, pp. 1231-1237, 2013.
12

---

Published as a conference paper at ICLR 2025
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXii:2406.112799 2024.
David Ha and Douglas Eck. A neural representation of sketch drawings. CoRR, abs/1704.03477, 2017. URL http:/t:///axxi wttp://..wrg/bb//1/..0000
Ankur Handa, Viorica Päträucean, Simon Stent, and Roberto Cipolla. Scenenet: An annotated model generator for indoor scene understanding. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 5737-5743. IEEE, 2016.
Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. Forgerynet: A versatile benchmark for comprehensive forgery analysis. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4360-4340 4369, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09663 2021.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint WxXiv:2404.063// 2024.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Mille Labeled faces in the wild: A database forstudying face recognition unconstrained environments. In Workshop on faces in Life Images: detection, alignment, and recognition, 2008
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. the / Ith International Conference on Learning Representation (ICLR 2023), 2023.
Harsh Jhamtani and Taylor Berg-Kirkpatrickk Learning to describe differences between pairs of similar images. arXiv preprint arXiv: 10584, 2018.
Baoxiong Jia, Ting Lei, Song-Chur Lang-Chun// Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343-3360, 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0mm 0040888 2024a.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.0/483 2024b.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17xxvv/775.... 17050000000 2017.
Muhammed Kocabas, Chun-Hao P Huang. Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J Black. Spec: Seeing people in the wild with an estimated camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11035-11045, 2021.
Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Roman Pfuugfelder, Koni-Kristian Ka- marainen, Martin Danelljan, Abdelrahman Eldesokey, Gabriel Fernandez, Alan Lukezic, et al. The sixth visual object tracking vot2018 challenge results. Proceedings of the European Con- ference Computer Vision Workshops, pp. 3-53, 2018.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
13

---

Published as a conference paper at ICLR 2025
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- nextinnex--nteereeeee Tackling multi -maat, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024a.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen. Yinan He, Zhangwei Gao, Erfei Cui, et al. Omnicorpus: An unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.088: 08418, 2024b.
Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi- modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26763-26773. 2024c.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2041-2050, 2018
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07571 2023.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 10467-10485, Online and Punta Cana, Dominican Republic, November 202 la Association for Computational Linguis- org/ 2021 emn 818
tics. BuRp:http:://acaattto///
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b.
Hong Liu, Yue Liu, Mengyuan Wang, Yuyan Chen, Limin Shen, and Qinghua Zhu Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. app.1176 1174-1181.
Jayyng Liu, Dejia Xu, Wenhan Yang, Minhao Fan, and Haofeng Huang. Benchmarking low-light image enhancement and beyond. International Journal of Computer Vision, 129:1153-1181 2021b.
Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma. A deep learning-based approach to progressive vehicle re-identification for urban surveillance. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp- 869-884. Springer, 2016.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model all-around player? arXiv preprint arXiv:2307.0628/ 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Haoyu Lu, Wen Liu. Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vi: towards roal-world vision- -aaiuane understanding. arXiv preprint 2003 2024a.
14

---

Published as a conference paper at ICLR 2025
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang. Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110. 2021.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0225.5 2023.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation mobile devices. arXiv preprint arXiv:2406. 08451, 2024b.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao I. Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022.
U-V Marti and Horst Bunke. The :am-database: an english sentence database for offline handwriting recognition. International journal on document analysis and recognition, 5:39-46, 2002.
Laurent Mertens, Elahe' Yargholi, Hans Op de Beeck, Jan Van den Stock, and Joost Ven- nekens. Findingemo: An image dataset for emotion recognition in the wild. arXiv preprint
arXiv:2402.01335, 2024.
Morris Moscovitch, Lynn Nadel, Gordon Winocur, Asaf Gilboa, and R Shayna Rosenbaum. The cognitive neuroscience of remote episodic, semantic and spatial memory. Current opinion in neurobiology, 16(2):179-190, 2006.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEEE, 2008
OpenAI. Gpt-4o. http:///ppenaa.ccomm///e////eeee
2024.
Paritosh Parmar and Brendan Morris. Action quality assessment across multiple actions. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1468- 1476. IEEE, 2019.
Paritosh Parmar and Brendan Tran Morris. Learning to score olympic events. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 20-28, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine-Horunng and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi- modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4542-4550, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Amir Rosenfeld, Markus Solbach, and John K Tsotsos. Totally looks like-how humans compare, compared machines. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 1961-1964, 2018
Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nicßner. Faceforensics+#: Learning to detect manipulated facial images. In Proceedings of the IEFF/CVF international conference computer vision, pp. 1-11, 2019.
15

---

Published as a conference paper at ICLR 2025
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1555.00855 2015.
Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pp. 4477-4486, 2018.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019, 2016.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. arXiv preprint 2.ki::240..888 2024.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference machine learning, pp. 843-852, PMLR, 2015.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223, 2017.
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13636-13645, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Saan-Bapiiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 18805 2023.
Qwen Team. Qwen2. 5: A party of foundation models, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2377,11,0,,,97,,,, 2023,
Mikaela Angelina Uy, Quang- Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Ki Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1588- 1597,
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint 24X0::2406009411 2024.
Limin Wang, Yu Qiao, Xiaoou Tang, al. Action recognition and detection by combining motion and appearance features. THUMOS14 Action Recognition Challenge, 1(2):2, 2014.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purposs. foundation models on low-level vision. arXiv preprint arXiv:2309. 14181, 2023.
16

---

Published as a conference paper at ICLR 2025
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Putlern Recognition (CVPR), pp. 9777-9786, June 2021.
Binzhu Xie, Sicheng Zhang. Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint arXiv:2306. 14899, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296. 2016.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lylm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023.
Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with language-guider diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9452-9461, 2024.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408. 08872, 2024.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint 18X9::88090003 2018.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5188-5197, 2019.
Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.166. 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-disciplnne multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, and Song- Chag- Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effecctiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 586-595, 2a.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. From facial expression recogni- tion interpersonal relation prediction. International Journal of Computer Vision, 126:550-5699 2018b.
Lirui Zhao, Tianshuo Yang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Rongrong Ji. Diffree:: Text-guided shape free object inpainting with diffusion model. arXiv preprint arXiv:2407.1600 2024.
17

---

Published as a conference paper at ICLR 2025
Wenliang Zhao, Yongming Rao, Yansong Tang, Jie Zhou, and Jiwen Lu. Videoabe: A real-world video dataset for abductive visual reasoning. IEEE Transactions on Image Processing, 3 31:6048- 6061, 2022.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pp. 11 1116-1124, 2015.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Luowei Zhou, Nathan Louis, and Jason Corso. Weakly-supervised video object grounding from text by loss weighting and object interaction. arXiv preprint 8050024805.02234, 2018.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4; An open, billion- scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
18

---

Published as a conference paper at ICLR 2025
A
MMIU DETAILS
A.1 MULTI-IMAGE RELATIONS
Overall, inspired by cognitive psychology, MMIU encompasses three broad types of image relation-
ships: semantic, temporal, and spatial. Furthermore, we refine all detailed types as follows:
Low-level semantic relations: This mainly refers to multi-image comparisons of low-level visual features, such as lighting, quality, and saturation.
High-level semantic relationship (objective): This refers the objective assessment of high-level image features, such as objects (e.g., dog), attributes (e.g., number), and rela- tionships between objects (e.g., person serving ball. person catching ball).
High-level semantic relationship (subjective): This refers the subjective assessment of high-level image features, such as thematic association (e.g., determining whether set of images conveys theme) or emotional association (e.g., identifying the emotions expressed in the images).
Discrete time (event) temporal relationship: Compared to continuous video frames, this mainly refers to discrete event/time sequence image tasks, such the analysis and reason- ing of multi-step tutorials.
Continuous time temporal relationship: mainly refers to video frame sequence tasks, including perception (e.g., action classification) and reasoning (e.g., action prediction).
Two-dimensional spatial relationship: This mainly refers to two-dimensional spatial multi- image relationships, such as rotation, translation, and symmetry.
Three--dmensioonl spatial relationship: This mainly refers to multi-image relationships in tiree-dimensional spatial contexts, such as different perspectives and depth variations.
A.2 HIFRARCHICAL, STRUCTURE OF MMIU
Image relationships and corresponding tasks. We present all 7 types of image relationships in the distribution of tasks for each type of image relation-
MMIU, totaling 52 tasks. ship.
Table 4: Details of tasks classified by image relationship our MMIU.
Image Relationship
Task
Number
Two-dimensional spatial relationship
raven-progresive-maatiis
9
jissa--puzllessolvnng, image-
captioning with-spattil-connex..
icon-question nonn-uuestion-asserrnng -iistriingwithh
spatial-context, image-text-
sarrivaal-withhspaaaal---omm
image-spatial- tranaiialrrassoommation
estimation, homography-
estimation, point-tracking,
single-ohject-tracking
19

---

Published as a conference paper at ICLR 2025
Table continued from previous
page
Image Relationship
Task
# Number
Three-dimensionall spatial relationship
scceme-reconsttructinn
11
JureeDoojject-ettectie
egocentríe-video-uuustio
answering, threeID-object-
tracking, threelD-osse-stimaatie
multiview-reasoning, multiview-
action-recognition, threeD-
depth- depth-estimation,
threeD-
question-ansee answering.
threed-
cad-recognition, threed-indoor- recognition
Discrete time (event) temporal relationship
visual-coherence, textual-cloze,
gui-anp--ecoonition, gui-next-
action-prediction, visual-cloze, visual-ordering
Continuous time temporal relationship
general-action-recogiiin
video-captioning. next-img-
prediction, temporal-ordering,
-imme-iideo--ndeestannnngg.
action-quality -uality-assessmentt
temporal- localiizaon,alnnnnnn,, mevis
Low-level semantic relations
soaul-guality-ussessmee forensic-detection
High-level semantic relationship (objective)
13
image2image-retrieaall sketch2image-retrivaal. vehicle-
retrieval, text2image-retrieaal
face-retrieval, handwritten-
retrieval, person-reid, spot- the-diff, spot-the-similarity,
visual rssual-correspoddece semantic- correspondence. functional- correspondence
High-level semantic relationship (subjective)
emotion-recooniition, casuality- 3
reasoning, multiple-image- captioning
Tasks and corresponding datasets. To introduce MMIU more thoroughly, need introduce all 52 tasks in MMIU, their specific descriptions, and which datasets they come from. Table Table [] and Table 7 respectively show the specific descriptions and data sources of tasks corresponding temporal relationships, spatial relationships, and semantic relationships.
20

---

Published as a conference paper at ICLR 2025
Table 5: Task descriptions and corresponding datasets for multi-image tasks in temporal relation- ships
Task Name
Task Description
Dataset
Action Quality Assessment
Action Quality Assessment involves evaluating the quality of an action or movement depicted in a se- quence of natural images. Given sequence of natu- ral images capturing the action, the task requires as- sessing the quality of the action movement.
Olympic
(Parmar
Tran Morris
2017). AQA-
7 (Parmar &
Morris 2019
Action Recogni- tion
General Action Recognition is vision task that in- volves recognizing and classifying the actions or ac- tivities depicted in a sequence of natural images.
Kinetics Kay al. 2017)
Meme Video Un- derstanding
Meme Vidco Understanding task involves under- standing and interpreting the content and context of meme video, where the visual input consists of sequence of synthetic images. The task requires pro- viding explanation of the meme video content or context.
FunQA Xie et al. 2023
Mevis
MeVIS involves localizing objects of interest within series of natural images.
McVIS (Ding
et al. 2023)
Next Image Pre- diction
Next Image Prediction refers to predicting the image at the next moment based given series of images in chronological order.
Moving MNist
Sri-
vastava et al.
2015)
Temporal Local- ization
Temporal Localization involves identifying the in- stance or target in sequence of frames or video at specific time or time range. The task requires ana- lyzing sequence of natural images and determining the identifier of the target instance in the sequence.
YouCook2
(Zhou et al.
2018). THU-
MOS14
(Wag et al. 2014)
Temporal Order- ing
Temporal Ordering is vision task that involves ranging a sequence of shuffled natural images in the correct temporal order.
Penn-Action
(Chiu et al. 2019)
Video Captioning
Video Captioning involves generating textual de- scriptions for sequence of video frames, providing narrative or informative explanation for the visual content.
MSVD
(Chen &
Dolan 2011).
MSRVTT
(Xu et al.
2016)
visual close
Visual cloze style questions test a skill similar to that of textual cloze task with the difference that the miss- ing information in this task reside in the visual do- main
RecipeQA
Yagcioglu
et al. (018)
textual close
Textual cloze style questions test the ability to in- fer missing text either in the title or in the step de- scription by taking into account the question's 's con- text which includes a set of illustrative images be- sides text
RecipeQA (Yagcioglu et al. 2018)
visual coherence
Visual coherence style questions test the capability to identify an incoherent image in an ordered set of images given the titles and descriptions of the corre- sponding recipe the context.
RecipeQA (Yagcioglu et al. 2018
21

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
visual ordering
Visual ordering questions test the ability of a system in finding a correctly ordered sequence given a jum- bled set of representative images of recipe. As in the previous visual tasks, the context of this task con- sists of the titles and descriptions of recipe
RecipeQA Yagcioglu
et al. 2018)
gui app recogni- tion
Identify and analyze the applications utilized in the graphical user interface (GUI) segment of the episode.
GUI-Odyssey (Lu et al.
2024b)
gui next action prediction
Predict the subsequent action based on the infor- mation provided in the previous screenshot and the given graphical user interface (GUI) navigation in- structions.
GUI-Odyssey
(Lu et al.
2024b)
Table 6: Task descriptions and corresponding datasets for multi-image tasks in spatial relationships
Task Name
Task Description
Dataset
Raven's Progres- sive Matrices
Raven's Progressive Matrices is a visual reasoning task involving synthetic images. Given a set vi- sual patterns, the task requires identifying the miss- ing pattern from a set options.
RAVEN
Zhang et al.
2019). PGM
(Santoro
et al. 2018)
Jigsaw Solving
Puzzle
Jigsaw Puzzle Solving task involves solving a jigsaw puzzle made up of natural images. The visual input consists of shuffled patch of natural image. and
MSCOCO
(Lin et al.
2014).
the instruction asks to rearrange the patches to recon- struct the original image. The patches can be fed as set of images.
WikiArt
Saleh
Elgammal
2015)
Image Spatial Transformation Estimation
Given pairs of images depicting scenes before and after spatial transformation (e.g., rotation, transla- tion), predict the type and magnitude of the transfor- mation that occurred.
MSCOCO
Lin
al.
2014
Image Caption- ing with Spatial Context
Given a set of images (in NLVR, each sample can be split into images), generate one sentence consistent with all images terms of spatial context.
NLVR Suhr
ct al. 2017)
Icon Question Answering with Spatial Context
Answer a multi-choice question in an icon image context.
IconQA Lu
1021
(a subset of addresses spatial rea- soning with multi-image.)
Image Text Re- trieval with Spa- tial Context
Given text addressing spatial context, identify the matched image within candidates.
SPEC Ko-
cabas et al.
2021)
22

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
Homography Es- timation
Computing the 3x3 homography matrix that maps the coordinates points in one image to their corre- sponding coordinates in another image. (Two images of the same planar.)
HPatch
(Balntas
et al. 2017).
Kaggle
for
HPatch
Single Tracking
Object
Visual Tracking involves following an object or re- gion of interest across series of images or frames. Given a query natural image with visual annotations, the task is to track the specified object region in subsequent natural images.
TAP-Vid-
DAVIS, TAP-
Vid-RGB-
stacking
Doersch
ct al. 2022)
Point Tracking
Point Tracking involves locating and tracking spe- cific point of interest within a natural image. Given a query natural image with a visual mark indicating the initial position of the point, the task requires finding the same point within another natural image.
Mevis Ding
et al. 2023)
3D Classification -CAD
3D classification CAD involves classifying 3D im- ages into specific categories based their content and features.
ModelNet40
Wu al.
2015)
3D Classification Indoor Point
3D classification indoor Point Cloud involves cate- gorizing indoor scenes based on 3D point cloud data.
ScanObjectNN
(Uy et al.
Cloud
2019)
Multi-view Rea- soning
This task is centered evaluating the multi-view reasoning capabilities of models. The objective is to deduce the relative camera motion based two im- ages of an object captured from different viewpoints.
BLINK Fu
et al. 2024b)
3D Object Detec- tion and Pose Es- timation
Detect objects and estimate their poses in 3D space using multiple views of the scene. Input Format:
ScanNet Dai
al.
2017).
Set of RGB images captured from different view- points, and a query image. Output Format: De- tected objects with their 3D bounding boxes and poses based on the query image.
SceneNet
(anda
et al.
2016).
SUN
RGB-
D
(Song
et al.
2015).
nuScenes
(Caesar et al.
2020)
3D Scene Recon- struction
Reconstruct the 3D geometry of scene. Input For- mat: An RGB image and depth image. Output For- mat: A set of images captured from different view- points for this scene.
ScanNet Dai
et al. 2017).
Matterport31
(Chang et al. 2017). SUN
RGB-D
(Song et al.
2015)
3D Object Track-
Sequences of RGB-D images capturing ob- ject motion over time. Task: Track the movement of objects in 3D space across multiple frames. Output: Trajectories paths of objects in 3D space (e.g., sequence of 3D poses (position and orientation)..
Input:
KITTI
ing
(Geiger
let al. 2013).
nuScenes
(Caesar al.)
2020)
23

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
Multi- View Ob- ject Instance Seg-
Estimate the instance-level segmentation map for query image based on multiple images captured from different viewpoints. Input Format: A Set of RGB images captured from different viewpoints, and query image. Output Format: A corresponding instance-level segmentation map for the query image.
ScanNet (Dai
et al.
2017).
mentation
SceneNet
(Handa
et al.
2016).
NYU Depth
Dataset
Siiberrman
al. 2012).
SUN RGB-D
(Song al.
2015)
Multi-View Depth Estimation
Estimate the depth map for query image based on multiple images captured from different viewpoints. Input Format: Set of RGB images captured from different viewpoints, and a query image, Output For- mat: A corresponding depth map for the query im- age.
MegaDepth
(Li
&
Snavely
2018).
SceneNet
(Handa et al.
2016). SUN
RGB-D
(Song al.
2015)
Multi- View Ac- tion Recognition
Recognize human actions activities in ing information from multiple views. Input Format: A set of RGB images from multiple views. Output Format: Action labels/categories.
NTU RGB+D
Shahroudy
al. 2016).
PKUMMD
Liu et al.)
3D Question An- swering
Given inputs the point cloud and a question about the 3D scene (real life), the model aims output the correct answer.
ScanQA
(Azuma
et al.
2022).
NuScenes-
QA
(Qian
et al. 2024).
SQA3D (Ma
al.
2022)
Egocentric Video Question- Answering
Egocentric Video Question-Answering (EgoVQA) is a task that involves understanding and reasoning about activities and events from the first-person per- spective. In this task, the model is presented with sequence of egocentric (first-person) videos, typ- ically captured by wearable cameras such as head- mounted cameras. The goal is to answer questions related the content and context of the videos.
Ego
TTa et al.
2022
24

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
Visual Naviga-
Given series of images captured by robots or drones in different locations, the model outputs navigation commands robot actions based its spatial rea- soning about the environment. Outputs may include directions for navigation, obstacle avoidance strate- gies, or object manipulation instructions.
DriveMLM
tion and Robotics
(synthetic),
YouTTbe--
VIS
Yang
2019). et al.
DAVIS
(Pont-Tuse Tuset
et al. 2017).
VOT2018
(Kristan
al. 2018)
Table 7: Task descriptions and corresponding datasets for multi-image tasks in semantic relation- ships
Task Name
Task Description
Dataset
Visual Quality Assessment
This task is to evaluate the visual quality of two images, such as resolution, brightness, and clar- ity.
Q-bench
(Wu al.
2023
VE-LOL-L
(Liu et al.
2021b)
Forensic Detection
This task involves multiple images and requires determining which image fake and not authen- tically composed.
FaceForensics++
(Rossler et
al.
2019
ForgeryNe Net
(He et al
2021
Visually Grounded Reasoning
This task involves giving pair of images and checking the sentence description matches the image pair.
NLVR v2
Suhr et al.
2017).
MaRVL
(Liu al.
2021a)
Image-to-Image Retrieval
Image-to-Image Retrieval involves retrieving the candidate image ID that most similar the query image.
places365
(Zhou al.
2017). tiny-
imagenet
(Le
Yang
2015
Sketch-to-Image Retrieval
Sketch-to- Imaee Retrieval involves retrieving candidate images that are most similar to given sketch image.
quickdraw
(Ha
&
Eck| 201 7).
DomainNe Net
(Peng et al.
2019
25

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
Text-to-Image Retrieval
Text-to-Iman Image task involves generating an image based on a given textual description. The visual input consists of natural images, and the task in- struction example could he 'Generate an image based on the provided text description." The out- put provides the identifier of the generated im- age.
CUB220-
2011 (Wah
et
al.
2011
Flowers 102
Nilsback
Zis-
serman
2008
Person Re-ldentification
Person Re-ldentification involves identifying and matching person's 's appearance across dif- ferent camera views or over time. The task re- quires comparing query image of person with multiple candidate images to determine if the same person appears in the candidates.
Market-
1501-v15
(Zheng
et al. 2015)
Vehicle Re- Re-Identification
Vehicle Re- leeeeitifficatoon involves identifying specific vehicle from set of candidate vehicle images based on a given query image of the ve- hicle.
veri-776
(Liu al. 2016
Face Verification
Face verification involves recognizing the iden- tity of a query face image by comparing it with each support face image with an annotated iden- tity.
LFW
(Huang
et
al.
2008).
CelebA
(Liu et al.
2015)
Handwritten Text Retrieval
Handwritten Text Retrieval and Verification in-
IAM Marrt
volves retrieving and verifying handwritten text from a query image against candidate images containing handwritten text.
& Bunke
2002
Spot the Difference
Spot the Difference task involves identifying the numeric value corresponding to the number of differences between two natural images.
spot-
the-diff
(Jhamtani
& Berg-
Kirkpatrick
2018
Spot the Similarity
Spot the Similarity involves identifying the simi- larity between multiple images and providing an explanation for the judgment.
TLL
RRoee-
feld et al.
2018
DISC21
(Douze
ct al. 2021
Visual Correspondence
This task involves providing several images from different angles and finding the same points in different perspectives, such as specific pixels.
BLINK
(Fu et al.
2024b).
ScanNet
(Dai et al.
2017
26

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Task Name
Task Description
Dataset
Semantic Correspondence
The task requires providing several images of different species and identifying semantically identical points across the different species, such as the head of horse and the head of a human.
BLINK
(Fu et al.
2024b),
MISC210K
Functional Correspondence
The task requires providing several images of different tools and identifying functionally iden- tical points across the different tools, such as the handle a broom and the handle of tooth- brush.
BLINK
(Fu et al.
2024b),
FunKPoint
Emotion Recognition
The task is provide multiple images, most of which depict the same emotion, and identify the image that represents different emotion.
FindingEmo
(Mertens
et al.
2024
ExpW
(Zhang
et
al.
2018b)
Casuality Reasoning
The task is provide multiple images from video and ask about the cause leading to a spe- cific result, such "Why did the little girl stop the car?" The answer might be: "She stopped to wait for her mom."
NeXTQA
(Xiao et al.
2021).
VideoABC
(Zhao al.
2022
Multi-image Captioning
The task is to provide multiple images of dis- crete events and require a title for them.
SSID (Ab-
delhamed
et al. 2018)
A.3 TASK ABBREVIATIONS
Due to the large number of tasks and models evaluated the benchmark, we use abbreviations to streamline the manuscript. Table lists the abbreviations used throughout the paper.
A.4 CONSTRUCTION
Metadata. We organize the dataset of each collected task into metadata format. Specifically, the metadata is organized as a dictionary, with keys categorized into two main groups. The first group comprises essential keys for task information, including task name, task description, input format and output format. The second group consists of keys that are unique to individual samples and may vary accordingly, including data sources. input image path, input context, question, visual components and output (i.e., the ground truth). This structured format helps us easily convert it into multiple-choice questions without losing information. The specific metadata format can be referenced in T
Based on the above predefined template, we obtain the detailed information of metadata primarily following two steps. In the first step, we create a Python script for each dataset to extract relevant keys directly from the original data, such as the image path, data source, and the ground truth. Since samples within dataset tend to share similar features, co-authors predefined elements such as the specific question template, input context template, and detailed visual components for each dataset. In the second step, our co-authors manually conduct a sample review focusing sample-specific keys, including the predefined question templates and the output information.
Ensuring Challenging Distractor Options in Samples. To create difficult distractor options, primarily rely on prompt engineering with GPT-4o, utilizing tailored prompts for specific tasks and applying in-context learning (with 2-shot or 3-ohot examples). The generated options thoroughly
27

---

Published as a conference paper at ICLR 2025
Table 8: The Abbreviations of terms mentioned in this paper and their corresponding full terms.
Abbreviation
Abhreviation
Tasks
12IR
Image2Inage Retrieval
S2IR
Skeich2lmage Retrieval
VR
Vchicle Retrieval
FD
Forensic Detection
CR
Causality Reasoning
T2IR
Texi2maaee Retrieval
FR
Face Retrieval
ER
Emotion Recognition
Functional Corresunndence
Handwiillen Retrieval
VCor
Visual Corresenndence
VGR
Visually Grounded Reasoning
Spot the Difference
VQA
Quality Assessment
MIC
Multiple Image Captioning
PR
Person Re-ID
Semantic Conespondence
STS
Spot Similarity
GAR
General Action Recognition
AQA
Action Quality Assessment
NIP
Next Image Prediction
Meme Vidco Understanding
TT.
Temporal Lncalization
MFV
McVis
Textual Cloze
GuAR
GUI App Recognition
GNAP
GUI Next Action Prediction
VO
Visual Ordering
Visual Coherence
VidCap
Captioning
TO
Temporal Ordering
VClz
Visual Cloze
Multiview Action Recognition
Homography Estimation
3DOT
3D Object Tracking
ICSC
Image Captioning with Spatial Context
Multiview Reasoning
Image Text Retrieval with Spatial Context
IQASC
Icon Question Answering with Sputial Context
3DE
3D Depth Estimation
RPM
Progressive Matrices
3DPE
3D Pose Estimation
3DSR
3D Scene Reconstruction
JPS
Jigsaw Puzzle Solving
3DCR
Rocognition
3DOD
3D Object Detection
ITTE
Image Spatial Transformation Estimation
EVQA
Egocentric Video Question Answering
3DIR
3D Indoor Recognition
3DQA
3D Question Answering
Point Tracking
SOT
Single Object Tracking
Table The example of the metadata.
Metadata Example
Task Info: {
TaskName: Name of the task, TaskDescription: Description of the task Input Format: Input data formats, such as text and images,
Output Format: Output data format, such text image.
Samples:
Source: The data set source of this sample,
Input: {
Input Image: The path of the input image, in list format
Input Context: The context needed to solve the problem, in text form.
Question: Input question or instruction.
Visual Component:
Image type, such as depth image, natural image.
Output: The actual textual output of the problem, which may be text (caption task) image path (retrieval task), etc.
reviewed, and any inaccuracies are corrected. Notably, GPT-4o is capable of producing highly chal- lenging options two key ways. Firstly, the distractors are often semantically similar the ground truth (GT), making them particularly confusing in classification tasks such as 3D Indoor Recogni- tion. For example, if the correct answer is "sink," the distractors might include "cabinet," "bed or "bin," all of which share similar semantic background with the GT. Secondly, the distractors can exhibit ambiguity in their relation to visual content. For captioning tasks, we leverage GPT-4o to analyze images and assist in generating plausible incorrect options, thereby mitigating the hallu- cination issues common with text-only LLMs. In tasks involving object detection, we introduce perturbations the GT using Python scripts to create distinct yet challenging options. Overall,
28

---

Published as a conference paper at ICLR 2025
Table 10: LPIPS between MMIU and MuirBench, MilesBench
LPIPS
0.1
0.2
0.3
0.4
MuirBench MilesBench
0.1
1.8
6.4
11.8
1.6
3.3
7.8
12.9
believe that MMIU generates options with appropriate level of difficcllyy, enhancing the robust- ness of our evaluation tasks.
Visual Grounding of MMIU Answers. To ensure that the generated ground truth answers in MMIU are visually grounded, we adopt the following strategies. First, during the question design process for each sample, meticulously craft specific question templates. We avoid embedding overly specific information within the templates, ensuring that the questions heavily require reliance on visual content for accurate answers. Second, when generating answer options, we create more complex distractors to prevent models from bypassing visual information and providing responses solely based on correlations between options or prior knowledge.
Unanswerable Set. We consider five strategies for modifying an answerable instance into its unanswerable counterpart with minimal changes. The five strategies include replacing key words, replacing the answer image, replacing other images, shuffling all images, and using irrelevant ques- tion/image sets. each task, we select the most suitable strategy or combination of strategies to construct the corresponding unanswerable task. The specific construction methods for each task can be referenced in Table
The difference with other benchmarks. MMIU is comprehensive benchmark designed evalu- ate the multi-image capabilities multimodal large models across wide range of tasks involving semantic, temporal, and spatial relationships among multiple images. Compared to other multi- image benchmarks such as MuirBench, MMIU offers broader task coverage and more in-tepth analysis of results. Moreover, MMIU distinet from these benchmarks in its data, representing completely independent benchmark. We assessed the similarity between the images MMIU and those in MuirBench and MilesBench using LPIPS (Zhang al., 2018a) for visual similarity. Many image editing works adopt LPIPS as metric, and based on their reported results, an LPIPS value below 0.1 be considered as indicating that the images relatively similar (Zhao et al. 2024 Xu et al. 2024). As shown in Table 10 when threshold of 0.1 is applied, the image duplication rates are low 0.1% and 1.6%, respectively.
Table 11: Causes of Unanswerable Tasks Considered in Their Construction: Note that single task often corresponds to multiple causes for being unanswerable.
Task
replace key word
replace answer image
replace other images
shufile all images
irrelevant question/image
CR
Yes
No
No
No
Yes
T2IR
Yes
Yes
No
No
Yes
VCo
No
Yes
No
No
No
VO
No
No
No
No
GAR
Yes
TL
Yes
No
No
Yes
TO
No
No
No
Yes
No
VidCap
No
No
No
Yes
Yes
HE
No
No
Yes
Yes
Yes
IQASC
Yes
Yes
Yes
No
Yes
ITTE
No
Yes
IRRSC
Yes
No
Yes
JPS
No
No
Yes
Yes
Yes
MAR
No
No
Yes
No
Yes
3DE
No
Yes
Yes
Yes
Yes
3DOD
No
Yes
Yes
No
3DPE
Yes
Yes
3DSR
Yes
Yes
Yes
3DCR
No
Yes
No
No
Yes
3DIR
No
Yes
No
No
Yes
29

---

Published as a conference paper at ICLR 2025
B EXPERIMENT DETAILS
B. MODEL DETAILS
Table T2 provides overview of the LVLMs utilized in this study, detailing their parameter sizes, visual encoders, and LLMs. It is important to mention that the evaluation process was carried out according the protocol established by OpenCompass. (Contributors 2023
Table 12: Model architecture of 24 LVLMs evaluated on MMIU.
Models
Parameters
Vision Encoder
LLM
GPT-4o OpenAT
2024
Geminil.5 Hash Team ét al. 2023
Geminil.0 Pro Viston Team atai.
2023
Claude3.5-Somne (Anthropic)
Mantis-idefics2 Jiang al.
2024b
8B
CLIP VIT-L/14
Mistral-7B-v0.1
Mantis-SigCLIP (Jang of 20241
8B
CLIP VIT-L/14
LLaMA-3-8B
XGen-MM (Xee et al. 2024
4B
XGen-MM- Mite--MM-vssonnneccodrr
Phi-3-mini
LLaVA-NexI- Vicut-iicn---/ (Ltu ct al. 2024a
7.1B
CLIP VIT-L/14
Vicuna-v1.5-77
SENNA-Next-Inteiiine (Lio et al.
20246
7.1B
CLIP VIT-L/14
Vicuna-v1.5-75
IntemVL2-Pro (Chen al. 20224
IntemVL2 Chen ct
20248
8B
Intern -00-300-448pp
Internlm2-5S76-chat
IntemVL-Chaa-Vi..> (C
al.
2024a
26B
Intem
MttenL.M--Chat--00
Mini-InternVL-Chhll (Che et al
2024a
2B
IntemVVT--300 VYT-300M
InternL.M-CChtt-1.81
Mini-InterWVL-Chatt [Chnn ct al
2024a
4B
Intern VTT-30UM
Phi-3-mini
DespScck-VL-1338 [Lu al. 20:4a
1.3B
SAM-B & SigLIP-L
DeckSock-1.3B
DeepSeck-VL-79 (Lu el 2074a
7.3B
SAM-B SigLIP-L
DeekSeek-7B
Monkey-chat Li
20/22
9.8B
CLIP-VES -uge
Qwen-7B
XComposer2 Dong al. 20224
7B
CLIP VIT-L/14
M2-77
XCumpuser2- 8b Dong al. 2024
1.8B
CLIP VIT-L/14
InternL
ShareGPT4V Chen al. 2025
7.2B
CLIP VIT-L/14
Vicuna-v1.5
SharedCaptioner (Chen al. 2023
8B
EVA-G
Internl M--7B
LLaVA-vi.5-7BB
el 20/410
7.2B
CLIP VIT-L/14
Vicuma-v 1.5-7B
Qwen-Base (Bai ct
9.6B
CLIP VIT-G/16
Qwen-7B
Qwen-chat Bar al.
2124)
9.6B
CLIP VIT-G/16
Qwen-7B
Idefics2-8h Laurengon el al
2024
8B
SigLIP-L
Mistral-7B
Idefics-bb-instrucc [5a el al
2023
9B
CLIP-VIT-H--4
Llama-7b
Monkey-chat al. 220240
9.8B
Vir-BigG
Qwen VT.
MiniCPM-Llamuuss-..
[1l
2024
8.4B
SigLip-L
Llama3-8B
V2 (Aaadalla
9.6B
CLIP VIT-G/16
Qwen-7B
GLM-4V-9B (GLM al. 2024
13B
GLM-4-9B
Llama3. Instructt (Dubey el al.
2024
8B
M2.55LL2..--nssmiiic (Car et al.
2024
7B
Qwen2.5-Instruue (Team 2124
7B
Table 13: The complete prompt template for MMIU
Model Prompts
Context: (CONTEXT)
Question: [QUESTION]
Choices:
(A) [OPTION A)
(B) (OPTION B)
(C) (OPTION C)
(D) (OPTION_D)
Hint: Please answer the option directly like A, B. C, D...
B.2 MODEL PROMPTS
According to Vista (Lu al. 2023). our prompt consists of four parts: the question, options, the hint indicating the answer format, and the context of this task (e.g. Your task is to track the movement of objects in 3D space across multiple frames, select from the following choices.). For
30

---

Published as a conference paper at ICLR 2025
Table 14: Quantitative results across 52 tasks on test set are summarized. Accuracy is the metrie, and the Overall score is computed across all tasks. The maximum value of each task is bolded. Notice that although Intern VLL.5-chaa supports multiple image inputs, its training phase did not incorporate multi-image data. The full term of task abbreviation can be found in Table iin Appendix.
VIR
128
NEY
313
113
218
153
280
D1B
72B
26.3
110
31A
968
343
153
26A
25.
315
313
9.3
63.
313
913
25A
1911
143
36.
123
31A
EhA
343
V6A
1911
124
115
123
164
Mul
63.
363
560
136
96
455
875
25.5
91.
645
3633
13A
66.8
51.1
345
260
12A
+4.3
31.5
164
153
66.
128
6.66
56n
4.8
3/2
6m
713
817
918
143
1014
22
312
968
230
803
2011
203
113
314
043
40.
61:
361
365
214
365
303
T1A
350
3-5
213
214
TLA
140
21.
345
214
314
203
315
918
Noaia
466
615
255
510
24
71
308
06.
06.5
21.8
615
T1A
514
165
734
1.5
214
214
364
234
80n
818
15.
131
140
110
11.
1301
1411
1011
201
110
911
819
110
138
240
576
271
760
03
313
51a
250
21.5
15.
21.
E1A
156
11.
163
003
180
360
430
36
401
340
361
116
201
736
160
150
156
196
566
786
21-
30
22:
135
301
013
618
540
180
53.
212
100
315
350
196
186
136
211
314
130
314
Misttariiieaaa 557
215
150
360
255
11K
213
100
308
303
201
158
738
301
13
150
315
8-3
131
111
213
333
1391
215
250
150
180
280
318
888
20
280
11
153
03
75
114
153
313
183
Dreplia
43
136
33
11.
10
308
8.
150
145
368
3631
13
154
11.0
760
500
18.
08
13
663
33
0.3
254
Das agodd
26-
246
214
212
345
190
16-
258
196
543
248
183
381
300
182
25
30
60
113
350
145
35
313
10
3M
145
319
21
315
301
15
153
120
608
3-3
133
59
112
143
18
130
123
26
83
279
110
18
185
180
36
1is
53
13'
13
203
113
013
E0
30.2
16
113
345
151
46
160
315
60
174
254
201
48
3M
25
166
56x
31.5
24
146
153
63
312
12.
223
114
22
133
50
11.
30
323
201
133
X1
313
365
120
76
13
60
47A
176
250
134
45
18
161
21.
18.
12
21
213
19
112
181
201
283
49
10
833
633
E0
2011
1961
2011
1.8
115
2011
100
300
213
31
218
254
412
138
230
154
258
130
41
36
46
750
190
369
160
380
416
343
156
143
150
166
16n
180
56
246
156
304
J80
28n
118
20
186
302
312
206
186
358
312
616
116
196
186
186
194
25.
224
106
118
OWIES
380
085
300
150
30
015
428
300
50
150
100
201
158
430
100
180
180
350
405
80
130
380
300
20
103
133
613
413
134
343
130
254
200
348
138
813
images, we insert them into the text to form coherent prompt. The complete prompt is as shown in Table 13
B.3 MULTITASK ANALYSIS
Differences in model capabilities across various image relationships. As shown in Figure g we visualize the average performance of all models across 7 specific image relationships. Detailed analysis can be found in Section of the main text.
Differences in model capabilities in testmini set. To mitigate the computational cost caused by the large number of samples. randomly sample 20 examples from each of the 52 tasks, resulting in testmini set with 1,040 samples while preserving the original diversity of MMIU. As shown in Table evaluate model performance on the testmini set across seven types of image relationships.
15
31

---

Published as a conference paper at ICLR 2025
Table 15: Quantitative results across 7 image relationships on testmini set are summarized. Accuracy the metric, and the Overall score is computed across all tasks. The maximum value of each task bolded.
High-evel-sub High-level-oby
Frequency Random
30.9
29.1
29.2
38.3
29.2
36.1
26.9
27.1
22.3
24.9
33.2
21.1
32.9
24.5
28.3
Cloved-source LVLMs
GPT-4o
55.6
56.7
55.6
87.5
75.0
Clunde3.5
54.3
Geminil
Geminil 10
11.2
18.3
50.0
$7.5
63.3
37.5
28.9
36.4
Adequate Multi-Image SPT LVLMs
Mamis-ideles2-SB
45.3
40.0
43.1
53.8
99.2
52.5
37.8
42.3
Mamis-SigCLIPP-a
35.8
XCrn-MMM-merlvved---
Multti Image impur I.VEMs
IntemVL2-PPr
49.8
59.2
45.5
73.8
75.8
57.3
31.2
37.7
Intemel1.5
18,7
15.0
50.5
41.2
63.3
27.3
35.6
35.5
IntemVL2-488
34.0
35.0
Mini chatt1
1.5---1--222
11.8
26.7
44,4
27.5
42.5
28.5
25.0
32.7
idefler2-bb
27.2
29.2
16.9
40.0
45.8
26.7
26.1
27.7
XCGen MM-single-4B
25.4
25.8
25.5
18.8
34.2
23.8
27.2
24.1
DeepSeck-A1-770
346
13.3
8.8
28.7
358
38.1
32.2
16.4
23.8
XCompuser-77b
Damingow2
22.7
17.5
21.9
27.5
XChmposer2-1 18hh
22.0
30.0
33.1
20.0
39.2
18.3
18.9
12.3
Qwen-chal
18.0
13.3
21.9
26.2
27.5
17.1
11.7
20.0
Abrliss-ttinstmet
13.2
26.7
5.6
12.5
35.8
9.0
11.1
11.8
Base
14.2
5,0
9.2
2.9
3.3
5.0
Single-Image Impar LVLMs
GL.M-4v-hh
26.6
20.8
37.5
22.9
23.3
25.0
28.7
39.2
21.5
18.3
20.9
MincCPMM-LLaaa-
21.7
20.0
52.5
27.5
40.8
14.8
10.6
25.9
1,577
18.7
20.8
16.2
17.5
31.7
19.6
16.1
16.8
sharegnii-77b
18.4
stanecapiioner
monkey-chul
14.3
10.0
13.8
11.2
25.0
Pure TEXT LLMS
GPT-4e
21.8
29.2
21.9
6.2
45.8
12.7
21.7
25.0
Caption
43.9
53.8
67.5
Qwen2.5
30.1
Caption
IntemL.M25
32.1
32.3
4022
$1.5
26.2
Caption
398
41.7
45.0
47.5
46.7
30.0
36.1
35.5
Llama3.1
20.1
30,0
32.5
36,3
21.7
28.8
26.1
31,5
Caption
38.3
42.5
46.2
53.8
39.2
38.1
28.9
35.0
Figure 8: The average performance comparison of some LVLMs on seven specific image relation- ships on test set.
32

---

Published as a conference paper at ICLR 2025
The results demonstrate that performance the testmini set effectively reflects the trends observed in the full test set.
The questions in MMIU are visual-dependent. MMIU, comprchensive benchmark for eval- uating multi-image understanding capabilities, heavily relies on accurate visual comprchension and cannot be solved using text alone, demonstrating distinct multimodal characteristics. Specifically, as shown in Table 3
we observe that when using state-of-the-arr large language models (LLMs) with only questions and options input, their accuracy is comparable to random selection. This indicates that MMIU fundamentally depends on visual information and that the design of distractor options robust, preventing inference of the correct answer based solely on the options. Furthermore, we investigate extracting captions for each image using GPT-4o and then performing reasoning with the L.LMM Experimental results show a significant improvement using this method, highlighting that accurate image understanding is foundational for solving MMIU. However, even with this approach, performance remains inferior to directly leveraging images for answering. For instance, the aceu- racy of GPT-4o with direct image input (55.5%) surpasses the performance achieved by caption extraction followed by reasoning (48.7%). This demonstrates that certain tasks in MMIU require visual processing rather than purely semantic understanding.
Why some models are less accurate than random selection. In MMIU, as shown in Table the accuracy of some models is even lower than that of random selection. Similar result also exist in MuirBench (Wang et al. (024) and MilesBench (Song et al. 2024) We have conducted a detailed investigation and analyzed the reasons.
1) For multi-image input models, errors occur frequently due to limitations in model capability or citrucctoonfollowinng ability, leading significantly lower accuracy. We summarize the proba- bility of "Z" answers for several models in the Table T6 Our findings are as follows: 1) The highest probability of generating "Z" answers occurs in spatial relationship tasks. This is because spa- tial data is relatively scarce, leading to insufficient model training. Additionally. spatial tasks tend to be more complex (some involve visual tasks), and models with limited spatial ability tend to gener- ate irrelevant responses. As shown in Table 3 even for powerful closed-source models, performance on spatial tasks remains below the overall average. 2) The lowest probability of "Z" answers oc- curs in temporal relationship tasks. This is because video data is the most readily available during training, enabling models to develop relatively strong capabilities in this area. As shown in Table 3. for closed-soune source models, performance on temporal tasks tends to be above the overall average. 3) instruction-following ability is equally important. As the model size decreases, instruction- following ability weakens, leading off-topic errors. In the error case analysis in Section [B.4 Mantis' instruction-following ability is significantly lower than that of closed-source models and larger open-source models like Intern VI.2-Pro. This results in situations where the model provides answers not in the options. II) For single-image models, lower accuracy or random selection errors primarily result from insufficient model capability or difficulty in generalizing long con- texts. We summarize the probability of "Z" answers for models in the Table T6] 1) When we use the concatenated image method, we find that "Z" answers are less frequent in single-mmaee mod- els. Instead, these models tend to select incorrect answers. This is because the models struggle to interpret concatenated images. 2) When we test with the concatenated token method, we observe similar issues to multi-image models. This is due to long context causing off-topic or irrelevant responses. For typical single-image models, even if long context input is supported, the training samples mostly consist of single images, which limits generalization. Specifically, average number of images per class for different image relationships: 10.02 for Temporal, 5.94 for Semantic, and 7.04 for Spatial. As the number of images increases, the probability of generating "Z" answers also increases.
calculate the
Table 16: Performance comparison of various models with multi-image input data on different tasks.
Model
Temporal
Semantic
Spatial
Overall
XComposer2-77
13.6
20.1
46.1
28.3
flamingov2
19.1
10.7
17.3
15.5
xgen-mm-single
15.1
29.8
35.8
27.9
Ilava-v1.5- (concat img)
0.6
0.1
0.9
0.5
sharecaptioner (concat img)
15.1
3.3
8.6
8.5
Bava-vv1.5-7b (concat token)
55.0
5.1
38.1
29.5
sharecaptioner (concat token)
38.9
25.0
34.1
32.2
33

---

Published as a conference paper at ICLR 2025
High resolution and dynamic resolution matter. High resolution and dynamic resolution critical factors. Research has shown that increasing resolution and employing dynamic resolution training significantly enhance image understanding performance (Lin et al.) 2023) This is because low resolution tends to overlook critical visual details, while training with a fixed resolution can lead to image distortion or inefficiencies in training and inference.In multi-image tasks, we observe that lop-performing models (c.g., VnternVL2-8B Mantis-idefics2, adopt high-resolution settings (e.g., 448, 384, and 336 respectively) combined with dynamic reso- lution training strategies. This approach allows the models to better comprehend image content and adapt to various input scales.
Next-Interleave) consistently
Table 17: Performance comparison of various models with multi-image training data in different stage on MMIU.
Model
Mulii-image Pretrained
Mulii-inage SFT
Size
Semantic
Temporal
Spalial
Overall
No
No
21
30.4
36.2
26.55
30.5
Mini IntemVL11.5.
No
No
4B
32.2
36.8
28.5
32.1
InternVT.I.S.ch
No
No
26R
33.4
45.2
35.5
37.4
LLaVA-Nee
No
No
7B
23.0
24.1
20.3
22.2
No
Yes
7B
35.7
32.2
31.1
32.4
Mantis-SigCLA
No
Yes
8B
53.6
38.4
35.7
42.6
XGen--MMPPhi3
Yes
No
4B
25.1
25.7
23.1
24.5
idefies2
Yes
No
8B
31.6
20.9
29.2
27.8
T1ternV1.2
Yes
Only video
39.5
31.2
Mantis-idsfies2
Yes
Yes
8B
56.3
40.8
39.2
45.6
AGmn-MM-Phi3-Imerlleed
Yes
28.7
29.1
Powerful vision encoder matters. The Intern series models demonstrate outstanding perfor-
mance MMIU. For instance, both Mini-meem 1.-- and xGen-MM- -M---MMMreeeeeeee-44B use Phi3-mini (Abdin et al., 2024) as their LLM, yet Intern VL achieves significantly better results. No- tably, the performance of Wwwe--eteen---ha----- 7B. We attribute this advantage partly to InternViT (Chen et al., (2024b). the vision encoder trained for Intern VL. As a powerful encoder, Intern ViT outperforms various CLIP-releated models across multiple benchmarks. This highlights the critical role of a strong vision encoder in improving per- formance on multi-image tasks.
even matches that of LLa Vaaa-eex--nttrrleaee
Discussion on the effectiveness of multi-image training. As shown in Table 17 by categorizing these representative models based on whether multi-image data included during pretraining or SFT, we observe the following: 1) Multi-image SFT effectively improves performance regard- less whether multi-image pretraining conducted. On the one hand, LLLaV--Net-Innterleee which applies interleaved visual instruction tuning LLaVA-Next (without multi-image pretrain- ing), still achieves performance gains. On the other hand, XGen-MM-phi3 with multi-image pretraining, also benefits significantly from multi-image SFT. 2) Multi-image SFT achieves greater improvements when combined with multi-image pretrain- ing. Both Mantis- and Mantis-idesfies2 use the same multi-image SFT dataset, but the latter outperforms the former because its model incorporates multi-imaae pretraining. 3) Multi- image pretraining alone is insufficient. Models such as XGen-MM-phi3 and idefics2, which only undergo multi-image pretraining without substantial multi-image SFT, do not outperform mod- Mantis-SigCLLI
built on
els with just multi Mallte-mmggeeeea SFT (e.g., LLaVA- -exxt Mext-Interleave the potential of multi-image pretraining requires multi-image SFT to be fully realized. 4) Models still exhibit some generalization multi-image tasks even without multi-image data. Surpris- ingly, the Mini-Intern VL.15 series, trained without any multi-image data, performs relatively well on multi-image tasks. This can be attributed to Intern ViT as a strong visual encoder and configurations like dynamic resolution, enabling superior image understanding (e.g., Mini-Intern LLI.5-chat-44 matches the performance of LLaVA-Next-333 still lag significantly behind those SFT with multi-image data (e.g., Mantis).
This indicates that
on MME (Fu et al., (023)). However, these models
Practical Applicability Discussion. MMIU involves plenty of tasks highly related to real-life ap- plications. To this end, we summarize representative LVLMs' performance tasks highly related to practical applications in Table TableT8Jane, an] detail our discussions from the following four perspectives. 1) MMIU provides extensive exploration of the visual capabilities needed for high-level planning in embodied systems and their application in robotics, making it highly significant in practical terms. For instance, shown in Table 18 on the task of Egocentric Video Question Answering
'GGen-MMPPhi has only few multi-image SFT data related spot the difference
34

---

Published as a conference paper at ICLR 2025
Table 18: Performance of models on various tasks related practical applications.
Model
Egocensi Video Questin Azswering
3D Depti Esimation
Veticle Revieval
Penon Re
GUI Ayp Recogniton
GUT Next Action Prodiction
GPT-4o
$7.5
24.0
68.0
61.5
73.5
46.5
Cladee 5-Sonnect
1910
77.5
M5.5
55.3
340
IneraVLL2
GPT-4o
Internvi2-Pro-
Semantic
37:9
GPT-4o Temporal
Claude3.5 Temporal
Intermvl2-Pro Temporal
Mantis Temporal
29%
4.75
119
20.8
GPT-4o Spatial
Claude3.5 Spatial
Intermvi2-Pro- Spatial
Mantis Spatial
125
K
62N
2J3
Perception Error
Reasoning Error
of Capability
Figure 9: Distribution of error types for GPT-4o, Claude3.5 Sonnet, Intern VL2-Pro and Mantis- idesifes2 towards three main image relationship.
(EVQA), VLternVL.2-Pro outputformed other LVL.Ms, demonstrates its development potential for physical interactions with the environment. 2) MMIU also contains spatial tasks playing pivotal roles in autonomous driving. For example, 3D depth estimation can provide vehicles with curate, real-time spatial awareness and situational understanding of dynamic environments. Among evaluation results in Table 18 find that most LVLMs faced difficulties delivering strong performance in this task, indicating their limitations on spatial reasoning capabilities. 3) MMIU consists of semantic tasks within the surveillance domain, such Vehicle Re-Identification and Person Re-ldentification, which plays a crucial role in enhancing the effectiveness and capabili- ties of surveillance systems. In our evaluations of Table [8] Intern outputformed other LVLMs in these two tasks, including the closed-source model GPT-4o and the multi-image SFT model Mantis, showcasing its potential applications within the surveillance domain. 4) MMIU has significant practival value for autonomous control applications. Because it includes temporal tasks like gui app recognition and gui next action prediction. In Table [8 we find that Claude3.5 demontrate relatively better performance compared to other LVLMs in these tasks, highlighting its significant growth potential for future development. sive benchmark encompassing diverse range of tasks, offering critical insights into the strengths and limitations of current LVLMs across spatial, semantic, and temporal dimensions, ultimately driving advancements in real-world applications.
conclusion, MMIU provides a comprehen-
35

---

Published as a conference paper at ICLR 2025
Performance Various
Internvi1.5-che Performance
Parformance
Performance
T2AR
VCJ
VO
viocp
ISTE
mms
MAN
30E
1000
20P6
2058
SOCK
201
T2RR
VCa
vo
GAR
10
we KIASC
ESTE
masc
aue
2000
soch
Figure 10: Comparison of GPT-4o and InterVVL1.. on unanswerable and answerable questions, with the red line representing the model's average accuracy across all tasks.
B.4 ERROR ANALYSIS
To more vividly demonstrate the performance of MMIU and the models on different tasks, we analyze cases for the four best-performing models: GPT-4o, Claude3.5-Sonnet, Intern Pro and Mantis-idefics2. Through an analysis of error types in MMIU, we categorize them into four types: Fail to Follow Instruction, Lack of Capability, Perception Error, and Reasoning Error. Specifically, Perception Error refers to instances where LVLMs struggle with object recognition, classification, detection in images due to limitations in the representational capacity of their vi- sual encoders, making it the most common eπrоr. Reasoning Error occurs when models correctly recognize and interpret visual content, but errors arise during the reasoning process, leading correct answers. Lack of Capability refers to cases where models refuse to answer questions, citing insufficient information despite having all necessary inputs. Fail to Follow Instruction indicates that LVLMs misinterpret instructions, resulting in incorrect responses.
As illustrated Figure we make the following observations regarding different types of visual re- lationships (e.g., semantic, temporal, and spatial relationships): 1) Semantic tasks tend to exhibit more perception errors compared to temporal and spatial relationships. We attribute this to the greater diversity often inherent in semantic tasks (c.g., image retrieval), which poses additional challenges for imati understanding. In contrast, temporal and spatial tasks typically involve images with inherent consistency, such variations in time viewpoint. 2) Temporal and spa- tial tasks are more prone to reasoning errors. Temporal tasks often require models to infer the dynamics of objects over time (e.g., ordering tasks or next image prediction). while spatial tasks We observe
demand spatial cogmition and reasoning (e.g., that models consistently struggle with these forms of reasoning. 3) Spatial tasks exhibit signifi- cant number of errors caused by lack of capacity. This is particularly evident in tasks requiring advanced visual understanding, such as 3D pose estimation. Current models often produce irrelevant or incoherent responses in these tasks, likely due insufficient training for specialized visual chal- lenges. On the other hand, when analyzing performance across models, we find the following: 4) Open-source models are more prone to perception errors compared to closed-source models. As discussed in Appendix B.3] we believe the capability of the visual encoder is critical for multi- image understanding. Enhancing the performance of visual encoders could effectively mitigate these errors. 5) Smaller open-source models (e.g., Mantis) are more likely to fail to follow instruc- tions. This can be attributed to the limited size of the language model, which impairs its ability to handle long-context tasks involving multiple images. This suggests that improving LLM capacity, particularly for long-context understanding, is crucial for better performance in multi-image tasks.
Finally, for each type of image relationship, we visualize one error case. Since the model responses or questions might be lengthy, we mostly extract the important parts. The images are shown in Figure Figure T6] Figure 17 and Figure 18
Figure Figure Figure
B.5 ABLATION STUDY
Impact of Unanswerable Questions on Model Performance. We have constructed 19 tasks, each including 40 questions. We tested series of models these questions, with full results refer-
36

---

Published as a conference paper at ICLR 2025
Table 19: Quantitative results for LVLMs across 21 tasks with unanswerable and answerable ques-
tions are summarized. Accuracy is the metric, and the Overall score is computed across all tasks.
Amoerabble
542
705
92.
21.
50%
25.0
12.5
215 16.0
35.0
175
91.0
233
268
405
854
220
09.0
400
07.
555
675
105
115
VIA
330
R20
61K
115
110
715
165
702
Mausa
369
46.
140
206
218
210
1541
210
06.
238
139
71.4
273
233
340
22.
25.0
510
46.
01.5
335
26.5
31.
145
15.0
230
18.5
1910
26.0 40.0
255
155
913
195
225
135
355
250
30.5
593
63.7
135
213
285
235
255
273
330
233
23:0
11.5
163
555
23.0
10.0
160
11.3
35.3
16.0
6.5
25.1
9.0
140
145
055
35
463
25.5
325
VL-Il
255
315
31.
245
14.5
7.5
58.5
XOrmpoont-118 12b
193
293
25.
59.5
24A
13.0
66.5
055
20-
312
7.0
160
15.5
11.3
24.
301
23.0
33.0
300
26.4
20.0
25.0
46.5
24.0
31.0
260
31.0
19.5
45.
250
1.2
Fariegon3
10.0
145
13.5
18.7
17.5
25.0
22.1
18.5
33.5
181
13.0
0.5
51,0
17.1
6.5
1600
5.0
28.5
6.5
3.0
22.6
6.5
4.0
4.0
800
955
42.0
250
225
XCxepase
59.5
155
20.6
165
350
M6
100
515
174
705
176
50
K5.5
0.
75.
00
142.
60
81
375
150
41
LLVA 5-Th
148
.90
215
40
23.
7.6
36.0
75
265
5.0
45
25.6
9.5
R0
60
145
34.5
26.5
14.1
135
140
75
360
761
310
Sanccapireer
113
135
145
114
14.
19.1
53
21.5
7.0
25.5
54
200
16.1
40
23
155
5.5
61
16.5
90
307
91.5
55
470
155
411
135
145
15
335
Terllyg thee
11
120
130
74
155
191
65
1955
70
27.
50
816
40
51
50
55
75
360
8.5
34
58
50
1000
30
108
08
10K
Urunoweaalle
CPP--
134
13.5
735
50
65.0
513
75
754
91.5
12.5
37.5
57.5
300
775
5.0
63.5
95.0
50
75.0
7.5
17.5
Maris
483
175
23.5
62.5
61.5
0.0
500
75.0
57.5
91.0
5.5
35.0
25
87.5
63.5
57.5
57.5
87.5
67.5
73.5
Inervil S-chr
150
75.0
55.11
35.9
21
701
310
10.0
77.5
80.0
65.0
12.5
17.5
264
53
91.
100
3011
20.5
21
$2.9
08
48.9
225
25.1
0.0
1961
31.8
140
11.
21.0
32.8
35.0
41
5t
10.0
160
13.5
34.2
60
104
0.0
15.
000
75
0.0
0.0
110
15.5
160
08
10.0
300
$1.3
33.5
XChrpponet 120
239
275
0.0
23
25.0
2.6
523
47.5
52.5
47.5
47.5
27.5
27.5
23
02.3
625
55.0
27.
95.0
150
35%
Bequint Niola
275
32.5
1661
150
15.9
320
3531
154
162
Fasdegen2
232
178
13
71.
0.0
12.5
33.0
62.5
220
0.0
100
12.
41.0
52.3
31.2
166
3.0
0.0
00
215
0.0
0.5
5010
0.0
0.5
0.0
7.
0.0
0.0
00
3.0
10.0
47.
35
92.
Misk Wan
[3]
3.9
15.0
100
43.5
23.1
0.6
47.5
13.0
7.5
10.0
0.0
5.0
0.0
000
000
100
0.0
415
200
235
HaaWvr S.7
50.0
0.0
57.5
20.0
13.5
15.0
12.5
2.5
850
463
193
00
430
0.5
150
7.5
17.5
12.5
10.9
0.0
500
000
1500
72.5
$7.5
438
Ansosplioeer
10.
3.0
0.0
00
25
0.0
0.5
100
0.0
2.5
0.0
0.0
25
0.0
0.
12.5
00
3.0
70.0
500
362
10.0
175
175
33.5
satdey-daa
122
0.0
0.0
00
255
010
0.6
300
75
125
0.0
25
13.0
0.0
250
5.0
100
10.0
425
150
252
17.5
53.5
8000
45.0
60.0
70.5
125
000
Qoer-Baae
1144
735
07.5
37.5
65.1
74.5
611
150
105
510
310
65.
95.0
635
100
50
25
50
61.
enced in Table 19 in the Appendix. As shown in Figure T0 we selected GPT-4o and Intern as representative models for analysis. We observed that for some tasks where the models generally performed well, such as GAR (General Action Recognition), both GPT-4o and Intern VL1.5 experi- enced performance degradation. However, for tasks that are inherently challenging for the models, as indicated by tasks below the red line in the figure, there is no significant pattern in the change of accuracy between answerable and unanswerable questions. We believe the reasons are as follows. 1) For tasks with high accuracy, introducing unanswerable questions confuses the models, increasing difficulty and thereby reducing accuracy. 2) For tasks with low accuracy, since the models already struggle with the original questions, the addition of unanswerable options might lead the models to directly choose the unanswerable option when uncertain, or the increased difficulty might further hinder their performance.
Impact of Different Testing Methods on Model Performance. We test the effectiveness of single- image input models in completing multi-image tasks using concatenated visual tokens or concate- nated images, and we record the final results in Table 20
C
TASK MAP
A task map determines the similarity between tasks based on their inherent characteristics. By com- bining the task map with model performance, we aim to analyze which types of tasks current models perform well or poorly alone, providing a more comprehensive conclusion through complementary methods. Thanks to the extensive number of tasks in MMIU, we can construct comprehensive multi-image task map. As
This approach avoids the bias introduced by using meta-task analysis
37

---

Published as a conference paper at ICLR 2025
Table 20: Quantitative results for single-image LVLMs across 52 mtasks with token-concat or image-coneat are summarized. Accuracy is the metric, and the Overall score is computed across all tasks.
Goid
ONAY
VO
VIWN
VANA
122
54
F15
213
80
63
20
103
207
18a
16
318
255
211
28
-20
74
144
115
76
294
183
53
14
301
233
95
13
56
28
83
232
655
80
440
363
300
101
232
189
201
84
190
30
84
101
176
20
03
137
22
x3
59
76
161
161
90
13
25.
30
15
13
90
120
10
100
100
115
181
113
13
15
08
15
115
36.5
63
80
80
69
63
08
09
03
08
71a
80
12.5
06
145
5.
184
218
13
199
264
114
23
663
bia
124
140
45
150
03
63
M6
114
113
11.
134
16.
408
284
255
69
03
90.
183
00
10x
101
11
3:5
15
115
10
50
Advanced semantic association reasoning
Spatial vieual perception and operation tasks
Spatial reasoning understanding tocks
Creas
netrieval
reconstruction perception tasks
Temperal sequence visual reasoning tasks
understanding reasoning
Action and interaction context reasoning tasks
Cias
Clac.
Figure 11: Visualization of task maps and hierarchical clustering along with the task map.
shown in Figure I1 using the task map, we cluster all tasks into 8 categories. The tasks in each category, along with detailed descriptions, provided in Table 21
C.I CONSTRUCTION
Inspired by the methodology outlined in TaskVee (Ilharco et al. 2023 and benefiting from the extensive and diverse set tasks in MMIU, we aim to analyze tasks and model performance across different tasks using task map. Specifically: 1) We extract task vectors similarly the approach Task Vec. Using Qwen VL-chat as a probing network, we fine-tune it on the multi-choice VQA samples of 52 tasks in MMIU. Formally, task vector is defined by the weight variation between the
38

---

Published as a conference paper at ICLR 2025
weight fine-tuned task data D' and the initial weight Wo of a probing model with the minimum loss, as given by
V = arg min ('W Wa
(1)
W
Notice that for most tasks, we train for 20 epochs, while for subset tasks with lower accuracy after initial training, we extend the training 60 epochs obtain accurate task vectors. Given the large parameter size of QwenVL-chat, we use LORA (Hu et al., 202T) for fine-tuning, which reduces the length of the task vector from 9.6B to 3.5M and consumes fewer storage resources. 2) For any two task vectors V* and V we compute the cosine distance between their task vectors, where G 1 - cos( V*, V²). This process results
52x52 task map. 3) As shown in Figure I1 the task map for MMIU reveals that similar tasks cluster together, such GNAP and GuAR, which are related to GUI, and VO, TO, and VCo, which involve designing image sequence reasoning tasks, which show that the constructed task map aligns with intuition. A comprehensive breakdown of task map can found in Table 22 and Table 2T
Table 21: Details of task clustering on the task map of our MMT-Bench.
Cluster ID
Tasks
Number
Overall Description
emotion-
Advanced semantic association and rea- soning tasks: This cluster focuses on semantic understanding and func- tional correspondence within visual data. Tasks include emotion recog- nition, semantic and visual correspon- dence, and visually grounded reason- ing, emphasizing deep semantic infer- ence from static images.
recognition, forensie-detection,
visual-quality- assessment,
reasoning, visual- correspondence,
semantic-
correspondence,
functional-
correspondence,
2
spot-the-diff,
Spatial visual perception and operation tasks: Tasks in this cluster involve spa- tial perception and object tracking, such as multi-image captioning, pose estima- tion, and point tracking. They require models to interpret spatial changes and structures while interacting with multi- ple images to achieve operational goals.
Multiples-imang -magl- captioning,
Homography- estimation, single-object-
tracking, point- tracking, jigsaw- puzzle-solving, threeD-Pose- Estimation, Image- Captioning-with Spatial-Context, Image-Spatial
Transformation-
Estimation
39

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Cluster ID
Tasks
# Number
Overall Description
next-img- prediction, spot- the-similarity, ravens-progressive- matrices, threed- indoor-reconnition,
7
High-level spatial reasoning and 3D scene understanding tasks: This clus- emphasizes 3D scene reasoning and multi-view analysis, including depth es- timation and object recognition. Tasks focus on deriving object properties and spatial relationships from 3D structures or multiple perspectives, showcasing advanced spatial reasoning.
threed-cad- recognition, Multiview- reasoning, threeD- Depth-Estimation
temporal-
10
Cross-modal retrieval and association tasks: This cluster addresses cross- modal associations between vision and text, covering tasks like image-to-text retrieval, handwritten retrieval, and icon-based question answering. The fo- cus is on efficiently mapping semantic and visual features for precise retrieval and understanding.
localization, visual- cloze, person-reid, text2image-
retrieval, vehicle-retrieval,
face-retrieval,
sketch2image-
retrieval,
image2image-
retrieval,
handwritten-
retrieval, Icon-
Question-
Answering-with-
Spatial-Context
5
mevis, threeD- Scene-
3D scene reconstruction and visual per- ception tasks: Tasks in this cluster specialize in 3D scene perception and reconstruction, such as object detec- tion, tracking, and scene reconstruction. These tasks demand precise spatial un- derstanding and the ability process 3D geometric structures.
Reconstruction,
threeD-Object-
Detection, threeD- Object-Tracking
6
visual-coherence,
3
Temporal sequence visual reasoning tasks: This cluster involves temporal reasoning in visual data, including vi- sual coherence, ordering, and tempo- ral sequence analysis. Tasks challenge models to infer temporal relationships between visual elements while main- taining logical consistency in sequential data.
visual-ordering. temporal-orderine
40

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Cluster ID
Tasks
# Number
Overall Description
meme-vedio-
8
Multimodal video understanding and semantic reasoning tasks: This cluster focuses multimodal semantic infer- ence and temporal video understanding, including video captioning, action qual- ity assessment, and cross-modal seman- tic retrieval. Tasks require models in- terpret dynamic scenes and synthesize temporal and semantic information.
understanding,
action-quality-
assessment, video- captioning, general- action-rocognition,
textual-cloze,
casuality- reasoning, Image- text-retrieval-wir
Spatial-Context, Egocentric- Video- QuestionAnswering
gui-next-action-
Action and interaction context reason- ing tasks: This cluster emphasizes con- textual reasoning in user interaction, such as GUI operation, action recogni- tion, and 3D question answering. Tasks demand models to comprehend and pre- diet dynamic actions within interactive scenarios.
prediction, gui- app-recognition, Multiview- Recognition, -hreeD-question- answering
C.2 ANALYSIS
We perform hierarchical clustering the task map and analyze each cluster. Unlike the previ- ous method of classification through multiple relationships, this approach leverages Qwen-Vt as probe network, allowing for more objective segmentation based on the intrinsic attributes of the tasks themselves. Combining the model's performance in each cluster, as shown in Figure 4 with the task map presented in Figure demonstrates strong performance. Next, we examine the out-domain tasks where the model un- derperforms. Finally, we propose using Taskmap to assess task difficulty, guiding future model development.
begin by analyzing the in-domain tasks where the model
In-Domain Tasks Analysis. In-domain tasks are tasks that most current multimodal large models can handle correctly. For multi- image tasks, the model generally struggles to achieve satisfactory results, with most models performing worse than random selection. Con- sequently, the model can only achieve good per- formance on a limited number of tasks. Specif- ically, as shown in Table 22 for tasks in clus- ters and some tasks in cluster which involve recognition captioning, action recognition), the model per- forms relatively well. We believe this is because these multi-image tasks focus on overall image perception, requiring less comparison and reasoning between images. Additionally, the model has already demonstrated strong capabilities in high-level perception tasks involving single images.
Table 22: The number of tasks within each clus-
ter after hierarchical clustering, and the Spearman correlation r between the average performance of the model on these tasks and the overall perfor- mance of the model.
+ Cluster
6
$
* Tusks
10
Wr Ace
094
0.35 092
0.96
0.83
0.93
0.74
0.73
captioning (e.g., video
27.9
34.7 263 268
202
33.7
Out-of-Domain Tasks Analysis. Out-of-Domain Tasks refer to tasks where most models perform poorly. Specifically, shown in Table 22 we find that models struggle with tasks in clusters 4, 5, and 6. Upon analysis, we discover that tasks in clusters 4 and 6 involve modeling semantic relationships or sequential order among multiple images, which requires strong memory capabilities and advanced perceptual and reasoning skills. Most open-source models underperform on these tasks, especially in image sequencing problems (e.g., temporal ordering tasks), where even closed-
41

---

Published as a conference paper at ICLR 2025
source models struggle achieve satisfactory results. Tasks in cluster 5 pertain visual tasks involving 3D spatial relationships, such as detection and tracking. Although models show some proficiency in handling 2D visual tasks, they lack optimization for 3D data, making them capable of handling only simple 3D tasks that involve basic semantic understanding, but inadequate for accurately modeling complex 3D visual tasks.
Analysis of Different Model Categories. As shown in Figure we categorize the models into four types. It is evident that across different task categories, the performance trend follows: closed-source models outperforning models with extensive multi-image SFT, which in turn outperform multi- image input models, and finally single-image input models. This indicates: 1) Extensive multi- image SFT is crucial for multi-image performance. A more detailed analysis of this is provided Appendix [B. While some single-image models (e.g., GLMM-VV can generalize multi- image tasks to certain extent, their average performance remains inferior to models that support multi-image input. This suggests that most single-image models struggle to generalize to multi-image tasks and require instruction fine-tuning with multi-image data.
D COMPUTAIONAL RESOURCES
Table 23: Resource consumption of some models evaluated on the testmini set of MMIU.
Model
Batch Size
Resources
Times
Memory Utilization Per GPU
GPT
USD
0.51h
V.aVA--NextInterleeve.
x A100-80GR
0.04h
6038/MiB
Mantis idefics2.88 8B
x ATOD SOGB
0.15h
73896MiB
XGen-MM-4B
I A100-80GR
0.14h
75082MiB
InterVVL. Chat V1.5 26B
I ATOD SOGB
0.89h
71619MiB
T1temVV1.--8B
I A100-80GR
0.97h
79292MiB
LLaVA-L-5-BB
X A100-80GB
0.02h
64462MiB
As shown in Table 23 benchmark enables efficient evaluation across all tested models, regard- less of their size or complexity. The results show that most models can be tested on a single A100 card within hour; even the closed-source model GPT-4o can be tested within hour with only 8.57 USD. Additionally, our framework's 's support for multi-GPU distributed inference ensures that even resource-intensive models achieve high efficiency. This makes our benchmark suitable for com- prehensive multi-model comparisons while maintaining practical evaluation times across diverse architectures.
42

---

Published as a conference paper at ICLR 2025
Low-Level Semantics: VQA
Question: Compared to the second image, how is the lighting situation in the first image?
Options:
A)
Similar
B)
Slightly better
C)
Slightly worse
GPT-4o: C. The first image is blurry with poor sharpness, likely due to suboptimal lighting or motion issues. In contrast, the second image has well-distributed artificial lighting, providing clear visibility and detail, with distinct objects, colors, and reflections..
Claude3.5: C. The first image noticeably blurry and lacks sharp focus, which suggests poor lighting conditions or camera shake due to low light.. In contrast, the second image has much clearer definition and vivid colors, particularly the bright reds and yellows of the train interior...
InternVL2: C. In the first image (Image-1), The primary light source seems natural, possibly from an overcast sky or shaded area, which results in less contrast and a generally darker atmosphere. the other hand, Image-2 shows artificial lighting with bright overhead lights that cast direct illumination on various elements within the scene...
Figure 12: Error cases in low-level semantic image relations (visual quality assessment). Red high- lights the wrong answer.
43

---

Published as a conference paper at ICLR 2025
High-Level(obj) Semantics: S2IR
Question: Please retrieve the most similar image to the Query Image in the candidate images. The query image is the first image..
Options:
A) the second image
B) the third image
C) The forth image
GPT-4o: A. The query image contains sketch-like drawings notebook, primarily of flowers and petals. the second image, which is an illustration of a house with somewhat simple, illustrated style, is relatively closer in terms of being illustrative drawing. The other images (a streetlight and medical scenario) do not share this illustrative nature.
Claude3.5: 55: C. The query image (first image) shows hand-drawn sketches of various objects, including flowers and what appears be pinwheel windmill- like shape. Among the candidate images, the fourth image the most similar because: It also depicts hands, which aligns with the hand-drawn nature of the query sketch...
InternVL2: C. The first image, which appears to be collection of simple line drawings including flowers and leaves, has artistic illustrative style. Among the given options, Image-4 also shows elements that can considered illustrations or diagrams related to human anatomy (showing a hand being bandaged). This visual representation aligns more clasely with the artistic nature of the first image compared to the other two images, making it the most similar terms its illustrative quality....
An error case in high- high-level(ob)) semantic image relations (sketch2image retrieval). Red highlights the wrong answer.
Figure 13:
44

---

Published as a conference paper at ICLR 2025
High-Level(sub) Semantics: ER
Question: Now you are given four images, please examine the details and tell which one of them has emotions of the characters that are different from the others. Which image is most likely to show a different emotion from the other images?
Options:
A)
the first image
C) the third image
B)
the second image
D) the forth image
GPT-4o: C. The first, second, and fourth images seem to portray moments with more intense serious emotions, likely involving protests, official interactions, or children deeply engaged. The third image, however, shows students in what appears to be an educational setting, listening attentively and interacting calmly. This setting and the visible emotions differ from the more charged formal emotions in the other images...
Claude3.5: D. The fourth image depicts young children playing with colorful balls in what appears to be daycare preschool setting. The children seem engaged in calm, focused play activities. Their body language interactions suggest curiosity, contentment, and peaceful exploration as they handle the toys. In contrast, the other images show more tense or serious situations:...
conflict protest. The individuals involved appear to engaged intense confrontation, with visible signs of tension and aggression. **Image (B)***The second image shows two standing outdoors, seemingly engoged in conversation or perhops observing something together. **Image (C):** The third image presents group people being addressed guided through what appeors museum exhibif. **Image (D):**The primary emation captured here is that of innocence, playfulness, and happiness. Image (D) stands out because it captures emotions related to childhood playfulness-jun curiosity, and comfort-whin starkly controsts with the more serious, tense, investigative moods evident the other images.
InternVL2: D. **Image (A):** This image depicts
zoo
An enror in high-level(sub) semantic image relations (emotion recognition). Red highlights the wrong answer.
Figure 14:
45

---

Published as a conference paper at ICLR 2025
Discrete Temporal: GNAP
You given GUI navigation task that includes current screenshot images, historical screenshot images, and corresponding actions. The task is: Open the Firefox Browser to search for the best video blogs on travel vlogs. Then, go to the Setting app to turn up the brightness your phone. Thehe historical actions are... The last image represents the current screenshot and the preceding images are historical screenshots. Which action should do next?
Question:
Options:
A)
SCROLL: LEFT
C) SCROLL: DOWN
B)
SCROLL: UP.
D) CLICK: (31, 960)
Input Images
Next step Image (Not Input)
GPT-4o: D The next action should be to adjust the brightness your phone by going the Settings opp, as per the task instructions. After the previously listed actions, which included opening the Firefox Browser screenshot shows mobile home screen, with apps including the Settings app visible. Considering the task involves navigating the GUI and focusing brightness settings, the most logical next step to click the Settings app icon. the selection associated with clicking the Settings app is: CLICK: (31, 960).
searching for travel vlogs,
current
Claude3.5: D Explanation:1. The current screen shows a YouTube channel page for travel vlogs, which indicates we' ve already completed the first part of the task (searching for travel vlogs)... 5. This action will allow you to navigate away from the YouTube app and move towards completing the next part of the task, which is accessing the Settings to adjust brightness. Therefore, clicking the Home button (option D) is the most appropriate next action.
InternVL2: D. Coordinates (31, 960) aligns with typical placement for system navigation button or perhaps a menu/options area that could provide access to other applications like Firefox. Therefore, based on common smartphone interface design principles and the location indicated by option D, clicking coordinates (31, 960) would be the logical next step to proceed towards opening the Firefox browser...
An error case discrete temporal image relations (GUI APP action prediction). Red highlights the wrong answer.
Figure 15:
46

---

Published as a conference paper at ICLR 2025
Continuous Temporal: TO
Question: Please predict the order of the following pictures, and give each picture sequential index. This index starts from O. The larger the index, the later the order.
Options:
A) 2,0,1,3
B) 1,0,3,2
C) 3,1,0,2
D) 0,2,1,3
GPT-4o: C: Following this sequence, The correct choice
is :C: [3, 1, 0, 2]
Explanation:
Picture 3 shows the player in the initial preparation.
Picture shows the player starting to swing.
Picture O shows the player in mid-swing.
Picture 2 shows the player finishing the swing.
Claude3.5: B: Image 1 (index O): The player is in the initial stance, preparing to serve. Image (index 1): The player has tossed the ball up and is reaching back with the racket. Image (index 2): The player at the peak of the serve motion, with the racket raised high.Image 2 (index 3): The player has completed the serve, with the racket following through.
InternVL2: C: Image-3 (Index O): The player begins initiating the backhand swing. Image-1 (Index 1):As the swing progresses, he transitions into the follow-through preparation phase. Image-2 (Index 2): Following the initial swing setup, he completes the backhand stroke with full extension...
Figure 16: An error case in continuous temporal image relations (temporal ordering). Red highlights the wrong answer.
47

---

Published as a conference paper at ICLR 2025
2D Spatial: JPS
Question: The patches in the middle of the image might be disordered. Please state the correct order of the number indexes based the given patches, following the sequence: top left, top right, bottom left, bottom right.
Options:
A) 1, 2, 3,4
B) 2, 4, 1,
4, 3,2
D) 2, 1, 3,3
GPT-4o: D Patch shows some vertical lines that seem match the top center lines in the larger image. This makes it appropriate for the top left position. Patch 4 shows portion of the head and hat, matching the top right the larger image. Patch shows less identifiable features but the patterns and shading might fit the bottom left corner the larger image. Patch shows straight horizontal and vertical lines that fit well into the bottom right corner of the larger image, matching the sketch patterns there...
Claude3.5: D Top left: The patch lobeled '2' shows faint sketch lines, which matches the top left corner of the main image. Top right: The patch labeled '1' contains sketched the top right corner of the main image. Bottom left: figure wearing hat, corresponding heavily shaded The patch labeled '4' shows the bottom left corner of the main image.
with curved lines, which aligns with
Bottom right: The patch labeled '3' depicts what appears to be a sketched body or torso, fitting the bottom right corner of the main image...
part of what appears be onimal or reptile's 's body with scales. The orientation and details suggest should placed the top right. Top Right (Patch 2)**; This patch also contains similar scaled textures but seems to logically as continuation from previous piece, indicating that belongs the top left. Bottom Left (Patch 3)** This patch has detailed lines and features consistent with the texture seen the other pieces, suggesting continuity from the top sections. It fits best in the bottom right. Bottom Right (Patch 4): This patch includes more defined edges and structures which align well when placed after the third piece, fitting correctly in the bottom left position....
InternVL2: D Top Left (Patch 1his patch
Figure 17: An enror case in 2D spatial image relations (jigsaw puzzle solving). Red highlights the wrong answer.
48

---

Published as a conference paper at ICLR 2025
3D Spatial: 3DIR
Question: What the category of the point cloud based on the multi- view of the point cloud?
Options:
A)
Cabinet
B) Bin
Bed
D) sink
GPT-4o: C: The multi-view images of the point cloud show structure that looks fairly flat and elongated with some irregularities one side. This shape most consistent with the features of a "bed" (Option C). A cabinet (Option A) would typically have more rectangular block-like appearance from multiple views.
A bin (Option B) would likely appear more box-shaped cylindrical and less elongated horizontally...
Claude3.5: B: Based on the multiple views of the point cloud provided, we can deduce that the object represented most likely a bin. Here's why: Shape: The point cloud shows object with a generally elongated, rectangular shape in most views, which consistent with the typical form of bin container. Open top: Several views, particularly the top-down perspectives, show an open top area...
The point cloud in the images displays several characteristics that suggest represents a cabinet: **Structure** The overall shape appears to be boxy with distinct edges and corners, which is typical for cabinet. Cabinets often have rectangular or square shapes with flat surfaces for doors and compartments **Proportions**; The proportions shown in the multiple views indicate height and depth consistent with common cabinet sizes. Cabinets are designed to fit against walls and provide storage space...
InternVL2: A:
Figure 18: An error case in 3D spatial image relations (3D indoor recognition). Red highlights the wrong answer.
49


Reviews:
[{"review_id": "lpTlLTmXMU", "reviewer": "Reviewer_bVMZ", "summary": "This paper introduces the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess Large Vision-Language Models (LVLMs) across a wide range of multi-image tasks. MMIU encompasses seven types of multi-image relationships, including semantic, temporal, and spatial aspects, and consists of 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions. The authors evaluate 24 popular LVLMs, including both open-source and proprietary models, on the MMIU benchmark and identify significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Through multi-faceted analytical experiments, they provide valuable insights for future model and data improvements.", "questions": "1. The authors mention how they collected and standardized the data to create the MMIU. However, they do not provide detailed information about the data collection process, such as which data sources and data processing techniques they used. \n2. The strong capability in single-image understanding is the foundation of multi-image comprehension. As we know, the ability of single-image understanding comes from powerful language models, such as MMMU benchmark, can you give some performance of pure language models.\n3. I hope to provide some model performance that add interleave data or video data in the pre-training or post-training stage.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and clearly presented, with a detailed background section, methodological descriptions, and experimental results. \n2. MMIU is the first benchmark to comprehensively evaluate the performance of LVLMs on multi-image tasks. It covers a wide range of multi-image relations and tasks, including semantic, temporal and spatial relations, as well as various image modalities.\n3. The design of the MMIU allows it to be easily extended to new multi-image tasks and image modalities. This makes it a useful tool for future research.", "weaknesses": "1. MMIU is large in scale, including 77,659 images and 11,698 multiple-choice questions. This may require a lot of computing resources to run experiments and perform analysis.\n2. The paper shows that multi-graph understanding requires the capabilities of semantic, temporal and spatial relations. However, most current models do not model the latter two capabilities in the pre-training phase, which can lead to a lack of practical significance in the comparison of various models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DY2bJB1m6w", "reviewer": "Reviewer_avPS", "summary": "This work introduces MMIU, a benchmark to evaluate a LVLM's capability of understanding information when given multiple images simultaneously. MMIU covers a wide range of established cognitive tasks on semantics and compositionality in fine categories.", "questions": "Please find my concerns and suggestions in the Weakness section above.\n\nTypo: Line 1389, Table reference is missing.\n\nUpdate (Nov 23): Raising my ratings according to the authors' response", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work offers meticulous details to the design of the benchmark. The authors have shown great effort in the data curation process while following a logical main plot line. I particularly find the results that show the LVLMs pre-trained with only single-image content may outperform their multi-image counterparts of great research insight.", "weaknesses": "Please find my two major concerns down below:\n\n1. My bigger issue about this paper is regarding the presentation of experimental results. **I find the benchmarking results are presented in a very cumbersome way.** Table 3 is the best example to illustrate this major weakness. The authors put in all the fine-category results by every single subtask. Although the comprehensiveness is appreciated, such overkill-style of presentation makes it hard to follow, so that we readers may not obtain a clear intuition about where and how well each baseline model thrives or lags against one another. This 'way-too-much-details' problem is also present in Figure 4 of the task maps. I highly recommend the authors show more summarized performance on the coarse categories, such the 3 big ones of Semantic/Spatial/Temporal in the main text, while leaving the finer categories in the Appendix.\n\n2. **Potential position bias in the unanswerable set is unaddressed.** . MMIU's QA test cases are all multiple-choices. However, according to Zheng et al, 2023, the position bias is a significant issue when constructing benchmarks for evaluating LLMs (esp. on non-GPT methods). Basically speaking, the position bias will be prominent if the 'None of the above' option for the unanswerable set is always Option D, while known LLMs tend to favor the first option. This means models intend to give away deflated results shown in Table 12 for the unanswerable tasks. Have the authors tried shuffling the option orders? I am not seeing particular evidence for addressing the position bias in Section 3.2 as supposed to be.\n\nZheng et al, 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PDqx1REpq", "reviewer": "Reviewer_4Ca4", "summary": "This paper presents MMIU, a comprehensive benchmark designed to evaluate large vision-language models (LVLMs) on multi-image tasks, addressing limitations in existing single-image-focused evaluations. MMIU includes 52 tasks spanning seven image relationship types, such as semantic, spatial, and temporal, with a total of 77,659 images and 11,698 multi-choice questions.", "questions": "- Clarification on GPT-4o’s Role in Data Generation: Could you elaborate on how GPT-4o’s limitations in multi-image reasoning are managed during data generation? Specifically:\n\n- How do you ensure that GPT-4o’s generated questions and answer choices are unbiased and accurately reflect the task complexity, given that it might have inherent reasoning limitations?\n\n- To what extent was human verification involved, especially in preventing GPT-4o's possible biases from influencing the multiple-choice options? Further clarification on this process would strengthen confidence in the benchmark’s objectivity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The paper introduces a novel benchmark, MMIU, specifically designed for evaluating large vision-language models (LVLMs) on multi-image tasks. This fills a critical gap, as existing benchmarks predominantly focus on single-image understanding. The inclusion of diverse image relationships—semantic, spatial, and temporal—is a creative approach that highlights areas where LVLMs struggle, especially in tasks requiring complex multi-image reasoning.\n\n- Quality: The experimental methodology is robust and thorough. The authors test 24 popular LVLMs, both open-source and closed-source, providing a comprehensive view of current capabilities. Their use of multi-dimensional analysis tools, including supervised fine-tuning (SFT) and task maps, provides depth to the evaluation, highlighting model strengths and weaknesses across various task types. This approach demonstrates rigorous scientific methodology and enhances the reliability of the conclusions drawn.\n\n- Significance: MMIU has high potential impact for the vision-language research community, particularly given the increasing relevance of multi-image tasks in real-world applications (e.g., video understanding, 3D reasoning). By providing a benchmark with a broad set of tasks and image relationships, MMIU allows for a more nuanced understanding of LVLMs’ strengths and limitations. The insights from this benchmark can guide future LVLM improvements, pushing the field towards more sophisticated models that handle complex visual contexts and multi-image data effectively.", "weaknesses": "- Reliability of GPT-4o-Generated Data: Although MMIU’s large dataset is a significant contribution, its reliance on GPT-4o for generating questions and responses raises concerns about potential bias. Given that GPT-4o itself may have inherent preferences or limitations in multi-image reasoning, this could inadvertently affect the construction of questions and answer choices. Additionally, if GPT-4o has limitations in multi-image reasoning (as noted by the authors), this brings into question whether the generated tasks and responses are fully reliable. While human verification can ensure the accuracy of correct answers, it may not catch subtle model biases embedded in the answer choices, potentially skewing results. A clearer discussion or additional safeguards against GPT-4o’s influence on task design would improve the benchmark’s objectivity and robustness.\n\n- Limited Discussion on Practical Applicability: While MMIU’s tasks are inspired by real-world applications, the paper could strengthen its relevance by addressing practical implications more explicitly. For instance, the authors could highlight specific MMIU tasks that align with high-stakes applications—such as high-level reasoning and long-term memory tasks relevant to autonomous driving, surveillance, and medical imaging. Connecting benchmark tasks to these domains would make the work more relatable and impactful for applied researchers. Additionally, discussing how MMIU might be adapted or extended to cover emerging application areas would further enhance the benchmark’s future-facing value and adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZX8k1uTkF", "reviewer": "Reviewer_JjzH", "summary": "The paper introduces the Multimodal Multi-Image Understanding (MMIU) benchmark, which evaluates large vision-language models (LVLMs) on complex multi-image tasks, including spatial, semantic, and temporal relationships. Covering 52 task types with 77,000 images and 11,000 curated questions, MMIU is a comprehensive benchmark that reveals significant challenges, particularly in spatial and temporal tasks, for leading models such as GPT-4o.", "questions": "This paper is overall good, it would be better to present deeper analysis on model performance variations across specific multi-image relationships, particularly spatial, temporal, and high-level semantic tasks. Currently, the analysis highlights general performance trends but could benefit from a more granular examination of how and why certain models excel or struggle with specific relationship types.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Multi-image understanding is indeed a meaningful and impactful problem within multimodal research, this paper effectively addresses this need by establishing MMIU, a benchmark specifically designed to evaluate large vision-language models (LVLMs) across diverse multi-image tasks.\n\n2. The presentation is clear and well-structured, figures and tables are effectively used to enhance understanding.", "weaknesses": "1. The benchmark evaluates numerous LVLMs but could benefit from deeper comparisons across models with distinct visual encoding strategies. This would provide more specific insights into how different architectures impact performance on complex spatial and temporal tasks, which is particularly valuable for the multi-image context.\n\n2. While MMIU successfully identifies gaps in LVLM performance, the paper does not sufficiently explore or propose alternative methods to address these gaps, such as novel training techniques or architectural adjustments. This limits the benchmark’s practical utility for guiding future model development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YY7mkaM0bS", "reviewer": "Reviewer_g41s", "summary": "The article introduces the MMIU benchmark for evaluating large vision-language models on multi-image tasks. It includes various tasks and a large dataset with over 77K images and 11K questions. The study finds that even advanced models struggle with complex multi-image understanding", "questions": "1、Are there any duplicates between the samples in your dataset and the current multi graph dataset?\n\n2、Why do models without multi-imgae SFT perform better than those with multi-imgae SFT on your benchmark?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、Comprehensive Evaluation: The MMIU benchmark provides a thorough assessment of LVLMs by covering a wide range of multi-image tasks and relationships, making it one of the most extensive benchmarks in this domain.\n\n2、Diverse Task Coverage: It includes a variety of tasks that require different cognitive processes, from low-level visual perception to high-level reasoning, offering a holistic view of a model's capabilities.\n\n3、Insightful Analysis: The article presents detailed analytical experiments that identify key performance gaps and limitations in current models, providing valuable insights for future research and development in the field of multimodal AI.", "weaknesses": "1、Typo: 173 line multimodal large language models (LVLMs), abbreviation error.\n\n2、The details of writing can be further polished, such as using Fig. in some places, Some places use Figure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)