MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
Authors: Sihan Yang
Runsen Xu
Yiman Xie
Sizhe Yang
Mo Li
Jingli Lin
Chenming Zhu
Xiaochen Chen
Haodong Duan
Xiangyu Yue
Dahua Lin
Tai Wang
Jiangmiao Pang
Venue: ICLR 2026 Poster (2026)
Keywords: Spatial Intelligence
MLLM
VLM
VQA
Benchmark
3D Understanding
Primary area: datasets and benchmarks
TL;DR: We propose a novel benchmark for multi-image spatial intelligence.

Abstract: Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail to assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple-choice questions from over 120,000 images, each paired with carefully designed distractors and a stepwise reasoning process. We conduct extensive experiments and evaluate 37 open-source and proprietary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30\% accuracy and OpenAI's GPT-5 reasoning model reaches 40\%, while humans score 97\%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide an automated error analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstruction errors, (3) situation-transformation reasoning errors, and (4) spatial-logic errors, offering insights for advancing spatial intelligence.

Published as a conference paper at ICLR 2026
MMSI-BENCH: A BENCHMARK FOR MULTI-IMAGE SPATIAL INTELLIGENCE
Sihan Yang" Runsen Xu Xu
Yiman Xie Sizhe Yang! 1.2 Mo Li 14
Jingli Lin 1,5 Dahua Lin
Chenming Zhu Xiaochen Chen"
Haodong Duan' Xiangyu Yue
Tai Wang Jiangmiao Pang"
Shanghai AT Laboratory
The Chinese University of Hong Kong *Zhejiang University 'Tsinghua
University
"Shanghai Jiaotong University Universityy Hong Kong 'Bejjing Normal University
Equal Contribution Project Lead Corresponding Author t Baiaang me@gma i com
(Position)
Answer:
$1.2
Image you already facing bed?
Reasoning: Image artiring the whele south, the book les ahead and the bed to the front right. incloaling But the book poothoned enat
(Motion) images drection
Accurasc (%)
Fumm Imsgge to Image 57
Error Analysis with MLLM
(Attribute) What
Cresttin HAshh Arsenae
38.7%
18.5
roto wighh-hoeeh
room?
Random
(Multi- Stop Reasoning) Q: After from the chall doot how first 30 bedede sclk comerr and iren the window?
13.2% aaal
Human
GDT-
GPC-
Qweed S-
GPT-
Internys
Fully Human Annotated
Diverse and Chaltenging Questions
Reasoning Provided
Automated Error Analysis
Figure The MMSI-Bench benchmark. Left and middle: Human experts carefully select image sets to create unambiguous and challenging multi- spatial reasoning questions. Human- annotated reasoning processes are also provided for correctness verification and further facilitate automated error analysis. Right: Comparison of models and human performance MMSI-Bende highlighting the benchmark's 's difficulty and the substantial gap between current MLLMs and human- level spatial intelligence.
ABSTRACT
Spatial intelligence is essential for multimodal large language models (MLLMs) operating in the complex physical world. Existing benchmarks, however, probe only single-image relations and thus fail assess the multi-image spatial reasoning that real-world deployments demand. We introduce MMSI-Bench, a VQA benchmark dedicated to multi-ing multi-image spatial intelligence. Six 3D-vision researchers spent more than 300 hours meticulously crafting 1,000 challenging, unambiguous multiple- choice questions from over 120,000 images, each paired with carefully designed distractors and stepwise reasoning process. We conduct extensive experiments and evaluate 37 open-source and proprictary MLLMs, observing a wide gap: the strongest open-source model attains roughly 30% accuracy and OpenAl's GPT-5 reasoning model reaches 40%, while humans score 97%. These results underscore the challenging nature of MMSI-Bench and the substantial headroom for future research. Leveraging the annotated reasoning processes, we also provide automated analysis pipeline that diagnoses four dominant failure modes, including (1) grounding errors, (2) overlap-matching and scene-reconstrruction errors, (3) situation-transsoomeee transformation reasoning errors, and (4) spatial-logic errors. offering insights for advancing spatial intelligence.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Multimodal large language models (MLLMs) have rapidly advanced their ability to ground language
and vision in the physical world (Liu et al. 2024a Black et al. 2024 Peng et al. 2023). making them a promising stepping seeppingstoone stone toward embodted artificial general intelligence (AGI). core capability of MLLMs deployed in the physical world is spatial intelligence, the skill to understand where things are, how they move, etc. To steer progress toward this goal, the community needs rigorous, diverse, and challenging evaluation benchmarks that can serve as north stars for progress.
However, most existing benchmarks probe only single-imag image spatial understanding and restrict themselves to straightforward spatial relations (Liu et al., [003]] Tong et al.] 2024, Johnson et al.) 2017 Cheng et al., 2024 Chen et al., 2024a). Real-world spatial understanding is more complex: models must reason across multiple images to track object and ego motion, and relate entities that
never co-occur in a single frame (Xu et al. 2024, Wang et al., 2024bb Lyu et al.) 2024). Without benchmarks that demand such multi multi-image reasoning, cannot reliably measure improve an MLLM's spatial competence.
In this work, we introduce MMSI-Bench, multiple-choice VQA benchmark dedicated to Multi- iMage-based Spatial Intelligence (Figure []. MMSI-Bench is built on a comprehensive taxonomy of fundamental spatial reasoning tasks that span the positions, attributes, and motions of three key spatial clements-camera (observer), object, and region in real-world scenes. Beyond these atomic tasks, construct multi-step reasoning split that chains them into long-horizon questions.
Crafting a benchmark that is simultancously diverse, accurate, and challenging is far from trivial. Prior benchmarks (Chen al., 2024a| Cheng al. 2024; Yang et al. 2024 Fu al. (024) populate templated questions from annotated or estimated spatial metadata. These automatic pipelines sulfer from limited diversity and difficulty. We therefore adopt a fully human-centric design. For each question, one of six 3D-vision researchers scours large image collections, selects set of images that exhibit non-trivial spatial relations, and creates novel, difficult, and unambiguous multiple-choice question whose answer can be derived only by reasoning across all selected images. Every item comes with carefully designed distractors and a step-by-step reasoning process that explicitly leads to the correct answer, second annotator then audits each example to ensure clarity and correctness.
Over the course of 300+ researcher-hour we inspected more than 120,000 candidate images and distilled them into 1 1,000 high-quality Q&A pairs covering broad spectrum of real-world scenes, from indoor scans (Dai et al.) 2017] Chang et al.) (017) and outdoor driving footage (Sun et al.) 2020)] Caesar et al., 2020) to robot manipulation (AgiBot- Conrd-Contributorss al. (025) and everyday ac- tivities (Grauman et al.) 2022). We evaluate 37 widely used MLLMS on MMSI-Bench. The strongest open-source model attains about 30% accuracy, while the best proprietary system, the OpenAl GPT-5 reasoning model (OpenAI 2025c), reaches only 40%, compared with human performance at 97% accuracy. To the best of our knowledge, MMSI-Bendh shows the largest performance margin between SOTA models and humans among existing spatial intelligence benchmarks.
This pronounced gap underscores both the difficulty of MMSI- and the headroom for progress in spatial intelligence. Our eтror analysis groups failures into four categories: (1) grounding errors, reasoning errors, and (4) spatial-logic errors. Because every question is paired with a reference reasoning process, we also release a novel automated analysis pipeline that contrasts a model's response with the reference to pinpoint eтror types efficiently across the entire benchmark, providing actionable insights for future work on talti-imaae spatial intelligence.
(2) overlap-matching and scene-reconstruction errors, (3) sttuation-transformaatit
RELATED WORK
Multi-image VQA benchmarks. Driven by the rapid progress of MLLMs, VQA benchmarks have
shifted from single-image reasoning (Hudson & Manning 2019) to multi-image settings focusing on various skills. Visual Haystacks (Wu et al. (025b) probes long-context retrieval, asking models to locate relevant content within large image collections. MIBench (Liu et al. 2024b) stresses cross- image reasoning that combines instruction following with knowledge retrieval. ReMi (Kazemi et al.. 2024) broadens the scope to math, physics, and code across two to six images, blending GPT-generated
items with human audits. BLINK (Fu al. 2024) zeroes in on inter-image relationships, while
2

---

Published as a conference paper at ICLR 2026
MuirBench (Wang et al. 2024a) adds robustness and diversity to multi-imag reasoning. Existing benchmarks show that contemporary MLLMs still trail behind humans as either the number of images or the reasoning complexity grows (Li et al. 2023). In this work, instead of building a general benchmark, we focus on comprehensively evaluating MILMS multi-image spatial intelligence.
Benchmarks for spatial intelligence of MILMMs Recent progress in MLLMs has enabled them
to potentially serve as the "brains" of embodied agents, e.g., the Decision-making modules of autonomous vehicles (Sima et al. (024) or robots (Kim al. 2024] Team al. [225a]] [lack et al. 2024). where reliable spatial intelligence is indispensable. To measure this capability, a variety of
benchmarks have been proposed, but the majority are designed to assess only single-imag -image spatial
understanding (Liao et al. 2024; Liu et al. [2023: Shiri et al. 2024; Zhou et al. 2025 Ma et al.
2024, Song et al., 2024] Du al.) 2024; Tong et al. 2024] Zhang et al. 2024). Beyond a single
view, associating information across multiple images is even more critical in practice, yet existing multi-image benchmarks remain fragmentary. BLINK (Fu al., 2024) and MuriBench (Wang) al. 2024a) contain just a few spatial sub-splits within otherwise general VQA suites, and thus lack comprehensive coverage. LEGO-Puzzles (Tang et al. (025)) probes multi-step reasoning in a synthetic LEGO LEGO-assembly setting, overlooking real-world scenes. Efforts such UniQA-3D (Zuo et al. 2024 MMIU (Meng et al.] 2024). SAT (Ray et al. (024)) MultiSPA (Xu et al.) 2025). ) and VSI-Bench (Yang et al. 2024) generate template-basee craplate-based based maseb questions from existing annotations/metadan. or simulators, which constrains question diversity and difficulty. By contrast, MMSI-Bench is fully human-curated, unconstrained by templates, and features challenging, diverse questions with accompanying reasoning annotations. ERQA (Team et al. (2025b) is likewise human-curate but comprises only 400 samples, of which just 113 are multi-image. Detailed quantitative comparisons (e.g., diversity) are provided in Appendix D
MMSI-BENCH
In this section, we present MMSI-Bench, a comprehensive and challenging benchmark designed to evaluate the multi-image spatial reasoning capability of MLLMs. Specifically, we introduce the concept of multi-imaae spatial reasoning and outline its associated sub-tasks in Section [.1 In Section 3.2
we provide detailed description of the benchmark construction process.
3.1 OVERVIEW OF MMSI-BENCH
Table I: Taxonomy of Multi-Image Spatial Reasoning.
Main Category
Sub-Category
Description
Camera
Camera-Camera Camera
Relationship between two camera viewpoints.
Camera- Object
Relationship between camera and object.
Positional Relationship
Object-Object
Camera-Regaa
Relationship between two objects.
Relationship between the camera and a semantic region (e.g.. kitehen).
Object-Region
Relationship between an object and a somantic region.
Region- Region
Relationship between two different regions.
Attribute
Appearance
Measurement
Geometric properties (e.g.. length, size) of elements.
Visual characteristics (c.g.. shape) elements.
Motion
Camera
The camera's 's movement (e.g., direction). movement of objects within the scene.
Object
Multi-Step Reasoning
Complex tasks requiring sequence of reasoning steps (e.g. navigation).
Humans possess the remarkable ability to infer spatial information about their environment by integrating visual inputs from different viewpoints or moments in time. We refer this process as multi-image spatial reasoning, in which information from multiple images is combined to reconstruct the underlying scene and enable reasoning to answer complex spatial questions. To evaluate the multi- image spatial reasoning capability of multimodal large language models, we introduce MMSI-Bench. We further discuss additional motivations for multi- multi-image input in Appendix C
We categorize tasks based three fundamental spatial elements: camera (the agent that captures the scenes), object (entities other than the agent within the scenes), and region (semantic areas within the scenes, such as a kitchen). Fundamental spatial understanding requires not only recognizing the positional relationships among these elements but also their attributes and potential motion. Guided by this taxonomy, we define spatial reasoning task types and one multi-step spatial reasoning
3

---

Published as a conference paper at ICLR 2026
category, which flexibly combines these basic types. The specific categories presented in Table Examples for all task categories are provided in Figure Notably, our questions are designed to be human- answerable;; thus, do not include questions about camera attributes (parameters). Additionally, since regions are generally static, we do not include motion-related tasks for regions.
Position (Cam.-Cam.)
Position (Cam.-Obj.)
Position (Cam.-Reg.)
Wheee yeu trok Image where sow the currers for
1) Which direction ic the blokk door rebviton to me whec La baktig Itsaue 27 lletnnd
Di When you tiking the bud Image, which dieciio fic peinting, area relative you? A. Rear Pharz C. Right
Imsge 2. relativc to you?
Left
Right
Right
Reasoning lmagg becwn bag
ovclupping innial
right
Ovcrlapping uurpda
pricing
Position (Obj-Obj.)
Position (Obj.Regg.
Position (Reg-Resp -Reg.)
states Footheesst Northast Northwed Southood Beassnnng The avarlappiza door indieates samo rorm the mumi lac sonthwet the covaee on it norh the deer poctiorned
Q:
painting Northeast Southeus Northwed Southeest
sleeping
A. Southwest B. Noeth Southwost Northeart Beasoning The pirk bunch
Hessoningg The boclakddia is the cast; the bitther eachh the cxme site line in the stulys
the bed
The white wall line to the soeth This the bart moom the enurheser curnee dispping region.
coraer
Motion (Camera)
Motion (Object)
front, indlevinng lofwwrd canerro cutation from Image to Image
tathhr tmage
indiating forward and froee lott
movemett.
continucus inages dredion camoro rotating?
Q
facing which
aoving?
Hight
Attribute (Measurement)
Attribute (Appearance)
Reasening: The imags ahore newrlopping woll pomeen the paining spars 14 wall stripes the mustel soase painiing
Reasoning: A vattial boock
aopeore image and two bocisontal uces lrage buildings top vawo mosemble
Wiich Isnaen the mill painting mand bencah the mirret Carnet deternined It. The eil pointing E. The came legth D. The montel
@. Whot le the top-stew shape the steible part the hulkiing?
A.I stage B, alape slupe D. stape
Multi-Step Reasoning
dock
pucitio??
A Directly fost To loft C. the foot right
from left Reseening dhedopping elements interme that the person eeners through the doar lmage a laces pepperdionaa to Bour colling window Image which postionnd
eitateed skgh beyund position.
left, placing
Figure 2: Representative AMSS-Bench samples from each category. Please zoom in to inspect image details. Questions and rationales simplified for brevity; the complete versions appear in Appendix G Correct answers are highlighted in green.
Table 2: Dataset statisties.
can Can thk Dan Day
0.43
onl Ast.
Postionna Relationable
Statistic
Value
k.m
2%
Total questions
1,000
Number of unique images
1,990
Average question length
130.28
Attrtani
10.0%
Mult- Slop
rB geu
Reavaning
Average reasoning process length
252.83
m%
Average image count per question
2.55
Motion
Maximum question length
449
Maximum reasoning process length Maximum image count per question
1,118
10
Data sources: ScanNet. nuScenes. Matterport3DD Ego4D, AgiBot-Worfe DTU, DAVIS 2017, Waymo.
Figure Distribution of cate- gories in MMSI-Bench.
In conclusion, MMSI-Bende consists of 1,000 multiple-choice question answer pairs based on
multiple images, each also accompanied by annotated reasoning processes explaining the answer.
4

---

Published as a conference paper at ICLR 2026
Question Type and Image Database Construction
Image Selection
QA Design
Uy lepneeeeee nothe bestrom feceegeenenn where
kai b-dad's Wost
Position
Motion
Attribute
Cam--Cam,
Obj-OMt
Heasurement
the
Cam-Ott:
Obj-feeg.
Object
poprrnncce
When scectrage Lehers was camen past uns A.I.
Cam.dRag
Multi-Step Reasoning
uol latur recoeee une terugn
Manually Selected
Reasoning Annotation
Image Database
Quality Control
Yes
Sources,
All satisfied?
251 Scenarios, 12K Images
Unambiguous
No
Challenging
Save
Figure 4: Illustration of the MMSI-Bench construction pipeline: images collected from diverse real-world datasets, relevant image sets are carefully selected, complex QA tasks and detailed reasoning processes are manually annotated, and all data undergo rigorous quality control.
Every question is specifically designed such that cannot be answered using single image alone. Except for the Multi-Step Reasoning category, all other categories contain only two images per question. Detailed statistics on the proportion of questions in each category and other relevant information about MMSI-Bent -escch are presented in Figure and Table 2 respectively.
3.2 BENCHMARK CONSTRUCTION PROCESS
Our detailed benchmark construction process illustrated in Figure
Data collection. To comprehensively evaluate the capability of MILMs across variety of real- world scenarios, it is essential to curate data from diverse and representative sources. To ensure the complexity and authenticity of the evaluation, we exclusively utilize images captured from real-world scenes. Accordingly, we leverage images from diverse collection of real-world datasets: indoor 3D scene datasets (Matterport3D (Chang et al., (017)) and ScanNet (Dai al., (2017)). local scene reconstruction dataset (DTU (Jensen et al. 2014)), autonomous driving datasets (nuScenes (Caesar al., 2020), Waymo (Sun et a020)) a robotics dataset (AgiBot- World (AgiBot- World-Contributors et al. 2025)), a video object segmentation dataset (DAVIS 2017 (Pont-Tuuse et al. 2017)), and an egocentric video dataset (Ego4D (Grauman et al. 2022)). These diverse data sources enable the construction of various question-answer pairs, ensuring that our benchmark captures the complexity and diversity inherent in real-world environments.
Question design and reasoning process annotation. Using the collected datasets, six 3D vision researchers annotate QA pairs spanning the above eleven distinct task categories. Annotators explicitly instructed create diverse set of questions that as novel and challenging as possible for MI.LMs, aiming to push the boundaries of current model capabilities. At the same time, annotators are instructed ensure that all questions are accurate, unambiguous, and answerable by humans. For each question, annotators begin by visually inspecting the image database to choose of relevant images. They then construct question that cannot be answered based on any single image in isolation; rather, the question requires integrating information from multiple images in order to infer the actual scenario. To ensure diversity question formats, no templates are used, and all questions are created in free form. Each question is designed four-option multiple-choice problem, with only one correct answer. The incorrect options are also carefully crafted to serve as plausible distractors. In addition, annotators provide detailed reasoning processes alongside each answer, which are used to facilitate filtering incorrect samples during quality control and to enable automated analysis of MLLMs* reasoning processes, described in Section 5.2
Quality control. To ensure the quality of the data, three additional reviewers, independent of the original annotators, systematically examine all data and remove any instances containing ambiguous questions (due to linguistic ambiguities or insufficient visual information). incorrect answers, or questions that be answered using single image common sense alone. This rigorous quality
5

---

Published as a conference paper at ICLR 2026
control ensures benchmark provides reliable, high-quality evaluation. We annotate the difficulty level of the data based on human answering time; detailed information is provided in Appendix F.6
4 EVALUATION ON MMSI
4.1 EVALUATION SETUP
We evaluate 37 open-source and proprietary MLLMs on MMSI-Bench. For consistency, all models evaluated with temperature of zero and maximum output length of 2,048 tokens.
For comparison, we include three baselines: Random Guessing, Human Performance, and Blind GPT-4o. Random Guessing reflects the expected accuracy under equal-probability selection from all options. Human performance is measured as the average accuracy of five independent evaluators not involved data annotation, representing human-level spatial intelligence on this benchmark. Blind GPT-4o denotes the performance of GPT-4o without access to image inputs.
We report accuracy (%) using exact match between answers extracted from model outputs and ground-truth answers for our multiple-choice questions. If a model fails to generate an answer in the required format, we adopt the LLM-based fallback strategy from VLMEvalKit (Duan et al. 2024) to extract the intended response. More experimental details are provided in Appendix
4.2
MAIN RESULTS
Table 3
presents the primary results of our evaluation. We summarize our main findings as follows:
Current MILMs struggle with multi-image spatial reasoning. Despite recent progress, the most advanced MILMS exhibit significant limitations in multi-image spatial reasoning. Most models achieve relatively low average scores, often only marginally outperforming random guessing. and are far from human-level performance. For example, the best-performing proprictary reasoning model, GPT-5, achieves an average accuracy of only 41.9%, while human performance exceeds 97%, highlighting a substantial gap in spatial intelligence between current MLLMs and humans. Additionally, the accuracy of Blind GPT-4o is close that of random guessing, indicating that our tasks indeed require genuine visual-spatial reasoning and cannot be solved by language priors or commonsense knowledge alone.
Advanced open-source models still trail proprietary counterparts. We observe that leading open-source models generally underperform compared to proprietary models. For instance, the best open-source model, Qwen2.5-VL-72D achieves average score of 30.7%, which is notably lower than that of the top proprietary model. This underscores a notable performance gap between cutting-edge open-source and proprietary MLLMs this challenging benchmark.
Multi-step reasoning and camera motion comprehension are challenging. For most models, performance multi-step reasoning (MSR) tasks is lower than single-step tasks (i.e., the average performance on Positional Relationship, Attribute, and Motion categories). indicating that integrating information across multiple reasoning steps remains challenge for current MLLMs. In addition, most models, particularly open-source ones, perform poorly on camera motion tasks, suggesting that MLLMs, embodied agents, struggle to understand their own movement. One potential reason is that open-source models generally lack access to annotated first first-person motion data during training, while proprictary models may leverage more diverse datasets containing such information.
Scaling up model size brings limited gains on MMSS-Bench. We observe that, for models within the same series, increasing the number of parameters leads to only limited performance gains on MMSI-Bench. For example. Qwen2.5-VL-777 VL-32B, and Intern VL3-78B outperforms Intern VL by just 1.5%. Moreover, NVILA-15B surpasses most models with over 70 billion parameters, ranking second only to the best-performing open-source model, Qwen2.5 5. VL-72B. These results suggest that, at present, the bottleneck for advancing MLLM performance complex spatial reasoning tasks may lie primarily in data quality and diversity, rather than model size. How to best leverage data, model size, and architecture in synergy remains an open and important question for future research.
achieves only 3% higher accuracy than Qwen2.5-

---

Published as a conference paper at ICLR 2026
Table Evaluation results for 37 MLLMs on MMSI-Bench. For each category, the best-performing proprietary model and the best-performing open-source model are both indicated in bold.
Mudels
Positional Relationship
Attribute
Motion
MSR
Cam -amn-CamnCCo Oli-Clij. Reg-Roy Cam-Otij -Rey Cam-Rey,
Mos.
App:
Ckty
Proprictary
GPT-5
43.0
35.1
32.1
45.8
42.4
51.8
60.9
M6.4
32.4
36.8
42.0
41.9
GPT-4.5
34.4
29.8
39.5
51.2
47.1
55.4
39.1
33.3
41.9
40.8
36.4
40.3
GPT-4.1
27.2
36.5
24.2
32.9
28.8
GPT-4o
34.4
24.5
23.5
19.8
37.6
27.7
32.8
31.8
35.1
36.8
30.8
30.3
Cemini-2.5..0
39.7
31.9
39,5
45.3
35.2
43.3
51.5
21.2
36.4
30.2
34.3
36.9
Claxde-3.7-Sonne
35.5
29.8
29.6
31,4
32.9
34,9
37.5
25.8
23.0
31.6
25.8
30.2
dots.vim
33.3
31.9
38.3
27.9
34.1
42.2
62.5
36.4
23.0
25.0
30.8
34,1
38.7
29.6
40.0
28.8
32.9
30.3
Douhao 1.5-pro
25.8
31.0
33.1
33.7
49.4
39.8
37.5
31.8
27.0
26.1
29.8
33.0
SeedL.5-VL
Open-source
In1emVL.3-8BB
34.4
23.4
32.1
12.8
37.6
26.5
37.5
19.7
28.4
31.6
29.3
28.5
Inlmm\L2..578B
27.3
Qwen2.5-VL-728
25.8
34,0
34.6
23.3
34.1
36.
45.3
27.3
27.0
30.3
27.3
30.7
LlaVA..OneVision-75
43.0
31.9
33.3
37.6
38.6
28.1
19.7
13.5
32.9
28.4
IntemVL3-38B
21.5
20.2
33.3
23.3
35.3
25.3
39.1
21.2
16.2
31.6
25.8
26.3
IntemWL2..5-3BB
18.3
22.3
35.8
22.1
38.8
37.5
25.8
14.9
38.2
25.3
27.9
Owen2.5-VL-322)
24.7
26.6
29.6
22.1
32.9
31.3
31.2
24.2
35.5
27.8
27.7
1ntemmV1..5-268
24.7
19.1
29.6
33.7
31.8
37.3
35.9
30.3
10.8
31.6
26.8
28.0
IntemVL3-114B
19.4
24.5
24.7
23.3
37.6
24.1
31.2
22.7
24.3
31.6
29.3
26.8
Llamm-3.2-11B-Viis
21.2
IntemV1.3-98
18.3
25.5
32.1
29.1
31.8
22.9
29.7
24.2
16.2
38.2
26.8
26.7
IntemV1.3-8B
25.8
31.9
37.0
25.6
35.3
23.4
24.2
16.2
32.9
25.7
IntemVL2..58B
32.3
27.7
29.6
32.6
24.7
32.5
26.6
27.3
16.2
31.6
30.3
28.7
NVILA-8B
17.2
24.8
24.7
30.2
22.4
34.9
34.4
25.8
25.7
34.2
29.8
28.1
Qwen2.5-VL-7B
24.7
24.1
25.6
29.4
26.5
25.0
18:2
20.3
25.8
25.9
LLaVA--neeWiiion-P
20.4
33.0
29.6
29.1
25.9
30.1
29.7
25.8
18.9
34.2
11.6
24.5
21.0
31.6
Qwen2.--VL3B
26,9
27.7
30.9
29.1
28.2
34,9
31.2
16.7
17.6
27.6
23.2
26.5
IntemVV13-28
26.6
28.0
21.7
24.1
37.2
29.4
36.
43.8
15:2
21.6
31.6
26.8
29.0
IntemVL1311B
24.7
35.1
22.2
29.4
30.1
32.8
28.8
19.7
26.3
27.0
IntemVVL2.5.BB
23.7
26.6
24.7
25.6
31.8
25.3
31.2
30.3
17.6
25.0
26.3
26.1
DeegSock-VL2
23.7
31.9
22.2
36.0
30.6
22.9
28.1
15.2
28.4
26.3
28.3
27.1
DeepSeek- V1.2-Small
24.7
28.7
18.5
38.8
27.7
28.1
25.0
29.8
DoepSeck-VL2-THn
29.0
27.7
21.0
23.3
17.6
31.3
14,1
24.2
14.9
25.0
27.3
24.0
Baseline
GPT-40
29.4
21.8
29.0
Random Guessing
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
25.0
Level
Table 4: Performance of models fine-tuned on opatial-reasoninn data and their base models on MMSI-Bench.
Models
Positional Relationship
Attribute Meas.
Motion Cam.
Avg.
Com.
Reg.
Cam.
Cam. Reg.
Oljj
Obg
27.7
30.9
29.1
28.2
34.9
31.2
16.7
17.6
27.6
23.2
26.5
Spatial WW al, 2025a
23.6
26.6
37.0
34,9
32.9
24.1
18.8
34.9
34,9
25.0
28.8
27.7
IntemVLL. 5-88
32.3
23.7
29.6
32.6
24.7
32.5
26.6
27.3
16.2
31.6
30.3
28.7
InterSppatial Deng 2025
20.4
27.6
28.4
29.0
30.5
26.5
29.6
27.2
14.8
23.6
31.3
26.9
24.7
26.6
29.6
22.1
32.9
31.3
31.2
24.2
18.9
35.5
27.8
27.7
RoboBrain2.0 0 32B Team et al. 2025al
21.5
28.7
27.1
22.0
27.0
32.5
31.2
36.3
36.4
23.6
29.2
28.5
Current spatial fine-tuning yields limited benefits on MMSI-Bench. We evaluated models fine- tuned on spatial-reasoning data on MMSI-Bench (detailed results are provided in Table and found only marginal gains over their base counterparts, underscoring the benchmark? 's high difficulty and stringent demands for spatial reasoning generalization. likely reason that their training data, built from limited templates, fails to generalize the diverse and challenging scenarios in MMSI-Bos -Bechh.
4.3 LIMITED EFFECT OF PROMPTING TECHNIQUES ON MMSI-BENCI
Prompting techniques are widely adopted to improve the reasoning and problem-sol solving capabilities of large models. Building these advances, investigate whether such methods similarly

---

Published as a conference paper at ICLR 2026
enhance the spatial reasoning abilities of MLLMss on MMSI-Bench. Specifically, investigate both linguistic and visual prompting approaches (see Appendix E.2 for further details):
Linguistic prompting: We adopt the widely used Zero-Shot Chain-of-Thouge (CoT) method. Following prior work (Wei et al. 2022), we prepend "Let's think step by step" to the prompts to encourage reasoning.
spatial understand- ing, fundamental requirement is the ability to identify overlaps between images in order to establish relationships across them. However, MILMSs may struggle to identify overlaps between images. To explicitly provide this cue to the model, we employ the popular image matching method
Visual prompting:
For
1.7
moddommnne Change (8)
PATS (Ni et al. 2023) to establish sparse correspondences between pairs of images, which involves concatenating multiple images and drawing lines between corresponding points to explicitly indicate cross-image relationships. As baseline, we use concatenated images without correspon- dence lines as visual inputs, and compare their results with those obtained using visual prompt.
Method
Ungulttic prompting
Visual prompting
GPT.
GPG4.5
Gwan2.546-728
www-OneViale--Ta
Figure 5: Impact of linguistic and visual prompting on MMSI-Bende
We conduct experiments using 4 representative models. As shown in Figure 3 the linguistic prompting approach (CoT) leads to modest performance improvement only for GPT-4o, while resulting in performance degradation on the other models. This limited effectiveness of linguistic prompting may be attributed to the models' inherent lack of spatial reasoning capabilities; simply encouraging "thinking" does not address their fundamental limitations in spatial understanding.
Similarly, the visual prompting approach yields only slight performance improvements for two models, while resulting in performance degradation for the other two. One possible explanation is that most models still lack the basic spatial intelligence required to benefit from such cues: they may not recognize the need to identify overlaps between images to establish spatial relationships, or, even when overlaps are identified, they struggle to reconstruct the actual scene. These findings indicate that further progress in the spatial reasoning ability of MLLMs may depend on advances in model architectures or training paradigms, such as the incorporation of domain-specifie data, rather than relying solely improved prompting strategies.
ERROR ANALYSIS
This section analyzes the eтrоrs made by MLLMs on MMSI-Bench. First, Section 5.1 presents manual categorization of error types observed in the reasoning processes of the representative model. Then, in Section 5.2 we introduce an enhanced automated eтror analysis method for systematically examining the reasoning errors of various MLLMs with the help of our annotated reasoning processes.
5.1 ERROR TYPE CATEGORIZATION
To identify the main bottlenecks of MLLMs on our benchmark, we manually analyze the reasoning processes exhibited by representative model, GPT-4o, when prompted with chain-of-thougg (CoT) instructions. Specifically, we randomly sample 100 instances from MMSI-Bench and categorize the observed errors into four distinct types (as illustrated in Figure
Grounding errors. The model fails to correctly identify or localize relevant objects or details within the image. For instance, it might misclassify object incorrectly determine the position of an object, errors that undermine its ability to anchor visual reasoning in the actual image content.
Overlap-matching and scene-reconstruction errors. The model fails to identify and match corre- sponding points that represent the same locations or objects in the real scene across different images, and struggles to implicitly reconstruct the underlying scene based on these cross-image relationships. For example, cannot recognize that a tree appearing in both images is actually the same tree, leading to incorrect reconstruction of the scene layout.
8

---

Published as a conference paper at ICLR 2026
Grounding Error
Overlap-Matching and Deene-Reconstruction Error
Model Output:
Model Output:
imsage ladder visiblle the left... Thus, the deceiiion movement back left
has turned away from the shelf. "he shell bahind the viewer and right
Q What is your movemeent direction relative to the Racing direction in image 1? Hacloward B: right
Q: The images are taken continuously. in image 2 where the shelf relative to you? back left Brfront
Wionngt ln image the
Wrongr In figure the shelf is in frunt of you right
back right.
SituationTtranstormatio Reasoning Error
Spatial-Logic Error
Model Outpat:
Model Output: closer to the camera, which moans the camera is moving forward.
ght side of the TV
Thus, the bedroom back right. relative TV.
Q- the TV's frant faces the vieuser, where Ss the bestromm located relative the TV?
Q: With the comero fartng froet, in which direction the cumerr moviny?
Wiring The artworrk the
Wrong object shouldin reffeeecc..
Frunt right Dack right
Backward D: Right
Figure 6: Illustration of four error types identified in MLLM spatial reasoning on MMSI-Bench.
GPT-4.5
GPT-4o
Qwen2.5-VL-772
21.5%
29.9%
37.5%
42.3%
48.7%
13.5%
9.8%
10.7%
14.6%
15.2%
1.8%
1.5%
Grounding
Spatial-Logic
Overlap-Match Scene-Reconstr.
Figure Distribution of correct and error types across three representative MIL.Ms.
reasoning errors. The model makes mistakes when reasoning about spatial directions relative to different reference objects, or when converting between relative directions (such left and right) and absolute directions (such north, south, east, and west).
Spatial-logic errors. The model exhibits errors in spatial logical reasoning, such as (I) hallucinating spatial relationships that do not exist in the given context; (2) failing correctly apply the transitivity of spatial relations, for example, given that east B and is east C, the model incorrectly infers that A is west of C; and (3) making mistakes in identifying the correct reference object during motion reasoning, such as using another moving object as the reference point when determining an object's absolute motion status (whereas the correct approach would be use stationary object like the ground as the reference).
5.2 AUTOMATED ERROR ANALYSIS
Error analysis of model outputs can provide valuable insights for improving MLLMs' spatial capabil- ities. However, manually reviewing each output is time-consuming and impractical at scale.
We try to give the model the correct answer and ask reflect on output and identify the primary eπror type following Section 5.1 In experiments with GPT-4o. it correctly labels the error type for only 53% the selected samples, indicating that the correct answer alone is insufficient for evaluating the reasoning process. When also supply the human-annotated reasoning process, the accuracy rises to 78%, highlighting the effectiveness and essential role of our reasoning annotation in achieving reliable automated error analysis.
We then apply this approach to analyze the reasoning processes of representative models across all MMSI-Bench questions, with results presented in Figure which reveals the distribution of error

---

Published as a conference paper at ICLR 2026
types across different models and highlights specific directions for improving their spatial reasoning abilities. Notably, we observe that even when the final answer is correct, the reasoning process still contain significant emrors. Specifically, the reasoning accuracy of GPT-4.5, GPT-4o, and Qwen2.5-VI.-72B 37.5%, 29.9%, and 21.5%, respectively, all lower than their corresponding multiple-choice accuracy. In partieular, Qwen2.5-V1--22B' reasoning accuracy is about 10% lower than its multiple-choice accuracy, revealing its insufficient spatial reasoning capabilities. Among the various eтrоr types, overlap-matching and scene-reconstruction errors account for the largest proportion of mistakes across all models. This suggests clear direction for future progress as MLLMs strive for human-level spatial intelligence.
CONCLUSION
To summarize, we introduce MMSI-Bench, a challenging and comprehensive benchmark specifically designed to assess the multi-image spatial reasoning capability of MLLMs. Our evaluation across 37 state-of-the-art models reveals substantial gap between current MLLMs and human-level spatial intelligence. Leveraging expert-annotated reasoning processes, we further propose an automated error analysis pipeline that enables scalable and systematic diagnosis of model failures, highlighting concrete directions for improving spatial reasoning future models. We hope MMSI-Bench will serve as a valuable resource for the community and accelerate progress toward more spatially capable and robust multimodal Al systems. We further discuss the LLM usage and limitations in Appendix
ACKNOWLEDGMENTS
This work is funded in part by the National Key R&D Program of China, and Shanghai Artificial Intelligence Laboratory.
REFERENCES
Agibot-world: The large-scale manipulation platform for scalable and intelligent embodied systems.
Davis 2017 httpa///w...aa........ooooooao/a
Dtu data terms of use.
https::/www.bbit
Ego4d licenses draft. http:///sseooooo...a//////////./
F
Habitat http://ww.w...ao..o.////ooo/a//a
dataset license,
nuscenes terms of use. httpg//ss////w/.nnceoaaac/////er
Scannet license. https: http://tt.....aacccccccccccaca/a
Official waymo open dataset aiceeneeeegeeenne hetpeeeeeeeeeeeeoooooooooooooooe
LgiBot-Wordd-Cnnribuue Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang. Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint 2503::553300666 2025.
10

---

Published as a conference paper at ICLR 2026
Anthropic. Claude 3.7 sonnet and claude code. https://ww..nthhntt ude 2ode-3---sonne, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar 20w.:2222.13233, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri. Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. PiO: A vision-languunge-actionn flow model for general robot control. /sttss://physicalll 2024.
ence. www.ee/cccoaa////l////ogg//
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-116311 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1e0:::7090.555 2017.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568 2021.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In CVPR, 2024a.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277 2024b.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang. and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
Google DeepMind. Gemini 2.5: Our most intelligent ai model. ht https::/bboo..oooggg
bechhoooogy/gogglaeeo 2025/] 2025.
Nianchen Deng, Lixin Gu, Shenglong Ye, Yinan He, Zhe Chen, Songze Li, Haomin Wang, Xingguang Wei, Tianshuo Yang, Min Dou, al. Internspatial: comprehensive dataset for spatial reasoning in vision-language models. arXiv preprint arXiv:2506. / 18385, 2025.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-bench. Benchmarking spatial understanding for embodied tasks with large vision-language models. DrXii:24006.5756 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao. Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11211111121, 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li. Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In ECCV, 2024.
11

---

Published as a conference paper at ICLR 2026
Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18995-19012, 2022.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.0706.. 2025.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, arXiv e-prints, pp. ar 2507, 2025.
Drew A. Hudson and Christopher D. Manning. Gqa: A new datast foal visual reasoning and .0002..95000 09506
compositional question answering, 2019. URL.htpps///rzxivoor/////
Bi Huo, Bin Tu, Cheng Qin, Da Zheng, Dehing Zhang, Dongjie Zhang. En Li, Fu Guo, Jian Yao, Jie Lou, et al. dots. I1ml technical report. arXiv preprint arXiv:2506.05767, 2025.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, 2014.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo. Sreenivas Gollapudi, and Ahmed Qureshi. Remi: A dataset for reasoning with multiple images, 2024. URL http::httsssatt //wpvv..a/ivvv//a org. /
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source Aasionlanguage-actiinn model. In CoRL, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 033266 2024.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: 2: Benchmarking multimodal large language models, 2023. URL htttps:///aas org/abs/ 2311 17092
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vasion-language models. arXiv:2409.0977 2499889788 2024.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. TACL, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, and Weiming Hu. Mibench: Evaluating multimodal large language models over multiple images, 2024b. https::t::////// //tp://aaavvvv///v///a org/ abs / 2407 15272
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang. Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng- Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, and Yao Lu. Nvila: Efficient frontier visual language models, 2024c. URL https: // arxiv..ii........ org/ abs/ /412 04468
Ruiyuan Lyu, Tai Wang, Jingli Lin, Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chemming Zhu, Dahua Lin, and Jiangmiao Pang. Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations. In NewrIPS, 2024.
12

---

Published as a conference paper at ICLR 2026
Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. arXiv preprint arXiv:2412.07825, 2024.
Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large 2024.
vision-language models. arXiv preprini
Meta. Llama 3.2: Revolutionizing edge and vision with open, customizable models. https meta. devices/ 2024.
Junjie Ni, Yijin Li, Zhaoyang Huang, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pats: Patch area transportation with subdivision for local feature matching. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17776-17786, 2023.
OpenAl. http:///tppgpppaaaaaao//////////
2024.
OpenAl. Introducing gpt-4.1 in the http:///peeea...o/////e/////g/// 2025a.
OpenAl. Introducing http:///ggg....t///aaaaaaaaaaaaa attpap//opeaaaaccoo///daa/aaaaaa 2025b.
OpenAI. Introducing httpss//oohhaaa.co////rra///rrrr 2025c.
OpenAl. Introducing o3 and o4 mini. http://s/.gaaa...ooo//////tttttr
104--44-ini]] 2025d.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint
2017.
Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2412.07755, 2024.
Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont, Laion coco: 600m synthetic captions from laion2b-en, 2022. URL http://ss//aaaooaa///bi.ooo//
cococ
ByteDance Seed. Doubao- 1.5-pro httpp//ss//eeeeooocccaooo//////a
ao 5_ S_pro] 2025.
Fatemeh Shiri, Xiao- Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. 2024.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph vi- sual question answering. In ECCV, 2024.
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. arXiv:2411.16537, 2024.
Pei Sun, Henrik Kretzsehmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

Published as a conference paper at ICLR 2026
Kexian Tang, Junyao Gao, Yanhong Zeng. Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. Lego-puzzles: How good are milms at multi-step spatial reasoning? arXiv preprint arXiv:2503.1999) 2025.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint mwXi::2507.02029, 2025a.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptisee Alayrac Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020 2025b.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, al. Cambrian-l: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint S441::2406.666666 2024.
Fei Wang. Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai- Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprchensive benchmark for robust multi multi-image understanding, 2024a.
URL htttt:ttt/axiiiiiie http::/aaxivv.oroo abs/ 09411
Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embodied- scan: A holistic multi-modal 3d perception suite towards embodied ai. In CVPR, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm Boosting milm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505..23 237447, 2025a.
Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual haystacks: A vision-centric grodle-in-ahhasstaee benchmark, 2025b.
URL httph::t///// //ap:///aaivvvvi org/ /2000//22477.. 13766
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao. Yiyang Ma, Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao. Kang Guan. Aixin Liu. Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12. Mixture-of-experts vision-language models for advanced multimodal understanding, 2024. URL httph:tts://rrr
org/abs/2440 2412. 10302
Runsen Xu, Zhiwei Huang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Vlm -lm-grounder: A vlm agent for zero-shot 3d visual grounding. 2024.
Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Mat Feiszli, and Kevin J. Liang. Multi-spatialmllm: Multi-frame spatial understanding with multi-modal large language models. arXiv preprint arXiv:2505. 17015, 2025.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Think- ing in space: How multimodal large language models see, remember, and recall spaces. 2412 2024.
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedhench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.0955 2025.
Wenyu Zhang, Wei En Ng, Lixin Ma. Yuwen Wang, Jungqi Zhao, Boyang Li, and Lu Wang. Sphere: A hierarchical evaluation on spatial perception and reasoning for vision- -ssoun-aanguage models. arXiv preprint ar a 12693, 2024.
14

---

Published as a conference paper at ICLR 2026
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong. Chi Zhang, Pengwei Wang. Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision- vision-language models for robotics. 0rXiv:2506.04308. 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv:2504. 10479, 2025.
Yiming Zuo, Karhan Kayan, Maggie Wang. Kevin Jeon, Jia Deng, and Thomas L Griffiths. Towards foundation models for 3d vision: How close are we? arXiv preprint arXiv:2410. 2024.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX OVERVIEW AND ORGANIZATION
This appendix provides supplementary details to support and extend the main paper. The organization of the appendix is follows:
1. LLM Usage, Limitations, Social Impact, and License & Access (Section B:
This section outlines not only our LLM Usage, the limitations of our work, but also discusses its potential social implications, associated risks, and ethical considerations. In addition, it clarifies the licensing terms and access conditions for MMSI-Bench, ensuring responsible and transparent use of the dataset.
2. Motivation for Multi-Image Input (Section C) We discuss the unique advantages of multi-image input for evaluating spatial intelligence.
3. Comparison with Other Spatial Question- Answering Benchmarks (Section D): In this section, we compare MMSI-Bene with existing spatial question-anae answering benchmarks, highlighting the unique value of our benchmark.
4. Additional Implementation Details and Experimental Results (Section E This section provides supplementary technical details regarding the implementation as well as more experimental results.
5. Details of the Data Curation Process (Section This section elaborates on the procedures and criteria used in curating the MMSI-Bench dataset, including data sourcing, annotation protocols, and quality control measures, as well as the methodology and results of the difficulty-level annotation.
6. Representative MMSI-Bench Samples from Each Category (Full Version) (Section G In this section, we present full set of representative samples from each category MMSI- Bench, without any truncation simplification. This is intended to give the reader complete view of the dataset's 's diversity and coverage.
7. Additional Case Studies (Section H:: We provide further case studies illustrate the model's reasoning process on MMSI-Bench, supplementing those already discussed in the main text.
Each section is intended to provide additional transparency and insight, ensuring the reproducibility and broader contextualization of our research.
B
LLM USAGE, LIMITATIONS, SOCIAL IMPACT, AND LICENSE & ACCESS
B. LLM USAGE
In the preparation of this manuscript, Large Language Models (LLMs) used writing- assistance tool. The primary purpose of their use was to aid in polishing the language, which included improving grammar, refining sentence structure, and enhancing the overall clarity and readability of the text. All core research concepts, methodologies, analyses, and conclusions were conceived and formulated exclusively by the authors. The authors have reviewed the final text and take full responsibility for the entire content of this paper, ensuring its scientific accuracy and originality.
B.2 LIMITATIONS
To ensure higher data quality and richer content, we chose to manually annotate question-anm answer pairs. This approach may limit scalability because it requires substantial human effort, which we acknowledge as potential limitation of MMSS-Bench. Nevertheless, MMSS--eench current scale (1,000 samples) is sufficient for meaningful assessment, given that today's 's MLLMs still have substantial headroom for improvement on this benchmark. Its size also comparable to widely used VQA benchmarks, such MMStar (1,500 samples), MM- Vet (218 samples), and LLaV LLaVA-in -V---inthhe-WWe -nil (60 samples). When performance begins to saturate, larger benchmark may indeed be necessary, but reaching that point will likely require new research breakthroughs more human annotation effort.
Notably, MMSI-Bench's 's scale is competitive among spatial benchmarks. For reference: ERQA has 400 samples, MuriBench 478, UniQA-3D 500, SpatialVI.M 546, BLINK 572, LEGO- 1,100,
16

---

Published as a conference paper at ICLR 2026
SpatialRGPT 1,406, and VSI-Bench 5,000. MMSI-Bench contains 1,000 samples, of which only the complate-basek VSI-Bench is substantially larger.
B.3 BROADER IMPACT
The introduction of MMSI-Bench brings both potential benefits and risks to the research community and broader society. On the positive side, MMSI-Bench provides a challenging and comprehensive benchmark specifically designed for evaluating the spatial reasoning capabilities of vision-language models. By facilitating more nuanced and rigorous assessment, MMSI-Bench may help drive advances in embodied AI, assistive technologies, spatial robotics, and other domains where spatial understanding is critical.
However, we also recognize several risks and challenges associated with the development and release of such datasets. First, improvements in spatial reasoning enabled by MAMSI-Bench may have unintended applications, such as enhanced surveillance or automated peciison-making systems that could impact privacy or disproportionately affect certain groups. We encourage users of MMSI-Bendh adhere strictly to ethical guidelines and to consider the societal implications of deploying models benchmarked using dataset.
From an environmental perspective, we acknowledge the significant computational resources required to evaluate modern vision-language models. By releasing MMSI-Bench and its associated scripts publicly, we aim to promote reproducibility and reduce redundant efforts, thereby minimizing unnecessary environmental impact.
In summary, while MMSI-Bench is intended to advance responsible and transparent research in spatial reasoning for vision- lision-language models, urge the community to remain vigilant regarding both its positive and negative societal impacts.
B.4 ETHICS STATEMENT
As this research uses only publicly accessible pre-trained models, it does not present any ethical concerns. All procedures strictly conform to the guidelines established by the ICLR Code of Ethics.
B.5 REPRODUCIBILITY STATEMENT
Our experiments follow the baseline configurations established by prior evaluation benchmarks the original testing protocols of each respective model. Detailed implementation information is provided in the Evaluation Setup and Appendix sections. We will release all data and code, along with thorough documentation, to facilitate faithful reproduction of our main experimental findings. All aspects of this research are fully compliant with the ICLR Reproducibility Requirements.
B.6 LICENSE AND ACCESS
The MMSI-Bench dataset is distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). For all images included in the dataset, their original licensing terms (dtul scal
nus
hab
cgo
agi
day
way
remain fully in effect.
We release MMSI-Bench under the CC-BY license and Terms of Use, and require that any use of the dataset for model evaluation be properly disclosed. This license supplements but does not override the original licenses of source materials; users must also comply with all relevant legal requirements concerning data subjects. This statement clarifies the obligations and liabilities associated with using this benchmark. While we strive to ensure the accuracy and legality of all samples, we do not guarantee their absolute completeness correctness. We assume responsibility for any legal or other issues that may arise from the use MMSI-Bench, including but not limited to copyright infringement, privacy violations, the misuse of sensitive information.
By accessing. downloading, or using MMSI- you acknowledge that you accept this statement and agree to comply with the full terms of the CC-BY license. If you do not agree with these terms or the CC-BY license, you are not permitted to use this benchmark. MMSI-Bench will hosted and maintained on GitHub and the Hugging Face Hub platforms.
17

---

Published as a conference paper at ICLR 2026
C MOTIVATION FOR MULTI-IMAGE INPUT
We expect that general spatial intelligence will, like humans, operate on video inputs. However, we start with a multi-image setting for three reasons:
Foundational to video. Multi-image understanding is the basis of video understanding. Analogous to SLAM, system reconstructs scene and estimates camera pose over stream, but the front-end module (VO) still estimates the delta pose between consecutive frames before constructing a holistic perception of the scene. Given current limitations of MLLMs in spatial understanding, models must first handle multiple images before they can reliably understand full videos.
Clearer diagnosis. Multiple images allow more systematic analysis of current MILMs' failure modes. With videos, if model answers incorrectly, it is hard to localize the cause, e.g., long- limits, image retrieval over long sequences, or spatial reasoning itself. The multi-image setup is more spatially focused and easier for error analysis, which we believe accelerates community progress.
Relevant proxy for video. Multi-image benchmarks also reflect capabilities needed for video-based systems. Our hypotheses are: (i) although humans learn spatial intelligence from videos, they can solve MMSI-Bench, suggesting our tasks capture a necessary condition for general (video-based) spatial intelligence; and (ii) multi-image setting can be viewed as multi-frame, low-FPS special case of video.
We hope MMSI-Bench will inspire future video-based benchmarks.
D COMPARISON WITH OTHER SPATIAL BENCHMARKS
Table 5: Comparison of spatial guestion-answering benchmarks. Some benchmarks contain QA pairs outside spatial understanding: the number of spatial QA is shown in parentheses. Auman-AI performance gap defined as the accuracy difference between the best- performfffmm model reported in the corresponding paper and human performance; for benchmarks without reported human results, the gap is calculated with respect to 100% accuracy and is marked as "<value%". " Note: Scenario entries labeled as "Indoor & Outdoor" or "Indoor" refer real-world images.
Benchmark
Annotation Method
Visual Input Modality
Number of QAs
Seesario
Human- Al Gap
VSR ct al_ 2023
Auto & Human
Single Image SinglaMuli-Immge
10K
Indoor Outdoce
25.0%
BLINK
2024
Auto Humsn
3,807 (572)
Real-world Synthetic
44,4%
UniQA-3D Zaktet 21p4
Auto Human
Single/Mull-tmage
500
Real-world Synthetic
MuriBench
Wing el
40
Auto & Humnon
Mulli Maali--mmag
2,600 (478)
Real-world & Synbbeti
25.2%
Sparia M
Anto Human
Single-tmage
Indoor Outdoor
SparalkGPT (Cheni al MP2
Auto
Singlo-Image
1,406
Real-warld & Synthetic
<4.2%
MMIU
Meng
Auto
Multi Image
JJK (2904)
Real Real-world Synthetic
<45%
ERQA
Team Qlail
Human
Bougle/Mutti-nnagge
400
Indoor Outdoor Syuthetic
<35%
LEGO
Pazzle [inng al 2005
Human
Singla/Multi Image
IK
39.5%
VSI-Bench
Yong ala 2021
Auto
Video
Indoor
MMSS-Benchh
Human
Multi Image
1,000
Indoor & Outdoor
55.3%
Despite significant advances in MILMs, robust evaluation of their spatial inteligeece-especcii in multi-image, real-world scenarios-reman limited. As summarized in Table most existing benchmarks focus narrowly on single-image spatial understanding or offer only limited multi-image coverage minor components within more general VQA suites (e.g., BLINK (Fu et al. (024), MuriBench (Wang et al. 2024a)), rather than serving as dedicated spatial reasoning benchmarks. Furthermore, several datasets rely heavily on template- or rule-based question generation (Ray et al.) 2024. Zuo et al. 2024 Meng et al. 2024: Yang et al. 2024), which restricts both the diversity and depth of spatial reasoning challenges. Synthetic settings such as LEGO-Puzzle (Tang et al., 2025) further lack the complexity and contextual richness of real-world environments.
In contrast, MMSI-Bench a fully human-curated benchmark, meticulously annotated by six 3D vision researchers without the use of templates. It offers comprehensive coverage of spatial relationships and glements-including camera (agent), objects, and regions-actte across ten well-defined task types plus multi-step reasoning category. All scenarios derive from diverse, worldworld sources, spanning indoor 3D scene datasets, local scene reconstructions, autonomous driving datasets, and roboties data, encompassing both indoor and outdoor environments. Notably, MMSI- -enchh
18

---

Published as a conference paper at ICLR 2026
Table 6: Quantitative comparison of question diversity against major non-single-image spatial reasoning benchmarks. Distinct words: Counts the total number of unique words after lowercasing and tokenization across all questions, measuring vocabulary richness. Unique n-grams: Counts the number of unique consecutive n-word sequences (here, n=3 and 5) after tokenization, measuring phrasal diversity. Syntactic structures: Calculated by parsing the syntax of each sentence, abstracting it into tree structure by removing specific words, and then counting the number of unique structures. This measures the diversity of sentence constructions.
Statistic
VSI-Bench
MMSI-Bench
ERQA (Multi-Img)
LEGO-Puzzle
Distinct words Unique 3-grams
398
1,108
559
275
4,482
8,499
2,483
1,023
Unique 5- 3-grams
12,972
13,010
3,318
1,373
Syntactic structures
473
993
310
51
provides detailed, human-authored reasoning chains for each question, enabling rigorous assessment of both answer accuracy and reasoning quality. This benchmark reveals substantial gap between current MLLMs and human-level spatial intelligence, and sets higher standard for multi-image spatial reasoning evaluation in realistic settings.
To further quantify the linguistic diversity of MMSI-Bench, we conduct comparative analysis against representative non-single-imal spatial reasoning benchmarks, including VSI-Bench, the multi-image subset of ERQA, and LEGO-Puzzle. We analyze the complexity of the questions across these datasets at the vocabulary, phrase, and syntactic levels. The results are summarized in Table 6
The analysis demonstrates that MMSI-Bench exhibits superior diversity across all metrics. At the vocabulary level, MMSI-Bench features 1, 1,108 distinct words, significantly more than the other benchmarks. At the phrase level, leads in the number of unique 3-grams and 5-grams, suggesting wider variety of phrasal combinations and expressions. Most notably, at the syntactic level, MMSI-Bench contains 993 unique syntactic structures-nearly double that of VSI-Bench and more than triple that of ERQA. This linguistic and structural complexity stems from fully human- curated annotation process, which is driven by experts and avoids any reliance on templates. This ensures that models challenged not only with spatial reasoning tasks but also with understanding and processing diverse, naturally-phrase questions that reflect real-world linguistic distribution. Consequently, higher bar for evaluating model's generalization capabilities.
E ADDITIONAL. IMPLEMENTATION DETAILS AND EXPERIMENTAL RESULTS
E. BENCHMARK MODELS.
We conduct comprehensive evaluation of wide range of MLLMs, encompassing both proprietary and open-source families, diverse model scales, and recent architectural advances. For proprietary models, we includes GPT-5 (OpenAT 2025c), o3 (OpenAT 2025d), GPT-4.5 (OpenAl 2025b),
GPT-4.1 (OpenAl 2025a). GPT-4o (OpenAll 2024). Gemini 2.5-Pro (DeepMind 2025 Claude- 3.7-Sonnet (Anthropic 2025). dots.vim (Huo al. 2025). GLM-4.5Vtthikking (Hong al.) 2025). Doubao- 1.5-pro Seed 2025), and Seedl V.. (Guo et al. 2025). On the open-source side, (Bai al.) 2025). LLaVA- Vision (Li et al. (024),, Intern (Zhu et al. (2025), Intern VL.2.5 (Chen et al. 2024b), DeepSeek- VL2 (Wu et al. 2024). Llama-3.2-1 Vision (Meta 2024), and NVILA (Liu al. 2024c).
we systematically assess state-of ofte-oo-thheare models such as Qwen2.5-v
E.2 IMPLEMENTATION DETAILS FOR MODEL EVALUATION
We report accuracy (%) using exact match between answers extracted from model outputs and ground-truth answers for multiple-choice questions. If a model fails to generate answer in the required format, we adopt the L.LM-based fallback strategy from VLMEvalKit (Duan et al. 2024) to extract the intended response.
19

---

Published as a conference paper at ICLR 2026
We evaluate models using three prompting strategies: Direct, Linguistic (Zero-Shar Chain-of- Thought), and Visual. The exact prompts used for each method are shown Table
Table 7: Prompts used for different evaluation settings.
Strategy
Prompt Template
Direct
(images) ) [question]
Answer with the option's letter from the given choices directly, Enclose the option's 's letter within
Linguistic (Zero-Shot CoT)
[images] ) [question]
Let's think step by step and then answer with option's letter from given choices. Enclose the option's i letter within
Visual
(image with visual prompt] [question) Answer with the option's letter from the given choices directly, Enclose the option's 's letter within
For the visual prompting strategy, "(image with visual prompt]" refers to the image input combined with visual prompt drawn using PATS (Ni et al. 2023). as illustrated in Figs.
Figure 8: Illustration of the visual prompt utilized in experiments.
Figure 9: Illustration of the visual prompt utilized in our experiments.
The specific prompt used for evaluating the reasoning process is shown in Table
The prompt for evaluating reasoning processes with a human-annotated reference is shown in Table 9
20

---

Published as a conference paper at ICLR 2026
Figure 10: Illustration of the visual prompt utilized in our experiments.
Minin
nimum
Mutic
ALMONI NN MIX
ASIAN
Figure 11: Illustration of the visual prompt utilized in our experiments.
Table 8: Prompt used for evaluating the reasoning process in our study.
Prompt
[images] | {question}
[reasoning process to be evaluated) [the correct answer)
You are given images, question about the images, the correct answer, and reasoning process to be evaluated.
Your task to evaluate the reasoning process and determine whether it entirely correct. If there are any errors, identify the type of error that most critically led to the incorrect answer by selecting the corresponding category number. If the reasoning process is completely correct, select only category
Categories:
Grounding error: Failure accurately detect, locate, or reference objects or elements in the image.
2. Overlap identification and scene reconstruction error: Mistakes in identifying overlaps between elements errors reconstructing spatial relationships within the scene.
Situation transformation reasoning error: Errors reasoning about directions or spatial relations from the perspective of an entity other than the observer (i.e., allocentric transformation errors). 4. Spatial logic error: Flaws in spatial reasoning logic, such as fabrication, linguistic intelligence error, misusing reference objects in motion reasoning, or other errors not covered by categories
or
Completely correct reasoning.
Please output only the relevant category number.
21

---

Published as a conference paper at ICLR 2026
Table 9: Prompt used for evaluating reasoning processes with human-annotater reference in our study. Here, the structured format of the reasoning process means that, based on the original human- annotated reasoning steps, each step is further labeled with reasoning type. There are four reasoning types, Grounding, Overlap-Matching and Scene- Reconstruction, SiquationTTranffrmmatii Reasoning, and Spatial-Logic Reasoning. which correspond to four types possible errors, respectively. For example, a reasoning process can be transformed into the structured format as follows: "The pink bench is to the cast of the bed. The white wall lies to the south. Thus, the bathroom lies in the southeast corner of the sleeping region." will become "The pink bench is to the cast of the bed (Grounding). The white wall lies to the south (Grounding). Thus, the bathroom lies in the southeast corner of the sleeping region (Overlap-Matching and RecessRRcoottruction)
Prompt
[images] (question)
[reasoning process to be evaluated] [the correct answer)
[human- aanaa--nnnoattdd reference reasoning process with structured [ormat]
You are given images, question about the images, the correct answer, doaman-annotann reference reasoning process, and a reasoning process to be evaluated.
Your task evaluate the reasoning process and determine whether it is entirely correct. If there any errors, identify the type error that most critically led to the incorrect answer by selecting corresponding category number. If the reasoning process is completely correct, select only category
Categories:
I. Grounding enor: Failure to accurately detect, locate, or reference objects elements in the image.
Overlap identification and scene reconstruction error: Mistakes in identifying overlaps between elements or errors in reconstructing spatial relationships within the scene.
Errors in reasoning about directions or spatial relations from the perspective of an entity other than the observer (i.e., allocentric transformation errors). 4. Spatial logic error: Flaws in spatial reasoning logic, such as fabrication, linguistic intelligence error, misusing reference objects in motion reasoning, other errors not covered by categories 1, or 3.
3. Situation transformation reasoning
5. Completely correct reasoning.
Note: The human-annotated reference reasoning process is the only correct and is for reference only. Other reasoning paths may also be correct. You should judge the correctness the reasoning process and the error category based on the specitic images. Please output only the relevant category number.
E.3 HUMAN EVALUATION SETUP
Human performance is measured the average accuracy of five adult participants who were not involved in the data annotation process. During evaluation, human annotators are provided with both the questions and the corresponding images simultaneously and allowed unlimited time answer each question to the best of their ability. They may review the images as many times as needed to gather comprehensive information. The evaluation interface supports two modes of image viewing: all images can be displayed side by side for direct comparison, or the participant can switch between individual images as needed. This flexible setup ensures that evaluators can fully utilize the available visual information for accurate reasoning and decision makinn..
E.4 COMPLETE EVALUATION RESULTS PROMPTING TECHNIQUES ON MMSI-BENCH
We provide the complete evaluation results of prompting techniques on MMSI-Bench in Table 10 which correspond to the results presented in Figure 5 of the main text.
E.5 CORRELATION WITH DOWNSTREAM EMBODIED AND ACTIVE PERCEPTION TASKS
To assess the transferability of MMSI-Bench to downstream embodied and active perception tasks, analyze the correlation between model performance on MMSI- BASI-Bench and three representative tasks from AMBRODIDDEENCEE Yang et al. (2025): EBB-Navigttion, EB--Manpplltiion, and EB- Eab-Haa
22

---

Published as a conference paper at ICLR 2026
Table 10: Complete evaluation results of prompting techniques on MMSI-Bench.
Positional Relationship
Attribute Appr.
Motion
MSR
Models
Cam-Camiiti-ae -Oa, Rog-Rog Cam-Oiij Olj-Reg Cam-Raag Mess.
City
The original MMST-Bench setting without the use of any prompting techniques.
GPT-4.5
29.8
39.5
51.2
47.1
55.4
39.1
33.3
41.9
40.3
GPT-4o
34.4
24.5
23.5
19.8
37.6
27.7
32.8
31.8
35.1
36.8
30.8
30.3
Qwen2.5.VL.
LLavA-OneVision-
43.0
31,9
33.3
30.2
37.6
38.6
28.1
19.7
13.5
32.9
15.7
28.4
Linguistic prompting
GPI-4.5
44.1
28.7
35.8
32.6
40.0
43.4
57.8
31.8
41.9
36.8
42.4
39.6
4PT
Owen2.5-VL-722
28.9
30.8
34.5
32.5
24.6
30.0
40.5
28.7
22.9
31.5
23.6
29.0
LLaVA One Vision 72B
31.7
24.0
26.5
31.3
22.1
27.3
25.2
20.1
13.4
8.0
23.2
24.7
Concalenating multiple images into a single composite image without using amy prompting techniques.
GPT-4.5
30.8
34.6
33.7
43.8
27.3
GPT-4o
28.0
22.3
38.3
24.4
41.2
31.3
32.8
19.7
21.6
30.3
26.3
28.5
Qwen2,5-VLL772B
36.6
22.3
34.6
18.6
35.3
28.9
37.5
16.7
23.0
31.6
31.3
29.1
LLaVA OneVViionn
32.3
24.5
30.9
26.7
37.6
27.7
32.8
19.7
9.5
26.3
29.8
27.6
Visual prompting
GPT-4.5
36.6
31.9
30.9
31.4
42.4
27.7
37.5
31.8
24.3
28.9
30.8
32.1
GPT-4o
28.7
29.5
28.8
25.0
37.0
30.6
40.4
27.1
16.4
30.5
29.2
29.3
Qwen2.5. VL.
30.9
27.4
30.4
28.7
36.
32.1
30.5
25.0
19.7
29.7
27.1
28.8
29.0
33.0
29.6
31.4
38.8
33.7
29.7
21.2
14.9
26.3
28.3
29.0
EB-Navigation evaluates embodied agents' navigation abilities in simulated environments, where models are required to output low-level actions to reach target locations.
EB-Manipulation focuses on object-centric embodied manipulation skills, such as grasping or rearranging objects, and requires fine-grained spatial reasoning and interaction.
EB-Habitat is built on the Habitat 2.0 simulator and targets high-level active perception and planning: agents must plan and execute sequences skills to visit multiple locations and find user-specified items.
Correlation analysis. We compute Pearson and Spearman correlation coefficients between the average accuracy on MMSS-Bench and performance on each downstream task across a diverse set of frontier multimodal models. The results show consistently strong positive correlations:
MMSI-Bench EBBNavigation: Pearson 0.8147,
Spearman 00.7333.
MMSI-Bench E:B-Manipulation: Pearson 0.7402, Spearman 00.5500.
MMSI-Bench EBBHabitat: Pearson 0.7299, Spearman 0.6167.
These results indicate that models performing well on MMSI-Bench also tend to achieve strong performance on both downstream embodied control (navigation, manipulation) and active perception (navigation, Habitat) tasks. The particularly high correlation with -B-Navigation suggests that the spatial understanding and scene interpretation abilities measured by MMSI-Bench with the competencies required for effective embodied navigation. At the same time, the substan- tial correlations with EB- 1BB-annpulattin and EB-Habitat demonstrate that MMSS-Bennd captures capabilities that generalize to both low-level control and high-level, goal-directed active perception.
closely aligned
Per-model performance. Table presents the detailed per-model scores on MMSI-Bench and the three EMBODIEDBENCH tasks that underlie the above correlation analysis.
F DETAILS OF THE DATA CURATION PROCESS
F.1 DATASET SOURCES
ScanNet (Dai et al., 2017): ScanNet is large-scale RGB-D video dataset designed for 3D scene understanding in real-worke indoor environments. consists of 2.5 million RGB-D images across 1,513 scans from 707 unique spaces, each annotated with 3D camera poses, surface reconstructions,
23

---

Published as a conference paper at ICLR 2026
Table 11: Performance of different models on MMSI-Bench and three downstream tasks from EMBODIEDBENCH: EB-Navigation, EB-Manipulation, and EB- BB-Habitat. All values are reported as percentages.
Model
MMSI-Bench
EB-Navigation
FB-Manipulation
EB-Habitat
GPT-40
30.3
57.7
39.6
59.0
30.2
45.0
28.5
58.7
Claude-3.5 3.5-l-ee--S
31.3
44.7
25.4
68.0
Llama-3.2- 2-1 IB- -isiiiionsnns
25.4
21.4
0.9
25.0
28.5
30.7
10.6
49.0
Intern VL2.5-388
27.9
30.3
15.8
38.3
Intern WLtcm-LL..5-8B
28.7
21.3
7.0
11.3
Owen2.5-VL-72BB-B
30.7
40.0
16.2
37.7
Qwen2.5-VL-7B-Inn
25.9
20.0
9.6
14.3
and dense object-level semantic segmentations. The dataset was collected using scalable pipeline that combines easy data capture with crowdsourced annotations, making it one of the most richly- labeled and comprehensive resources for training and evaluating deep learning models on tasks such as 3D object classification and semantic voxel labeling.
nuScenes (Caesar al., 2020): nuScenes is multimodal autonomous driving dataset that features a complete sensor suite, including 6 cameras, radars, and 1 lidar, all with 360-degree coverage. contains 1,000 driving scenes, each 20 seconds long, fully annotated with 3D bounding boxes for 23 object classes and 8 attributes. nuScenes stands out for its diversity of sensor data, large scale of annotations, and support for advanced tasks such as 3D detection and tracking, making it a benehmark for research in perception for autonomous vehicles.
Matterport3D (Chang al. 2017): Matterport3D a large-scale RGB-D dataset capturing 10,800 panoramic viewpoints from 194,400 images of 90 building-scale indoor scenes. It provides surface reconstructions, globally aligned camera poses, and detailed 2D and 3D semantic segmentations at both region and object levels. The comprehensive panoramic coverage and precise global alignment enable wide range of computer vision tasks, including keypoint matching, semantic segmentation, and view overlap prediction, particularly in the context of indoor scene understanding.
Ego4D (Grauman et al., 2022): Ego4D an unprecedented egocentric video dataset, comprising 3,670 hours of first-person footage collected by 931 participants across 74 locations in countries. The dataset covers a wide array of diiey-life scenarios and includes supplementary data such as audio, 3D meshes, gaze, and multi-camera views. With millions of annotations supporting various benchmark tasks, Ego4D enables research into first-person perception, episodic memory, and social interactions, significantly advancing the field of egocentric visual understanding.
AgiBot-World AggBBot-WorldCoomee World--ontriibtoors et al. 2025): AgiBot-Worke a large-scale robotic manipulation dataset featuring over million trajectories across 217 tasks in five real-world do- by more seman-in-the-lowe verification to
mains- inciun-nnluuiing domestic, retail, industrial, restaurant, and office cuvironments-collee than 100 real robots. Data collection standardized and includes ensure high quality and diversity. The dataset supports a wide range of hardware, from simple grippers to dexterous hands with visuo-tactile sensors, and provides rich multimodal annotations. AgiBot- World enables the development and evaluation of generalist robot policies in complex, long-horizon, and diverse scenarios, moving towards scalable, general-purpose purpose robotic intelligence.
DTU (Jensen et al. 2014): The DTU dataset is large-scale multi-view stereo (MVS) benchmark designed for 3D reconstruction research. It contains scenes with significant variability in surface propertics and geometric complexity. Each scene is captured from 49 or 64 accurate camera positions and includes high-quality reference seans obtained via structured light. This dataset advances the scale and diversity compared to previous MVS datasets and serves challenging benchmark for evaluating the accuracy and completeness of stereo reconstruction algorithms.
DAVIS 2017 (Pont-Tuset al. 2017): DAVIS 2017 is benchmark dataset for video object segmentation, consisting of 150 video sequences and a total of 10,459 annotated frames with pixel- level masks for multiple objects per scene. Compared to earlier versions, DAVIS 2017 introduces
24

---

Published as a conference paper at ICLR 2026
PNtt-derch
colagedd imagey www.stnnskk...pp..
Quedee
Uneer
Diter
Draneptin
Galand
Celected
Dieezstinn
Cranepter
resseing
Figure 12: Screenshot of the data annotation interface, where annotators select images, design questions, and provide answers and reasoning.
greater complexity, including multiple objects, occlusions, small and fine structures, and fast motion. The dataset has driven significant progress in video object segmentation and is widely used for benchmarking and competition in the field.
Waymo Open Dataset (Sun et al., 2020): The Waymo Open Dataset one of the largest and most diverse multimodal datasets for autonomous driving. It comprises 1,150 scenes, each lasting 20 seconds, captured across multiple cities with extensive geographic coverage. Each scene includes synchronized and calibrated high-resolution camera images and LiDAR data, along with exhaustive 2D and 31) bounding box annotations and consistent object tracking IDs. The dataset supports research in detection, tracking, and sensor fusion, providing valuable foundation for developing robust and generalizable self-driving technologies.
MMSI-Bench is composed of questions from data sources: 463 from Matterport3D, 280 from ScanNet, 67 from Ego4D, 45 from AgiBot-World, 72 from DTU, 39 from nuScenes, 29 from DAVIS 2017. and from Waymo. We analyzed widely used VLM pre-training datasets. CC3M (Changpinyo et al. 2021 and LAION-COCE (Schuhmann et al. (022), which are commonly utilized by open- source VL.Ms such as LLaVA, NVILA, and Qwen- VI.. Our analysis indicates that outdoor scenes constitute only estimated 10% of the data in these foundational VLM datasets. To align our benchmark with the typical training distribution of these models, we have curated MMSI-Bench to maintain comparable proportion, with approximately 12% of our samples sourced from outdoor datasets. Recognizing that spatial understanding one of the most critical capabilities for autonomous driving, selected Waymo and nuScenes the primary sources for these outdoor samples.
F.2 DATA ANNOTATION AND QUALITY CONTROL INTERFACES
The interfaces for data annotation and quality control are illustrated in Figure 12 and Figure 13 respectively. During annotation, amnotators select an image, design question, provide the correct answer, and write the corresponding reasoning process. For quality control, reviewers can choose to discard a data entry if any issues are identified, providing the reason for rejection; otherwise, they can approve and retain the data.
25

---

Published as a conference paper at ICLR 2026
Cata Quality Check
Co Typ
Pamsorc
Optine semark
Optnn
Eedinatioo
Figure 13: Screenshot of the quality control interface, where reviewers decide whether to retain or discard data and specify the reasons for data issues.
F.3 GENERAL GUIDELINES
MMSI-Bend is designed to systematically evaluate the multi-image spatial reasoning capabilities of multimodal large language models (MLLMs) using human-readable questions. The following guidelines govern the data annotation process:
Human-Centric Design: All questions and answer choices are formulated in natural, readablee English, avoiding technical prompts such as direct requests for camera parameters.
Image Requirement: Every question is constructed so that it cannot be answered using single image alone; the information required answer must be synthesized from multiple images.
All questions are multiple-choice with four options, only one of which is correct. Questions are created in free form (no templates) to ensure diversity and challenge for MLLMs.
Question Type and Format:
Category Coverage: Each question assigned to one of eleven categories, covering positional relationships, attributes, motion, and multi-step reasoning among cameras, objects, and regions. This taxonomy is designed to comprehensively span the space of fundamental spatial reasoning types.
Difficulty, Diversity, and Clarity: Questions must be of high difficulty, unambiguous, and answerable by humans. Annotators are encouraged to maximize diversity in question design and avoid repetitive patterns. Annotations must be accurate, consistent, and adhere high standard of academic rigor.
Ethics and Licensing: All images and question sources must comply with copyright
and licensing restrictions. Annotators must not material from sources that prohibit redistribution.
Reasoning Process Annotation: For each question, annotators provide a detailed step-by- step reasoning process, documenting the logical path from the images to the answer.
26

---

Published as a conference paper at ICLR 2026
For the "Multi-Step Reasoning" type, ask annotators to design questions that are as challenging as possible and cannot be answered directly, and to explicitly write down the sep-by-step solution in the reasoning process. According to statistics on annan-annotated reasoning processes in this category, "Multi-Step Reasoning" questions involve between 3 and reasoning steps, with an average of 5.7 steps per question.
F.4 DATA FORMAT AND STRUCTURE
To ensure consistency and facilitate reliable evaluation and analysis, all benchmark entries conform standardized data structure:
Each question-answer pair stored as JSON object, with fields for question text, four answer options, the correct answer index, category, difficulty, image paths, and annotated reasoning process.
JSON Format:
Image Files: All referenced images stored separately and linked via file paths in the JSON entries.
Field Completeness: Each entry must have all fields completed: question, answer, category, image list, and reasoning process.
Multi-Image Structure:
Except for the Multi-Step Reasoning category, every question is associated with exactly two images. Multi-Step Reasoning samples may include more images according to task complexity.
Final Storage Format: For efficient downloading and usage, the entire dataset-includiin binary-encoded images and metadata is stored in the Parquet format.
F.5 QUALITY CONTROL AND VALIDATION
A rigorous multi-stage quality control procedure is implemented ensure the reliability and validity of the benchmark:
All annotated samples are independently reviewed by three expert reviewers who were not involved in the original annotation process.
Independent Review:
Error and Ambiguity Filtering: Reviewers systematically remove or revise questions exhibiting insufficient visual information, ambiguity (linguistic or visual), incorrect answers, vague/incomplete reasoning processes.
Non- non-Triviality Check: Samples that can be answered using a single image or general common sense (rather than spatial reasoning across images) are excluded.
Iterative Feedback: Annotators receive feedback on problematic items and are required to revise or replace such entries.
Final Audit: Only questions passing all review stages are included in the benchmark, ensuring high data quality and robust evaluation.
F.6 DIFFICULTY ANNOTATION
We recruited new human evaluators to answer MMSI-Bench and recorded their per-question comple- tion times. Each question was independently assigned two evaluators. We normalized completion times across evaluators and used the mean normalized time as the difficulty indicator, with longer times indicating higher difficulty. The resulting distribution exhibited a clear three-tier structure. We therefore labeled each question as Easy, Medium, or Hard. Any question answered incorrectly by the evaluators was directly assigned to the Hard tier. The Easy tier contains 605 questions with an average human completion time of approximately 40 seconds. The Medium tier contains 262 questions with an average time of approximately 1 110 seconds. The Hard tier contains 133 questions, defined as those taking longer than approximately 110 seconds average those answered incorrectly.
F.7 DETAILS ABOUT THE DISTRIBUTION OF DATA ACROSS THE 11 TASKS
For the Motion (Cam.) task, we use a total of 74 samples: 32 from Ego4D, 27 from AgiBot Moold, from ScanNet, and 7 from nuScenes (combining all nuScenes image subsets).
27

---

Published as a conference paper at ICLR 2026
For the Positional Relationship (Cam.-Dl Obj..) task, we 86 samples in total. Among them, 61 samples are drawn from Matterport33 (combining validation and test splits), and 25 samples are from ScanNet.
For the Attribute (Meas.) task, we use 84 samples: 40 from Matterport3D, 19 from ScanNet, and from DTU.
For the Positional Relationship Matterport3D and 2 from ScanNet.
task, we use 81 samples in total, of which 79 come from
For the MSR task, we use 197 samples overall. Specifically, 96 samples are from ScanNet, 79 from Matterport3lD, 10 from DAVIS 2017 (via the tapvid-based motion setup), from AgiBot-Workke World, 6 from nuScenes, and from Waymo.
For the Motion (Obj.) task, 13 from SgiBot-World, 6 from nuScenes, and 3 from Waymo.
76 samples. These include 35 from Ego4D, 19 from DAVIS 2017,
For the Positional Relationship (Cam.-Cam.) task, we use 92 samples in total. This consists of 60 samples from ScanNet, 20 from nuScenes, 11 from Matterport3ID, and 2 from DTU.
For the Positional Relationship (Cam.-Reg.) task, ScanNet and 46 from Matterport3D.
83 samples in total, including 37 from
For the Attribute (Appr.) task, we use 66 samples. We draw 65 samples from DTU (aggregating all DTU variants we use) and sample from ScanNet.
For the Positional Relationship (Obj.-Reg.) task, 85 samples in total, with 75 samples from Matterport3D and 10 from ScanNet.
Finally, for the Positional Relationship (Obj.-bjj) task, we use 94 samples, composed of 72 samples from Matterport3D and 22 from ScanNet.
G REPRESENTATIVE MMSI-BENCH SAMPLES FROM EACH CATEGORY (FULL VERSION)
For clarity and readability, questions and rationales in the main text are simplified; the complete versions for each category are provided in this section. We present one full example for each of the 11 categories: Position (Camera-Cameray Position Objert, Position CCamera-Region) Position (Object-Object), Position (Object-Region) Position (Region- (Region-Region Region), Motion (Camera), Motion (Object), Attribute (Measurement), Attribute (Appearance), and Multi-Step Reasoning. These representative samples are illustrated in Figures 14
H ADDITIONAL CASE STUDIES
In this section, we present additional complete reasoning processes of current multimodal large language models (MLLMs) to illustrate their spatial reasoning deficiencies more intuitively. As shown in Figure 25 Figure 26 Figure 27 and Figure 28 we highlight four types of errors in different colors: grounding errors (blue), ovarlap- and aceme-reconstruution errors (red), situation-transfore transformation reasoning errors (orange), and spatial-logic errors (green).
28

---

Published as a conference paper at ICLR 2026
Position Cam.-Cam.)
Question: Assuming am taking the first image, where is the camera likely positioned relative when taking the second image? Options: A: Left, B: Front, C: Back, Right
The first image captures the green chair and the backpack on the left and right sides, respectively. The second image captures the green chair and the backpack on the right and left sides, respectively, and can be seen that the brown leather chair, which was very close to the camera in the first image, farther from the camera and almost directly in front in the second image. forward.
Reasoning:
inferred that the approximate direction is
Figure 14: Full example for Position (Camera- (Camera-Camera) category.
Position (Cam.- -bbj)
When I took the second image, where was the black door in relation to me?
Question:
Options: A: Behind me, B: On my Left, C: In front of me, D: On my right
The images have overlapping areas the painting the right side of Image and the painting the left side of Image are the By aligning the overlapping parts, we can perform spatial reasoning. In Image 1, the painting is in front and to the right of the photographer, while the black door directly in front. In Image 2, the painting is in front and to the left of the photographer. Therefore, from Image Image 2, the photographer rotated 90 degrees to the right. As result, when taking the second image, the black door was on the photographer's left.
Reasoning:
Figure 15: Full example for Position (Camera-Objeee category.
29

---

Published as a conference paper at ICLR 2026
Position (Cam. (Cam.-Reg.)
Question: The images
sequentially from first-person
perspective. At the moment of taking the last image, in which direction is the file printing area relative to you? Options: A: Behind, front, C: Right, D: Left
The images were taken sequentially, and there is an overlap in the carpet between Image 1 and Image 2. Based on the relative position of the carpet in both images, it can be inferred that the camera rotated approximately 60 to 90 degrees the right. In Image the printing area is located in front and slightly to the left the photographer, so after the camera rotates to the angle in Image the file printing area ends up your left.
Reasoning:
Figure 16: Full example for Position (Camera- Regnoo)) category.
Position (Obj.-Obj.)
The deer head ornament is on the west side of the mythological which direction from the mythological statue is the mural the room?
Question:
Options: A: Southwest, B: Northeast, C: Northwest, D: Southeast
The door of the the room on left side of Image have overlapping parts, so they belong to the same room. The deer head ornament on the west of the mythological statue, and the mural in the room is further north than the deer head ornament. Therefore, the mural in the is the northwest the mythological statue.
Reasoning:
the right side of Image and the door
Figure 17: Full example for Position (Object-Obyf Object) category.
30

---

Published as a conference paper at ICLR 2026
Position (Obj.- Reg.)
The whiteboard in the study is hung on the north side of the study. which direction is the painting the wall relative the center of the study?
Question:
Options: A: Northeast, B: Southeast, C: Northwest, D: Southwest
The books the left side of Image and the books the right side of Image 2 have overlapping features. The whiteboard is on the north side of the study, so the bookshelf is on the east side. The painting is on the east side of the study, just like the bookshelf, but it is positioned further toward the south than the bookshelf. Therefore, the painting on the wall is in the southeast direction the study.
Reasoning:
Figure 18: Full example for Position (Object-Region, category.
Position (Reg.-Reg.)
The pink bench at the end of the bed is on the east side of the bed. In which direction is bathroom located the bedroom? Options: A: Southwest, Northwest, Southeast, D: Northwest
Question:
The flooring and walls on the right side of Image 1 and the left side of Image have overlapping features. The pink bench the end of the bed is on the east side the bed, so the white wall on south side of the bedroom. Image 1, the bathroom is on the same side as the wall, which the south side of the bedroom, but is positioned further toward the east. Therefore, the bathroom in the southeast direction the bedroom.
Reasoning:
Figure 19: Full example for Position Besgion-Regione category.
31

---

Published as a conference paper at ICLR 2026
Motion (Camera)
Question: The images are taken sequentially from a first-person perspective. In which direction is the camera rotating? Options: A: Left, B: Right, C: Up, D: Down
Reasoning: images were taken sequentially from a first-person perspective, and the orange sofa in the lower left corner of Image also appears in Image In Image the sofa the left side of the photographer; in Image the sofa is front the photographer, Therefore, from Image to Image 2, the camera rotated to the left.
Figure 20: Full example for Motion (Camera) category.
Position (Cam,-Cam.)
With the cameea facing backward and the images taken in sequence, which direction the race moving? Options: A: Left, Front left, C: Front right, D: Forward
Question:
Reasoning:
The camera is facing backward, and the images taken is
the photographer, indicating that it is moving forward. Compared to Image 1, has shifted to the left, so from Image to Image 2, the race car is moving toward the front-left.
sequentially, Image 2,
Figure 21: Full example for Motion (Object) category.
32

---

Published as a conference paper at ICLR 2026
Attribute (Measurement)
Question: Which is longer, the oil painting the wall the table below the mirror?
Options: A: Sometimes the table below the mirrar longer, sometimes the oil painting on the wall is longer, B: The oil painting on the wall, C: The same length, D: The table below the mirror
The right side of Image and the left side of Image 2 have overlapping features, indicating that the images were taken in the same room, with identical wall patterns. Using the striped pattern the wall a reference, length
Reasoning:
painting Image the width of 14 wall stripes, while the length of the table beneath the mirror in Image 2 is about 12 wall stripes wide. Therefore, the painting in Image is longer than the table beneath the mirror Image? 2.
approximately equal
Figure 22: Full example for Attribute (Measurement) category.
Attribute (Appearance)
What the shape of the top view of the visible part of the building in the images? Options: A: I-shape, B: T-shape, C: F-shape, D: L-shape
Question:
The yellow umbrella and the colorful rooftops in Images and 2 have overlapping areas, allowing for spatial reasoning when the two images are combined. Image vertical rectangle, while Image 1, there are two small horizontal rectangles on the same side of it. The vertical rectangle forms the main stem of the letter "F", and the two horizontal rectangles form the top and middle of "F". Therefore, the shape the top view visible part of the building is the shape of "F".
Reasoning:
houses are connected to
Figure 23: Full example for Attribute (Appearance) category.
33

---

Published as a conference paper at ICLR 2026
Multi-Step Reasoning
When open the door and come from outside, where is the clock located relative to me?
Question:
Options: Directly in front, On the left, C: Front right, D: Front left
Reasoning: The staircase appears in both Image and Image 2, the refrigerator in the lower right corner of Image 1 overlaps with the refrigerator the left side of Image and the tiled tabletop the left side Image overlaps with the right side of Image Therefore,
sequence the images as follows: from Image 2 to Image 1, the camera turns downward; from Image to Image 4, the camera turns to the right; from Image 4 to Image 3, the camera turns to the right. Based on the overlaps and the order the images, when open the door, come from outside, am facing perpendicular to floor-to-ceiling window seen Image 3. To reach the clock shown in Image 3, should walk straight and then turn left. Thus, the clock is located in the front-left direction relative
Figure 24: Full example for Multi-Step Reasoning category.
34

---

Published as a conference paper at ICLR 2026
Rock bot Roll Haps
g
ON
If the direction the is facing towards the viewer is considered the front, where is the bedroom located relative the TV? Options: A: Front left, B: Back left, C: Front right, D: Back right
GPT-40
To determine the location of the bedroom reletive TV, lat's analyze images step by step: First Image Analysis: The TV is mounted on the wall. There are decorative items and artwork around the TV. The reflection in the TV shows window and part of room with a couch. 2. Second Image Analysis: The second image shows a view of a bedroom through doorway. The artwork the wall same as
second image is the right side of the TV the first image, and the bedroom visible through doorway next this artwork in the second image, the bedroom is located to the right of the TV. 4. Conclusion: The bedroom is located to the right of the TV when facing it. Since the TV facing towards the viewer, the bedroom on the viewer's right side. bedroom located back right relative to the correct option
right
image.
Positioning:
GPT-4.5
Step-by-step reasoning: the first image,
TV facing towards the viewer, viewer looking front of the the reflection the TV screen, showing windows and sofa, indicating the room extends behind and to the left of the TV. 3. The second image shows the bedroom through a doorway, which is located the right side perspective).
TV (as the TV not visible this second image, it must behind viewer's perspective (opposite
right behind TV), the bedroom is located the back relative the TV. Therefore, the correct enswer is D
bedroom
Qwen2.5-2 5-VL-22B
analyze images step by step: Image 1: reflection of window and some furniture in the TV screen. This indicates that the TV is facing towards the viewer. 2, Image 2: The image shows poster on the wall and an open doorway leading to another room, The doorway on the right side the image, and through it, we can see part bedroom Image
determine location
relative
The iV mounted the wall,
bed and a lamp. Relative Positioning Since the facing towards the bedroom visible through doorway on right Image bed room must be located the right of the TV when considering the TV's orientation, Given these observations, the bedroom is located to the front right of the TV. *C
Figure 25: Illustration of situation transformation reasoning emrors (highlighted in orange).
35

---

Published as a conference paper at ICLR 2026
The pictures taken sequentially from first-person perspective. At the moment of the last picture, in which direction is the shelf relative you? Options: back left, B: front right, back right, D: front left
SF
GPT-40
To determine the direction of the shelf relative to the viewer in the last picture, let's anelyze the sequence of images: 1. First Image: The shelf visible on the right side of the image. The couch the foreground, suggesting viewer facing shelf is longer visible. The person walking away from the viewer, indicating the viewer has turned around. Given this sequence, in the last image, the viewer has turned away from the shelf. Therefore, the shelf is now behind the viewer and to the right. The correct option is C: right
Image:
viewer looking through doorway,
GPT-4.5
viewer, slightly right. The second image taken turning the view the left, indicated by the shelf no longer being visible and the door and hallway now being front. 3. Since the shelf was initially front and slightlyy to the right, after turning left, the shelf must be behind and to the right of the viewer. Therefore, the shelf located at the back right relative to the viewer. Anewer:
Step-by-step reasoning:
image
shelf directly
Qwen2.5 2---2-2---
person taking picture, analyze sequence shelf room with shelf the laft side. The person appears facing the of images: 1. the first image,
determine
shelf. 2. In the second image, the person has moved forward and turned slightly to the right, as indicated by the change the view movement: The shelf that was initially on the side the last picture, the shelf relative the
room and the presence of another person in the background. Given this
image now behind to the left of the V'AY".
second image. At the
Figure 26: Illustration of mvtrlap and sceme-reconstrrattin errors (highlighted in red), -patial-logic errors (highlighted in green).
36

---

Published as a conference paper at ICLR 2026
The images are taken sequentially from first-person perspective. In which direction is the camera moving (with the direction the camera is facing at the moment of image as the front)? Options: A: Backward, B: Back right, C: Left, D: Back left
GPT-4o
To determine the direction movement, let's analyze the changes between the two images: 1. Image 1: person holding a
on ground, ladder closer the ladder, which is prominently view. The toolboxes are still visible, but the perspective has shifted slightly to the left. The person seems to have moved closer to the ladder and slightly away from the gate. Based on these observations, the movement appears be towards the ladder and slightly away from the initial position, which suggests movement back relative initial facing direction. Therefore, the direction movement
standing near gate.
visible the left. 2.
2: The
left.
GPT-4.5
Step by-step reasoning: In the first image, observe the position of the gate, the toolbox, and the plants. The gate slightly open, and the toolbox directly in front of the person. 2. the second image, the gate and toolbox appear to have moved slightly forward and to right relative to the person's perspective, The plants, which initially the
directly in front. 3. This indicates that the moved backward and slightly to the laft relative to their original facing direction in image 1. Therefore, the correct direction of movement 'Back left". Answer: D
side,
has
Qwen2.5 5-0.555-L22-
To determine the direction movement, let's analyze the changes between the two images: 1. Image 1: The appears
person holding a perspective suggests they facing towards gate
workshop
materials around.
Image The person now ladder, and the view has shifted. The workshop area is still visible but from a different angle. The person appears to be moving towards the left side of the image, as indicated by the position of the ladder and the change the surrounding environment. Given these observations, the person seems to be moving to the left relative initial position Image
"C'
Figure 27: Illustratoon of grounding errors (highlighted in blue), spatial-logic (highlighted in green).
37

---

Published as a conference paper at ICLR 2026
Sequential shooting from first-person perspective, with the direction the camera is facing the front, which direction the camera moving?
Options: A: Left, Forward, C: Backward, D: Right
GPT-40
To determine the direction the camera is moving, let's analyze the two images: 1. First Image: The white car camera positioned behind
Second moved closer the The camera still positioned behind the white car, but the car now larger the frame. From these observations, the camera moving towards the white car, which means it is moving forward. Therefore, the direction camera moving B: Forward.
lane, moving The white
Image:
indicating that
GPT-4.5
Stap-by-step reasoning: the first image, the white car directly ahead and slightly distant, and the road markings indicate
image, white closer the camera, and the silver on the left side now visible and closer, indicating the camera has moved backward relative to the white car. 3. The relative positions of the vehicles and road markings clearly show that the camera moving away from the white car, meaning it is moving backeward Therefore, the
white
Qwen2.5 211-2..55-LL222
moving. analyze changes Position Vehicles: the second image, the white front appears slightly closer, indicating that the camera (and the vehicle it is attached to) moving forward. 2. Surrounding Environment: The buildings and trees on both sides of the road appear to be similar positions relative the camera, suggesting no significant lateral movement
images:
overail layout or right). Road Markings: The road markings Given these observations,
road consistent, further supporting the forwerd camera is moving forward. B"
Figure 28: Illustration of spatial-logic enrors (highlighted in green).
38


Reviews:
[{"review_id": "ckXZNeripU", "reviewer": "Reviewer_htiq", "summary": "This paper introduces MMSI-Bench, a large-scale benchmark for evaluating multi-image spatial intelligence in multimodal large language models (MLLMs).\nUnlike prior benchmarks focusing on single-image reasoning, MMSI-Bench evaluates an MLLM’s ability to reason across multiple images to infer spatial relationships, motion, and object-camera-region dynamics.\nThe dataset contains 1,000 multiple-choice questions (covering 10 atomic spatial reasoning categories and one multi-step reasoning task) curated by experts from over 120,000 real-world images drawn from datasets such as ScanNet, nuScenes, Matterport3D, and Ego4D. Each question includes human-written reasoning chains and carefully designed distractors.\nExtensive evaluations of 37 MLLMs (including GPT-5, Gemini-2.5, Claude-3.7, Qwen2.5-VL, and InternVL-3) reveal that even the best proprietary model achieves only ~42% accuracy, while humans reach 97%, showing a large performance gap. The paper also presents an automated error analysis framework identifying four dominant failure modes: grounding, overlap-matching, situation-transformation, and spatial-logic errors.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Novel Benchmark Scope: MMSI-Bench uniquely targets multi-image spatial reasoning — a critical yet underexplored capability for MLLMs and embodied AI systems. Prior works (e.g., BLINK, ReMI, MuirBench) only contain limited spatial sub-splits, while this benchmark provides systematic coverage.\n\nHigh-Quality, Human-Curated Data: Each question is manually designed and audited by multiple experts with reasoning explanations, ensuring clarity, difficulty, and lack of ambiguity. The benchmark’s construction pipeline (Fig. 4) and taxonomy (Table 1) are well-documented and rigorous.\n\nComprehensive Evaluation: The authors benchmark 37 models, analyze scaling trends, compare open-source and proprietary systems, and examine effects of CoT and visual prompting. This breadth enhances credibility.\n\nInsightful Error Taxonomy: The four-type categorization of reasoning errors (Fig. 6) — grounding, scene reconstruction, situation transformation, and spatial logic — provides clear direction for future model development.\n\nImpactful Findings: The results demonstrate that current MLLMs lack robust spatial reasoning and that scaling model size or prompt engineering yields marginal gains, implying fundamental architectural and data limitations.\n\nStrong Writing and Presentation: Figures and examples (e.g., Fig. 2’s diverse question types) are clear, and the organization is consistent with ICLR standards.", "weaknesses": "Manual Effort vs. Scalability: Although the manual curation ensures quality, it also limits scalability — future expansions may face bottlenecks unless semi-automatic generation or verification methods are introduced.\n\nMetric Simplicity: The benchmark reports only accuracy on multiple-choice tasks. Incorporating richer evaluation metrics (e.g., reasoning correctness or step alignment) could offer more granular insight.\n\nPotential Dataset Bias: While data diversity is claimed, the benchmark draws primarily from common 3D and driving datasets, possibly biasing toward indoor and urban scenes rather than outdoor natural environments.\n\nLimited Generalization Discussion: The paper does not test transfer to downstream embodied tasks (e.g., navigation, manipulation) where multi-view reasoning is crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ycikgUy9Ob", "reviewer": "Reviewer_HMkb", "summary": "The paper introduces MMSI-Bench, a benchmark designed to evaluate the multi-image spatial reasoning capabilities of multimodal large language models (MLLMs). The dataset was created manually by six 3D-vision researchers. It consists of 1,000 multiple-choice question-answer pairs and 1,990 unique images sourced from eight real-world datasets (Matterport3D, ScanNet, DTU, nuScenes, Waymo, AgiBot-World, DAVIS 2017, and Ego4D). The questions are categorized into 11 tasks, under 4 main categories - positional relationships, attribute, motion and multi-step reasoning. The questions revolve around 3 spatial elements - camera, objects, and region. Most questions, other than the multi-step reasoning questions, involve 2 input images. Another key contribution is that each question is accompanied by a human-authored, step-by-step reasoning chain.\n\nThe paper conducts a comprehensive evaluation of 37 MLLMs. The primary result is a massive performance gap: the best-performing model (GPT-5) achieves only 41.9% accuracy, while human-level performance is 97.2%. The results also show that multi-step reasoning is particularly challenging for models. The authors also report fine-tuning and prompting ablations both of which provide minimal to no benefit. Language prompting involved zero-shot chain-of-thought reasoning and visual prompting involves highlighting PATS correspondences between image pairs. Finally, the authors provide insights into the failure modes of the evaluated models.", "questions": "1. How are the multiple images fed into the models? Are they concatenated into a single image, as not all models natively support multiple image inputs?\n2. The order in which the images are presented to the model seems critical, especially for tasks involving motion. How is this temporal order preserved and communicated to the model during evaluation? Can the authors provide an ablation study on the effect of image ordering?\n3. Is the visual prompt (with correspondence lines) provided along with the normal image inputs? There is a concern that the lines themselves might occlude important details in the images, making them harder to see. That could be the reason behind marginal prompting gains.\n4. The authors should provide details about the distribution of data across the 11 tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The benchmark's core strength is its manual, expert-driven annotation process. The questions are linguistically diverse, non-trivial, and require spatial understanding. The problem of multi-image spatial reasoning is highly relevant and timely for advancing embodied AI and robotics, and this paper clearly demonstrates a critical capability gap.\n2. The paper evaluates an extensive suite of 37 models, providing a valuable and comprehensive snapshot of the entire SOTA. The inclusion of \"Human Performance\" (97.2%) is a strong baseline that effectively contextualizes the low model scores.\n3. The findings that both advanced prompting (CoT, visual prompting) and fine-tuning fail to provide significant gains are valuable. These negative results strongly suggest that the models' failure is not a simple problem but a more fundamental capability deficit.\n4. The paper is well-written making it easy to read and follow.", "weaknesses": "1. 1,000 samples is small size, especially when divided across 11 tasks. This limited scale is a direct trade-off for the high-quality manual annotation (300+ hours), but it makes the benchmark difficult to scale and creates a risk of models eventually overfitting to this specific test set.\n2. Blind GPT-4o is not a suitable baseline since the questions depend heavily on the images. Language priors are unable to capture the context of the problem unless the images are described in words and the accuracy is expected to be similar to random baseline. The authors should update the baseline to be something more suitable.\n3. The paper's analysis of the core problem is unclear. In model size ablations, the authors suggest the bottleneck lies in data quality and diversity. However, the paper's own experiments fine-tuning and prompting show that existing methods to bridge data gaps do not work. This suggests the bottleneck is more likely architectural or requires in-domain fine-tuning, which the paper does not explore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYIakVKOE1", "reviewer": "Reviewer_AFVc", "summary": "This paper presents MMSI-Bench, a benchmark designed to evaluate multi-image spatial intelligence in multimodal large language models (MLLMs). The dataset contains 1,000 human-curated, multiple-choice questions requiring reasoning across multiple real-world images, with detailed reasoning annotations. The authors categorize 11 spatial reasoning tasks covering position, motion, and attributes, and benchmark 37 models. Results show a large gap between current MLLMs (best at \\~42%) and human performance (\\~97%), revealing major weaknesses in spatial reasoning across viewpoints. They also propose an automated error-analysis pipeline that identifies key failure types such as grounding and scene-reconstruction errors.", "questions": "1. How was question difficulty calibrated beyond human answer time? Did annotators estimate complexity or confidence?\n\n2. Could reasoning annotations be used for training models (not just analysis)? If so, how does that affect overfitting?\n\n3. How consistent are human annotators across the four error types? Any quantitative measure?\n\n4. Did you observe differences between models trained with ego-centric data vs general web-image pretraining?\n\n5. How might MMSI-Bench interact with embodied datasets like Habitat or RoboBrain for active perception tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The focus on multi-image spatial reasoning fills a clear gap between single-image VQA and real-world embodied perception. The fully human-curated design adds credibility compared to prior template-based datasets.\n\nQuality: The taxonomy of spatial relations (camera, object, region) is systematic, and the annotation process with reasoning traces and multi-reviewer verification shows rigor. The large-scale evaluation across 37 models is comprehensive and carefully controlled.\n\nClarity: The paper is well organized with clear figures and strong examples of question categories. The error typology (grounding, scene reconstruction, situation transformation, spatial logic) provides insight beyond raw accuracy.\n\nSignificance: MMSI-Bench exposes a real bottleneck in MLLMs’ ability to perform grounded spatial reasoning. The benchmark can drive future work on embodied AI, robotics, and multi-view understanding.", "weaknesses": "The dataset is still modest in scale (1k QA pairs), which limits generalization analysis. It would help to report variability or cross-split reliability. Many questions rely on human interpretation of viewpoint or direction. Some ambiguity might remain even with expert curation, which could affect reproducibility.\n\nThe evaluation metric focuses only on answer accuracy; assessing reasoning trace similarity (e.g., using annotated rationales) could reveal finer-grained improvements. While the benchmark is thorough, the paper lacks concrete recommendations or model design principles derived from the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fjssrnUNr6", "reviewer": "Reviewer_FDLo", "summary": "This paper introduces MMSI-Bench, a new VQA benchmark for evaluating the multi-image spatial intelligence of MLLMs. The dataset consists of 1,000 challenging, multiple-choice questions meticulously crafted by 3D-vision researchers, which are designed to be unanswerable from any single image. Each question is paired with a human-annotated, step-by-step reasoning process and is categorized into one of eleven spatial reasoning tasks. An extensive evaluation of 37 MLLMs reveals a substantial 55-point performance gap between human (97.2%) and SOTA model (41.9%) accuracy. The authors also provide an automated error analysis pipeline, identifying \"overlap-matching and scene-reconstruction\" as the dominant failure mode for current models.", "questions": "1. You demonstrate that zero-shot linguistic and visual prompting fails to provide significant gains. Did you experiment with few-shot in-context learning by providing the full human-annotated reasoning chains as exemplars in the prompt?\n2. The error analysis in Figure 7 is aggregated across the entire dataset. Could you provide a breakdown of the error type distributions for the most challenging categories, specifically \"Multi-Step Reasoning\" and \"Motion (Camera)\"?\n3. The \"Positional Relationship\" category is very broad, covering six distinct sub-types. Do models show significant performance variation across these sub-categories (e.g., is \"Cam-Cam\" more difficult than \"Obj-Obj\")?\n4. What specific guidelines were given to the 3D-vision researchers for creating the \"Multi-Step Reasoning\" questions, and how did you ensure these tasks truly required a sequence of the basic spatial skills?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses a clear and important gap in MLLM evaluation, moving beyond existing benchmarks that often focus on single-image reasoning or use automated templates.\n2. The dataset's fully human-centric curation process, involving 3D-vision experts and 8 diverse, real-world data sources, produces challenging and linguistically varied questions.\n3. The authors provide an extensive evaluation of 37 MLLMs, establishing a robust baseline and highlighting a massive performance gap between SOTA models and humans.\n4. The inclusion of an automated error analysis pipeline, which categorizes failures into intuitive types, provides concrete and actionable directions for future research.\n5. The investigation into prompting techniques, including a novel visual prompting method using feature matching, provides strong evidence that current models have fundamental limitations in this domain.", "weaknesses": "1. The dataset size of 1,000 samples, while acknowledged as a result of costly human curation, is small. This raises questions about the in-depth diversity within each of the 10 sub-categories, which have ~100 samples or fewer on average.\n2. The \"multi-image\" claim feels overstated, as all categories except for \"Multi-Step Reasoning\" are explicitly constrained to using exactly two images.\n3. Key results that strongly support the benchmark's novelty, such as the poor performance of models specifically finetuned on other spatial datasets, are relegated to the appendix.\n4. The methodology for constructing \"Multi-Step Reasoning\" questions is underspecified, lacking detail on how annotators were guided to combine the basic task types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)