SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild?
Authors: Azmine Toushik Wasi
Wahid Faisal
Abdur Rahman
Mahfuz Ahmed Anik
Munem Shahriar
Mohsin Mahmud Topu
Sadia Tasnim Meem
Rahatun Nesa Priti
Sabrina Afroz Mitu
Md. Iqramul Hoque
Shahriyar Zaman Ridoy
Mohammed Eunus Ali
Majd Hawasly
Mohammad Raza
Md Rizwan Parvez
Venue: ICLR 2026 Poster (2026)
Keywords: Spatial reasoning
Vision–language models
Large languge models
Reasoning Models
LLM Evaluation
Spatial Understanding
Primary area: datasets and benchmarks
TL;DR: SpatiaLab presents a comprehensive benchmark for evaluating vision-language models' spatial reasoning capabilities across realistic, diverse scenarios, revealing significant gaps compared to human performance.

Abstract: Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision–language models (VLMs). Prior work largely relied on synthetic or LLM-generated environments with limited task designs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce **_SpatiaLab_**, a comprehensive benchmark for evaluating VLMs’ spatial reasoning in realistic, unconstrained contexts.
**_SpatiaLab_** comprises 1,400 visual question–answer pairs across six major categories: *Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation,* and *3D Geometry*, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 questions, supporting both multiple-choice and open-ended evaluation.
Experiments across diverse state-of-the-art VLMs, including open- and closed-source models, reasoning-focused, and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. In the open-ended setting, all models show a performance drop of around 10–25%, with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry.
By providing a diverse, real-world evaluation framework, **_SpatiaLab_** exposes critical challenges and opportunities for advancing VLMs’ spatial reasoning, offering a benchmark to guide future research toward robust, human-aligned spatial understanding. **_SpatiaLab_** is available at: https://spatialab-reasoning.github.io/.

Published as a conference paper at ICLR 2026
SPATIALAB: CAN VISION-LANGUAGE MODELS PERFORM SPATIAL REASONING IN THE WILD?
Azmine Toushik Wasi Wahid Faisal Abdur Rahman
Mahfuz Ahmed Anik12, Munem Shahriar Mohsin Mahmud Topu
Sadia Tasnim Meem 1.2 Rahatun Nesa Priti Sabrina Afroz Mitul
Md. Iqramul Hoque"2, Shahriyar Zaman Ridoy!4, Mohammed Eunus Alis,
Majd Hawasly Mohammad Raza", Md Rizwan Parvez"
Commpttttionll Intelligence and Operations Laboratory (CIOL)
Shahjalal University of Science and Technology (SUST) 'BRAC University
"North South University (NSU) 'Monash University
*Qatar Computing Research Institute (QCRI)
Correspondence to: azmine ooshikk wasi (maal. com, mparvez@hhbu.eeuiu
ABSTRACT
Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision- languale models (VLMs). Prior work largely relied on synthetic or LLM-ggnnrrted environments with limited task de- signs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce SPATIALAB, comprehensive benchmark for evaluating VLMs' spatial reasoning in realistic, unconstrained contexts. SPATIALAB comprises 1,400 visual question-answer pairs across six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 ques- tions, supporting both multiple-choice and open-ended evaluation. Experiments across diverse state-of-the-arr VLMs, including open- and closed-source models, cusooinn--fcussed and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, Intern VL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. the open-ended setting, all models show performance drop of around 10-25%. with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry. By providing a diverse, real-world eval- uation framework. SPATIALAB exposes critical challenges and opportunities for advancing VLMs' spatial reasoning, offering benchmark to guide future research toward robust, numan-alignen spatial understanding. SPATIALAB available htttps://sptte reasoning github.io io/
INTRODUCTION
Spatial reasoning is a fundamental aspect of human cognition that involves understanding spatial layouts, imagining and manipulating objects, and navigating environments. It supports applications ranging from robotics, computer graphics, autonomous driving and augmented/virtual reality to geospatial analyties, interaction, and education (Huang et al.) 2024bb Qi et al. 2024 2025). For aritificial agents, autonomous systems that perceive and act in dynamic environments, reliable spatial competence requires interpreting geometric relations, depth cues, occlusion patterns, and agent-centric perspectives in scenes that are visually noisy and structurally complex (Bar-Anan al. 2006; Cai et al.) 2025 Trope & Liberman 2010 Ramalho et al. 2018). Recent vision- languuee models have advanced multimodal representation and language grounding, but their spatial judgments
remain fragile in realistic environments (Chen et al. 2025 Kosoy et al. 2025
Pothiraj et al. 2025

---

Published as a conference paper at ICLR 2026
Limitations Existing Benchmarks
Excluation Tspe Mutiple Choice Guestarn
Desigs
Cpenwerded Guestions
EmeSpet Bece
Dota demahl Internet
Nanual Capture
Teak Image Complexity Analysia
Spel 61-MM
Man Osjecte
Van of Ctjects Partally Vistus (Agg)
Mamber Septh Lxpert (Arg)
[Agg]
SpatiaLab
2144
5.28
2 0
Lighing
Complisity
5 Calager 60
Complex
Contreat
25.7%
Patterned
TL365
Midd the Gap
Shadowas
35.79%
Lack of Real Marld Data Compleely
Davinant Spake: Retation
Stuckee
13775
Rellectiv
Scatered
48
Tranelucant
3.68
Algpad
Transparent
2.34%
Dae pmercle) B8A
Reqquinee?
Re-ading sciting ditantts R.iddorrdbraaitad
36.37% Fixw
Figure 1: Overview of SPATIALAB. The benchmark addresses limitations of prior datasets (left), introduces 1,400 visual QA pairs spanning 5 categories and 30 subcategories (center), and enables systematic evaluation through maultppe-chhice and open- tasks. It features diverse task and image complexity, with varied object counts, layers, lighting, textures, relations, and materials (right).
Existing spatial benchmarks, however, are narrow and simplified. Most focus on elementary tasks such as binary spatial relations, low-resolution depth categories, or schematic navigation cues, often
in synthetic or simplified settings that reduce visual clutter and homogenize objects (Du et al. 2024
Jia et al. 2025 |Szymanska et al.] 2024, Shiri et al.) [024]] Kamath et al.) 2023 Fu et al., 2024
Song
ct al. 2024] Yang et al., 2024; Qi et al. 2025; Cai et 2024). These controlled setups lessen perceptual and reasoning demands, leading to apparent saturation while masking failures under distribution shift. Critical challenges such as robust occlusion inference, scale-consistent judgments across viewpoints, and planning under partial observability remain under-sampled (Kosoy et al. [2025] Pothiraj et al., [025 Chen et al., 2025 Cai et al., (025). As a result, models that perform well on existing benchmarks often fail integrated, multi-step inference tasks, limiting safe deployment in real-world environments where spatial errors carry practical consequences.
meaningful evaluation must therefore span the all core axes of spatial reasoning: relative posi- tion, depth and occlusion, orientation, size and scale, navigation, and 3D geometry. Unlike current task-isolated setups, human cognition integrates these dimensions seamlessly under noise and ambi- guity (Driess et al. 2023 Brohan et al., 2023) Zitkovich et al., 2023 Collaboration et al.) [2023] Yuan al. 2024 Qi et al. 2025). Robust benchmarks should therefore test both choice and generation. benchmark against human performance, and draw from real-world scenes that reveal failures masked by simulator tattasets-priccplees that motivate our benchmark.
To address the above needs, we introduce SPATIALAB, a benchmark designed to assess spatial reasoning of vasion-language models in realistic, unconstrained visual contexts, as outlined in Figure 1 SPATIALAB provides balanced suite of visual question-answer items covering broad taxonomy of spatial tasks and supports both multiple-choice and open-ended evaluation formats. The benchmark emphasizes photographic diversity, complex spatial relational queries. and tasks that require integrating depth, perspective, and 3D structure. We evaluate a wide collection of statc-of-the- art VLMs, including open- and closed-source models and reasoning-foeuser focused systems, compare them to human performance, and perform extensive error analysis. Our key contributions are as follows:
1. We introduce SPATIALAB, large-scale benchmark of 1,400 visual question-answw pairs spanning six principal categories and thirty task types in realistic, unconstrained contexts. With at least 200 questions per category and 25 per subcategory, ensures balanced coverage and supports both multiple-choice (SPATIALAB-MCO) and open-ended ( (SPATIALAB- OPEN) evaluation.
2. We perform a comprehensive evaluation of 25+ state-of-the-art vision-e models: open- and closed- reasoning oriented, and spatially specialized reasoning models against human baselines, exposing a consistent and significant gap spatial reasoning.
crasoning-oriented,
3. We deliver in-depth error analyses and diagnostic studies that uncover systematic failure modes in depth perception, navigation, occlusion, and 3D geometry, yielding concrete insights and actionable directions for advancing robust, human-aligned spatial understanding.
2

---

Published as a conference paper at ICLR 2026
Table Comparison of spatial reasoning benchmarks.
Dataatt
E.
TCC.
Vencel
Pial
Al.
Md.
Boe Readine
R.B.P.
Domaiin
Scurce
A.
Sine
Count
DC:
Capp.
Yarr
Type
Depdl
ScanOA
Incksee
Closoerded
Spatial
R.W.
MCQ.BB.
LXMERT
70.10
What's
EMNLP'SSC
Inchace
2.K
3,68
ACL
MDQ
Qwes-VE-Max
40.11
Space Hench
Indecr
211
Mm
Mix
RAGHD Chat
66.30
N
12
Mis
E.D.
T
15KK
L.SC
Mmn
E
NowtPS 24C
MCQ.Bia.
M
L
GPT
$7.83
BLINK-Spriil
14
Mit
RDD.
M
75K
1.88
H
E
BOCY-MC
MIXQ
M
M
GPT-4o
59.01
Spersi-MM
Intenne
Internet
T
2.9K
Mmn
P'SC
Ojen
Hh
GPT-4 Vasion
63.82
RoboSpazial
Indoor
E.D.
T
134
3M
E
CVPR 25C
Open.Bnn
LEO
71.90
SpeisiVVLM
With
E.
Oyen
CIPE-eV
68.00)
VSI Hence
Indoer
M.E.D.
MI
CVPR 25C
Gemini- Pro
OrmiSpanial
Internet
Insornet
Mind Gap
1.88C
Ma
2025
Mis
48,63
VL.M0
Mit
ICUY'TS
MC12
Mix
Gemini-225 Pro
02.10)
SPAZILAE MyCQ
30
Mis
MILM
M
1.2K+
1.5K
H
2005
MCQ
H1
H
Mix
Lm.emVi..5-7288
54,90
SPATIALAB-OPEN
30
Mis
Mi..
M
1.2K+
1.5%
81
VHI
2025
Open
181
Mis.
CiPT-S-mii
40.08
4. We explore multiple strategies to enhance spatial reasoning in VLMs, including supervised fine-tuning (SFT), thain-of-thought (CoT) prompting, CoT with self-reflection, and multi-agent architectures (SPATIOXOLYe) with multi-step reasoning, analyzing their benefits and limitations across multiple-choic and open-ended tasks.
In the multiple-choice setting, leading models achieve roughly 50-55% accuracy, while human annotators exceed 85%. In open-ended generation, model performance drops by another 10-25 percentage points, with the best systems near 40% versus about 65% for humans. Error analysis reveals concentrated failures in occlusion inference, scale-consistency, and multi-step navigational planning. indicating limits in compositional spatial representations and reasoning. Our comprehensive study highlights which interventions provide robust gains, which amplify existing biases, and the critical role of perceptual grounding for generalizable spatial reasoning. By providing tasks, baseline results, and diagnostic analyses, SPATIALAB establishes foundation for progress and reproducible comparison in spatial reasoning evaluation.
2 PRELIMINARIES: SPATIAL REASONING IN VLMS
Visual-spatial reasoning, the ability to perceive, represent, and manipulate spatial relations among objects, plays a central role in human cognition and in enabling embodied agents to navigate and act in the world (Palmer 1999. Chabris et al.) 2006). Formally, we can describe a scene as a set of objects = {ol,.... )n with attributes (e.g.. position, orientation, size) in a world coordinate system W. Visual-spatial reasoning then seeks to learn a mapping f (I, Q) + R, where / is an image (or image sequence), Q is a linguistic query, and R is a structured representation of spatial relations R = {r(o,, | o 0j E O} capturing predicates such as left-of, occludes, supports. For vision-language models (VLMs), learning / is highly challenging because it requires integrating noisy visual perception with compositional linguistic abstraction under embodied constraints such as perspective, gravity, and physical interaction (Dai et al. 2017 Alayrac et al. 2022).
Early approaches to spatial reasoning in VLMs have largely relied on basic spatial relations or narrow task designs (Cheng et al. 2024 Du et al. 2024; Szymanska et al. (024). often situated in synthetic datasets like CLEVR (Johnson al. 2016) or structured question answering benchmarks such as
GQA Hudson & Manning 2019) or puzzle- like simulated setups such as MGT (Stogianmidis et al. 2025) and OmniSpatial (Stogiannidis et al. 2025), as we can notice in Table i While useful for controlled evaluation, these settings oversimplify spatial structure, omit environmental noise, and neglect dynamic factors such as temporal changes or viewpoint shifts. To overcome these limitations, we argue for the need to benchmark spatial reasoning in rich, real-world contexts that capture scene complexity, temporal variation, and multiple perspectives. Such settings expose VLMs to challenges they would encounter in embodied or interactive applications, including roboties, AR/VR, and autonomous navigation (Mu et al. 2023 Collaboration et al. (023).
In SPATIALAB, we organize spatial reasoning into six principal categories, Relative Positioning, Depth& & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 31) Geometry, each linked to a distinct cognitive faculty studied in psychology and spatial cognition (Baddeley 1998 Previe 1998).
Her,. E. Embodied, R.W. + Real World, TCC Tusk Calegories, E.D. Existing Duiaser, A. Amnotatoon T + Template. M Manual, P
Puzolos, Mudality Image.V Video), Rin. Bin. Nm. Nm, Difficulty Cmppl - Complexity (F Eayy - Medians, Hard
Shallow, Low, - Hi, VHIVH Very Hard/Wry High), A.I. - Analysis Level, R.R.P. Rest Raseline Perfommance.
3

---

Published as a conference paper at ICLR 2026
Muttiple Choice VOA
Orientation
Relative Positioning
Question: Which
Question: What object the cantered object?
Question:
the one is the beige bubbke toa near the dumpling backet the person's eloow, shows tet favering lett-handed orientation?
looking
en pink tool, the green tool, amalllest from their kind of items. is true?
bucket
Stabue Human child
I: The one the beige bubble toa green lea beside the person's clbow
Both di Nona
The
Man
Answer:
True
Not meaturable
Falio
visible
Open enndeeededd VQA
3D Geametry
Depth Occlusion
Spatial Navigation
Question: Could the potted plant be placed on one
Question: Which object the
Quection: The ambulance
image has entirely reflection
orange-yaliow car) direct towards the top side cars trua? Answer: False
thais??
image?
Answer:
everpass.
perfectly.
Figure 2: Representative examples from six categories in open-ended and MCQ Tasks.
and As shown in Figure and Appendix B each category is further decomposed into five subcategories, producing a taxonomy of thirty task types that isolate fine-grained reasoning skills, such as directional alignment, occlusion inference, stability prediction. This classification scheme is motivated by the need to capture the full spectrum of visual-spatial reasoning challenges. By systematically breaking down complex reasoning into measurable and testable units,
evaluate not only whether models "solve" spatial tasks but also which specific dimensions remain fragile or underdeveloped (Gardner
2011 Huang et al. 2022
Gupta& & Kembhavi 2023 Yang al. 2023).
Moreover, we adopt two complementary evaluation formats: multiple-choice and open-ended question answering. The multiple-choice format probes recognition and fine discrimination among distractors, while the open-ended format measures generative reasoning and compositional description (Driess) al. 2023 Liu al., 2025 Zheng al. 2022). Together, these formats reflect real-world reasoning demands and expose different strengths and weaknesses in model capabilities. By combining principled taxonomy. diverse real-world imagery. and dual evaluation formats, SPATIALAB provides structured framework for systematic henchmarking. This setup enables fine-grained model compar- ison, alignment with human-level performance, and deeper insights into the development of robust spatial cognition in artificial agents.
SPATIALAB BENCHMARK
Spatial reasoning remains persistent challenge for lision-langure models (VLMs), especially in naturalistic conditions where occlusion, perspective distortion, and noisy backgrounds co-occur, To address this, we present SPATIALAT benchmark designed to systematically evaluate VLMs' capacity for complex spatial cognition under realistic visual conditions. The benchmark comprises 1,400 visual question-ansee answer pairs organized into six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each further divided into five subcategories, yielding 30 distinct task types. Every subcategory includes at least 25 QA pairs, while each top-level category contributes a minimum of 200, ensuring balanced coverage across the taxonomy. SPATIALAB supports both multiple-chol assessment of discriminative accuracy well generative reasoning and explanatory competence, thus aligning with broader multimodal evaluation practices while emphasizing spatial-specific reason- ing. High visual diversity, fine-graine annotation, and rigorous quality control make SPATIALAB reliable and scalable testbed, and Figure Dillustrates representative examples from the six categories, showcasing both open-ended and multiple-choice formats that capture the richness and difficulty of real-world spatial reasoning tasks.
and open-ended evaluation, enabling
3.1 BENCHMARK CONSTRUCTION
Image Collection. The image corpus was curated using three complementary strategies: (i) automated large-scale web crawling, (ii) targeted online retrieval of scene-specific images, and (iii) manual snapshots collected in diverse indoor and outdoor environments (Figure 3. This multi-source pipeline ensures diversity across object categories, environmental conditions, and spatial configurations. To explicitly capture natural variability, images were profiled along six meta-dimensions: lighting condition (high contrast, low contrast, shadows, and reflective settings), texture complexity (uniform, patterned, and complex), edge complexity (sharp and smooth boundaries), dominant spatial relation
4

---

Published as a conference paper at ICLR 2026
Image Collection
Annotation
Review & Quality Control
WWW
Web Crawling
Online Search
Manual Snaps
Annotator Annotation & Ques Ans Training
Ques-Ans Review
and Verification Annotation
Processing
Pairs
with different objects situations concerning info
Collection ennotated spatial visual Duestion-Answer pairs
Collection of gold annotated spatial visual Question Answer pairs
Figure 3: Data creation pipeline for SPATIALAB, Images are collected via web crawling, targeted search, and manual snapshots, followed by structured annotation of spatial question-answer pairs.
(stacked, scattered, aligned), material type (transparent, translucent, opaque, reflective), and gravity constraints (normal, floating, and unconstrained), Each dimension was systematically represented in the final corpus, ensuring that the benchmark reflects a broad spectrum of real-world, in-the-wil wild scenarios rather than simplified synthetic setups. Complexity analysis of the corpus reveals that average images contain 21,48 objects, of which 1.88 are partially visible, distributed across 3.23 distinct depth layers. Spatial reference chains (e.g.. "object is left of B, which is behind C") contain 2.07 links, indicating the presence of multi-step relational reasoning. Also, 16.71% of the questions require mental rotation; 56.07% require object specificity (specific color. pattern, or other details). Such statistics posits that SPATIA captures cluttered, layered, and relationally rich scenes, which are characteristic of real-world spatial reasoning, underrepresented in synthetic datasets.
Annotation. Annotation proceeded in three phases. First, annotators underwent targeted training (derails are available Appendix C.2) to ensure consistency in interpreting spatial categories and subcategories. Second, each image was paired with one or more spatial QA tasks covering range of reasoning types such as proximity gradients, alignment patterns, occlusion inference, and stability prediction. Questions were designed to balance perceptual grounding with higher-order inference (e.g., predicting stability under gravity constraints). Third, QA pairs were encoded both multiple- choice (04 options) and open-ended formats, ensuring comparability across evaluation paradigms. In total, it produces around 1,500 spatial visual QA pairs, across six categories and 30 subcategories.
Review and Quality Control. To ensure reliability, annotation outputs underwent three-tier review and validation pipeline. In the first stage, QA items were checked for semantic validity and alignment with the designated subcategory. In the second stage, independent annotators verified correctness of both the question phrasing and the designated answer, focusing on eliminating ambiguity and enforcing task clarity. In the third stage, a gold-standard annotation round was conducted, establishing a set of high-confidence QA pairs for evaluation. All metadata was removed from the images, and the dataset was carefully curated to avoid copyright concerns, though minor oversights may still occur. This procedure yielded final benchmark of 1,400 validated QA items.
4 EXPERIMENTS AND EVALUATION
Models. We categorize the evaluated models into four groups. Proprietary models include GPT-4o-mini (OpenAl 2024), GPn---nano, Gemini-2-Flash and Gemini-2.5- (Team 2025a), and Claude 3.3 Haiku (Anthropic) 2024). Open-source models comprise Intern-Vi (IB, 28,
4B) (Zhu et al.) 2025), Qwen- VL2.3 (3B, 7B, 32B, 72B Instruct) (Bai et al. (025), GLM 4.5V.
3.06B and GLM-4.1 V-9B-Thinking (Team al. 2025). Gemma-3 (4B, 27B) (Gemma Team Google DeepMind, 2025), Llama-3.2 (IIB, 90B) Vision-Instruuet (Grattafiori et al.) 2024). and Step-3-321B-Moo StepFunn 2025). Reasoning models include 04-mini (OpenAl) 2025), Gemini-
2-Flash and Gemini-2.5. 2.5-Fass thinking variants (Team] 2025a), and Kimi-VL-AX Thinking-
(Team) (0255). Spatial reasoning specialists comprise SpaceOm, SpaceThinker--Qwee2..c (Chen et al., 2024a). For GLM-445V-106BBMM. GLM- 4.1 --hn--Thin,,gg, and Step-3-321B-M6 we only performed open-ended evaluation, as these models are not instruction-tuned and were trained primarily on open-ended tasks, resulting in excessive refusals during MCQ evaluation. For ease of comparison our results tables, we
3B, and SpaceQwen2.5-vi..
use color coding: proprietary open-source up to 7B
larger than 7B proprietary reasoning.
open-source reasoning spatial specialists and human baseline
5

---

Published as a conference paper at ICLR 2026
Table 2: Multiple Choice Evaluation Accuracy (%)(T) on SPATIALAB-MCO by Question Categories.
Dep. (#259)
Orientation (#202)
Relat. Posit. (4212)
Scale (#252)
Spati. Navig
Model
(4238)
(#237)
(#1400)
Proprictary Models
GPi-do-mini
47.06
39.00
47.03
47.17
49.60
49.79
46.50
GPT-5-minni
48.74
54.83
60.40
62.74
44.84
56.54
54.29
Gemini-2.00
55.21
58.02
46.84
52.50
Gemini-2.5 2.5--sshh
44.96
48.26
48.02
56.13
42.46
51.05
48.29
Gemini-2.55 P.5-Po
50.19
43.65
52.32
50.07
Claude 3.5 Haiku
42.44
42.08
46.53
46.23
35.71
45,99
42.93
Mistral Medium
49.81
47.52
61.79
41.77
47.93
Open-Source Models
Im
33.61
32.43
23.27
37.26
31.75
30.80
31,64
Int5mVL33,522
34.03
31,66
31.68
40.57
32.54
32.49
33,71
Qwen- VL2.5-3BBBBB-
41.18
35,52
46.04
40.09
47.22
39.24
41.43
IntemmVL33.-4BB
42.86
42.86
42.08
54.72
36.51
42.19
43.29
Gemma-3-4B-it
Qwen-V -L..2...--7B-Isstrreet
42.86
37,84
42.57
46.23
42,06
35.44
41,00
Mama Vi.
Gemma- 3-2BB it
43.28
40.15
48.02
$4.25
48.02
47.26
46.57
Qwen V1.2.5-22R-
IntemmVL3.5-722
50,00
57.14
53.47
66.04
49.21
54.85
$4.93
Qwen- 1tt.--LL..-777-mmttteee
54.25
48.86
LIama 20BB Vision Instruct
46.22
52.12
50.50
$8.96
46.83
48.52
50.36
Reasoning
63-mini
50.00
54,83
51.98
61.32
38.89
50.21
50.93
Gemini-2 2. 2-Flash-Thinking
37.82
41.31
41.58
45.75
50.40
43.04
43.36
Gemüni 2.5-Flasb Thinking
Kimi- VL-A3B-1 VL-A-B-Thinking-- Thinking-2506
42,86
41.31
40.59
$1.42
39.68
41.35
42.71
Spatial Roasoning Models
SpaceOm
42.44
38.61
48.02
37.74
42.86
39.24
41.36
SpaceThinker. Owen2.5V1L.3B
SpaceQwen2. 5. VL. Instract
31.51
35.14
37.62
37.74
50.79
47.26
40.14
91.51
Evaluation Process and Metrics. We measure accuracy as the primary metric for both MCQ and open- ended tasks. For MCQ evaluation, use direct prompting, where models return the option number corresponding to their selected answer. Responses are automatically checked against the correct answer. Detailed prompts and details provided in Appendix [.]] For open-ended evaluation, models are prompted to generate free-form answers, which are then assessed using large language model judge. Prompts and details are available Appendix D.2] To evaluate the performance of LLM Judge with humans, we performed LLM- M-Mumann agreement analysis too, as achieves substantial agreement with human annotators, with a Cohen's 's kappa of 0.738 against the majority vote and 1.68-00.795 against individual annotators. Raw accuracy against the majority vote is 0.880, while Fleiss' kappa among human annotators is 0.774, indicating strong reliability. For additional details on experiment setup, prompts, and evaluation procedures, see Appendix D
detailed in Appendix D.2 demonsstetssttrtees that the LLM judge (Gemini-2.5-Flaas
Improving Visual Reasoning Capabilities. To enhance spatial reasoning our diverse benchmark, we explore reasoning strategies including inherent reasoning in the VLMs (Appendix H.l Chain-of- Thoughts (CoT) (Appendix H.2 CoT with self-reflection (Appendix HK supervised fine-tuning (SFT) (Appendix H.4). and multi-agent systems (Appendix H.5). For SFT, specifically, Qwen- VL..5- 3B-Instruct fine-tuned on 40% of the dataset with stratified sampling. improving generalization across relational and geometric tasks. Evaluation on the remaining 60% showed reduced emrors in navigation and orientation, with additive gains when integrating CoT and SFT. However, agentic reasoning revealed that while orientation benefits substantially, other categories stagnate or degrade, underscoring that deeper perceptual and spatial grounding is required beyond multi-step reasoning.
FINDINGS AND ANALYSIS
OVERALL RESULTS
SPATIALAB-MCO. Table demonstrates model performances in multiple choice evaluation on SPATIALAB-MES We observe substantial variation in performance across models, tasks, and families: overall model accuracies span roughly 30-55% (random choice 25%). while the human baseline is 87.57%, indicating large headroom. Vnte5nV..3.5-722.. open source model, is the best

---

Published as a conference paper at ICLR 2026
Table Open-ended Evaluation Accuracy (%)(1) on SPATIALAB-OPEN by Question Categories.
Dep. (#259)
Orientation (#202)
Relat. Posit. (#212)
Scale (#252)
Spati. Navig (#237)
Overall
Model
(4238)
(#1400)
Propristary Models
GPT-4o-mini
23.53 .3
16,00
23.27
30.66
17.86
21.94
26.00
GPT-5-mini
Gemini 2.0 2.0.Faahh
31.93
24.32
27.23
31.13
26.19
24.47
27.43
Gemini-2.5 F.a.aah
Gemini 2.5. Pro
37.14
45.45
36.36
37.14
23.91
24.44
33.61
Claude
Mistral Medium 3.1
25.21
19.31
21.78
29.25
15.08
16.88
21.00
Open-Sowrce Models
InternV1.3.511
05.88
09.65
9.90
13.68
09.13
10.13
09.64
VB3.5223
14.50
Qwen VI V122.5-3B.Instrnc
15.55
8.49
15.35
10.85
18.25
9.28
12.91
In1.m\1.3.5.4B
17.76
-enma-33-B--1
20.17
13.13
14.85
23.58
15.08
19.83
17.64
VL.2.5.7B.Bh
1amma-3.2-118--
Vision-Instruct
16.81
16.99
22.28
25. 01
13.49
18.57
18.57
Gemma-3-27B-it
16.22
34,43
22.62
21.94
23.43
Qwen- Instruct
16.39
09.65
14.85
16.98
09.92
13.50
13.36
1mcmm\L33,57728
20.46
31.60
26.16
23.36
Qwen-V1.2.5-72
26.89
20.85
25.25
30. 66
24.60
20.68
24.64
22.69
23.17
28.30
27,00
GLM-4.5V.106BBMM
31.09
20.46
25.25
26.42
24.21
24.47
25.21
Reasoning Models
03-mini
39.08
30.12
29.70
39.15
41.27
30.38
35.00
04-mini-mediun
31.09
27.41
31.19
34.45
29.37
29.54
30,36
37.14
45.45
36.36
37.14
21.74
22.22
32.77
13:45
14,29
11.88
17.92
12.70
18.57
14.79
GL.M4, 1V-9B-TVToiii
21.43
19.31
26.73
24.53
19.84
22.78
22.21
Step 3:32 MoE
32.35
18.53
27.72
27.36
29.76
27.85
27.14
Sparial Reasoning Models
SpaceOm
12.61
06.95
15.84
11.79
18.65
12.24
12.93
Sieiikkrr-Qee2.5VELL SVL-3B
13.45
09.27
17.82
10.38
19.44
13.36
SpaceQwen2. 5.VI.-3 3R-Innttmet
12.61
03.86
13.86
11.90
11.39
10.36
Baseline
62.87
performing model, with 54.93% accuracy. Reasoning- and instruction-tunel models (e.g., 04-mini: 53.21%, %, Semmini2.5--Flah---hiikee source models (e.g., Intern VL3.5-72B: 54.93%, 1lama-3.2-90B: 50.36%) rank near the top, but model scale alone not determinative: for example, -.maa-3.2-1BB attains only 30.50% despite its substantial size. Category-wise Orientation and 3D Geometry are where many strong models excel (several models exceed 60% on orientation or 3D geometry), whereas Spatial Navigation, Depth Ocelusion, and Size & Scale are more variable and generally harder across the board, Notably, the spatially specialized models (SpaceOm, SpaceThinker SpaceQwen) achieve only mid-40s low-40s overall, so specialization does not automatically translate to higher MCQ accuracy across diverse question set. Depth & Occlusion
52.93%, GPT-5-mini: 54.29%) and very large open-
Size Scale show mixed results, some architectures handle occlusion relatively well while others fail, pointing to differing inductive biases or training data. These patterns imply that instruction tuning and reasoning capability particularly help geometric and orientation reasoning, while multi-step grounding tasks like navigation remain open weakness.
SPATIALAB-OPER Open-ended evaluation scores presented in Table are substantially lower than MCQ results and show a wide spread: overall accuracy ranges from about 9.6% (In.ernVL3.5. 1B) up to 40.9% (GPT-5-mini). while the human baseline sits at 64.9%, indicating a large gap to close. Proprietary models generally lead the leaderboard 40.93%, GPT-4o-mini 26.00%, Gemini-22.5Fllas 30.93%), but reasoning-tuned variants stand out in --a--aah--hii Mrinshhhhiiingg (32.77%) perform noticeably better than many baseline open-source counterparts, suggesting instructioon/CoT-sii improves generative outputs. Most open-source small models cluster at the bottom (many in the teens; Intern VL3355-B = 9.64%, Qwen-VLL..533B 12.93%), whereas some very large open models improve to the low-20s (e.g., JntemmVL3.5-72B 23.36%, Qwen-VL2.5-722 = 24.64%), showing that scale alone yields only modest gains. Spatial-speciale models (SpaceOm, SpaceThinker, SpaceQwen) attain only 10-13% overall, implying that architecture or task specialization does not automatically translate to robust open-ended generation across diverse question set. Across subtasks we observe relatively better performance on orientation and certain relative -osiiiiin items for top models (e.g., GPT-5-mini Rel.Pos 49.53%, Ori 37.13%). while depth & occlusion, size
open-ended tasks (e.g., GPT-5-mini
particular, the o4- 04-mini (37.86%) and Gemini-2.5 2.5--lhsh
tuning substantially

---

Published as a conference paper at ICLR 2026
& scale, and especially spatial navigation remain consistently hard (many models score <30% on these subtasks). This pattern reinforces the earlier finding that multi-step grounding and perceptual reasoning are the bottlenecks for open generative answers.
5.2 PERFORMANCE DROP IN OPEN-ENDED EVALUATION COMPARED TO MCQ
performance gap in SPATIALAB is 23.0% (a 5.5%), with subtask means ranging from 22.89% (spatial navigation) to 24.57% (3D geometry). Specialist spatial reasoning models exhibit the largest gaps (about 27%), particularly navigation (up to 36.68%) and orientation (34.44%), while reasoning-oriented models achieve smaller gaps (around 19%) and lower subtask variance, reflecting the stabilizing effect of instruction-tuning and CoT decoding. Correlations confirm that spatial navigation dominates overall disparity (Pearson r = 0.99), with orientation (r = 0.83) and 3D geometry (r = 0.79) also contributing, Negative or near-zero gaps (e.g., Llama-3.2- 1B: -1.98 in depth & occlusion; 04-mini: -3.18 in relative position) indicate that poorly designed MCQ distractors can misrepresent true competence. We hypothesize that format sensitivity arises from MCQ's structural advantage, specialization bias toward categorical tasks, sequential reasoning challenges in spatio-navigation, and calibration differences in generative output. These findings suggest that MCQ alone can overestimate practical spatial reasoning ability. We recommend complementing MCQ with open- open-ended evaluations, auditing distractors, employing stepwise generation instruction-tuning and reporting per-subtask diagnostics to capture true model performance and reduce format-dependent artifacts. More details are in Appendix G and Table T8
Across 25 models, the average MCQ-+open-eme
spatial
5.3 ERROR ANALYSIS
SPATIALAB-MCH Our analysis (Appendix E shows that VLMs achieve selective peaks but lack holistic spatial competence. Closed-soures models reach the highest scores (c.g., 85.7 % in Stacking Orientation), yet collapse to near-chance in Relative Size Comparison. Open-source scaling improves ceilings (80.56% in Corner/Angle Positioning) but does not prevent catastrophic failures (2.0% in Object Rotation). as in occlusion inference or navigation, but remain capped below 55% in many physical abstraction tasks (Gravity Effects, Stability Prediction). Error distributions further reveal systematic weaknesses in embodied reasoning (Tool Handedness). recursive relational chaining (Pathway Existence), and non-local cues (Reflective Surfaces), while stronger results in Obstacle Avoidance suggest shortcut exploitation rather than robust planning. Taken together, these patterns confirm that current VL.Ms rely heavily on surface-level correlations and lack stable encodings of orientation, physics, and compositional logic. Progress will likely require both richer training distributions (e.g., embodied and physics-driven data) and architectural mechanisms for geometric and reference-framt grounding.
-aaboonnnn-uggeneeee and spatially specialized models add localized gains, such
SPATIALAB-OPES OPEN. Error analysis of SPATIALAB-OPES (Appendix F) reveals that even top- performing closed-source models (e.g., GPT-5-mini: 14% in Directional Relations) collapse on tasks like Proximity Gradients, indicating that dominance remains task-specific. Large open- models achieve strong results in Depth & Occlusion (c.g.. 60.0%) but still fall to 9.3% on Proximity, showing that scale alone cannot ensure robustness. Smaller models exhibit isolated strengths (e.g., 26.0% in Relative Size Comparison) but often break down completely (e.g., 0.0% in Berweenness), reflecting the limitations of compact architectures. Reasoning-tune -unee systems show promise, Gemini- 2.5-Flash-Thhin Thinking reaches 75.0% in Tool Handedness, yet similar models fall to single digits, revealing fragile integration. Specialized spatial models perform worst overall, rarely exceeding 20% and frequently failing on core relational tasks. Failure patterns highlight systemic weaknesses in occlusion handling, orientation, and multi-step relational chaining, while partial success cues suggests reliance on superficial correlations. These results point to three root causes: missing representations for spatial analogies, architectural limits tied to scale and training diversity, and brittle reasoning pipelines underscoring the requirement for unified geometric encodings, physics-aware reasoning, and embodied data to move from isolated peaks toward consistent spatial competence.
Qualitative Error Analysis. To better understand model weaknesses, we conducted qualitative error analysis (see Appendix [ and found that failures cluster into a small number of recurring classes rather than being random noise. Common errors include spatial mislocalization, where models confuse referents in crowded scenes; perspective and scale mistakes, where reliance on object- size priors overrides image-based cues; and occlusion ordering failures, especially with thin or
8

---

Published as a conference paper at ICLR 2026
40
Spattblab MCQ
Apenutatt-ppen
2229
Category
MCQ
Open-enied eeneenneed
Accurscy (N) Bang
23
Before
After
Gain
Before
After
Gain
3D Geumetry
30.07
37.76
7.69
20.28
23.78
3.50
15
23
Depch & Occlusion
23.87
32.90
9.03
43.23
45.81
2.58
20
Oricatation
23.33
36.67
13.33
39.67
42.15
2.48
Relative Positioning
26.19
27.78
1.59
51.18
50.39
-0.79
13
Scale
-3.95
10
Spatial Navigation
22.38
38:46
16.08
36.62
39.44
2.82
Grand Total
25.63
36.59
10.97
34.40
35.48
1.07
1.0
2.0
Epoch
(b) Performance comparison (Before vs After) across cate- gories for MCQ and Open-ended formats.
(a) SFT performance trends over epochs.
Figure 4: SFT training results: (a) learning trends over epochs, and (b) final accuracy values.
partially hidden structures. We also observe attribute confusion, such as mixing perceptual properties with functional ones, and open-ended rationalizations that generate fluent but visually ungrounded narratives. These errors are consistent across architectures and prompting styles, suggesting structural biases in current VLMs rather than fataset-speciffe noise. Importantly, failure rates spike when multiple cues must be fused, such as combining depth ordering with relative size, and confidence calibration is especially poor in open-ended settings. Our diagnostic protocols confirm that models often ignore minimal but decisive visual features, rely on brittle heuristics, and fail to update when counterfactual edits are applied. The root causes point to insufficient object-centric binding, lack of geometric supervision, and training objectives that reward plausibility over grounding. Together, these findings highlight that current VLMs achieve strong coarse perception but struggle with multi- cue integration and grounded reasoning, underscoring the need for geometry-awan supervision, multi-scale feature retention, and verification pipelines.
5.4 PERFORMANCE IMPROVEMENT APPROACHES
Inherent Reasoning Mechanisms. We notice that reasoning-enabled models consistently outperform their baselines in both MCQ and open-ended formats, with the largest gains in relational and orientation tasks (c.g., +13.1% 1% in Relative Positioning for MCQ). While reasoning helps stabilize open-ended performance, improvements remain uneven across categories (notably declining in Size & Scale), underscoring that reasoning modules boost logical consistency but do not fully solve grounding or scale sensitivity. Detailed per-category comparisons are provided in Appendix H..
Chain-of-Though (CoT) Prompting. Across models, CoT prompting provides little benefit and often reduces accuracy (Table with orientation being the only category showing consistent gains. This indicates that while step-by-step reasoning helps with directional alignment, it fails in tasks requiring robust perceptual grounding such as depth, scale, and spatial navigation. Unlike in textual reasoning, CoT here tends to amplify flawed priors rather than correct them, underscoring that solving SPATIALAB requires perceptual understanding beyond logical chaining (details in Appendix H.2))
Chain-of Thouught (CoT) with Self- Reff-Refeectin. Adding self-reflection to CoT prompting, where models are explicitly prompted to review their previous reasoning stepp and correct mistakes (see Figure X in Appendix for prompt), yields modest benefits in multiple-choiee settings, especially for geometry and depth. However, this fails to generalize to open-ended tasks (Appendix (H.3). This confirms that reflective reasoning cannot compensate for missing perceptual grounding, highlighting a core limitation of current VLMs.
Supervised Fine-Tuning (SFT). As shown in Figure 4. supervised fine-tuning (SFT), conducted on Qwen2 5-VL-35 3B- Inst using a stratified 40% split of SPATIALAB (~560 samples), consistently improves MCQ accuracy across all spatial reasoning categories. However, it offers little, and sometimes negative, transfer to open-ended tasks. This divergence suggests overfitting to task-specific answer distributions, reinforeing concerns that SFT often teaches to the test rather than fostering generalizable reasoning (Wang et al., 2025). The sharp drop in open-ended performance points to biased internal representations, tuned for categorical discrimination but unstable in generative settings. A likely cause is catastrophic forgetting of linguistic priors during fine-tuning (Huang et al.) 2024a), compounded by objectives that favor discrete-choiee alignment over spatial reasoning in language. Overall, while SFT enhances performance in structured formats, it risks creating an algorithmic straightjacket, boosting accuracy in constrained tasks while stalling progress in naturalistic spatial reasoning. More details are available in Appendix H.4

---

Published as a conference paper at ICLR 2026
AI Agents for Spatial Reasoning (SPATIOXOLVER). To explore agentic capabilities, we employ SPATIOXOLVER, a multi-agent system adapted and extended from Xolver (Hosain et al.) 2025) designed to perform structured spatial reasoning images. This framework decomposes complex visual reasoning into specialized sub-tasks managed by dedicated agents, including modules for object segmentation, attribute extraction, spatial relation mapping, and transformation tracking. By consolidating these oulputs into a unified structured representation, the system attempts to propagate low-Ievel perceptual cues through higher-order relational inferences. We implemented the pipeline using Gemini-2, 5- Fashash with low temperature to ensure deterministic outputs during the multi- stage refinement process. Experimental results demonstrate substantial gains in the Orientation category, improving by +8.00% in MCQ and an impressive +36.00% in open-ended evaluation. However, this success is not universal; categories such as Depth & Occlusion and Spatial Navigation suffered severe declines of -24.00% and -12.00% in open evaluation, respectively. These drops indicate that when inherent perceptual criors-such as depth or occlusion cucs are weak, multi-step reasoning tends to reinforce misconceptions rather than correct them. Ultimately, these findings suggest that while agentic workflows enhance tasks reducible to sequential alignment, they cannot compensate for lack of fundamental perceptual grounding in complex 3D environments. More details are available in Appendix H.5
6 DISCUSSION
Spatial reasoning is central to embodied intelligence, yet most benchmarks rely on synthetic scenes, templated questions, or narrow task scopes that overlook real-world visual variability. We introduce SPATIALAB, a benchmark of 1,400 visual question-answe pairs covering six major spatial categories and 30 subcategories, including 3D geometry, occlusion, and navigation. The benchmark uses both Multiple-Choice and Open-Ended formats to separate option-based pattern use from genuine generative reasoning. Its quality is supported by a multi-stage review process and complexity checks covering lighting, texture, and spatial diversity. Evaluations of more than 25 contemporary models and human baselines show that SPATIALAB functions as a precise diagnostic tool for identifying weaknesses in current vision-language systems.
Detailed error analysis uncovers systematic failures in handling transparency, multi-step navigation, and occlusion, indicating that models often rely on surface-level texture cues rather than internalized 3D geometric representations. While interventions such as Supervised Fine- Fine-Tuning (SFT) yielded clear gains in structured MCQ tasks, they simultancously triggered instability in open-ended reasoning, suggesting a "catastrophic forgetting" of linguistic priors when optimizing for discrete answers. Furthermore, reasoning- based approaches like Chain-of-Thoon Thought proved effective for sequential tasks like Orientation but failed to correct fundamental perceptual errors in Depth and Scale, often amplifying hallucinations instead of resolving them. Agentic decompositions showed similar trade- offs, improving specific alignment tasks while degrading performance in holistic scene understanding contexts. These divergent behaviors confirm that scaling laws and reasoning scaffolds alone are insufficient; achieving human-level spatial intelligence will require architectural innovations that explicitly integrate geometric grounding and thysics-aware representations.
7 CONCLUDING REMARKS
Spatial reasoning remains core bottleneck for VLMs, particularly in visually complex, real-world settings. LAB introduces 1,400 diverse QA pairs across six spatial categories and thirty sub- categories, enabling fine-grained evaluation via both MCQ and open-ended formats. Despite applying several performance-inpee improvvng methods, our experiments reveal substantial gaps between state-of- the-art models and human performance, with open-ended generation showing an additional 10-25% drop relative to multiple-choice accuracy. Error analysis reveals persistent failures in depth, occlusion, 3D geometry, and multi-step navigation, reflecting reliance on superficial cues over grounded spatial representations. SPATIALAB exposes these limitations and provides a structured testbed for diagnos- ing spatial competence. Our benchmark enables holistic evaluation and provides actionable insights for developing VLMs with human human-aligned spatial understanding. Future progress will require richer training signals (c.g., embodied interaction, contioocondiiooned feedback) and architectures capable of geometric grounding, relational chaining, and consistent multi-scale reasoning.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have taken multiple steps to ensure ethical compliance throughout the development of this work. All metadata associated with the images was removed protect privacy, and any personally identifi- able features (such as human faces or license plates) were minimized or excluded wherever possible. The dataset was carefully curated to avoid copyright concerns, with diligent attention paid to sourcing and licensing, though minor oversights may occur despite best efforts. The benchmark construction process was designed with transparency and fairness in mind, ensuring that no demographic groups were singled out negatively represented. We adhered to the ICLR Code of Ethics and confirm that no human subjects or sensitive personal data were involved.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide detailed description of the dataset construction process, annotation pipeline, and evaluation protocol in the main text and appendix. All filtering, quality- control, and validation steps explicitly documented, with clear definitions of task categories and annotation criteria. Hyperparameters, model configurations, and training procedures are reported in the appendix support consistent replication of our results. SPATIALAB is available at: https //spatialab - wawssoninnngithuu..... github.ioo..
ACKNOWLEDGMENT
We sincerely thank Mahir Absar Khan and Enjamamul Haque Eram for their assistance with data collection. We also acknowledge the Qatar Computing Research Institute (QCRI) for providing computing resources and technical support.
REFERENCES
Jean- Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc. Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Anthropie. Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet, October 2024.
cwm/m/ cdddd0 Bace099983366444 67///7////// 004466//rrr/naaa Accessed: 2025-09-11.
Claude Model October Aaa pdf
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19139, 2022.
Alan Baddeley. Working memory. Comptes Rendus l'Académie des Sciences-Series Ill-Sciences de la Vie, 321(2-3):167-173 1998.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL org abs///////// / 2502 13923
bittps://arat
Yoav Bar-Anan, Nira Liberman, and Yaacov Trope. The association between psychological distance and construal level: evidence from an implicit association test. Journal of experimental psychology: General, 135(4):609, 2006.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishhan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian
11

---

Published as a conference paper at ICLR 2026
Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-worth control at scale. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.
Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao, Spatialbot: Precise spatial understanding with vision language models. CoRR, 1et/2400.136442. 2024.
Zhongang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wanqi Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng. Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, and Lei Yang. Has gpt-5 achieved spatial intelligence? an empirical study, 2025. URL httph:tt///// /ttpss///rxvvvoo////
abs/2508 13142
Christopher Chabris, Thomas Jerde, Anita W Woolley, Margaret E. Gerbasi, Jonathon Schuldt, Sean L. Bennett, J Richard Hackman, and Stephen M Kosslyn. Spatial and object visualization cognitive styles: Validation studies in 3800 individuals. Group brain technical report, 2(1-20):2, 2006.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12100 2024a. URL /ttpttttt////axx..oorggg abs/2401. 12168
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 14465. IEEE, 2024b.
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Nicbles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vims? an attention mechanism perspective on focus areas. CORR, abs/2503.01773, 2025.
An-Chieh Cheng. Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In Amir Globersons. Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Open X. Emmmboddiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Raj, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-
Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang, Haochen Shi, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x-embodiment: Robotic learning datasets and RT-X models. CORR, abs/2310 10.22330008664 2023.
Lee J. Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297-334, September 1951. ISSN 1860-0980. doi: 10.1007/6002100555. URL http:///x..x. doi. org/ 10..007/8F0231DS
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In 2017 IEEE Conference
12

---

Published as a conference paper at ICLR 2026
Computer Viston and Pattem Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pp. 1433-2443. IEEE Computer Society, 2017.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In ICML, 2023.
Chris Drummond. Replicability is not reproducibility: nor good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, volume 1, Montreal, Canada, 2009. National Research Council of Canada.
Chris Drummond and Nathalie Japkowicz. Warning: statistical benchmarking is addictive. kicking the habit in machine learning. Journal of Experimental and Theoretical Artificial Intelligence, 22(1):67-80, March 2010. ISSN 362-3379 doi: 10. 1000000092288330000010000 7/dx. doi org/ 10. 1080
URL htthtp
Mengfei Du, Binhao Wu, Zejum Li, Xuanjing Huang, and Zhongyu Wei. Embspatial- Bench- marking spatial understanding for embodied tasks with large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 Short Papers, Bangkok, Thailand, August 11-16, 2024, pp. 346-355, Association for Computational Linguistics, 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXIII, volume 15081 of Lecture Notes in Computer Science, pp. 148-166. Springer, 2024.
Howard E Gardner. Frames of mind: The theory of multiple intelligences. Basic books, 2011.
Gemma Team, Google DeepMind. Gemma 3 technical report, 2025. URL httttt:////ttrree googleapis. 09-11.
a Seeama/Gemmren 3e RRportt. pdf
Accessed: 2025-
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu. Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, J Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne,
13

---

Published as a conference paper at ICLR 2026
Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguer, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaclle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing I Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Elconora Presani. Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halper, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanann Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Tufanov, Ilias Leontiadis, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhic. Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman,
14

---

Published as a conference paper at ICLR 2026
Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. BRRpbttp:://aeee http://xxwv..craaa/////aaa /tw.a///2247.220 21783
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.
Charles Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathamiel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Marchan, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7785)))75537736666 September 2020. doi: URL https: / http://o....gg/////////// 1038
020- 2649-2
Md Tanzib Hosain, Salman Rahman, Md Kishor Morol, and Md Rizwan Parvez. Xolver: Multi-agent reasoning with holistic experience learning just like an olympiad team, 2025. URL https
//arxiv org/ /t..////2660222 14234
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https
//arxiv.orq////oo.oo wttp:r/gaabaaa..aaaaeee/s/ 09685
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1428, Bangkok, Thailand, August 2024a. Association for Computational I. Linguistics. doi: 10. 1865//w1/0224aclllongg
URL httpt::t:////ttttm acc aaaaaatclantooooc ogy.ooy..oooo.o............ wrw./org//2444 -allacllllnng 777
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CoRL,, 2024b.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700-6709. Computer Vision Foundation IEEE, 2019.
Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models, 2025. URL
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URL https::::://arr wat..org/aaaa abs 110..///111.1101000111111
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 9161-9117 Association for Computational Linguisties, 2023.
15

---

Published as a conference paper at ICLR 2026
Terry K. Koo and Mae Y. Li. A guideline of selecting and reporting intraclass comrllation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2):155-663, June 2016. ISSN jem
1556-3707. doi: 10. 10.1016f..cm.22166222.111 URL http:/ http://...d..r/11..100000 201 62.00.0.2
Eliza Kosoy, Annya Dahmani, Andrew K. Lampinen, Iulia M. Comsa, Soojin Jeong, Ishita Dasgupta, and Kelsey Allen. Decoupling the components of geometric understanding vision language models. CoRR, abs/2503.03840, 2025.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, abs/2301. 2023.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Trans. Assoc. Comput. Linguistics, 11:635-655 651, 2023.
Siqi Liu, Ian Gemp, Luke Marris, Georgios Piliouras, Nicolas Heess, and Mare Lanctot. Re- evaluating open-ended evaluation of large language models. In The Thirteenth International Conference on Learning Representations, 2025. hhttps:://ppereviess
dd=kbOATXKNςς
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. In NeurIPS, 2023.
OpenAL. Gpt-4v(ision) system card, 2023. gpt 4v- uo-systeem-ae card
URL http:://openaa..omm/rsaaaar
OpenAl. Gpt-4o system card, 2024. URL htl.atpps//aaxbboor////////111..
OpenAL Openai o3 and o4-mini system card, April 2025. URL/attps::/ppeea..oomm
o3-04- 04-00--miin
card/| System Cards. Accessed: 2025-09-11.
Stephen Palmer. Vision Science: From Photons to Phenomenology, volume 1. MIT Press, 01 1999.
Atin Pothiraj, Elias Stengel- -eenge--skkin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision language models via occluded object counting. CoRR, abs/2504.1588, 2025.
Fred H Previc. The neuropsychology of 3-d space. Psychological bulletin, 124(2):123, 1998.
Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part XLIII, volume 15101 of Lecture Notes in Computer Science, pp. 214-238. Springer, 2024.
Zekun Qi, Wenyao Zhang, Yufei Ding, Runpei Dong, Xinqiang Yu, Jingwen Li, Lingyun Xu, Baoyu Li, Xialin He, Guofan Fan, Jiazhao Zhang, Jiawei He, Jiayuan Gu, Xin Jin, Kaisheng Ma, Zhizheng Zhang, He Wang, and Li Yi. Sofar: Language-groume object manipulation. CoRR, 131/25 2025.
orientation bridges spatial reasoning and
Tiago Ramalho, Tomás Kociský, Frederic Besse, S. M. Ali Eslami, Gábor Melis, Fabio Viola, Phil Blunsom, and Karl Moritz Hermann. Encoding spatial relations from natural language. CoRR, abs/1807.00 Abb//800.01670, 2018
Fatemeh Shiri, Xiao- Yu Guo, Mona Far, Xin Yu, Reza Haf, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 21440-21455. Association for Computational Linguistics, 2024.
Patrick E. Shrout and Joseph L. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420 428, 1979. ISSN 0033-2909. doi: 10.103770033-2900 290..9....98.222200 URL Ut..tttpp//is..ddd.rr//00......0
86. 2200
16

---

Published as a conference paper at ICLR 2026
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. CoRR, abs/2411. 2024.
StepFun, Step-3 large yet affordable: Model-system co-design for cost-effective decoding, 2025. URL htR.httts:///xxiv http///xxvvv...gaaaa/ wrgp.r/////........ 19227
Ilias Stogiannidis. Steven McDonagh, and Sotirios A. Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision-language models. CoRR, abs/2503.19707, 2025.
Emilia Szymanska, Mihai Dusmanu, Jan- lan-Willem Buurlage, Mahdi Rad, and Mare Pollefeys. Space3d- bench: Spatial 3d question answering benchmark. CoRR, abs/2408. 16662, 2024.
Mohsen Tavakol and Reg Dennick. Making sense of cronbach's alpha. International Journal of Medical Education, 2553-55, June 2011. ISSN 2042-6372. doi: 10.5166/jmee4ddff..d... URL
http: doi org 5116/111111 jme.e...iemmmmm//// 4dfb. 8dfa
Gemini Team. Gemini: A family of highly capable multimodal models, 2025a. URL https
7/ //axiv.org/abs//31111 11805
Kimi Team. Kimi-vl technical report, 2025b. URL BRLLLttps///a211..////////00
Team, Wenyi Hong. Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang. Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang. Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang. Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang. Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang. Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du Zihan Wang. Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Minlie Huang. Yuxiao Dong, and Jie Tang. Glm-4.5v and fim--..1-thinkkngg Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. /Ul//ttps///rrxxv.o//////b/
01006
Yaacov Trope and Nira Liberman. Construal-level theory of psychological distance. Psychological review, 117(2):440, 2010.
Raphael Vallat. Pingouin: statistics in python. Journal of Open Source Software, 3(31): 1006, November 2018. doi: 10.21 1021105/joss.0105
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10.1003//44592-01999
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine -in-inning: Learning to critique is more effective than learning imitate, 2025. URLpstts:://axxii wwtpziiivcrrg/eeo /ttp//b/////5511/
Jihan Yang, Shusheng Yang. Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. CoRR,
abs/2411..41711, 2024.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasa,, Faisal Ahmed, Zicheng Liu. Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: prompting chatgpt for multimodal reasoning and action. CoRR, abs/2303. 11381, 2023.
17

---

Published as a conference paper at ICLR 2026
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision- lasiun-lagguage model for spatial affordance prediction for robotics. CoRR, abs/2406.10721, 2024.
Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. VImbench: A compositional benchmark for vision-and-language manipulation, 2022. URL https: pattpss//aate //arxiv
org/abs/2200 /206 08522
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang. Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang. Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: //aps:///axxvv org/ abs 2504. 10479
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang- Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez. Arenas, and Kehang Han. RT-2: Asiionlanguagg-actto action models transfer web knowledge to robotic control. In Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pp. 2165-2183. PMLR, 2023.
18

---

Published as a conference paper at ICLR 2026
Related Work
21
A.1 Detailed Analysis of Limitations in Prior Benchmarks
22
Task Design and Descriptions
22
B. Relative Positioning
23
B.2
Depth Occlusion
24
B
3
Orientation
24
4
Size & Scale
25
B.5
Spatial Navigation
25
B.6
3D Geometry
25
More Details on Benchmark Construction
26
Annotator Recruitment
26
Annotator Training and Protocol
26
More details on Experiments and Evaluation
26
D.I Multiple Choice Evaluation
26
D.2 Open-Ende Ended Evaluation
27
D.2.1 Evaluation of LLLM-as-a--udaee for Open-ended Evaluation
27
E
Error Analysis of SPATIALAB-MCQ
28
Sub-Category-wise Quantitative Error Analysis
28
Model Failure Analysis
33
Error Analysis of SPATIALAB-OPES
33
Sub-Category-wise Quantitative Error Analysis
33
F2
Model Failure Analysis
37
Analysis of MCQ vs. Open-Ended Performance Gaps
39
Quantitative snapshot
39
2
Patterns and observations
39
Hypotheses on root causes
40
G.4
Practical implications!
41
H
Details on Improving Visual Reasoning Capabilities
41
Inherent Reasoning Mechanisms
H.2
Chain-of-Thought (CoT) Prompting
41
Thought (CoT) Prompting with Self- Reflection
43
H.4
Supervised Fine-Tuning (SFT)
44
H.4.1
Setup and Implementation
44
H.4.2
Supervised Fine- Fine-Tuning (SFT) Performance
44
19

---

Published as a conference paper at ICLR 2026
H.4.3
SFT Dynamics Analysis
46
H.4.4
Performance Gain on External Benchmarks
47
H.5
AI Agents for Spatial Reasoning
48
H.5.1
Agents
48
H.5.2
Technical Implementation Details
49
H.5.3
Evaluation and Analysis
49
I
Qualitative Error Analysis and Error Patterns
50
Overview
50
1.2
Error taxonomy and analysis
51
I.2.1
Spatial mislocalization and reference confusion.
confisioo.......................
I.2.2
Perspective and scale mistakes,
51
I.2.3
Occlusion and ordering failures.
52
2,4
Attribute confusion and semantic swap,
52
12.55
Open-ended rationalization without verification,
52
Error patterns and cross-model signals
53
Mitigation strategies
53
J
Representative Benchmark Samples Across All Sub-Categories
58
Statistical Robustness and Dataset Stability
88
K.1 Model Robustness
88
K.
Multiple Run Averages and Deviations
88
K.
1.2
Intra-Class Correlation (ICC)
88
K.2
Dataset Stability and Internal Consistency
88
K.2.1
Resampling Study
89
K.2.2
Item-Level Consistency
89
K.2.3
Cronbach's 's Alpha
89
Empirical Results
90
K.3.
Model Evaluation Robustness
90
K.3.2
Dataset Stability and Internal Consistency
3.3.
Overall Discussion
92
20

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
Spatial reasoning in multimodal models. Spatial reasoning, the capacity to understand geometric structures and relational layouts in visual scenes, is central to embodied intelligence. Early multimodal benchmarks such CLEVR (Johnson et al., 2016) and GQA (Hudson & Manning. (019)) tested compositional reasoning with synthetic or curated templates, which allowed controlled analysis but provided limited ecological validity. With the advent of large vision- language models (VLMs) (Li et al., 2023; Alayrac et al.) 2022; OpenAl 2023). spatial reasoning has been revisited more naturalistic datasets, though systematic weaknesses remain. Despite strong performance on captioning and visual QA, these models often fail in tasks requiring depth ordering, occlusion inference, or consistent reference-frane alignment. This discrepancy highlights a broader problem: scaling alone does not guarantee progress in structured spatial understanding.
Spatial reasoning benchmarks. A range of benchmarks have recently been proposed to probe these gaps (Table SCANQA (Azuma et al. 2022) was among the first embodied QA datasets, requiring reasoning about 3D indoor environments, but its limited scale (800 scenes) and relatively shallow complexity restrict broader generalization. Datasets such as VISUAL SPATIAL (Liu et al. (023) and WHAT'S 's UP (Kamath et al., 2023) expanded scope to real-world or household contexts, but relied on template-driven questions, leading to reduced linguistic variability and an overrepresentation of simple binary or multiple-choice relationships. More recent efforts, including EMBSPATIAL-BENT (Du et al. (024) and SPACE3D D-BENCH (Szymanska et al. 2024), increased task variety yet remained constrained by scale (2.2K and 211 samples, respectively) and annotation style (template-hear introduced mixed domains and up to 14 categories, improving diversity, but still emphasized either static snapshots or limited multi-view simulation.
or manual but narrow).
BEAAAAGGPT-BBNNNEE and
Embodiment, difficulty, and diversity. The trend toward embodiment is visible in datasets like ROBOSPATIAL (Song et al., 2024). which scales up to 1M scenes with robot-centric questions. However, its template-based annotation makes questions less natural and easier to game with heuristic shortcuts. Similarly, SPATIALVLM (Chen et al., 2024b) offers unprecedented scale (billions of QA pairs), but its two-category coverage renders it unsuitable for fine-grained diagnostic evaluation. On the other end, high-quality manual datasets such as VSI-BENCH (Yang et al.) 2024) achieve improved annotation fidelity, but with only 288 scenes, they remain insufficient for training robust cross-model comparison. In terms of complexity, most existing benchmarks cap at "medium" (Mm in Table [T.. rarely extending to the "hard" or "extremely hard" levels that require multi-step causal inference. This leaves a wide evaluation gap for next-generation models.
Evaluation protocols. Equally important are evaluation types and analysis depth. Most prior work employs close-ended QA or binary judgments (e.g.. VISUAL SPATIAL, EMBSPATIAL-BESS BENCHH,, which simplify scoring but restrict the expressivity of model outputs. A few datasets such as ROBOSPATIAL. SPA and SPATIALAB -PEN introduce open-ended evaluation, but only SPATIALAB OPEN couples this with extremely high complexity, manual puzzle-style annotations, and comprehensive analysis of errors. This enables deeper understanding of systematic weaknesses beyond accuracy, including calibration and cross-model divergence, which existing datasets rarely capture. Without this granularity, performance gains be misleading, as models may overfit dataset-specific biases rather than generalizing robust spatial competence.
Positioning of SPATIALAB. SPATIALAB directly addresses these limitations. Compared to prior datasets that span between and 14 task categories, SPATIALAB covers 30 distinct categories across geometry, occlusion. orientation, relative positioning, size and scale, and navigation, offering un- precedented task diversity. Unlike template-basee benchmarks, all questions are manually authored and muszle-inspired, ensuring linguistic richness and reducing shortcut opportunities. Furthermore, SPATIALAB is fully embodied and multi-modal, including both static and dynamic visual con- texts, thus bridging gaps between robotics-style datasets (ROBOSPATIAL) and web-scale corpora LMM) As shown in Table [ even the strongest baselines, such as VL3.55L33357225 and GPT-5-mini. perform well below 60%, with the open-ended variant yielding only 40.93% accuracy. This large performance gap demonstrates the difficulty of the benchmark and its utility for driving future research.
21

---

Published as a conference paper at ICLR 2026
Toward next-generation evaluation. By combining high difficulty, broad coverage, manual annotation, and open-ended tasks, SPATIALAB extends the landscape of spatial reasoning benchmarks from template-driven narrow evaluations comprehensive, embodied testing. This design makes it possible not only to assess whether a model answers correctly, but also to identify systematic error patterns, probe causal and counterfactual reasoning, and measure calibration reliability. In this sense, SPATIALAB provides a rigorous foundation for the next phase of multimodal research, complementing datasets like MIND THE GAP and VLM4D that explore temporal or generative aspects but remain narrower in coverage. We expect future work to leverage SPATIALAB as high-bar diagnostic tool. pushing models beyond surface-level recognition toward genuine grounded spatial intelligence.
A.1 DETAILED ANALYSIS OF LIMITATIONS IN PRIOR BENCHMARKS
To situate SPATIALAB within the broader landscape, we conducted detailed examination of existing spatial reasoning benchmarks. Earlier efforts have contributed important foundations, yet they present structural and methodological constraints that limit their value for evaluating current VLMs. These constraints fall into three areas: synthetic-domain bias, limited task scope, and restrictive evaluation formats. Table summarizes the major limitations identified.
Synthetic and Template-Baseed Constraints. Benchmarks such as CLEVR and GQA rely on synthetic environments template-generann linguistic structures. Although these settings offer controlled conditions for isolating compositional reasoning, the visual domain lacks real-world complexity and the linguistic space is narrow. Models can rely on template regularities rather than grounded spatial interpretation. Related simulated benchmarks such as ROBOSPATIAL face similar issues, resulting in a large sim-to-real gap.
Domain and Scale Limitations. Datasets grounded in 3D reconstruction, including ScanQA and Space3D-Benne -ennhh, provide depth supervision but remain restricted to indoor scenes. This prevents assessment in outdoor or mixed environments that are essential for embodied agents. High-quality manual datasets such as VSI-Bench offer richer annotation but remain small in scale, limiting their usefulness for high-resolution subcategory analysis.
Taxonomic and Evaluation Narrowness. Large-scale datasets such as Spatial cover broad tasks but lack detailed taxonomies, making them unsuitable for diagnosing specific weaknesses in ori- entation, occlusion, or layout reasoning. Many recent benchmarks, including EMBSPATIAL- and W WHAT'S i UP, rely heavily on Binary or MCQ formats. These formats can inflate performance by enabling option elimination rather than requiring generative spatial reasoning. SPATIALAB mitigates these issues by combining MCQ and Open- -ndedd formats across detailed taxonomy of 30 spatial subcategories.
Table 4: Limitations identified in representative prior spatial benchmarks compared with SPATIALAB. Limitations are categorized by domain constraints, evaluation format, and task taxonomy.
Benchmark
Domain
Eval. Format
Limitations Ideacifeed
CLEVE and GQA
Syufletic
Closed
Relatioal
Antifiiaa visua dumain; templtee drvve language: proze to shorcet leaming.
ScanQA
Indocr 3D
Open Ended
Novigaion
Ressicaed indoce point cloud; limited scale (c 1& sccnes)
SpotialV1 I.
With This
Open-Padee
Genvral
Covers only a small number of high-lesel spucca enogreies.
Emlisquaiial-Bnncc
Indhor
MCQ / Riney
6 Calegarios
Relkss on diooal-cnaal fimmrce linital complesit in nccouring thrins.
KobeSSpotial
Simulator
Mtybed
Embodied
Template generated questions simuloted scenes produce sim to zeal pap.
Space Bench
Indocr
MCQ
5 Categones
Very snall scale (-2200 samples) prevents robust stamitical evaliation.
OmniSpatial
Mix
MCQ/ Hinary
Merric
Emphasmes metric oorputs. limited support explanatory ressninng
Spotialab ab (Ours)
the
e Open
Sub
Addresses steriieed linitatios with real world images, detalled riconoy, and diverie evoluation formass
B TASK DESIGN AND DESCRIPTIONS
Understanding spatial reasoning requires more than simple recognition of objects; it depends on how those objects relate, align, and interact in physical space. To capture this complexity, we introduce SpatiaLab, a benchmark designed to test fine-grained aspects of spatial understanding across diverse categories. The tasks in SpatiaLab reflect how humans naturally describe and reason about environments, ranging from relative positioning (e.g., What's to the left of the microwave?) to geometric inference (c.g.. Would this stack fall if another book is added?). Each category is further broken into subcategories that highlight subtle but essential dimensions of reasoning, such
22

---

Published as a conference paper at ICLR 2026
occlusion, orientation, navigation, and physical effects like gravity. By framiig each subtask in accessible, everyday terms, SpatiaLab encourages models to go beyond object detection and engage in structured spatial interpretation. This design allows to probe how well systems generalize across contexts that matter for robotics, AR/VR, autonomous driving, and human-computer interaction. The following sections provide short, task-level descriptions for every category and subcategory in SpatiaLab
Reflective Surfaces
Occiusion
Layening
Transparent Effects
Partia
Completa Occlusion Inference
Order
Comparison
Velume
Direction
Cardinal
50
50
70
Stability
Prediction
Facing
Direction
Depth usion
Spatial
Containment
Object
42
Rotation
3D Geametry (258)
Orientation
Shape
Projection
(202)
Stacking
50
Orientation
35
Gravity
Tool
50
Effects
Handedness
SpatiaLab
A ignment
43
50
Viewpoint
Pattems
Visiblitty
Spatial Navigation
Betweenness
(237)
Relationships
Spatial
Sequence
Pathway
Existence
(252)
Relations
43
Size Scale
50
50
Distance-Sh Size Correlation
50
Shadow- Size Projection
Perspestive Distortion
Relalive Size Comparison
Consistency
Scale
Figure 5: Categories and subcategories of spatial reasoning in SPATIALAB. Each category
decomposes into five subcategories, yielding thirty task types in total.
The following outlines the rationale and real-world utility guiding each task design, per category:
B.1 RELATIVE POSITIONING
Directional Relations. Directional relations describe how objects are positioned relative to one another, such as left. right, above, below. Understanding these relations enables models to interpret spatial layouts more naturally, just like how humans describe scenes. For example, identifying what lies immediately to the left of appliance helps in scene understanding and robot navigation.
Proximity Gradients. Proximity reasoning involves recognizing which objects are closest or farthest from reference point. This is key for questions like "Which object is nearest the doorway?" Models capable of estimating gradients of closeness can support navigation tasks and assistive technologies.
23

---

Published as a conference paper at ICLR 2026
Alignment Patterns. Alignment concerns whether objects follow certain orientation order, such as being stacked vertically or arranged in a straight line. Recognizing alignment patterns helps in understanding organization in environments like shelves, tables, or warchouses. Such reasoning is also critical for tasks like robotic sorting and automated inspection.
Betweenness Relationships. Betweenness focuses identifying objects that occupy the middle ground between two references. For instance, asking what sits between the lamp and the armchair requires reasoning about spatial continuity. This helps in navigation, retrieval, and situational awareness tasks.
Corner/Angle Positioning, Corners and angled positions often serve reference anchors in spatial reasoning. Questions such as "What's positioned in the back-right corner?" require the model to localize objects within constrained regions. This kind of reasoning is vital in robotics for mapping and placement.
B.2 DEPTH & OCCLUSION
Layering Order: Layering requires distinguishing which objects appear in front versus behind. For examplc. identifying the frontmost item on a table reflects depth reasoning. This is particularly useful in AR/VR applications where visual stacking affects user experience.
Partial Occlusion. Partial occlusion occurs when one object hides part of another from view.
Recognizing the hidden item behind coffee cup, for instance, requires filling in missing information. This strengthens robustness in perception systems operating in cluttered environments.
Complete Occlusion Inference. Sometimes objects are fully hidden, yet clues like shadows or bulges reveal their presence. Inferring what lies under blanket requires the model to use indirect evidence. Such reasoning mirrors human inference in uncertain visual conditions.
Transparency Effects. Transparent surfaces, such as glass or water, allow partial visibility of objects behind them. Understanding what's visible through a fish tank requires blending occlusion reasoning with transparency perception. This plays an important role in simulation, graphics, and embodied Al.
Reflective Surfaces. Reflective reasoning asks models to interpret mirrors, shiny metals, glass reflections. Identifying what is reflected in mirror involves indirect spatial mapping. This is essential for safety in autonomous vehicles and situational awareness robotics.
B.3 ORIENTATION
Cardinal Direction. Cardinal reasoning relates to compass directions like north, south, east, and west. Asking which way a front door faces requires integrating global orientation cues. This is critical for navigation and geospatial alignment.
Object Rotation. Rotation reasoning measures how much an object has been turned around an axis. Estimating that a chair is rotated by some degrees is crucial for accurate 3D modeling. This helps robots align objects correctly for manipulation.
Facing Direction. Facing direction specifies the orientation of an object's front side. For example, de- termining whether a monitor faces north requires mapping its surface orientation to global references. This is important in surveillance and robotics.
Stacking Orientation. Stacking involves understanding whether objects are piled vertically, horizon- tally, or in mixed orientations. Recognizing such layouts ensures stability in warehouses and homes. is also critical for safe robotic stacking.
Tool Handedness. Certain tools, like scissors gloves, designed for left- fight-handed use. Identifying handedness requires fine-graiie -ee-graided shape and orientation perception. This enhances assistive robotics in human-centered environments.
24

---

Published as a conference paper at ICLR 2026
B.4 SIZE & SCALE
Relative Size Comparison. Relative comparisons assess whether one object is larger, taller, or smaller than another. Asking a vase is taller than statue requires proportional reasoning. This is crucial for packing, fitting, and recognition tasks.
Perspective Distortion. Perspective can make distant objects appear smaller than nearby ones, even if the actual size differs. Identifying illusions like faraway looking smaller than nearby bicycle requires compensating for depth cues. This ability makes models more robust to camera perspectives.
Distance-Size Correlation. Here, size judgments depend distance estimation. For instance, nearby bicycle appearing larger than a far car should not confuse true size reasoning. This helps in navigation, AR/VR, and visual grounding.
Scale Consistency. Scale reasoning involves checking whether object sizes are reasonable in context. mouse larger than chair signals inconsistency, which models must detect. This strengthens realism in simulated environments.
Shadow-Size Projection. Shadows provide indirect cues about object size and light source position. Estimating how tall a light must be based on shadow length requires geometric reasoning. This is useful in graphics, architecture, and physics- based modeling.
B.5 SPATIAL NAVIGATION
Pathway reasoning checks if clear route exists between two points. For example, determining if one can reach the desk without moving chairs tests navigability. This ability underpins indoor robot planning.
Pathway Existence.
Obstacle Avoidance. Avoidance requires finding routes that minimize collisions. Asking for the clearest path from door to window directly maps to robotic and human navigation. It is essential in autonomous driving and indoor mobility aids.
Viewpoint Visibility. From a given viewpoint, certain objects may be blocked or visible. For example, checking what is obscured when sitting on the couch requires simulating perspective. This strengthens AR/VR realism and robotic vision.
Spatial Sequence. Sequence reasoning identifies the order of encounters along path. Walking from the kitchen to the balcony, one might pass the dining table first. Such reasoning mimics human navigation memory.
Accessibility Constraints. Constraints identify which objects are blocked from access due to obstacles. For instance, a cabinet may be unreachable because of table front. This is crucial for assistive AI and ergonomic design.
B.6 3D GEOMETRY
Volume Comparison. Comparing volume asks which object occupies more three-dimensional space. Judging whether box is larger than sphere helps in packing and containment tasks. This is important for logistics and simulation.
Stability Prediction. Stability reasoning predicts whether adding or moving objects will topple stack. For instance, checking if placing a book will collapse pile mirrors human intuition. This supports safety in robotics and construction.
Shape Projection. Projection tasks ask what shape an object would cast under certain rotations or lighting. For example, sphere always projects circle, while cube may project square or hexagon. Such reasoning strengthens geometric understanding.
Spatial Containment. Containment checks whether object fit inside another. For example, reasoning a ball can fit in a box supports packing and design. This task aligns with intuitive geometry.
25

---

Published as a conference paper at ICLR 2026
Gravity Effects. Gravity reasoning predicts motion outcomes under physical forces. For example, rolling ball may stop under a table due to friction and barriers. This strengthens real-world simulation capabilities.
C MORE DETAILS ON BENCHMARK CONSTRUCTION
C.I ANNOTATOR RECRUITMENT
We recruited annotators aged between 24 and 30 years, all of whom were engineering graduates. They participated voluntarily and worked independently without external incentives. Before starting the annotation process, we provided structured training sessions ensure clarity of task requirements and consistency across annotators. We also maintained all ethical and quality control standards throughout the process. This approach ensured that the resulting annotations were both reliable and aligned with the dataset design.
C.2 ANNOTATOR TRAINING AND PROTOCOL
Before beginning annotation, we designed a structured training program to ensure consistency and reliability across the full taxonomy of tasks. We first introduced annotators to the six core categories (Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry) and their thirty subcategories. For each subcategory, we provided curated examples that illustrated the corresponding spatial relations (e.g., directional relations, occlusion inference, stability prediction) along with counter-exampe examples to highlight common pitfalls. Beyond the taxonomy, we trained annotators to account for six meta-dimensions of image variability: lighting, texture complexity, edge density, dominant relation, material type, and gravity constraints, so that questions explicitly leveraged visual diversity. We demonstrated how these factors interact in real-world settings, such as reasoning about occlusion under low-contrast lighting or maintaining scale consistency under perspective distortion.
We also introduced the two evaluation modes: multiple-choice and open-ended. In multiple-choice settings, annotators learned design precise distractors to test fine-grained discrimination, while in open-ended settings they practiced formulating prompts that encourage compositional reasoning. To ground this training, we ran calibration sessions in which annotators generated both types of questions for sample images and iteratively refined their designs with feedback. To situate task design within the broader research landscape, we reviewed prior benchmarks (e.g., SpatialBot- Bench, EmbSpatial, SpatialMM) with annotators and analyzed their limitations. These resources often emphasize simplified relations, synthetic or staged scenes, and narrow reasoning types, while neglecting natural complexity such as shadows, reflections, or gravity-sensitive stability. This gap analysis underscored the need for richer, more ambiguous, and dynamically layered spatial contexts.
Finally, provided detailed demonstrations of the annotation protocol. These demonstrations specified how select images with sufficient spatial complexity, how to map questions unambiguously to subcategories, and how to avoid trivial formulations. We instructed annotators to prioritize tasks requiring multi-step reasoning, such as depth layering, relational chains, or cross-vice view scale inference, over binary labels. Through this systematic training and iterative feedback. we ensured that the final dataset balanced taxonomic coverage with real-world relevance.
D MORE DETAILS ON EXPERIMENTS AND EVALUATION
D.1 MULTIPLE CHOICE EVALUATION
The prompts used for the answer model are shown in Figure We set temperature to 0 and tokens to 10, while keeping all other generation parameters at their default values.
26

---

Published as a conference paper at ICLR 2026
SpatiaLab-MCO. Direct: Prompt Answering Model
spatial reasoning
nultiple question image,
Answer with single number (1-4) corresponding to the option.
Question:
(question)
Options:
(opti)
2. (opt2)
3. (opt3)
(opt4)
SpatiaLab- Open::pennnn Direct Prompt for Answering Model
spatial reasoning
Answer the following question in 2-4 sentences about the given inage. Question:
(question)
SpatiaLab-Open Direct: Prompt Judge Model
strict evaluator.
You will receive a question, the ground truth answer, model's answer. Return only If the model's 's answer correct, otherwise 0.
Question: (question)
Ground Truth Answer: @ground_truth)
Nodel
(nodel_answen answer)
Output only single digit: or 1.
Figure 6: Prompts used evaluation.
D.2 OPEN-ENDED EVALUATION
The prompts used for both the answer model and the judge model provided in Figure 6 We temperature to 0.7 and max.tokens to 1200, with all other parameters kept at their default values.
D.2.1 EVALUATION OF LLM-AS-A-JUDGE FOR OPEN-ENDED EVALUATION
To evaluate 2.mini-2..5-lashh as an automatic judge, we selected a balanced set of 240 samples covering all categories. Each sample included the evaluation setup (question, ground truth, and model output), which was scored independently by three human annotators. Annotators assigned binary scores: 1 for correct and for incorrect. We compared the LLM's judgments against the human annotations using multiple complementary agreement metrics, computed both overall and category. Specifically, we employed Cohen's Kappa, Accuracy with Majority Vote, and Fleiss' Kappa, defined as follows.
Cohen's 's Kappa. Cohen's 's kappa (k) measures pairwise agreement between two raters while cor- recting for chance agreement. For binary task, it is defined
Po-Pe where Po is the observed
I-pr
agreement and Pe expected agreement by chance: Pe = E.e (0,1) Pe pe (2) Here, pe
(1) and pe
are the marginal probabilities that each rater assigns class c. Ano 1: perfect agreement; λ = agreement equals chance; < O: systematic disagreement) We computed Cohen's ''s kappa between the LLM and each annotator individually (Kiumall KHummn22 (immmm3)) as well as between the LLM and the majority vote of the annotators (Kmmjoortt).
Accuracy with Majority Vote. We also measured raw agreement between the LLM and the annotator majority. Accuracy is defined Accuracy - Σ 1= ( ") where y is the LLM's decision for sample i, and y is the majority vote among annotators. Unlike Cohen's 's kappa, accuracy does correct for chance agreement; it simply reflects the proportion of identical labels.
27

---

Published as a conference paper at ICLR 2026
Fleiss' Kappa. To assess anter-annotator reliability, computed Fleiss' kappa (KF), which generalizes Cohen's kappa to multiple raters. Let Ray denote the number of annotators assign- ing category j to item i, and define P. n(n Eing(nu - 1), where n the number of annotators. The observed agreement is P N Ez P, and the expected agreement is 1
P-P.. Higher P. EG with Pj = Nn n.y. Fleiss' kappa then computed as Rp 1-P. values indicate stronger anner-annotator reliability beyond chance.
Table 5: Agreement metrics between Gemini-22.5Flas as an automatic judge and three human annotators across all categories. The table reports Cohen's 's kappa with the majority vote and each annotator, accuracy against the majority, and Fleiss' kappa among all annotators. Bold values highlight the highest agreement within each category.
Category
<Majority
xHumanl
kHuman2
kHuman3
Acc. (majority)
Fleiss x (all)
Overall
0.738
0.738
0.681
0.795
0.880
0.774
3D Geometry
1.000
L.000
0.875
1.000
1.000
0.916
Depth & Occlusion
0.805
0.805
0.899
0.697
0.909
0.871
Orientation
0.500
0,500
0.426
0.588
0.810
0.571
Relative Positioning
0.833
0,657
0.676
0.833
0.917
0.778
Size & Scale
0.417
0.571
0.417
0.696
0.714
0.704
Spatial Navigation
0.815
0.815
0.667
1.000
0,933
0.773
Interpretation Table 5 shows that the LLM judge (Gemini-2.5-Flash) demonstrates strong overall agreement with human annotators, achieving a Cohen's 's kappa of 0.738 with the majority vote and 0.681-0.795 against individual annotators. Raw accuracy against the majority is 0,880 overall, while Fleiss' kappa among human annotators is 0.774, indicating substantial consistency. Certain categories, such 3D Geometry, reach perfect agreement with 1.000 and accuracy of 1.000. Depth & Occlusion also shows high agreement (K. majority = 0.805, accuracy = 0.909). Categories like Orientation (x 0.500, accuracy 0.810) and Size Scale (x = 0.417, accuracy 0.714) show moderate agreement but remain within acceptable reliability ranges.
These results indicate that Gemini-2.5-Flass performs comparably to human evaluators in scoring cor- rectness, with Cohen's 's kappa values mostly in the substantial range (0.61-0.80) and some categories reaching almost perfect agreement (higher than 0.81). Fleiss' kappa of 0.774 confirms that human annotators themselves are highly consistent, validating the reliability of the reference scores. The L.M achieves perfect or near-perts pear-perfect agreement in clearly defined categories such as 3D Geometry (K 1.000, accuracy 1.000) and Spatial Navigation (k 0.815, accuracy 0.933), demonstrating accurate pattern recognition. Moderate agreement in categories like Orientation and Size & Scale reflects task complexity and inherent subjectivity rather than model shortcomings. Overall, it shows that LLM-as-judhe shows robust reliability and can serve as a practical automated scoring tool, closely mirroring human evaluation.
E ERROR ANALYSIS OF SPATIALAB-MCO
E. SUB-CATEGORY QUANTITATIVE ERROR ANALYSIS
Closed-source Models. flosed-source models Table 6) produce some of the strongest results across the benchmark. The best case is GPT-4o-mini, which reaches 85.71% 71% in Stacking Orientation, the highest score overall. Gemini-2.0-Flaat performs well in occlusion and projection, achieving 64.0% in Partial Occlusion and 64.71% in Shadow-Size Projection. Similarly, GPT-5-mini scores 68.0% Tool Handedness and 55.71% in Layering Order. Weaknesses remain, as Claude 3.5 Haiku only reaches 32.0% in Relative Size Comparison. On average, closed-source systems outperform open-source ones by 15 15-25 percentage points across several categories. However, within-mode variation remains large, often exceeding 50 percentage points. We therefore conclude that while closed-source training pipelines deliver higher ceilings, they do not solve the challenge of uniform spatial competence.
28

---

Published as a conference paper at ICLR 2026
Table 6: Multiple-choiee Evaluation Accuracy on SPATIALAB-MCO by Question Sub- Categorrss for Closed-source Models. We bold and underline the best score within each model category.
Calgoey
Sub-Cheeprry
CiP----mint
CPPT-S-mint
Cemmiat-2---Faae
Clemin-2.
Clunde Huiku
Visura Malium 3.1
Grivity Fifects
$2.0
46.0
52.0
52.0
46.0
Stopee Projection
48.0
46.0
60.0
Goometry
Special Contrnnent
36.0
46.0
34.0
18.0
Stability Prediction
$2.38
52.38
99.52
50.0
45.24
52. 33
Volume Comperieon
413
47.83
30:13
$4.78
34.78
36.95
Complets Dccluson Inference
61.54
69:23
61.54
61.51
61.54
46 15
Layaring Order
32.56
55.71
50.0
47885
12.8
45.71
Depthand Occlssion
Pantall Oeelusia
42.0
54.0
64.0
50.0
46.0
$3.0
Reflective Surfaces
34.0
55.0
46.0
48.0
48.0
44.0
Transparescy Lffects
32.0
15.0
50.0
44.0
36.0
62.0
Cadinal Direction
360
30.0
44.0
18.0
Facing Direcion
45:24
$7.14
$7.13
47.62
35.71
45.2
Orientaion
Object Rosatin
40.0
60.0
50.0
46.0
38.0
44.0
Stacking Oxizmatiin
85.71
77:34
62.86
65.71
64.57
62 M6
Tool Handedsess
52.0
68.0
64.01
04.00
56.0
56,0
Aligiment Patterns
44.19
58.14
53.49
46.51
$8.14
Beteresness Relationsips
51.00
63.83
70.21
63.83
44.58
61.7
Relative Positioning
Comme/Anple Pootsoning
47:22
63.39
50.0
53.53
50.0
77.78
Directional Relations
48.34
79,07
67.44
09.77
53.69
09.77
Prosiuity Gradiens
44.39
J8.54
44.19
54.85
37.21
44.19
Distance Size Coreeation
$6.0
$6.0
56.0
42.0
34.0
42.0
Perspective Distortion
$1.0
40.0
46.11
36.00
36.0
34.0
Size and Scale
Relative Size Compurison
42.0
44.0
58.11
46.0
32.0
3M.0
Scale Conestetcy
47.86
45.1
87.06
39.22
23.33
41.15
Strahw-SSre Projection
$2.04
39.22
64.71
40.02
41.18
52.90 13.94
Accessbility Coustraits
56.1
60.98
65.85
65.85
48.78
56.1
Ghotacl Aveidunce
35.0
48.0
34.0
34.0
46.0
40.0
Spucial Novigation
Palloaa Exisience
$4.35
41.3
43.48
45.65
39.13
36.95
Spccial Sequeace
66.0
78.8
48.0
62.0
56.0
44.0
Viewpeint Vicisiiity
36.0
34.0
48.0
50.0
40.0
34.0
Table 7: Multiple-choiee Evaluation Accuracy on SPATIALAB-MC by Question Sub- cateCraeessies for Open-source Large Models. We bold and underline the best score within each model category.
Cakgory
Sab-Chagpeyy
Octera--21B-4
Skips buggction
X2.0
42.0
44.0
20.
46.0
Gennary
Spotii Cansalanne
34.0
35.11
35.31
42.30
(170)
Velane Conqezious
13/4
3013
32.1
55.12
43.40
16:56
Gomples Usclaion
36.77
01.54
5637
06677
08:12
66,17
Leaanng Orter
56.29
36.99
37
25.6
808
15.51
Dipck
Reflenire Sarkos
25.0
380
15.0
216
500
04.
Trangateg Elick
430
400
Cartinal Diracioo
22.
48.0
3431
46.0
480
41.
Griceaaie
Ohject Butain Sacking ee
40.0
38.31
46F
46.0
44.5
08.57
05.1
65.11
(6.57
56
11.45
Alansaei Pezzze
51.01
33.40
Bebaiendyss
03:55
66.96
Canasthngi Postaeing
28.33
55.55
36.13
$0.50
500
75.0
Prscerty Grrieen
41.35
37.55
18.00
34:53
11:56
Prrppction Bessnivn
33.0
13.0
1531
46.
510
53.8
Suc and Soal
Campation
518
Scda
29-11
47.86
41.33
43.14
12.55
Skidkw Naw Trajetion
15.25
519
06.86
20.7
38,54
43.
75A1
53.96
43.5
Obouck
38.0
38.0
463
42.0
Spaid Norgatiee
Spatil Seqpence
115
1300
10.0
66.8
425
Open-source Large Models. When scaling to large open-source models Table we see clear improvements but also severe fragilities. The InternVL3.5-72B achieves 80.56% Corner/Angle Positioning. one of the highest open-source results. Both vision-instruct reach 66.67% in Complete Occlusion Inference. Yet weaknesses persist, such as the
and 1lama-3.2-90b-
scoring only 2.0% in Object Rotation. Gains are also evident in Stacking Orientation, where 1imma--.--00-visioo-nnntttt
scores 71. 71.43% but this comes with losses in other categories. The within-model spread often exceeds 70 percentage points, highlighting brittle specialization. Gravity reasoning remains modest at 56.0%. We find that scaling yields higher peaks but does not resolve structural weaknesses. Our conclusion is that size amplifies selective strengths while leaving foundational representation gaps intact.
Open-source Small Models.
We observe that source small models Table 8) display sharp reaches
contrasts in performance across sub-categories. For example, -wwen2..-vv-7b-innstt 74.29% in Stacking Orientation, while InterVV33.5-11 collapses to 12.0% in Cardinal Direction. Scores in Pathway Existence remain modest, with the best models tied at 34.78%. Gravity-related
29

---

Published as a conference paper at ICLR 2026
Table 8: Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub- Sub-Categories for Open-source Small Models. We bold and underline the best score within each model category.
Calaaeey
Sub Cutegety
IntemVL35-BB
Inten LL3.51L
CACD 2,5-1,1 intract
Genuna
qwen 2.5 i--uutrrutt
Gruvity Lfffects
46.1
36.01
45.0
50,0
50.0
Stapo Projecson
33.0
42.0
56.0
47.0
40.0
41.0
3D Geometry
Spatill Contaimnnnt
33,0
26.0
34,0
34.0
36,0
34.0
Subilit Prodiction
1095
14:11
90.0
$2.58
45:34
$1.08
Volum Compation
30.43
26,09
36.95
35.96
47.53
34.78
Complee
30.0
27:14
Dept Occlusion
Panial Occluioo
35,0
35,0
42.0
Refleztive Sarfaces
36,0
34.0
96.0
50.0
42.8
40.0
Trinppazscy Effects
25.0
42.0
35.0
40.0
32.0
46,0
Cectral Diirction
15.0
24.0
38.9
Facisg Direction
38.1
Orianbrrion
86
14.29
Handelness
40.0
Aligement Potteese
27.91
M0.75
19.55 93
46.19
41. 56
41 19
Betweencess Relicktships
35.3
38.3
46.91
65.96
46.51
53.15
Relative Proitiorrigg
Comzrikapi Angle Prailloning
Dirsctinal Relatiess
51.15
27.91
Procinity Urdbents
20M3
1M.53
37:21
Disoaas-Sia Sine Currlaatin
12.0
2010
46.0
56.0
58.0
36.0
Perspective Distatioon
25.0
40.0
42.3
58.0
44.0
44.0
Size and Scale
Rebaiie Stee Comparinen
34.0
32.0
46.0
58.0
5010
22.0
Scale Consistacy
27.45
35.29
41.18
51.37
27.45
41
Stadow Sizc Projection
66.67
39.32
47.06
47.06
Asczedivilty Cominintc
90.00
Obstacle Aorodance
24.0
36.0
Speiil Navigaius
Pallway Esiateras
34.78
30.45
12.1
34.78
34.78
36.26
Spatial Sequence
35.0
42.0
4000
56.0
130
44.0
Verpoiii Visiiliyy
35.0
34.0
35.0
00.8
36.0
26.0
reasoning is similarly weak, with only 50.0% achieved by InternVL3.5-4B and Gemma-3-4B- it. The within-mode spread often exceeds 60 percentage points, showing brittle generalization. These patterns suggest that smaller models are highly cue-dependent, performing well where visual templates are strong but failing in tasks requiring deeper spatial abstractions. We interpret this inconsistency as evidence that representation learning is uneven across sub-categories. Overall, we find that small models provide localized competence but lack robustness across the full spectrum of spatial reasoning.
Reasoning Models. Reasoning-enhancee models Table 9) show notable benefits in select tasks but remain inconsistent overall. 04-mini reaches 65.12% in Alignment Patterns and 56.0% in Gravity Effects, outperforming most open baselines. Gemini-2.0- 2.0iii--.--FFahhh-- F--laah-Thinking achieves 62.75% in Shadow-Sizz SSze Projection and 52.17% in Volume Comparison, demonstrating stronger geometric reasoning. The strongest results come from Gemmin-2.55-Flahh-TTin Complete Occlusion Inference and 70.21% in Betweenness Relationships. Yet the same family fails in others, such as 34.0% in Cardinal Direction and 34.0% in Shape Projection. This variance shows that reasoming mechanisms improve depth in certain sub-categories but do not broaden coverage. Within- model spreads frequently exceed 35 percentage points. We conclude that reasoning augmentation adds localized capacity but leaves structural gaps in spatial understanding. Overall, these models demonstrate promise but not generalizable competence.
which scores 71.79% in
Spatial Reasoning Models. Spatially specialized models Table 10 but lack generality. Sparee@wen255-V--B--nnnss
reveal targeted improvements achieves 48.0% in Pistance-Size Correlation
and 49.02% in Scale Consistency, clearly surpassing many open-source baselines in scale reasoning. The same model leads in navigation tasks, with 53.66% in Accessibility Constraints and 44.0% in Obstacle Avoidance. SpaceOm and Space Swwennkrrr-eee..SV--e. reach 50.0% in Stability Prediction, matching expectations for domain-specific reasoning. However, performance drops sharply in other tasks, such as only 24.0% for Spatial Containment. Rarely do these models exceed 55.0%, even in their strongest areas. The spread across sub-categories frequently surpasses 30 percentage points. We find that specialization drives narrow peaks but does not produce holistic competence. Our interpretation is that domain targeting enhances select abilities but fails to integrate them into a robust, general framework.
Table 10 Table 9). we observe consistent fragmentation. Small open-source models peak at 74.29% in Stacking Orientation but collapse to 12.0% in Cardinal Direction. Larger models raise ceilings, such as 80.56% in Corner/Angle Positioning, but still fail catastrophically with 2.0% in Object Rotation. Closed-source
Overall Discussion. Across all categories Table 8 Table 7
Table 6
30

---

Published as a conference paper at ICLR 2026
Table Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub- Sub-Categories for Reasoning Models. We bold and underline the best score within each model category.
Caeegury
Sub Category
of e4-mini
Gemin 2.00 P.a--lssh Thinking
Gemini 2.5-Flash ThinkingThiinnng
Kimi VL-A3B Thinking 2506
Gravity Fifretts
$6.0
38.0
46.0
50.0
Shape Projection
50.0
3600
40.0
34.0
3D Geometry
Sputial Containment
56.0
28.0
48.0
42.0
Stability Prediction
50.0
Volume Compurison
Coangltte Occlusion
71.79
Layering Order
57.14
3000
41.45
35.71
Depth and Occlusion
Partial Ouclusion
50.0
42.0
58.0
40.0
Reflective Surfaces
54.0
40.0
5200
42.0
Transpurnccy Effects
64.0
50.0
540
42.0
Cardinal Direction
54.0
50.0
Facing Direction
Orientation
Object Rotation
36.0
48.0
30.0
Stacking Orientation
71.43
57.14
57.14
65.71
Tool Hlandedness
56.0
$4.0
60.0
44.0
Alignment Patterns
65.12
41.86
44 19
34,88
Betweenness Rehaiimmipp
65.96
55.32
70.21
63.83
Relative Pusitioning
Coener/Angle Posltioning
55.56
Directional Relations
Proximity Gradients
51.16
32.56
41.5 1.85
48.84
Distance-Sire Conclation
4400
42.0
56.0
44.0
Perspective Distunton
40.0
44.0
50.0
36.0
Size and Scale
Relaive Size Compurison
46.0
46.0
58.0
32.0
Scale Consisency
22
56.86
52.94
43.14
Size Projection
Accessibility Constraines
53.05
Obstacle Avoicince
44.0
44.0
36.0
Sputial Navigation
Pathway Existence
45
41.3
47.83
34.78
Sputial Sequence
64.0
48.0
65.0
54.0
Viewpoinx Visibility
60.0
40.0
54.0
48.0
Table 10: Multiple-choice Evaluation Accuracy on SPATIALAB-M by Question Sub- Categories
for Spatial Reasoning Models. We bold and underline the best score within each model category.
Category
Sub-Category
SpaceOm
Spec/Thinkerr-Qwee2.WWW.. SpaceQwen2.55 5----r-n22.--------ssssu
Gravity Effects
48.0
46.0
32.0
Shape Projection
38.0
36.0
36.0
3D Geometry
Spatial Containment
40.0
36.0
24.0
Stability Prediction
50.0
Vulume Comparison
36.96
34.78
36.96
Complete Occlasion Inference
38.46
35,9
38.46
Layering Order
30.0
32,86
24.29
Depth and Ovclusion
Partial Occlasion
44.0
46.0
38.0
Reflective Surfaces
44.0
44.0
38.0
Transparency Effects
40.0
32.0
42.0
Cardinal Direction
38.0
Facing Dircction
$9.52
54.70 76
50.0
Orientation
Object Rotation
42,0
Stacking Orientation
62.86
62.86
40.0
Tool Handedness
40.0
Alignment Pattems
39.53
44.19
39.53
Betweenness Relationships
44.68
42.55
53.19
Relative Positioning
ComeerAAngl Positioning
41.67
38.89
Directional Relations
25.58
25.58
20.93
Proximity Gradients
Dislance Size Correlation
40.0
44.0
48.0
Perspective Distortion
40.0
38.0
42.0
Size and Scale
Relative Size Comparison
42.0
46.0
48.0
Scale Consistency
33.33
33.33
49.02
Shadow-Size Projection
58.82
54,9
66.67
Accessibility Constraints
48.78
46.34
$3,66
Obstacle Avoidance
32.0
Spatial Navigation
Patlway Existence
32.61
34.78
41.3
Spatial Sequence
44.0
Viewpoint Visibility
32.0
34.0
48.0
systems achieve the strongest scores, with GPT-4o-mini reaching 85.71% in Stacking Orientation, yet still fall to 32.0% in Relative Size Comparison. Specialized models show strengths in navigation (53.66%) and stability (50.0%) but remain capped below 55% overall. Reasoning models demonstrate gains, such as 71.79% in Complete Occlusion Inference, but others remain at 34.0%. Within-model spreads often exceed 50 percentage points, showing brittle internal consistency. Sub-categories
31

---

Published as a conference paper at ICLR 2026
Table 11: Distribution of question counts by the number of models (0-5) that correctly answered each item across categories and sub-categories in SPATIALAB-MCO "0" indicates no model answered correctly, while "5" indicates only five models succeeded to answer these among all models tested.
Number of Models Successfully Answered
Category
Sub-Category
3
Total
3D Geometry
Gravity Effects
9
7
3
2
1
25
Shape Projection
4
0
9
7
31
Spatial Containment
6
3
31
Stability Prediction
2
22
Volume Comparison
6
5
28
3D Geometry Total
30
33
19
21
20
14
137
Depth & Occlusion
Complete Occlusion Inference
0
6
17
Layering Order
9
7
8
5
43
Partial Occlusion
2
5
7
5
2
23
Reflective Surfaces
6
5
7
4
28
Transparency Effects
4
2
27
Depth Occlusion Total
21
20
30
23
138
Orientation
Cardinal Direction
1
10
7
4
32
Facing Direction
2
5
22
Object Rolation
3
Stacking Orientation
2
2
Tool Handedness
2
4
3
12
Orientation Total
11
20
29
20
13
11
104
Relative Positioning
Alignment Patterns
3
4
4
20
Betweenness Relationships
3
2
2
5
7
20
Comer/Angle Positioning
0
3
2
3
12
Directional Relations
1
2
13
Proximity Gradients
5
26
Relative Positioning Total
11
16
16
15
17
16
91
Size & Scale
Distance-Sine Correlation
6
1
26
Perspective Distortion
3
Relative Size Comparison
5
4
29
Scale Consistency
7
7
4
5
2
31
Shadow-Sin Projection
5
7
2
1
4
20
Size & Scale Total
31
13
137
Spatial Navigation
Accessibility Constraints
3
5
4
3
18
Obstacle Avoidance
2
9
6
10
35
Pathway Existence
2
10
4
1
4
24
Spatial Sequence
3
2
25
Viewpoint Visibility
2
9
Spatial Navigation Total
12
22
22
23
33
21
133
Grand Total
116
146
131
122
123
740
requiring physical abstraction, Gravity Effects, Stability Prediction, Spatial Containment, consistently score below 35-40% regardless of category. We attribute these failures to two root causes: the absence of stable encodings of orientation and physics, and an over-reliance on surface-level correlations. The wide variance, from 85.71% highs to 2.0% lows, suggests models are seatterr-maaching rather than reasoning robustly about space. Scaling, proprietary training, and reasoning augmentations each provide benefits but none produce general coverage. Instead, each approach yields isolated peaks that coexist with glaring blind spots. The collective picture is one of fragmented competence rather than integrated ability. We conclude that progress in spatial reasoning will require architectural changes aimed embedding consistent geometric grounding and physics-aware mechanisms. In sum, results show both remarkable selective gains and the persistence fundamental gaps holistic spatial intelligence.
32

---

Published as a conference paper at ICLR 2026
E.2 MODEL FAILURE ANALYSIS
The distribution across categories and sub-categories in Table reveals several notable patterns in model performance. Categories such as 3D Geometry and Size & Scale show relatively balanced distributions, yet both retain a significant proportion of failures where no model answered correctly. suggesting that even basic geometric invariances are not reliably internalized. In contrast, Depth & Occlusion exhibits a heavier tail in the mid-range (2-4 correct), pointing to partial but inconsistent reasoning about layered scenes. Orientation emerges as a weak category overall, with especially low totals in Stacking Orientation and Tool Handedness, which indicates that reasoning about subtle directional or embodied affordances is poorly captured. Relative Positioning presents a mixed picture: while some tasks like Proximity Gradients show moderate success, others such as Corner/ Angle Positioning have many near-zero correct counts, reflecting brittleness in compositional reasoning over reference frames. Interestingly, Spatial Navigation shows comparatively stronger performance, particularly in Obstacle Avoidance and Viewpoint Visibility, but still has significant errors in Pathway Existence where relational chaining is needed. When aggregated, no category achieves uniformly high accuracy, and the Grand Total shows an even spread across 0-5 bins, suggesting that success is highly task-dependent rather than reflecting a general spatial competence. Overall, models display isolated strengths but lack systematic reliability across spatial dimensions.
These results suggest that current vision-language models struggle with several root causes tied to representational and reasoning bottlenecks. High error rates in Stacking Orientation and Tool Handedness indicate limited grounding in physics-based and embodied reasoning, consistent with models trained primarily on static internet imagery rather than interactive task-driven data. The difficulty in Corner/Angle Positioning and Pathway Existence reflects inadequate multi-step relational chaining, pointing to weaknesses in their ability to handle recursive spatial logic. The inconsistency in Transparency Effects and Reflective Surfaces underscores the challenge of handling visual phenomena that require non-local cues, such secondary reflections occluded geometry. By contrast, compar- atively better performance in Obstacle Avoidance suggests that models may have learned shortcuts from dataset biases, such as frequent co-occurrence patterns of walkable paths, rather than genuine planning ability. Taken together, these failures highlight that model training regimes overemphasize correlation-driven vinual-language alignments while underrepresenting embodied, counterfactual, and physics- bessc-bsssed reasoning. Addressing these issues likely requires richer training distributions (e.g., embodied simulation data, physics-augmenteed learning) and architectural innovations that can explicitly model reference frames and relational constraints. Without such advances, models will continue to show brittle competence that breaks down under compositional or physically grounded tasks.
F ERROR ANALYSIS OF SPATIALAB-OPER
F.1 SUB- CUBB-CATGGRY-WOE WISE QUANTITATIVE ERROR ANALYSIS
Closed-source Models Table 12}. We observe that closed-source models achieve the strongest over- all performance in the open-ended evaluation, yet their results remain uneven across sub-categories. GPT-5-min is the clear leader, consistently securing the best scores in 3D Geometry and elative Positioning, with peaks of 58.14% in Directional Relations and 55.81 % in Alignment Patterns. In contrast, Gemini-2.55Flaas and Claude 3.5 Haiku register very weak results in Proximity Gradients (16.28%) and Perspective Distortion (12.0%). respectively. This wide gap highlights the fragility of even the strongest models when tasked with reasoning over subtle, continuous visual cues. We believe these disparities stem from the differential sophistication of models' spatial reasoning pipelines, where some architectures capture relational and geometric structure more effectively than others. Tasks such as Proximity Gradients are particularly challenging, as they require analogical, graded reasoning rather than discrete classification. The fact that GPT-5-mini succeeds where others fail suggests that scaling and proprietary training corpora confer distinct advantages. Nevertheless, even the leader shows vulnerabilities, pointing to a systemic issue: closed-source source dominance is relative, not absolute, and the models still fall short of robust, human-likk spatial reasoning.
Open-source Large Models Table 13). For large open-source models, we find a more distributed performance landscape, with different systems excelling in isolated sub-categorics. llama-3.2-4 3.2-90b- vision-instruct achieves 60.0% in Depth and Occlusion, while GL M.MM-.55-006--MM0 leads in Spatial
33

---

Published as a conference paper at ICLR 2026
Table 12: Open-ended Evaluation Accuracy on SPATIALAB-OPH by Question Sub-Categories for Closed-source Models. We bold and underline the best score within each model category.
Category
Sub-Category
GPT-S-mini
Gemini-2.0-Pash
Gemini-2255PPan
Clande 3.5 Haiku
Mistral Medium 3.1
Gravity Effecrss
18,0
36.01
30.0
24.0
20.0
Shape Projection
40.0
18.0
22.0
16.0
Geometry
Spatial Containment
36.0
Stahility Prediction
50.01
26.19
35.71
30.95
26. 19
Volume Comparison
39.13
413
32.61
23,91
28:26
Complete Occlusion Inference
46.15
35.9
38,46
28.21
25.64
Layering
[5.71
Depth and Ocelusion
Partial Occlusion
35,0
20.0
28.0
14.0
14.0
Reflective Surfaces
32,0
28.0
32,0
22,0
22.0
Transuuencyy Elfects
32,0
18.0
18.0
16.0
22.0
Cardinal Diroction
30.0
Facing Direction
28.57
28,57
26.19
19.05
14.29
Orientation
Objecl Rotation
42,0
14.0
28.0
30.0
14.0
Stacking Orientation
37.14
20.0
51,43
22,86
28.57
Tool Hundedness
48,0
24.0
Aligoment Pattens
34.88
Berweenness Relationships
48.94
23.4
31.91
23.4
21 11.28
Relative Positioning
Corner/Angle Positioning
$4.44
30.56
41.67
16.67
36.11
Directional Relations
58,14
37.21
Proximity Gradients
Distunce-Sizee Correlation
48.0
32.0
36.0
20.0
22.0
Perspective Distortion
45.0
24.0
22.0
20,0
12.0
Size and Scale
Relative Size Comparison
36.0
26.0
28.0
18.0
16.0
Consislency
9.8
Sludow- Size Projection
37.25
27.45
23.53
23.53
13:69
Accessbiiityy Constraints
41.46
31,71
39.02
24.39
21.95
Obstacle Avaidance
40.0
22.0
22,0
26.0
14.0
Spatial Navigation
Pathway Esistence
19.57
Sputial Sequence
$2,0
32.0
34.01
24.0
20.0
Viespoint Visibility
28.0
22.0
32.01
20.0
10.0
Table 13: Open-ended Evaluation Accuracy on SPATIALAB-OPER by Question Sub-Categories for Open-source Large Models. We bold and underline the best score within each model category.
Carpar
Carges
Earau
ojvra S5t. teathet
GLMMI Mall"
Qneitt Elide
22.0
12.5
21.3
26.0
25.0
Smpo Geeda
Coonaryy
351
180
177
Saiil Producion
16/2
31/1
36.15
38T
Complae aneeeee
Raraa Oader
15:1
820
14.09
15.11
1000
Dapt's Ouciodas
5.
303
15.0
The Ihropacci Bilice
19.0
140
312
22/
28,8
Cadiaal Disctes
525
36.
30.0
Racke Dioccis
3659
14.39
31-5
16.07
Sackig Oreeadion
12:14
23/11
1111
23:11
3771
$1.85
Algaasrr Penurn
2003
46.51
$1.33
Bokite Pookeees
OenanAngi Photieeng
22-16
26.11
11:11
20725
1115
16.52
Poolirly Ghadaen
03
116
9.3
185
15.5
24.55
Pengecto ticetio
10.
15.
93.
183
130
38.8
Sale Condirasg
11.36
11.7
300
1941
2151
11.55
Projetien
Acens by Coeansses
31:55
76.55
203
32.0
1037
21.24
633
17.50
1532
17.39
Veseeea Stilaity
150
15.0
120
123
268
116
Containment at 38.0%. Despite these highlights, several models collapse on Proximity Gradients, and qwen-2.5-w11722-innss
both scoring only 9.3%. This indicates that architectural scaling helps but does not guarantee coverage across difficult continuous-reasonie tasks. We hypothesize that much of this variation arises from heterogeneous training data: models exposed to richer depth and geometry corpora perform better in occlusion and containment tasks, while others generalize poorly. The persistent weakness on Proximity Gradients suggests that these systems are not optimized for subtle spatial analogies and distance scaling. We also observe higher variance in this group than among coosed-sourree models, underscoring the impact of open-source development heterogeneity. Overall, large open-source models show encouraging peaks but inconsistent coverage, suggesting that scale alone is insufficient without targeted pre- pre-training and fine-tuning.
with awwe-22.-vv1322-imm
34

---

Published as a conference paper at ICLR 2026
Table 14: Open-ended Evaluation Accuracy on SPATIALAB-OPER by Question Sub- Sub-Categories for Open-source Small Models. We bold and underline the best score within each model category.
Calaoey
Sub Cutegety
IntemVL35-18
CACD 2,5k1,5
Genuna
qwee 2.5-1.7b Pb-uutrrutt
Lfffects
4.1
6.0
14.0
16.0
22.0
Stapo Projecson
2.0
15.0
18.0
4.3
10.0
000
3D Geometry
Spatill Contaimnnnt
6.0
4.0
12.0
20,0
34,0
8.0
Subilit Prodiction
2:36
11.9
16.29
19.05
21
Volum Compation
15:22
32.61
28.26
30.43
3043
28.26
Complen
5.71
10.0
14:29
Depti Occlusion
Panial Occluioo
14.0
12.0
Refleztive Sarfaces
15.
16.0
6.0
14.0
15.0
26,8
Trinepazsyy Effects
13.0
20.0
4.0
25,0
14.0
X.0
Cectral Diirction
15.0
28,8
24.0
Facisg Direction
9.52
28.57
Orianbrrion
8.57
18:57
Handelness
8D
8.0
Aligement Potteese
15.95
22.91
12. 96
11.55
41.56
Betweencess Relicktships
12.77
17.02
0.0
19:15
17.00
25.53
Relative Proitiorring
Comzrikapi Angle Prailloning
2.78
19.44
Dirsctioal Relatiess
0.3
37.21
Precinity Urdbents
18.6
0.3
15.6
Disoaas-Sia Sine Currlaatin
600
10.0
14.0
16.0
20.0
13.0
Perspective Distationn
800
12.0
15.0
16.0
12.0
16.0
Size and Scale:
Rebaiie Stee Comparinen
10.0
16.0
26.0
24.0
10.0
16.0
Scale Consistacy
7.84
9.8
11.76
13.76
7.84
11.76
Stadow Projection
13.73
11.75
21.37
13.75
25.49
Asczedivilty Cominints
7.32
Obsnacle Aorodance
12.0
26.0
Speiia Navigaius
Pallway Esiateras
0.0
8.7
2.87
8.7
17.30
87
Spatill Sequence
120
22.0
80
26.0
2017
25.0
Verpoiti Visiiliyy
15.0
18.0
15.0
34.0
14.0
16.0
Open-source Small Models (Table 14). Small open-source models predictably lag behind their larger counterparts, but their results reveal noteworthy localized strengths. qwwen22.5-v133b-insttt achieves 18.0% in Perspective Distortion and 26.0% in Relative Size Comparison, while Gemma-3- 4B-it leads in Spatial Containment (24.0%) and Stability Prediction (21.43%). Yet these successes are offset by catastrophic failures: Intern InternVL3.5-2 2B scores 0.0% in Betweenness Relationships, and Intern V manages only 2.0% in Shape Projection. We attribute this extreme variability to limited capacity and restricted training diversity, which prevent small models from learning robust representations of geometric and relational principles. Sub-categories that demand precise reasoning about orientation and hierarchy trigger complete breakdowns, as small misinterpretations translate into total failure. Nevertheless, their ability to occasionally outperform larger models in narrow tasks suggests that even small-scale training can instill isolated competencies. This points to the value of specialization but also confirms the structural ceiling of compact architectures. In short, small open-source models demonstrate pockets of promise but remain highly brittle, with overall accuracy limited by size and data constraints.
Reasoning Models Table 15). Reasoning-oriented models present a sharper internal divide, with Semmni-2.5--lash-hhmme standing far above its peers. It records the best scores in nearly every sub-category, including Tool Handedness at 75.0% and Complete Occlusion Inference at 57. 17%. By contrast, Kimi- LL-A3B-Thhnning 2506 struggles dramatically, scoring just 6.0% in Spatial Containment and lagging behind across the board. We believe this gap arises from differences in how these models integrate multi-modal streams into coherent reasoning frameworks. Gemini-2.5- flash-thinking appears capable of forming and querying internal world models, enabling it to infer object interactions and physical propertics more effectively. In contrast, weaker models rely heavily on surface-level pattern matching, which proves inadequate for compositional causal reasoning. The consistently high performance of Gemini-2.5 across categories suggests that explicit reasoning mechanisms offer tangible benefits for spatial understanding. However, its failure to dominate Shape Projection and Betweenness Relationships shows that even advanced reasoning modules are not yet universally effective. Thus, while reasoning models hold promise, the category itself remains heterogeneous and far from solved.
Spatial Reasoning Models (Table 16). Spatial reasoning models, though explicitly designed for the task, record the lowest results overall. Performance is highly fragmented: several models tie at 16.0% in Gravity Effects, 8.0% in Shape Projection, and 14.29% in Stability Prediction. SpaceQwen2.55 VL- 3B-Instruct stands out slightly with 23.91% 1 in Volume Comparison and 13.73% in Scale Consistency, but these scores remain far below competitive thresholds. At the other extreme, Betweenness Relationships and Comer/AAgle Positioning collapse entirely, with models recording 0.0% and
35

---

Published as a conference paper at ICLR 2026
Table 15: Open-ended Evaluation Accuracy on SPATIALAB-OPEN by Question Sub-Categories for Reasoning Models. We bold and underline the best score within each model category.
Caregory
Sub-Category
gernini-2 Semini-2--daarh--hiiiknn
gemini-2---fast--iinnng
Kimi-VE--A38. -.L-AA--Thinking-- Trinkihi-i2i000
Step-3
Grvvity Fiftcts
32.0
40.0
14.0
38.0
Shape Projection
24.0
0.0
10.0
30,0
3D Goometry
Spatial Containment
36.0
30.0
Stailiity Prediction
26.
42.86
14.29
$3.33
Volume Comparison
36.96
14.44
23.91
30.43
Complete Occlasion Inference
33.33
57.14
17.95
33.33
Layering
Deyth and Ocelusion
Partial Occlusion
28.0
50.0
8.0
14.0
Reflective Surfaces
34.0
63.64
10.0
16.0
Trassurency Effects
26.0
57.14
22.0
20.0
Facing Direction
35.71
37.5
14.29
28.57
Orientation
Object Rotation
18.0
40.0
4.0
22.0
Stacking Orientation
22.86
0.0
5.71
34.29
Tool
Alignment Pattems
16.31
66.67
25.58
41.86
Beiweenness Relationships
27.66
20.0
12.77
25.53
Relative Positioning
Cornes/Angh Positioning
36.11
50,0
11.11
11.11
Dinectional
Proximily Gradients
23.26
16.67
11.63
18.6
Distance Size Cornelatinn
32.0
57.14
14,0
22.0
Penseetivee Distertion
18.0
11:11
10.0
30,0
Size and Scale
Relarive Size Comparison
28.0
36.0
Scale Consisenccy
33.33
20.0
11.76
31.37
Shudow-Sie Size Projection
35.29
18.18
13.73
29.41
Accessibiiity Constraints
31.71
45.45
17,07
41,46
Obsttele Avoidance
26.0
18.0
Spatial Navigation
Pathway Existence
Sputial Sequence
34.0
12.5
30.0
36.01
Vicupoint Visibiliry
31.0
12.5
16.0
26.0
2.78%, respectively. These results suggest that the models' smaller capacity and narrowly focused training severely constrain their representational flexibility. We interpret the failures as evidence that narrow specialization without sufficient capacity or data breadth cannot yield generalizable spatial reasoning. Particularly on geometric and positional tasks, even slight representational errors lead to total breakdowns. Consequently, despite their focus, spatial reasoning models are least effective, underscoring the necessity of both scale and diversity in achieving robust performance.
Overall Discussion. Across all categories, closed-source TTblle 12p, open-source large Table open-source small (Table 14), reasoning TTble 15), and spatial reasoning Tabll 16). we find consistent pattern fragmented competence and systemic weaknesses. Closed-source models perform best overall, with GPT-5-mini achieving highs of 58.145 58.14% in Directional Relations and 55.81% in Alignment Patterns, yet even they collapse on Proximity Gradients. Large open-source models show impressive peaks, such 60.0% in Depth and Occlusion, but still fail at just 9.3% in Proximity Gradients, illustrating that scale without specialization remains insufficient. Small open-source models confirm this further, with rare localized strengths (e.g., 26.0% in Relative Size Comparison) but catastrophic lows like 0.0% in Betweenness Relationships. Reasoning models demonstrate that explicit reasoning mechanisms can significantly improve results, with Gemini- 2.5-flas--thinkinn reaching 75.0% in Tool Handedness, yet peers like Kimi-VL-A3BB fall to single digits, exposing the fragility less-integrated reasoning frameworks. Spatial reasoning models, despite their specialization, underperform the most, with scores often below 15% and total failures on Betweenness Relationships. Taken together, these results highlight three root causes: (1) a lack of robust representations for continuous and analogical spatial cues, (2) structural limitations tied to scale and training diversity, and (3) brittle reasoning pipelines that prevent models from generalizing across tasks. We conclude that no current approach, whether proprietary scaling, open-source expansion, reasoning integration, or specialization, delivers comprehensive coverage. The frequent performance gaps exceeding 50 percentage points within single model further confirm that progress piecemeal. Future work must thus focus on unifying robust geometric encodings, physics-infeee informed reasoning, and cross-task transfer, moving beyond isolated gains achieve consistent spatial intelligence.
36

---

Published as a conference paper at ICLR 2026
Table 16: Open-ended Evaluation Accuracy on SPATIALAB-OPER by Question Sub-Categories for
Spatial Reasoning Models. We bold and underline the best score within each model category.
Category
Sub-Category
SpaceOm
Sine/Thiinkrr-Qeeen22VWWW..
Gravity Effects
14.0
12.0
16.0
Shape Projection
8,0
3D Geometry
Spatial Containment
8.0
10.0
10.0
Stability Prediction
14.29
14.29
7.14
Volumee Comparison
19.57
23.91
23.91
Complete Occlusion Inference
7.69
10.26
7.69
Layering Orler
8.57
11.43
4.29
Depth and Occlusion
Partial Occlusion
12.0
Reflective Surfaces
4,0
4.0
2.0
Transparncy Eiflects
8.0
2.0
Cardinal Direction
Facing Direction
23.81
23.81
16.67
Orientation
Object Rotation
Stacking Orientation
8.57
8.57
17.14
Tool Handodness
4.0
Alignment Pattems
32.56
32.56
23.26
Betweenness Relationships
4.26
2.13
0.0
Relative Positioning
ComnerAAnle Positioning
2.78
2.78
2.78
Directional Relations
11.63
6.98
11.63
Proxinity Gradients
Distance- Size Correlation
18.0
18.0
16.0
Perspective Distertion
20.0
Size and Scale
Relative Size Comparison
22.0
20.0
8.0
Scale Consistency
11.76
13.73
Shadow-Size Projection
27.45
27.45
19.61
Accessibility Constraints
7.32
7.32
9.76
Obstacle Avoidance
8.0
Spatial Navigation
Pathway Existence
6.52
6,52
8.7
Spatial Sequence
12.0
Viewpoint Visibility
20.0
16.0
8.0
F.2
MODEL FAILURE ANALYSIS
The distribution in Table 17 highlights persistent weaknesses categories, with notable skew toward the lower bins (0- models correct). 3D Geometry broadly challenging, with Shape Projection (19 items with zero correct) and Stability Prediction (10 items with zero correct) showing especially poor outcomes, suggesting models lack reliable physical and geometric reasoning. Depth and Occlusion the hardest category overall, accumulating the highest zero-correct counts (72), particularly in Layering Order (20 zero) and Transparency Effects (16 zero), both requiring nuanced handling of occlusion and non-local cues. Orientation exhibits similar struggles: while Cardinal Direction achieves some mid-range success, Stacking Orientation (1 zero) and Tool Handedness (5 zero, none with perfect success) underscore the brittleness of embodied directional reasoning. Relative Positioning reveals widespread inconsistency, with tasks such Betweenness Relationships and Proximity Gradients rarely solved by more than two or three models, while Corner/Angle Positioning shows low but somewhat balanced performance. Size and Scale appears relatively stronger, with multiple subcategories (e.g., Scale Consistency with 6 full successes) showing that some models internalize coarse perspective and scaling cues, though failures remain high Shadow- Size Projection (17 zero). Spatial Navigation is split: while Spatial Sequence and Obstacle Avoidance show occasional full-model success, Pathway Existence (16 zero) and Accessibility Constraints (10 zero) highlight persistent difficulty in multi-step relational chaining. Overall, the grand total reveals that across 356 items, zero-correct dominates (356), while only 74 items are solved by all five models. confirming that consistent spatial competence remains elusive.
These findings indicate that the root causes of failure stem from dataset design and more from the representational and reasoning limitations of current models. First, the dominance of errors in Depth and Occlusion reflects a reliance on superficial appearance features rather than structured scene representations: for example, models may latch onto surface textures rather than reasoning about hidden layers, leading to systematic errors Layering Order. Failures in Orientation tasks such as Tool Handedness expose the absence of embodied affordance modeling, since pretrained internet imagery often lacks explicit annotations of left/right handedness or stacking physics. The brittleness in Relative Positioning and Spatial Navigation points to deficiencies in chaining multi-hop spatial
37

---

Published as a conference paper at ICLR 2026
Table 17: Distribution of question counts by the number of models (0-5) that correctly answered each item across categories and sub-categories in SPATIALAB-OPER "0" indicates no model answered correctly, while "5" indicates only five models succeeded to answer these among all models tested.
Number of Models Successfully Answered
Category
Sub-Category
4
3D Geometry
Gravity Effects
13
1
Shape Projection
19
Spatial Containment
10
3
2
3
Stability Prediction
10
7
0
2
Volume Comparison
11
4
2
3D Geometry Total
63
22
23
18
12
10
Depth & Occlusion
Complete Occlusion Inference
5
Layering Order
20
3
Partial Occlusion
14
4
2
Reflective Surfaces
15
4
2
5
Transparency Effects
16
1
Depth & Occlusion Total
72
38
27
21
15
14
Orientation
Cardinal Direction
5
Facing Direction
13
3
2
Object Rotation
12
10
1
2
Stacking Orientation
11
1
Tool Handedness
2
0
Orientation Total
49
34
13
12
12
7
Relative Positioning
Alignment Patterns
Betweenness Relationships
11
1
3
Corner/Angle Positioning
7
2
3
2
Directional Relations
2
0
Proximity Gradients
10
10
2
Relative Positioning Total
40
30
19
11
9
10
Size & Scale
Distance- Siience--izzeeee Correlation
1
Perspective Distortion
12
5
2
Relative Size Comparison
14
4
2
4
Scale Consistency
14
3
6
Shadow-Siz -ize Projection
Size & Scale Total
64
29
28
21
14
19
Spatial Navigation
Accessibility Constraints
10
1
Obstacle Avoidance
15
8
2
5
Pathway Existence
16
1
Spatial Sequence
14
3
Viewpoint Visibility
Spatial Navigation Total
68
22
31
13
10
14
Grand Total
356
175
141
96
72
74
relations; for instance, Pathway Existence requires integrating local connectivity with global layout, which is not well captured by attention-basee co-occurrence priors. On the other hand, partial success Size and Scale suggests that models exploit statistical regularities perspective distortion or scale consistency, though they collapse under less frequent cases like shadows. These patterns collectively show that current models succeed when cues are salient and correlate with training priors but fail when reasoning requires counterfactual, compositional, issues will require architectural advances that explicitly model reference frames, occlusion layers, and physical dynamics, well training data enriched with embodied simulations and counterfactual augmentations. Without these, progress in spatial reasoning will remain fragmented, with models performing well surface-level cues but breaking down structured, real-wor cal-world spatial logic.
embodied inference. Addressing these
38

---

Published as conference paper at ICLR 2026
Table 18: Performance gap between MCQ and Open-ended (%) (L) across spatial categories on SPATIALAB. A lower value indicates that model performs comparably in both setups, whereas higher value reflects greater disparity. We bold and underline the highest gap within each model group.
3D Geom. (4238)
Dep. & Ocen. (#259)
Orientation (#202)
Relat. Posit. (#212)
Size & Scale (#252)
Spati. Navig (#237)
Overall (#1400)
Model
Proprictary Models
GPT-to-mini
22.40
23.76
16.51
31.74
27.85
20.50
23.53
GPT-5-mini
20.08
23.27
13.21
2.38
19.41
13.36
3.36
Gemüni-2 2.0-Fash
30.89
26.73
26.89
28.18
22.37
25.07
15.13
Gemini-2 S-Flash
21 6.62
16.34
17.45
15.87
21.51
17.36
10.93
Claude Haiku
21.78
15.47
20.29
16.39
Mistral Medium 3.1
30.50
25.74
32.54
26.59
24.89
26.93
21.43
Open-Source Models (Small)
22.78
13.37
23.58
22.62
20.67
22.00
27.73
Ln35mVL3
20:46
20.79
16.99
20.64
14.35
19.21
21.85
Qwen- VL2t5-3B-nnstrcct
27.03
30.69
29.24
28.97
29.96
28,50
25.63
In35mVVL3.5-B
25.10
26.24
34,91
20.24
23.20
25.
23,53
Gemma-3-4BBii
21.23
31.68
22.17
22.22
18.14
22.93
23.53
Qwen- Dwon-WL22.5-7B-Inntte
22.01
22.27
18.40
26.19
15.61
22.14
27.73
LJama-3 3..e--1.21111 [B- La-B--Viioon-nstrree
13.51
-1.98
17.92
17.07
13,50
11.93
9.66
Gemma-3 3r2m-a
Qwen-V Secn-VV1..5-32-Innseee
30.50
31.68
28.30
35.32
28.27
29.85
24.79
Intemm113.557221
Qwen VL2.5-72B- Instrucct
27.80
26.73
23.59
19.05
28.27
24.22
20.17
Vision
Reasoning Models
o4-mini
22.77
10.50
Gemini 2 Flaah-Thhinning
13.90
10.39
11.32
21.03
13.50
13.00
6.73
Sommini-.5-Flash-Thiiiee
Kimi VL- VL--A3B-Tinnking Thinking 2506
27.02
28.71
33.50
26.98
22.78
27.92
29.41
Sparial Reasoning Models
SpaceOm
31,66
32.18
25.95
24.21
27.00
28.43
29.83
Space Ohinker-Onnn.2..5 5VL-3B
29.21
27.83
23.81
27.84
27.
26.89
SpaceQwen2.55 V. -L.-33R-Tnstras
31.28
23.76
28.31
38.89
35.87
29.78
18.90
G ANALYSIS OF MCQ VS. OPEN-ENDEL PERFORMANCE GAPS
In this section provide detailed analysis of the performance gap between multiple-choice (MCQ) and open-ended evaluation formats across spatial reasoning categories SPATIALAB, as demonstrated in Table 18
G.1 QUANTITATIVE SNAPSHOT
Across 25 models, the average performance gap between MCQ and open-ended formats is 23.0 % with standard deviation of 5.5%. Dataset-wide means per subtask (24.57), depth & occlusion (3..4)), orientation (23.70), relative positioning (23.11), size & scale (23.42), and spatial navigation (22.89), yielding an overall average of 23.01. The overall gap ranges from minimum of 11.93 to a maximum of 31 31.57 (median 2 23.34). ) At the family level, spatial reasoning models exhibit the largest average gap (27.03), while reasoning oriented models show the smallest (19.11). Proprietary models have lower gaps on average (21.09) compared to many long open-source models (24.51). Notably, the human baseline shows an overall gap of 22.64. The highest gaps are observed in Intern VL.3.551....5722 (31.57%), Qwen-V1.2.5 V1.2.5-322 (29.85%). and SpaceQwen2.55 VL--3B-Instruet (29.78%). The lowest belong to L.lam-3.2211BB (11 .39%), Gemini-2-Flashe --Faah--Thinking (13.00%), and GPT-5-mini (13.36) Correlational analysis reveals that spatial navigation dominates the overall gap (Pearson r = 0.99), with orientation (r 0.83) and 3D geometry (r = 0.79) also showing strong alignment.
follows: 3D geometry
marienen--oreented
G.2 PATTERNS AND OBSERVATIONS
Three broad patterns emerge. First, specialist spatial reasoning models exhibit the largest MCC-~xppe-endee gaps, suggesting they optimized for classification-style supervision and less robust to free-form generation. By contrast, reasoning-focusee models display smaller gaps, reflecting the stabilizing effect of chain-of-thought-styl training generative outputs. Second,
39

---

Published as a conference paper at ICLR 2026
spatial navigation is the strongest predictor of overall disparity: models that falter open endeeddddd navigation reasoning drive the aggregate gap. Orientation and 3D geometry subtasks also contribute significantly, while relative position and size/scale vary more idiosyncratically across models. Third, scale alone does not explain robustness: smaller models such as Llama-3.2-1 1B achieve among the lowest gaps, while very large models such Intern VL3.5-72B show the highest. Finally, several negative or near-zero gaps appear (c.g., depth & occlusion for Llama-3.2-11 3.2-- at - 1.99,, relative position for 04-mini at -3.18), indicating that some cases open-ended evaluation more reliable than MCQ, likely due to misleading distractors.
G.3 HYPOTHESES ON ROOT CAUSES
We hypothesize several interacting factors that underlie the observed MICQ-oope-eended performance gaps in SPATIALAB, each supported by statistical observations:
1. MCQ structural advantage. MCQ format constrains the output space, allowing models exploit surface cues or eliminate distractors. Across models, the mean MCQ score is systematically higher than the open-ended score, with the average gap at 23.0% and σ 5.5%. Negative gaps observed for Llama-3.2- 1BB in depth & occlusion (-1.98) and o4-mini in relative position (-3.18) indicate that open- open-ended generation can occasionally outperform MCQ when distractors are misleading, reinforcing the notion that MCQ can artificially inflate or distort measured performance.
Recommendation: To address this issue, we suggest complementing MCQ evaluations with open-ended tasks and subtask-level diagnostics, which can more accurately capture model competence and reduce overestimation caused by option-based cues.
2. Specialization bias in spatial reasoning models. Spatial reasoning models exhibit the largest average gaps (27.03%) despite being optimized for spatial tasks, suggesting that they may have been trained primarily on categorical synthetic selection tasks rather than free- form generation. For instance, VnterrV133.57222 a 72B parameter spatial reasoning model, shows the highest overall gap (31.57%) particularly in spatial navigation (36.68%) and orientation (34,44%), highlighting a reliance on classification-lik supervision. Correlations between subtask gaps and overall gap (SpNav: r 0.99, Orientation: r 0.83) indicate that these format -eeead-de failures are concentrated in sequential and geometric reasoning tasks.
Recommendation: To mitigate this bias, we recommend fine-tuning or instruction tuning on open-ended spatial generation tasks, which can improve model robustness to generative formats and reduce reliance on categorical supervision.
3. Instruction-ttning and stepwise decoding. drasoning--rientee models (e.g., Gemini-2- Flash-Thinking 04-mini) demonstrate smaller average gaps (around 19. 1911%) and lower variance across subtasks. For example,
has an overall gap of 13.0% and consistently modest subtask gaps (Difference in 3D Geometry: 13.9%. Difference Relative Positioning: 21.03%), suggesting that bhain-of-thouggee or stepwise decoding helps models produce stable open-ended responses. This is further supported by reduced correlation between subtask gap variance and overall gap within this family, implying that instruction-tumie reasoning mitigates format sensitivity.
Recommendation: To address format sensitivity, we suggest adopting chain-of-thought or stepwise decoding strategies and instruction-tunm -uning for open-ended spatial tasks, which can improve generative consistency and reduce MCQ dependence.
4. Spatio-navigation stresses sequential grounding. Spatial navigation gaps are the strongest predictor of overall MCQ-+open-ended disparity (Pearson r = 0.99). Models with high performance gaps in spatial navigation, such as SpaceQwen2.5.v... V-----------------tt-rr
(Spatial Nav- igation gap = 35.87%). also exhibit large overall gaps (29.78%) indicating that multi-step referential reasoning is disproportionately affected in open-ended formats. This suggests that MCQ options simplify the reasoning chain, masking underlying weaknesses in sequential grounding and relational computation.
Recommendation: To better capture sequential reasoning abilities, we recommend including dedicated open open-ended supervision, which can improve grounding and reduce overestimation by MCQ.
tasks and employing targeted multi-step reasoning
40

---

Published as a conference paper at ICLR 2026
5. MCQ distractor artifacts. Some negative distractors can suppress measured MCQ performance, For example, 04-mini shows a negative relative position (-3.18% while maintaining positive gaps in other subtasks, implying that distractor quality can artificially depress MCQ accuracy. Conversely, models like Qwen- 11.25-332B-Instrue exhibit very high MCQ->oper gaps (overall 29.85%) across multiple subtasks, suggesting that MCQ distractors can also exaggerate perceived competence selection-based tasks.
gaps indicate that poorly designed
Recommendation: To address this, we suggest careful auditing and refinement of MCQ distractors and complementing them with open-ended evaluation, which can yield more accurate and unbiased measures of model spatial reasoning ability.
G.4 PRACTICAL IMPLICATIONS
Our findings show the importance reporting both MCQ and open-ended results. Exclusive reliance on MCQ inflates estimates of practical spatial competence, with average disparities around 23%. Subtask-level breakdowns are essential, as spatial navigation disproportionately drives observed differences. Diagnostic evaluations that disentangle classification versus generative reasoning can better isolate error modes. Moreover, instruction-tuning and decoding strategies that encourage stepwise reasoning appear promising for reducing MCQ dependence. Finally, careful design and auditing of MCQ distractors are necessary to ensure fair comparisons across formats.
H DETAILS ON IMPROVING VISUAL REASONING CAPABILITIES
To enhance spatial reasoning performance on our diverse visual question benchmark, we explored multiple complementary strategies.
H.1 INHERENT REASONING MECHANISMS
We next examine the role of built-in reasoning mechanisms by comparing standard models with their geasoning-enable counterparts Table 2 Table 3). Across MCQ evaluations, reasoning consistently boosts performance, though the magnitude varies by category. For instance, Gemini-2.5-Fia Thinking improves
its non-reasoning variant by +4.64% overall (52.93% 48.29%), with particularly strong gains in Relative Positioning (+13.1%) and Orientation (+8.6%). A similar trend holds for 14-mini-medium. which reaches the highest overall MCQ accuracy (53.21%) suggesting that explicit reasoning scaffolds are especially effective structured, discrete-answer settings.
Open-ender evaluation reveals more nuanced picture. Here, reasoning reasoning-equippeed models again outperform their baselines, but the gains are uneven. Gemini-2.55Flash-Tie Gemini-2.5-Flash (32.77% vs. 30.93%), but the largest leap is observed in 04-mini-medium (37.86%), which significantly surpasses all non-reasoning models. These results indicate that reasoning mechanisms help mitigate the instability seen in free-form generation, especially in complex categories such as 3D Geometry and Spatial Navigation. However, gains are not universal, reasoning models sometimes show regressions, e.g.. Gemini-2.55Flass 1-----Fa--Thinkkee Thinking drops sharply in Size & Scale (21.74% vs. 26.59%), echoing our earlier finding that generative settings amplify brittleness.
improves slightly
Overall, these comparisons highlight that explicit reasoning modules substantially enhance structured MCQ performance and can stabilize open-ended responses, but their benefits are uneven across categories. The results suggest that reasoning mechanisms strengthen logical consistency and relational chaining, yet they do not fully resolve the challenges of scale sensitivity, occlusion, and free-form linguistic grounding.
H.2 CHAIN-OF-THOUGHT (CoT) PROMPTING
To examine whether structured reasoning improves spatial performance, further evaluated models under Mfain-of-Thousee (CoT) prompting. For each of the 30 sub-categories SPATIALAB, we sampled 5 items (seed fixed to 42) construct balanced test set covering all major spatial phenomena. This design ensures fair coverage while keeping evaluation computationally tractable. Example CoT prompts are shown in Figure which illustrate the step-by-step reasoning scaffolds provided to
41

---

Published as a conference paper at ICLR 2026
SpatiaLab-MCO. CoT: Prompt for Answering Model
You are spatial reasoning assistant. Given nultiple choice question about an image, step by step answer single number (1-4) corresponding to the option exactly one fresutt>..</resuuul tag.
Question:
(question)
Options:
(ept1)
2. (apt2)
(opt3)
4. (opt4)
Final answer fornat: <resultsik<ressutt>
SpatlaLab-Open: CoT Prompt for Answering Model
You are spatial reasoning assistant.
nultiple choice question about inage, step by step Following question
Question: (question)
Figure 7: Chain-of-thought (CoT) Prompts.
the models. The setup allows to isolate the contribution of CoT reasoning from model scale training.
Table 19: Performance with CoT prompting across categories for InternvVI- 3 3-78b and Gemini-2.. Fasshsh Values are accuracies (%) and Gains represent the difference between with and without CoT prompting.
InterneVL.3-790 (MCQ)
InterneVL.3372.
Interne VL.3-78h (Open)
Gemini-1.5-Flash (MCQ)
Gemini-2.5.Faash (Open)
Category
w/o CoT
with CoT
Gain
wio
with Col
Gain
w/o CoT
with CoT
Gzan
wio
with CoT
Guin
3D Geometry
52,00
52.00
0.00
28.00
28.00
0.00
48.00
20.00
-25.00
60.00
26.00
-32.00
Depth Occlusion
Oricatation
60.00
64.00
4,00
24.00
28.00
4.00
24.00
-32.00
0.00
32,00
Relative Positioning
M0.00
72.00
-8.00
44,00
56.00
12.00
61.00
36.00
-24.00
50.00
32.00
-18.00
Size Scale
56,00
11.00
-12.00
28.00
20.00
-8.00
48.00
60.00
12.00
25.00
8.00
-17.00
Spotial Navignion
Overall
62.00
59.33
-2.67
31.33
30.67
41.66
54.67
36.67
-18.00
38.24
24.00
-14.24
Our analysis Table 19) shows that orientation is the only category where CoT prompting reliably improves performance, while most other categories experience either stagnation or decline. This suggests that reasoning helps when the task is reducible to logical alignment of directions, but fails when more complex perceptual priors (e.g., depth, scale, relative positioning) are required. We anticipate this happens because models lack inherent knowledge about 3D perception and geospatial relationships; they are not fundamentally "aware" of these concepts in the way humans are. As result, when CoT prompting is applied, the step-by-step reasoning does not correct misconceptions but instead reinforces them, propagating and amplifying errors across reasoning steps.
This finding highlights that our benchmark is non-trivial: cannot be solved merely by adding multi-step reasoning to existing models. Unlike textual reasoning tasks where CoT reliably improves outcomes, our benchmark exposes deeper architectural limitations in perceptual and spatial grounding. In categories like depth and size, incorrect priors mean that every additional reasoning step moves the model further from the correct solution. The sharp performance drops, particularly for Gemini in MCQ settings, demonstrate how fragile reasoning becomes when it is not supported by robust perceptual knowledge.
Thus, our task design is strength, it goes beyond surface-level reasoning and probes whether models possess a genuine understanding of geospatial and 3D relational concepts. Solving it requires more than chaining logic; demands embeddings and training that capture aspects of perception and spatial cognition. In this way, our benchmark challenges models at a fundamental level and highlights an important gap in current architectures.
42

---

Published as a conference paper at ICLR 2026
Spatialab- aMbb CaT Self-Refiection Prompt for Answering Model Selff-Reflection)
You are a spatial reasoning assistant. Review your previous reasoning and response below for the given question and options. If there step-by-step reasoning and the final answer in exactly one <result>..<<rreults tag. If there are no mistakes, repeat reasoning and confirn the final answer.
nistakes reasoning in the chosen option, correct then. Provide corrected
Question: (question)
Options: (opt1) (opt2) (opt3) (opt4)
Previous model output:
(prev_output) Provide corrected reasoning followed the final answer tag, and nothing else.
SpatlaLab-Open Self-Refiection Prompt Answering Model (Primary Ane)
You are spatial reasoning assistant.
Given multiple choice question about an inage, think step by step and answen
Provide your reasoning in 2-4 sentences, then provide the final answer on a separate 1ine like this:
Reasoning: <your reasoning here> Answer: syour answer here>
Question: (question)
SpatiaLab-Open Self-Refiection Prompt Answering (Self Reflection Ans)
spatial reasoning assistant and strict self-reflector.
You will receive the question, the nodel's reasoning, and the nodel's answer,
Check the reasoning and
for any mistakes. If mistakes exist, correct then.
Provide the corrected reasoning and answer in the same fornat:
Reasoning:
ccorrected reasoning Answer: <corrected answer here>
Question: (question)
Model reasoning:
(heason)
Model answer: (answer)
Figure Chain-of-thought (CoT) with Self-Reflection Prompts.
H.3 CHAI--OF-THOUGHT (COT) PROMPTING WITH SELF-REFIECTTIO
We further extend the CoT setup by adding a self-reflection stage, where the model prompted to review and revise its initial reasoning before finalizing answer (see Figure 8 The evaluation protocol remains identical to the standard CoT setting, using 5 balanced samples per sub-category (seed 42). This design isolates the effect of self-reflection while keeping the comparison directly aligned with baseline CoT performance.
Table 20: Performance with CoT CoT Self-Reflection (CoT-SR) across categories for InternvVL-33 3-b and Gemini-2 (in.ii..-FFeee. 2-5F-h Values are accuracies (%) and Gains repre- sent the difference between CoT-SR and CoT.
Category
VL-3-78h (MCQ)
VL-3-78b
(Open
Geminii22.-PPaahh (MCQ)
Gemini-2.5-PFlash (Open)
Coll
Col-SR
Gain
Coll
CoI-SR
Gain
CoT
CoT-SR
Gain
Col
CoT-SR
Gain
Geometry
20.00
32.00
Depih & Occlusion
60.00
44.00
-16.00
12.00
16.00
4,00
28.00
52.00
24.00
16.00
16.00
0.00
Orientation
Relative Positioning
72.00
64.00
-8.00
56.00
28.00
-2.000
36.00
72.00
36.00
32.00
32.00
0.00
Size & Scale
56.00
20.00
12.00
Spatial Navigation
64.00
56.00
-8.00
40.00
24.00
-16.00
52.00
64.00
12.00
28.00
28.00
0.00
Overall
59.33
56.67
30.67
22.00
20.66
Adding self-reflection to Chain-of- Thought (CoT) prompting produced mixed and highly model- dependent outcomes. For Interny VI--3-78b, overall accuracy declined slightly (-2.66% MCQ.-8.67% -8.67%
43

---

Published as a conference paper at ICLR 2026
open-ended), with sharp drops in categories like depth, relative positioning, and navigation, suggesting that reflection often reinforced errors instead of correcting them. Some localized gains were observed, such +12% in Size & Scale (MCQ), but these were inconsistent and offset by large degradations (-28% in relative positioning open-ended). In contrast, Gemini-2.5-Flash showed strong MCQ gains (+20.66%), with particularly large improvements in 3D Geometry (+32%) and Depth & Occlusion (+24%), though open-ended improvements remained negligible (+0.67%). These results highlight key asymmetry: reflection helps in multiple-choice contexts where models can prune unlikely options, but does not transfer to free-form generation where uncertainty propagates into ungrounded rationales. We also find that the categories most improved by reflection are those where geometric consistency is central (orientation, depth, size), while tasks requiring relational integration across multiple objects (relative positioning, navigation) degrade. Overall, reflection functions as stabilizer in structured choice settings but fails to reliably enhance naturalistic reasoning.
This observed inconsistency mirrors our earlier findings with plain CoT prompting: reasoning enhancements only succeed when the model has stable perceptual anchors ground its steps. Self- reflection adds an explicit verification pass, but this mechanism is only as strong as the model's internal priors. When depth cues, occlusion ordering, coordinate frames are poorly encoded, the second pass amplifies misconceptions rather than repairing them. This explains why Gemini, which exhibits stronger perceptual grounding, benefits under MCQ conditions, while suffers more frequent degradations. The broader root cause lies in a pretraining mismatch: current VLMs import reasoning templates from language modeling but lack the embedded 3D and geospatial representations that humans leverage. Without these anchors, reflection cannot perform true error correction; instead, functions as a linguistic filter that polishes reasoning syntax without improving perceptual grounding. As a result, reflective reasoning loops provide modest utility in discrete answer formats but collapse under open-ended demands where factual verification is absent.
These findings confirm that our benchmark shows fundamental gap in current architectures: surface- level reasoning layers alone cannot compensate for missing spatial cognition, and meaningful progress will require deeper integration of geometric and perceptual structure. Moreover, because VL.MMs are commonly trained on web data, the concepts that are well-represented online are those where models tend to perform better, whereas poorly represented spatial and 3D concepts suffer, reinforcing these performance disparities.
H.4 SUPERVISED TUN-TTNNIG (SFT)
H.4.1 SETUP AND IMPLEMENTATION
We fine-tuned the Qwen 2.5 VL 3B instruction-following model Sonsloth//wenng
VVV--3B-Inttructt using a parameter-efficient approach with LoRA (Hu et al. 2021) To accommodate limited GPU resources, we loaded the model in 4-bit precision [load.in.bbitTTen) and inserted LoRA adapters into attention and feed-forward network projection layers. The LoRA hyperparameters, including rank, scaling factor, dropout, and target modules, are sunmarized in Table Gradient checkpointing was enabled to reduce memory usage, and fixed random seed ensured reproducibility.
For tokenization, we followed the Qwen-2.5 chat template to maintain consistent formatting between training and inference. Each training example converted into a single text sequence, and dynamic padding was applied with lat.aoollooooooose222e. (per-device batch size with gradient accumulation of 4). Trainer and optimization hyperparameters. including learning rate, optimizer, weight decay, and scheduler, are listed in Table [2] The training loop consisted of four passes, each spanning a single epoch, yielding four saved checkpoints.
orroorreeeeee,, Training used an effective batch size of 4
Overall, this setup allowed us to efficiently fine-tune a large vision-language model on limited GPU resources while maintaining high adaptation capacity.
H.4.2 SUPERVISED FINE- TINE-TNNIGG (SFT) PERFORMANCE
The learning trends in Figure 4a reveal notable differences between the MCQ and Open-ender evaluation modes over training epochs. We observe that MCQ accuracy begins a lower baseline of 25.63% but rises consistently, converging near 36.59% by the fourth epoch. This steady improvement suggests that the model benefits strongly from additional supervised fine-tuning, particularly in tasks
44

---

Published as a conference paper at ICLR 2026
Table 21: LoRA Hyperparameters for SFT
Hyperparameter
Value
Notes
LoRA rank (r)
16
Low-rank update capacity
LoRA alpha
16
Scaling factor
LoRA dropout
No dropout applied
Gradient checkpointing
Enabled
Library-specifie flag
LoRA random seed
3407
Adapter initialization
Table 22: Trainer and Optimization Hyperparameters for SFT
Hyperparameter
Value
Base model
unsloth(Qwee2.55VV-333
Max sequence length
2048
Load in 4-bit
True
Per-device train batch size
1
Gradient accumulation steps
4
Effective batch size
4
Learning rate
2e-4
Optimizer
adged. _Sbit
Weight decay
0.01
LR scheduler
linear
Logging steps
1
Seed
42
Tokenizer template
Qwen-2.5
that constrain answers within a limited candidate space. By contrast, Open-ended evaluation starts higher at 34.40% but declines sharply by the second epoch, dropping to as low as 12.62%. Only in later epochs does partially recover, reaching 35.48% at the end. This instability points sensitivity in open-ended generation that is not present in the MCQ setting, where the structure of multiple-choie questions provides more stable optimization signals. We interpret the fluctuations in open-ended performance as evidence that fine-tuning may overfit to MCQ-like reasoning patterns at the expense of free-form language generalization. The dotted baselines the plot further highlight how the MCQ mode achieves gains beyond its initial standing, whereas Open-ended remains approximately stagnant. Overall, the figure illustrates that SFT predominantly favors MCQ tasks in terms of stability and consistent accuracy gains.
The tabulated results in Table 4 complement the figure by providing a detailed before-and-after breakdown across six spatial reasoning categories. Across MCQ tasks, find that performance improved in every category, with gains ranging from a modest 1.59% in Relative Positioning to a substantial 17.11% 1% in Size and Scale. The Grand Total gain for MCQ stands 10.97%, confirming that the fine-tuning procedure effectively enhances structured-ansee Open-ended results are far more mixed, with both positive and negative changes. Categories such as 3D Geometry and Spatial Navigation show small but positive gains (3.50% and 2.82%, respectively). yet Size and Scale exhibits significant drop of -3.5%. Relative Positioning also decreases by -0.79%, reflecting that fine-tuning can inadvertently harm certain reasoning modes in free-form contexts. The overall Open-ended gain is only 1.07%, which is statistically negligible compared to MCQ improvements. These results confirm that the effectiveness of SFT is highly sensitive to the evaluation paradigm. By isolating the categories, we underscore where training helps, where it fails, and how performance benefits differ between constrained and unconstrained outputs.
performance. In contrast,
deeper inspection reveals striking disparities between categories and formats. For MCQ, Size and Scale and Spatial Navigation yield the highest improvements, suggesting that models trained with multiple-choice prompts are especially effective at learning tasks requiring scale discrimination and navigational reasoning. However, in Open-ended form, Size and Scale actually declines, showing that the same learned representations do not transfer well to generative contexts. Similarly, Relative Positioning barely improves in MCQ and worsens in Open-ended, highlighting consistent weakness
45

---

Published as a conference paper at ICLR 2026
in relational reasoning regardless of format. Conversely, Orientation shows strong MCQ gains (+13.33%) and small Open-ended improvements, suggesting that some categories generalize more robustly. We hypothesize that the divergence between the two formats may stem from the structural scaffolding MCQs provide, which reduces the need for the model to generate precise linguistic formulations. Open-ended tasks, by contrast, require simultaneous mastery of reasoning and natural language fluency, which amplifies weaknesses. The trends indicate that while fine-tuning strongly enhances pattern recognition in discrete-choice settings, introduces volatility and even degradation when free expression is required. This duality makes clear that training interventions optimized for MCQs may not generalize seamlessly to unconstrained reasoning.
The implications of these findings are twofold. First, we show that SFT as currently applied provides strong benefits for structured tasks but limited or even negative transfer to open-ended reasoning. This limitation is not only an artifact of fine-tuning, but also symptomatic of how contemporary VLMs are pretrained. Existing pretraining pipelines heavily emphasize broad image- alignment, object recognition, and captioning, yet they provide little explicit supervision on compositional spatial structures, occlusion layers, or geometric transformations. As a result, models learn to excel at surface-level associations while failing to acquire robust representations of depth, orientation, or scale. The instability we observe in open-ended ended spatial reasoning suggests that the model's ' learned priors fragile: without multiple-choice scaffolding, they revert to shallow meitern-matching instead of grounded inference. Categories such as Size and Scale and Relative Positioning, where declines are most pronounced. point missing inductive biases in pretraining, biases that humans develop through embodied interaction with 3D environments. Another contributing factor is the reliance on synthetic or eestruction-folloinng datasets that flatten spatial diversity into puzzle-like abstractions, preventing models from generalizing to real-world clutter, noise, and complexity. These findings underscore the need rethink both pretraining and fine-tuming fin-ttuning for spatial reasoning, incorporating datasets and objectives that reflect naturalistic visual-spatial challenges. Ultimately, we argue that without targeted reforms, current VLMs risk reinforcing an algorithmic straightjacket: they appear competent under constrained evaluations yet remain brittle and unaligned with the embodied, flexible reasoning that humans bring to world spatial understanding.
H.4.3 SET DYNAMICS ANALYSIS
To investigate the stability of Supervised Fine-Tuning (SFT) on spatial reasoning tasks, we conducted multi-seed validation study (N 5, seeds A-E) using the Qwen 5- Inst model. The training trajectory, summarized in Table 23 reveals a distinct divergence in performance dynamics between Multiple-Choice Question (MCQ) and Open-Ended evaluation formats.
Divergent Learning Trajectories. As illustrated in Table 23 and Figure 9 performance on SPATIALAB-MES MCCQ exhibits a monotonic improvement, rising from a baseline of 25.63% to an average of 35.74% at Epoch 4. This indicates that the model successfully adapts to the discriminative nature of the task, learning to map visual features to constrained output options with increasing reliability.
In sharp contrast, LPATIALAB-OPEN displays a pronounced U-shaped trajectory across all seeds. Rather than steady improvement, the model suffers an immediate and significant performance collapse in the early epochs (dropping from 34.40% to an average of 19.26% at Epoch 1). While performance recovers in later epochs (converging sx34.82% at Epoch 4), the final gain over the zero-shot baseline is negligible (+0.42%).
Representational Brittleness and Alignment Tax. We attribute this phenomenon to the fragility of spatial representations in current VL.Ms when subjected fine-tuning. The initial collapse in open-ended accuracy suggests catastrophic forgetting of the linguistic priors required for fluent, grounded generation. As the model optimizes for the specific instruction instruction-following format of the training data, it overfits to the structural logic of the tasks (benefiting MCQ) the expense of the flexible, generative spatial grounding required for open-ent open-ended descriptions.
The subsequent recovery phase represents the model realeinn articulate spatial concepts within the new distribution, yet it fails to surpass the initial baseline significantly. This dynamic underscores critical finding: SFT is effective for teaching models select correct spatial answers, but it does not
46

---

Published as a conference paper at ICLR 2026
Table 23: Multi-seed validation of SFT dynamics (N 5). While MCQ performance improves
steadily, Open-Ender performance exhibits systematic "U-shaped" collapse and recovery, indicating representational instability in generative spatial tasks.
Epoch
Type
Seed
Seed
Seed C
Seed D
Seed
MCQ
25.63%
25.63%
25.63%
25.63%
25.63%
(Base)
Open
34.40%
34.40%
34.40%
34.40%
34.40%
MCQ
29.56%
35.83%
24.13%
27.63%
26.53%
1
Open 22.50%
23.20%
12.50%
14.00%
24.10%
MCQ
34.33%
37.76%
27.33%
31.83%
34.43%
2
Open 12.62%
25.30%
28.80%
23.80%
16.65%
MCQ 34.68%
37.33%
32.86%
32.23%
34.73%
3
Open 28.69%
22.22%
30.34%
32.90%
28.30%
MCQ 36.59%
38.48%
34.43%
34.36%
34.83%
4
Open
35.48%
34.59%
33.92%
34.70%
35,40%
SpatiaLab-MCQ
SpatiaLab-Open
40
40
30
30
Accuracy
Accuracy
25
20
20
Seed
Seed
15
Seed
15
Seed
Seed
0.0
0.5
1.5
2.0
2.5
3.0
4.0
0.0
0.
1.5
2.0
2.5
3.0
3.5
Epoch
Epoch
Figure 9: SFT Dynamics Analysis.
fundamentally enhance the underlying generative spatial representation needed for robust open-ended
reasoning.
H.4.4 PERFORMANCE GAIN ON EXTERNAL BENCHMARKS
To assess whether the representations learned from SPATIALAB generalize beyond the benchmark s specific format, we evaluated the transferability of our fine-tuner model three external spatial reasoning datasets: OMNISPATIAL SPACE (Multimodal), and MIND THE GAP. We employed the Qwen2. 5-VL--BB-Inst--e model, fine-tuned on SPATIALAB via the SFT protocol detailed in Appendix H.4.1, and evaluated it on these external benchmarks in zero-shot setting.
Table 24: Transfer learning performance of Qwen2 --V--3B-Instruet on external benchmarks after fine-tuning on SPATIALAB. The consistent gains indicate that the model leams generalizable spatial representations rather than merely overfitting to the source dataset's 's format.
Benchmark
Base Accuracy
After SFT
Improvement
OmniSpatial
40.30%
47.35%
+7.05%
SPACE (Multimodal)
23.43%
28.67%
+5.24%
Mind The Gap
35.86%
47.56%
+11.70%
As shown in Table 24 fine-tuning on SPATIALAB yields substantial and consistent performance improvements across all evaluated domains, ranging from +5.24% +11.70% The most significant
47

---

Published as a conference paper at ICLR 2026
gain observed on MIND THE GAP (+11.70%) suggesting that our benchmark's emphasis on occlusion and spatial contimuity effectively addresses the reasoning gaps targeted by that dataset. OMNISPATIAL (+7.05% and SPACE (+5.24%) confirm that the
Similarly, improvements geometric and relational concepts encoded in our taxonomy align with broader definitions of spatial intelligence in the literature. These results validate the quality of the SPATIALAy data. While the primary purpose of this work is to serve as rigorous diagnostic benchmark, the strong transferability demonstrates that the dataset does not merely encourage overfitting to a specific question template. Instead, the diversity of the 30 sub-categories forces the model to acquire robust, transferable spatial representations that generalize to unseen distributions and task formats.
H.5 AI AGENTS FOR SPATIAL REASONING
We developed SPATIOXOLVER, a multi-agent system adapted and extended from Xolver (Hosain et al., 20255), to perform structured spatial reasoning on images within the SPATIOLAB evaluation. The system decomposes complex visual reasoning into specialized sub-tasks, each managed by a dedicated agent, allowing for focused processing of objects, attributes, relations, symmetries, and transformations. This modular design not only improves accuracy and interpretability but also supports multi-step reasoning, enabling the system to propagate low-level perceptual cues through higher-order relational and transformational inferences. The architecture, agent implementation, and their coordinated interaction are detailed in the following sections, with the full set of prompts provided in Figures T0
11 and 12
H.5.1 AGENTS
Base Visual Analysis Agent. Base Visual Language Model (VLM) agent initiates the pipeline by generating a detailed textual description of the input image, capturing object types-ineluding geometric shapes, natural elements, and artificial icons-alongside attributes such as size, orientation, color, and shading. Relative positions like leftmost, center, and top are encoded explicitly, and the prompt emphasizes exhaustive detail without summarization. Images are converted to data URIs and sent with the prompt Gemini-2.5. 2.ii-225-bbcdd model. whose output is parsed into structured bullet-list using regex-based extraction. This structured description serves as the foundation for downstream agents, providing comprehensive perceptual anchor that encodes both explicit and implicit spatial cues. The approach ensures that even subtle visual features, such as partial occlusions or minor shape variations, are retained for later relational and transformational reasoning.
Object Segmentation Agent. Object Segmentation agent converts the verbose textual description into a discrete list of uniquely identified objects, differentiating between visually similar instances. Each object receives a stable identifier (e.g., Obj1, Obj2) to maintain consistent references across subsequent reasoning steps. The agent uses carefully formatted prompts and calls the same Gemini- 2.5 model to structure the description, while post-processing splits lines by object ID to enforce consistency. This structured representation reduces ambiguity in downstream attribute and relation extraction, enabling precise mapping of complex scenes. By explicitly isolating objects, the agent also facilitates detection of overlaps, repetitions, and hierarchical groupings that may not be evident in raw descriptions.
Attribute Extraction Agent. Attribute Extraction agent translates each object into a structured
ISON capturing core properties such as shape, size, color, shading, orientation, and positional descriptors. Prompts enforce strict JSON output, and responses are parsed using ison. coads (), with fallback defaults to ensure robustness against model inconsistencies. This structured encoding allows quantitative and qualitative comparisons of objects, supporting later stages of spatial reasoning. By preserving both absolute and relative measures, the agent maintains fidelity the original perceptual content, while also normalizing descriptors for computational consistency. The JSON structure enables programmatic downstream operations, such as calculating symmetries or detecting transformations.
Spatial Relation Agent. The Spatial Relation agent encodes object interactions triples (Ob- Leler,Reeaiionnbeecee capturing directional relations (c.g.. left of, above, inside) and higher-order alignments like rows, grids, repetition patterns. Consistent object IDs are enforced prevent
48

---

Published as a conference paper at ICLR 2026
relational ambiguity, and parsing splits output lines starting with parentheses, followed by automated consistency checks. This step transforms static object descriptions into a relational graph that encodes spatial topology and hierarchy. The resulting structure allows reasoning over both pairwise interac- tions and more complex scene configurations, which is critical for tasks like symmetry detection, grouping, and multi-step navigation of spatial layouts.
Grouping and Symmetry Agent. Grouping and Symmetry agent abstracts higher-order patterns from object lists and spatial relations, identifying clusters, rows, grids, and vertical, horizontal, rotational, and translational. Outputs requested in structured JSON with keys groups and symmetries with empty structures as fallbacks to maintain downstream compatibility. This agent captures emergent scene regularities that are not apparent the object or pairwise relation level, providing critical context for pattern recognition and reasoning. By formalizing structural redundancies and symmetries, the agent supports both perceptual inference and anticipatory reasoning for dynamic transformations.
Transformation Tracking Agent. For multi-frame sequences, the Transformation Tracking agent logs object-level changes including translations, scaling, rotations, color/shading shifts, and appear- ance or disappearance events. Structured logs are extracted line-by-line using regex patterns that capture directional arrows, producing temporal representation of the scene. This encoding allows the system to track not just static spatial configurations but also dynamic evolution, supporting reasoning about motion, causality, and object interactions time. The agent ensures that transformations linked to consistent object IDs, enabling coherent cross-frame reasoning and predictive modeling.
Representation Standardization Agent. Finally, the Representation Standardization agent con- solidates all outputs into unified JSON format containing ob jects, relations, groups, symmetries. and trans formations Prompts enforce consistent object IDs and structured formatting, while JSON parsing failures revert empty dictionaries to ensure pipeline robustness. By centralizing all perceptual, relational, and temporal information, this agent creates a comprehensive scene representation that is both human-interpretable and machine-readable. The resulting unified structure serves as a versatile foundation for downstream tasks, including reasoning evaluation, spatial pattern analysis, and multi-frame prediction.
Open- Been-Ended  Spatial Reasoning Agent. The standardized representation feeds into the reasoning agent, which answers multiple-choice or pattern recognition questions. Inputs include objects, attributes, relations, groups, symmetries, and transformations. The model considers geometric patterns, relational alignment, and dynamic transformations to generate or select the correct answer.
H.5.2 TECHNICAL IMPLAMENTATION DETAILS
For our technical implementation, we use Gemini-2.5-F with a low temperature (0. to ensure deterministic outputs and perform multiple iterations for refinement. API calls are handled via OpenRouter, supporting both text and image inputs, while parsing of model responses relies on regex and JSON-based extraction to generate structured outputs. All interactions are logged in JSON format for reproducibility, and the final output is a unified JSON per image containing comprehensive information on objects, relations, symmetries, and transformations. This multi-agent decomposition allows each specialized model to focus on a constrained sub-task, enhancing overall accuracy, interpretability, and robustness in structured spatial reasoning.
H.5.3 EVALUATION AND ANALYSIS
The evaluation protocol remains identical to the standard CoT and CoT-SR setting, using balanced samples per sub-category (seed 42), making total of 150 samples.
The evaluation of Gemini-2.5-Flaat with agents highlights both strengths and limitations of agentic reasoning perceptual-geometricc tasks. Orientation is the only category where agents provide consistent and substantial gains, improving by +8.00% in MCQ and an impressive +36.00% in open evaluation. This pattern suggests that agent-based multi-step reasoning is effective when the task can be decomposed into sequential alignment of directional cues, where structured deliberation offers a clear advantage over single-step outputs.
49

---

Published as a conference paper at ICLR 2026
Table 25: Agent Performance Analysis
MCQ
Open Evaluation
Category
Gain
Gain
Normal
With Agents
Normal
Agent Results
3D Geometry
48.00%
44.00%
-4.0%
60.00%
48.00%
-12.00%
Depth & Occlusion
48.00%
44.00%
-4.00%
40.00%
16.00%
-24.00%
Orientation
56.00%
64.00%
+8.00%
0.00%
36.00%
+36.00%
Relative Positioning
60.00%
60.00%
0.00%
50.00%
36.00%
-14.00%
Size & Scale
48.00%
52.00%
+4.00%
25.00%
24.00%
-1.00%
Spatial Navigation
68.00%
64.00%
-4.00%
40.00%
28.00%
-12.00%
Overall
54.67% %
57.33 %
+2.66 %
35.83%
31.33%
-4.50 %
In contrast, most other categories exhibit negative or neutral gains, revealing that agent reasoning does not universally enhance performance. For example, depth occlusion suffers severe -24.00% drop in open evaluation, while spatial navigation decreases by -1..0%%. These tasks require deeper perceptual grounding in three- dineessimnii space, and the absence of such priors means that reasoning steps tend to reinforce misconceptions rather than correct them. Similarly, relative positioning shows stagnation MCQ and a -14.00% decline open evaluation, indicating that agent loops fail add robustness when relational understanding is inherently weak.
Some categories show marginal improvements, such as size & scale (+4.00% MCQ), but the gains not stable and do not transfer to open evaluation. Even in 3D geometry, agents fail to provide a boost, with -4.00% and -12.00% drops in MCQ and open evaluation, respectively. These results underscore that our benchmark is complex and not solvable simply by layering multi-step agentic reasoning on top of existing models. Instead, requires a deeper integration of geospatial understanding and perceptual representations, which current architectures lack.
Overall, the mixed results emphasize that agentic workflows alone are insufficient for advancing performance in this domain. While orientation tasks benefit from structured reasoning, most categories expose the models' lack of inherent awareness of perceptual concepts. This makes errors propagate through multi-step reasoning, amplifying performance drops. Consequently, benchmark serves as strong stress test, highlighting that future progress will require embedding genuine perceptual and geospatial priors rather than relying solely on agent-based reasoning.
I QUALITATIVE ERROR ANALYSIS AND ERROR PATTERNS
This section presents a focused qualitative analysis of model failures observed in our visual reasoning benchmark. We explain each error class, provide diagnostic root cause analysis, identify recurring patterns, and describe verification protocols. The discussion aims to be both interpretable and actionable, so it can inform future architectural and evaluation design.
I.1 OVERVIEW
Across the sample set, models achieve strong coarse perception, they can reliably recognize object categories, colors, and simple binary relations, but they break down systematically when multiple cues must be integrated. Failures are not random noise but cluster into a small set of interpretable classes: spatial mislocalization, perspective and scale mistakes, occlusion and ordering errors, attribute confusion, and ungrounded open-ended rationalization. These classes recur across architectures and prompting strategies, which suggests that the underlying causes are common inductive biases in current Mision-language pipelines rather than isolated training artifacts. Importantly, the same error often appears in both large-scale proprictary systems and small open-source baselines, indicating structural weaknesses shared across the model family. the following, we unpack each error class with representative examples and root-cause hypotheses.
50

---

Published as a conference paper at ICLR 2026
1.2
ERROR TAXONOMY AND ANALYSIS
1.2.1 SPATIAL MISLOCALIZATION AND REFERENCE CONFUSION.
Observation. When queries require distinguishing one object from set of visually similar candidates, models frequently misidentify the target. For example, if asked to point out "the leftmost red cube among three stacked cubes," the model may incorrectly output the middle cube, influenced by lighting highlights or central positioning. This problem becomes particularly acute in cluttered environments such kitchen counters or crowded table scenes, where multiple overlapping objects share the same color and shape. In such cases, the model's errors are not random: it tends to privilege objects that appear most salient due to size, brightness, centrality, even when these cues contradict the explicit spatial instructions in the query. These consistent patterns highlight that the failures are systematic, not accidental.
* Analysis. The recurring mistakes suggest that models lack a robust object-centric representation, or what cognitive science would call an "index" that persists across reasoning steps. Instead of maintaining stable pointer to the correct candidate throughout the reasoning chain, models appear to drift, often reassigning attention to whichever object seems most visually striking at intermediate layers. This leads to reasoning errors such as starting with the leftmost cube but ending the chain with the brightest or largest cube, The inability to anchor spatial reference consistently makes tasks involving relative comparisons, like "the ball behind the second chair", particularly fragile. In essence, the model operates more like a salience detector than a referential reasoner.
@ Root cause. At the architectural level, feature pooling operations, whether via attention or convolution, tend to mix contextual information from neighboring regions. While this is beneficial for global context, it erases the fine-grained, object-specific identity needed distinguish between similar candidates. Once pooled, features of two adjacent red cubes may be nearly indistinguishable in the latent space. Without an explicit mechanism, such as slot-based representations or pointer tokens, the network collapses distinct entities into blended embeddings. This structural limitation prevents reliable referent tracking across reasoning steps and explains why models consistently confuse dense, overlapping objects.
1.2.2 PERSPECTIVE AND SCALE MISTAKES.
Observation. Tasks involving metric reasoning about relative size, fit, scale are inconsistently handled. On simple comparisons like "is car larger than a bicycle," models typically succeed because such relationships align with strong statistical priors. However, when faced with nhee-thre decisions such as "will the small sphere fit inside the slightly larger cylinder," errors become frequent common pattern reliance on canonical object sizes: for example, predicting that a door must be taller than a truck even in a rendering where the truck is scaled down. Another failure occurs in unusual visualizations, such as toy miniatures or architectural models, where familiar proportions are deliberately inverted. In these cases, models cling to their priors and ignore the actual geometric evidence visible in the image.
and unpredictable,
* Analysis. These behaviors indicate that models rely heavily distributional statistics of object categories rather than performing true geometric inference. Instead of reasoning about projection geometry, relative bounding box sizes, or depth cues, they substitute their world knowledge of typical scales. This shortcut works well in everyday contexts but fails catastrophically in edge cases that deviate from the training distribution. For instance. when an unusually small airplane rendered next to person, the model still predicts the airplane as larger. Such errors highlight the lack of grounding in pixel-level metric cues and the absence of consistent use of perspective or vanishing points.
& Root cause. Current architectures do not include explicit geometric reasoning modules that preserve projection-awe awwere relationships. Supervision for scale and fit tasks is limited, so models learn correlations between object category labels and size but not the underlying geometry. Furthermore, training objectives prioritize overall accuracy rather than fine- mme-grained metric consistency, meaning that scale-sensitive errors under aedal--en Without inductive biases for 3D reasoning targeted supervision, the model defaults to memorized priors about canonical sizes rather than interpreting the actual perspective geometry in the image.
51

---

Published as a conference paper at ICLR 2026
1.2.3 OCCLUSION AND ORDERING FAILURES.
Observation. Determining which object is in front, behind, or partially hidden remains a core weakness. Models often misreport the foreground object when two animals overlap. when semi- transparent barriers like glass or fences are involved. For instance, in an image where a cat sits partially behind curtain, models may incorrectly describe the curtain as being occluded by the cat. Thin or fragile structures, such as wires, tree branches, or railings, are particularly error-prone: they are frequently omitted altogether, leading wrong answers in questions like "what object lies behind the fence." " These mistakes occur consistently, especially when occlusion cues are subtle or when the depth difference is small.
* Analysis. Such failures indicate that models treat depth and occlusion as weak, secondary signals. While the pixel data contains clear occlusion cues (e.g., edge boundaries, T-junctions), the intermediate features learned by the model do not retain them with sufficient resolution. Aggregation layers blur boundaries and downweight thin structures, effectively removing the very cues necessary to resolve ordering. This is why models succeed in gross depth distinctions (foreground vs. background) but collapse in fine mene-grained cases where partial occlusion must be inferred. In practice, they guess rather than infer, with systematic biases toward treating the visually larger brighter object as foreground.
Root cause. The architecture lacks explicit mechanisms for multi-scale feature retention and has no dedicated supervision for depth or occlusion reasoning. Thin or partially occluded signals are easily lost when features are pooled across large receptive fields. Furthermore, training data often underrepresents subtle occlusion cases, especially those involving transparency or fine boundaries. As result, the learned representations flatten layered scenes into 2D appearance maps, preventing robust construction of depth-aware relational reasoning.
1.2.4 ATTRIBUTE CONFUSION AND SEMANTIC SWAP.
Observation. Models sometimes conflate perceptual attributes (such as color, texture, or material) with functional or semantic roles. A classic example is misjudging whether a crosswalk safe: models may base the answer solely on the presence of painted stripes while ignoring the state of the traffic light or approaching vehicles. Similarly, a surface with a metallic texture may be misclassified as "steel" even if it is painted plastic. In tool-use contexts, the presence of handles or familiar shapes may lead the model to infer functionality that the object does not possess. These confusions are frequent in tasks requiring fine-grained visual grounding of attributes.
* Analysis. These errors reveal that language priors strongly bias the model's predictions when visual input is ambiguous. The decoder favors plausible narratives based on co-oceurrence in the training data: "crosswalks are safe when stripes are visible." " "shiny textures correspond to metal," and so on. This shorteut allows models to produce semantically coherent answers even when they do not inspect the relevant visual evidence. The result is superficially correct reasoning in familiar contexts but systematic errors in atypical or adversarial scenarios. In practice, this manifests overfitting to textual stereotypes rather than grounding in perceptual details.
Root cause. The underlying training objective, maximizing likelihood of fluent and plausible text, does not enforce visual grounding. As long as the generated answer is semantically reasonable, the loss minimized, even if the reasoning is incorrect. Consequently, the model is incentivized to substitute common-sense priors for genuine perceptual inference. Without auxiliary losses or verification steps to tie textual claims to image regions, models will continue to swap perceptual and semantic attributes.
I.2.5 OPEN-ENDED RATIONALIZATTON WITHOUT VERIFICATION.
Observation. For explanatory or why-style questions, models often generate coherent but un- supported narratives. For example, when asked "why the object leaning," " a model might answer "because the surface is slippery" even though slipperiness cannot be visually inferred. In another case, a tilted shelf might be explained as "due to an earthquake," a claim entirely outside the evidence in the image. These errors highlight tendency to fill gaps with imaginative but ungrounded rationales, producing fluent explanations that mislead users about the true evidence.
* Analysis. The issue stems from decoupling between text generation and visual grounding. The model generates high-quality language that reads convincingly, but there is no mechanism ensuring that these rationales are tied to actual visual features. Confidence calibration compounds the problem:
52

---

Published as a conference paper at ICLR 2026
models often assign high certainty to fabricated explanations, giving the false impression of reliability. As a result, open-ended outputs may look authoritative while being factually baseless. This mode of failure is particularly concerning in safety-critical domains where plausible but false rationales could misguide decision-making.
Root cause. Training is dominated by language-modeling objectives that reward fluency and co- herence but not factual grounding. Decoders are optimized to produce the most probable continuation of text given the prompt, without constraints linking the explanation back to specific image evidence. The lack of an explicit verification layer or grounding requirement means that models default to linguistic plausibilitt over perceptual truth. Without additional architectural or training constraints, open-ended rationalizations will continue to prioritize surface coherence at the expense of evidential accuracy.
1.3 ERROR PATTERNS AND CROSS- SIGNALS
When comparing failures across different models, several consistent patterns appear, revealing insights into the underlying mechanisms of errors. One clear trend that disagreements between models are systematic rather than random: some models consistently rely on geometric cues such as relative size, depth ordering, and occlusion relationships, while others fall back on linguistic priors or common- sense assumptions derived from training text. This leads to predictable divergences, for example when two models give different answers about which object "in front" despite seeing the same scene. Errors also spike when multiple cues must be integrated simultaneously. Tasks that require reasoning about occlusion, relative scale, and perspective together are particularly challenging; for instance, understanding whether larger object partially occluding smaller one is in front or behind often triggers consistent mispredictions. Calibration is another noticeable issue: high-confidence predictions frequently correspond to incorrect answers, especially in open-ended explanations or complex spatial queries. These patterns indicate that errors arise from how models balance perceptual grounding and linguistic plausibility. Different architectures exhibit distinct tendencies, some over- rely on visual geometry, others overfit to language correlations, so failures are not just isolated mistakes but reflect deeper trade-offs in reasoning strategies. This also implies that any attempt to improve performance should account for both sources of bias simultaneously rather than treating errors as independent events.
1.4 MITIGATION STRATEGIES
Given these еггог patterns, several targeted strategies can be pursued to reduce failures without requir- ing complete architectural redesign. For instance, geometry-aware supervision can be incorporated by adding auxiliary tasks like depth prediction, occlusion ordering, or overlap estimation. This encour- ages models to respect spatial constraints and improves consistency across predictions. Multi-scall feature retention is another approach: by using edge-aware pooling or attention mechanisms, models can maintain information from thin or partially occluded structures that would otherwise vanish in standard downsampling. Coordinate anchoring can help resolve confusion between viewer-centric and object-centric frames by explicitly representing reference points for relative positioning. Coun- terfactual augmentation can also be applied, where distractor objects systematically swapped or perturbed, forcing the model to rely on the true referent cues rather than heuristics. Lightweight verification modules can be added on top of existing outputs, cross- choss-hh claims against visual or geometric constraints to filter implausible predictions. Each of these strategies can be implemented incrementally, providing clear paths to reduce systematic failures while keeping the core model largely unchanged.
In addition to these architectural and training augmentations, supervised fine-tuning (SFT) offers a practical pathway to instill more reliable spatial reasoning. By curating datasets that explicitly highlight failure cases, such as near-threshold scale comparisons, occlusion ambiguities, or reference resolution traps, SFT can directly expose models to the edge scenarios where they otherwise default to heuristics. Carefully designed instruction-style fine-tuning further encourages step-by-step reasoning, prompting the model to verbalize intermediate spatial relations rather than skipping guess. This not only strengthens grounding but also improves interpretability, as errors become traceable to spe- cific reasoning steps. Moreover, SFT can be paired with counterfactual or adversarial augmentations, ensuring that the model does not simply memorize but instead learns invariances to distractors and context shifts. Importantly, these interventions require wholesale changes the architecture: they
53

---

Published as a conference paper at ICLR 2026
adapt existing models by reshaping their inductive biases through supervision. In this sense, SFT serves as a lightweight yet powerful complement to geometry-aware and verification- strategies.
In this work, additionally performed SFT using 40% of the training data and observed clear performance gains, particularly on multiple-choice questions. This suggests that targeted supervision helps models better internalize spatial cues and resolve distractor confusions when answer options are well-defined. However, improvements on open-ended responses were more limited. The model continues to produce fluent but weakly grounded explanations, highlighting that SFT alone cannot fully address rationalization without verification. These findings reinforce the need to combine SFT with geometry- aware supervision, counterfactual augmentations, and lightweight verifiers achieve balanced progress across both structured and free-form tasks.
Our analysis shows that failures are arbitrary but structured and interpretable, emerging from recurring patterns rather than random noise. This indicates that VLM errors follow systematic trajectories tied the inductive biases of their architectures and training regimes. For example, geometric reasoning often collapses when occlusion cues are weak, and in such cases language priors take over, producing answers that sound plausible but lack visual grounding. Similarly, scale inference drifts when canonical object size priors dominate over pixel-level metric cues, leading to consistent misjudgments in near-threste threshld scenarios. These findings highlight that the gap is not simply one of capacity but of representation: current models lack explicit mechanisms anchor visual features to grounded geometric relations.
By adopting the proposed verification protocols and targeted mitigation strategies, researchers address these weaknesses directly. Geometry-aware supervision and counterfactual augmentation would help models resist shortcuts, while multi-scale feature retention ensures thin structures and partial occlusions are not discarded. SFT, when carefully applied, can further strengthen structured reasoning by aligning model predictions with task constraints, though it must be combined with grounding objectives to avoid overfitting to discrete-choice formats. Importantly, verification modules provide a lightweight way to enforce consistency between claimed relations and image evidence, counteracting the tendency to rationalize without checking. Taken together, these steps chart practical pathway toward systems that reason more like humans: robust to clutter, interpretable in their failures, and capable of explaining not only what they predict but why.
54

---

Published as a conference paper at ICLR 2026
Base VLM Agent Prompt
Visual Describer. Produce raw, exhaustive description the
Input inage:
(inage_input)
focus:
1. Object Identification:
Shapes (circle, square, triangle, polygon, etc.)
elements (tree, sun, water, etc.)
Artificial objects (arrows, patterns, icons)
2. Attribute Noticing:
Size (snall, nediun, large, relative comparison) Orientation (upright, rotated, tilted) Shading or texture (salid, striped, shaded, transparent)
Color (if available)
3. Positioning:
Relative
(leftmost, rightnost, top, botton, center)
Proxinity to other objects
4. Level Detail:
Do not summarize
explicit every visible
Object Segmentation Agent Prompt
You are an Object Segnentation Specialist. Convert verbose inage description into structured list of distinct objects.
Input description:
(wisuaa__escriptioon)
Your focus:
Object Isolation:
Treat each unique instance as separate object
objects sinilar, assign separate
Labeling:
Assign IDs: 0bj1, 0bj2, Obj3...
Keep descriptions short but complete
3. Example Output:
obji: Snall shaded circle (leftnost) obj2: Large unshaded square (center)
obj3: Triangle rotated 45* (rightnost)
Attribute Extraction Agent Prompt
Attribute Extraction Specialist, object,
properties structured
Your focus:
Attributes to Capture:
Shape
Size (relative necessary)
Color or shading
Orientation rotation
(leftnost, rightnest, center, top, botton, etc.)
Output
J5ON array, e-g-:
(("ID": "Obj1", "Shape": "Circle", "Size": "Snall", "Shading": "Yes", "Rotation": "g"", "Position": "Leftmost") })
(('ID": "Obj2", "Shape": "Square", "Size": "Large", "Shading": "No", "Rotation: "45"", "Position": "Center") }
Figure 10: Prompts for SPATIOXOLVER (Part 1).
55

---

Published as a conference paper at ICLR 2026
Spatial Relation Agent Prompt
Spatial Relation Specialist, Extract spatial relationships objects.
Your focus:
Relations:
_eft_of, right_of, above, below inside, surrounding
touching, overlapping
aligned, centered
2. Precision:
Always specify directionality (objA left_of obja not obja left_of 0bjA)
Use consistent object IDs
Structural Patterns:
Detect repeated relations (e-g-, "each circle is above square") spacing and alignnent
4. Output Forma:
Triples
(ObjectA, Relation, Object8)
Example: (obja, left of, 0bj2) (Obj3, inside, 0bj4)
Grouping Symmetry Agent Prompt
You are Grouping and Symnetry Specialist. Identify higher-order structures from objects and relations.
Relations:
(relation_ list)
Your focus:
1. Grouping:
Clusters objects (rows, columns, triangles, grids)
Repeated patterns
2. Symetrry
VVertical, horizontal, rotational
Mirror or reflective symnetry Translational repetition
3. Output Fornat:
350N with two fields: "groups" and "symnetries" Exanple:
"groups": ["Obji, 0b32, 0bj3 forn horizontal row'), "symetries": ("Vertical symetry
Transformation Tracking Prompt
You are Transiorration Tracker. Analyze changes across multiple franes of an image sequence.
Your focus:
Types Transformations:
Movenent (left, right, up, down)
Scaling (larger, smaller)
Rotation (degrees)
Shading or color change
Shape change (square circle)
Appearance/dssapearracc
2. Output Format:
Per-object transformation logs:
(rotated clockuise)
06]2: Frane2 Frane3 (noved right)
Figure 11: Prompts for SPATIOXOLVER (Part 2).
56

---

Published as a conference paper at ICLR 2026
Representation Standerdization Agent Prompt
are Symbolic Abstraction Specialist. Consolidate perception results standardized structured fornat.
Inputs:
Objects: (object_list)
Attributes: (attributes_jseg
Relations: (relation_
Groups Symetries: sgruuping_syrmeery)
Transformations: (transformations)
1. Create unified 1S0N representation containing:
Objects with attributes
Relations as triples
and symetries
Transformations across
consistency in and formatting
Example Output:
<<
"objects": [...]>
"relations": [...b
"groups": [...],
"symetries": [...] "transformations" [...])}
MCQ Spatial Option Prompt
Reasoning Expert. Analyze image
option nultiple-
choice question.
Inage Analysis Data:
Objects and Attributes: (attributes_jsom
Spatial Relations: (relations list)
Groups & Symetries: (grouping_symmettee sysnery_sontt
Transformations
Franes: (transforaations_ssen)
MCQ:
Question: (question_ text)
Options: (options_text) text)
Instructions:
Consider shapes, positions, shadingg rotations, symetries, and transformations.
Identify pattern
sequence images.
Select only the correct option Only (1, 2, 3, 4
provide explanations, reasoning, extra text. Only option letter.
Output format: "coption_number" number>"
MCQ Spatial Option Prompt
You are Spatial Reasoning Expert. Analyze the inage data and give correct answer for the question.
Image Analysis
objects and Attributes: bettributes_json
Spatial Relations:
(relations _iastonns_isst
Groups & Symetries: (grouping synnetry_jsooeeeee
Transfornations Across Franes: (transfornations_json)
MCQ:
Question: (question_text)
Instructions:
Consider shapes, positions, shading, rotations, symetriis, and transformations.
Identify the pattern or rule in the sequence of images.
Figure 12: Prompts for SPATIOXOLVER (Part 3).
57

---

Published as a conference paper at ICLR 2026
J REPRESENTATIVE BENCHMARK SAMPLES ACROSS ALL SUB-CATEGORIES
We present representative benchmark samples for all 30 sub-categories SPATIALAB, illustrating the diverse spatial reasoning challenges covered. These samples highlight the variety of visual phenomena, including 3D geometry, occlusion, orientation, relational positioning, size and scale, and navigation tasks. Together, they provide concrete view of the types of reasoning each model must perform to succeed across the benchmark.
3D Geometry: Gravity Effects
Q Question: From which direction would the wind cause the liquid in the glass which is the front to spill onto the ground first?
Options
1. From the left side
2. From the right side
From the front
4. From the back
Gold MCQ nsser::
Gold Open ended A nsser: From the left side
MCQ Answers:
G.P--5mmnn::
Gemini-2.5-Flass
InternVL33.5722B
Bon--VLL-ABBThiinkinn
pen-ended A nswers:
GPT-5-mini: gust blowing from the back of the patio (the top of the image, toward the viewer) would push the liquid forward over the near edge and make spill first. Winds from the sides would tend to slosh it sideways rather than directly over the front rim.
the right side of the coffee table. Given its position, wind blowing from left (from the direction of the wooden cabinet and tall plant) would likely cause the liquid to spill first. This is because the glass is positioned with its opening facing somewhat towards the left, making it more susceptible to gust. from that direction.
Semini-2.5-Flash The glass in the front is the
InternVL.3.5 1.B.-72B The wind would need to come from the upper- right direction to cause the liquid in the glass the front right of the table to spill first. This is due to the positioning of the glass towards the front edge the table, closest the right side of the image.
Kimi-V -L- L- 3B-TBBTTT Tee-hhinning-22 2506: Okay, let's tackle this question. The user asking about the direction the wind would have to come from to spill the liquid in the front glass. Looking at the position of the glass: it's the left front edge of the coffee table. The table seems to have flat surface. If the wind blows from the left side (from the viewer's left, which would be the front of the image if the viewer.
Figure 13: Some examples from the benchmark (3D Geometry: Gravity Effects) and evaluation results.
58

---

Published as a conference paper at ICLR 2026
3D Geometry: Shape Projection
Question: If the hanging light casts shadow on the wall, which outline might appear?
O Options
1. A rectangle narrowing at the bottom From the right side
2. A vertical cylinder
A circle with fuzzy edges
A triangle pointing downward
Gold MCQ nswer:
Gold Opn ended Answer: A vertical cylinder
MCQ nswers:
GPT---mmni:
Gemini-2.5-Flash
InternVL3.55 L3.5-72B:
Kimi-VL-A ----VL---B-TTTT 3--Thinkiing2550000
pen-ended A nswers:
GPT-5-mini: You would see pendant's dome-shapes profile: rounded semicircle or oval shadow with
thin vertical line or narrow neck above where the cord hangs. The shadow would fall on the wall to the right the window, roughly over the nightstand.
Gemini-2.5- 2.5-Flash: The hanging light is simple, cylindrical pendant. If it casts a shadow the wall, the outline would likely be circular elliptical shape, depending on the angle of the light source relative to the wall. Given its position
the window, the shadow might also be influenced by ambient light or other
objects in room.
InternVI. InternVL.3.57725 1.3.5-2B: The hanging light fixture is likely to cast shadow resembling its own shape onto the wall, Given its cylindrical form, the shadow would appear as soft-edged, elongated circle or oval, depending on the angle of the light source relative the wall.
Okay, let's tackle this question. The user asking which outline hanging light would cast on the wall it casts shadow: The wall near the window light- colored, so the shadow would be visible. Since the shade is cylindrical, the shadow might be a cylinder. ut maybe the outline of the lampshade's perimeter. A Itemativell, if the light is hanging from chain, the shadow could be more elongated. But based on typical light fixtures, cylindrical shade would cast a circular oval shadow. The answer should be the outline of the hanging light's shade.
Kimi-VL- L- 3B-T hhinking-2000
Figure 14: Some examples from the benchmark (3D Geometry: Shape Projection) and evaluation results.
59

---

Published as a conference paper at ICLR 2026
3D Geometry: Spatial Containment
Question: Look the tall pot the background with the wide mouth (top left). could the small jug from the center of photo completely inside it?
O Options
1. Yes, easily with room to spare
2. No, the small jug hand will stuck
No, the small jug tall. Yes, it would perfect fit with no room left
Gold MCQ nswer:
Gold Opn ended Answer: No, the small jug hand will stuck
MCQ nswers:
GPT--5mmii:
Gemini-2.5-Flash
InternVL3.55 L3.5-72B:
Kimi-VL-A 3----BB-hiiniie 2e-Thhnkinng25066
pen--ended nswers:
GPT-5-mini: No the tall pot's mouth appears too narrow compared with the small jug's widest part and its
handle. The jug wouldn't pass through the opening to sit completely inside.
Gemini-2.5-Flash No, the small jug
center of photo could not completely inside the tall pot the background. While the tall pot has wide mouth, the jug's overall height and width appear to be tno large to be fully contained within the pot's interior: The jug's handle would also likely prevent it from fitting snugly. InternVI. L.5. 3.5--2B: The tall pot in the background with the wide mouth appears large enough potentially accommodate the small jug from the center the photo. Considering the proportions and the wide opening of the tall pot, it likely that the small jug could completely inside it.
Kimi-VI. V--A A-A 3B-TBTTTT Thinking-2553 2506: Okay, let's see. The question is about whether the small jug in the center the photo could fit completely inside the tall pot in the background with the wide mouth (top left). However, without explicit scale references, this conclusion relies typical proportional relationships between similar pottery forms. Based on typical proportions, the small jug height likely shorter than the tall pot's, 's,s, and width seems narrow enough to
Figure 15: Some examples from the benchmark (3D Geometry: Spatial Containment) and evaluation results.
60

---

Published as a conference paper at ICLR 2026
3D Gomeery:
Volume Comparison
Q Qustion:: Compare the internal cooking volume of the microwave internal cooking volume the larger ov en below the stove.
A
Options
They both have same volume vertical cylinder
2. The microwave has slightly more volume
3. The oven below the stove has much more volume
4. The oven has more width, but the microwave has more height, they equal
Gold MCQ A nswer:
Gold pen ended Answer: The oven below the stove has much more volume
MCQ nswers:
GPT--5mmin:
Gemimi-22.5Flaash
InternVL..55 L3.5-72B:
Kimi-VL-A ---i-VL-A---TTIIA T-33-BBinning-255000
pen-ended A Aswers:
GPT-5-mini: The microwave's cooking cavity is much smaller than larger oven below the cooktop it's only small fraction of the oven's 's internal volume. The countertop microwave is sized for single plates or small dishes, wheraa the built-in oven can hold full baking trays and multiple racks for larger meals.
Gemini-2.5 2.5-Flashh The internal cooking volume of the microwave appears to be significantly smaller than that of the larger oven below the stove. The microwave is compact countertop model, suitable for reheating or cooking smaller items..
InternVi 3..5-72B: The internal cooking volume the microwave on right likely smaller larger oven located below the stove. This is because microwaves typically have more compact dimensions to fit on countertops, while ovens have greater intornal capacity to accommodate laryer dishes and multiple items. The size and design visible in the image suggest the oven has deeper and wider cooking area compared the microwave.
of
Kimi-VL. VL- 3B-T 18-Thinkinn-- 2506: Okay, let's tackle this question. The user wants me to compare the internal cooking volumes of the microwave and the larger oven below the stove in the provided image. So, the intemal cooking volume of the oven below the stove is larger than that of the microwave. Based on their mllaive sizes in the image, the internal cooking volume of the oven below the stove substantially larger than that of the microwave
Figure 16: Some examples from the benchmark (3D Geometry: Volume Comparison) and evaluation results.
61

---

Published as a conference paper at ICLR 2026
3D Geometry: Shape Projection
Q Question: If you look at the seating arrangement from above, what shape does it form?
Options
1. A perfect square
2. A round shape
A rectangle
An irregular shape
Gold MCQ nswer:
Gold Pen ended Answer: An irregular shape
MCQ nswers:
GPT--5mmii:
Gemini-2.55Flash
InternVL3.5-720 L3.5-72B:
Kimi-VL-A35 3---Thininn--2500
pen-ended A nswers:
GPT-5-mini: Viewed from above, the seating is arranged a U-shape (a horseshoe) around the central fire
pit. The L-shaped sectional plus the two facing chairs create that U configuration.
Gemini-2.5-Flash The seating arrangement, when viewed from above, primarily forms an L-shape with the large sectional sofa. This L-shape is complemented by two individual chairs placed opposite the open ends of the sectional. Together, furniture creates U-shaped configuration around the central fire pit, designed for communal gathering.
InternV 1.3.5-2B: From above, the seating arrangement forms U-shape around the central fire pit The sectional sofa curves around the pit, leaving an open side opposite the tall fence, creating semi- enclosed space.
Kimi-Vi ML-ii--LAAAA 3B-T Teinkihii-iinni 2506: Okay, let's tackle this question. The user wants know the shape formed by the seating arrangement when viewed from above. W When viewed from above, the configuration clearly forms a right- anglengled L-spape, combining two parallel sections at a 90- dogrege angle. The symmetry of the layout and the placement of fumiture around the fire pit reinforce this geometric form. The seating arrangement forms right- angled L-shape when viewed from above.
Figure 17: Some examples from the benchmark (3D Geometry: Shape Projection) and evaluation results.
62

---

Published as a conference paper at ICLR 2026
Spatial Navigation: A ccessibility Constraints
The notebook at the leftmost corner of the table is inaccessible because of which object?
Question:
O
Options
1.
Chair
2. Vase
Table
Mug with pens and pencil it
Gold MCQ Answer:
Gold O pen ended Answer: Mug with pens and pencil in it
MCQ nswers:
GPTT-5mmii:
Gemini-2.5-Flast
InternVL3.557200 L3.5-72B:
Kimi-VL-A 1---AA--hiinkin TBB-Tinnkin--500
Open-ended A Answers:
GPT-5-mini: The leftmost comer notebook is blocked by wooden chair: The chair sits directly in front of the desk, preventing easy access to the notebook.
Semmiii-2.5-Flas The notebook the leftmost comer of the table is inaccessible because of the chair: The chair positioned directly front the notabook, blocking direct access it. To reach the notebook, one would first need to move chair of way.
InternV L.3.5VV....57775 L.3.--7BB: The notebook at the leftmost corner of the table is inacoessible because of the chair, which positioned in front of it. The chair's backrest and seat block direct access to the notebook.
Kimi-VL- L-nu--V----A 3B-T 2BB-Tinkiing225 2506: A lright, let's tackle this spatial reasoning question. So, the question is asking why the notebook the leftmost corner of the table inaccessible. The chair positioned near the leftmost. comer of the table, and given its placement, it's reasonable conclude that the chair is blocking access to the notebook. The chair's presence in that space would make difficult to reach the notebook from the left side without moving the chair first. The chair is the object that's making the notebook at the leftmost comer inaccessible.
Figure 18: Some examples from the benchmark (Spatial Navigation: Accessibility Constraints) and evaluation results.
63

---

Published as a conference paper at ICLR 2026
Spatial Navigation: Obstacle Avoidance
Question: There is clear direct path to the end of the hallway without colliding with any object.
Options
1. TRUE
2. FALSE
Maybe
Unable to determine
Gold MCQ nsswr:
Gold O pen ended Answer: TRUE
MCQ nswers:
GPTT-5mmii:
Gemini-2.5-Flash
InternVL..55 L3.5-72B:
Kimi-VL-A T
pen-ended Answers:
GPT-5-minl: Yes. The central aisle is unobstructed from the foreground to the far end, with shelving stacked
along both sides and items blocking the floor. You can walk straight down the hallway without colliding
with any objects.
the hallway without colliding with any object. The image shows wide, unobstructed aisle between two rows of tall shelving units filled with boxes. The floor is clear indicating safe passage for movement.
Gemini-2.5-Flass Yes, there is a clear direct path to
InternV VL 3.5--2BB: The image shows large warehouse with high shelving units full of boxes both sides of clear and unobstructed, leading directly the end the hallway. There are no objects or obstacles the pathway, ensuring direct clear route to the end the aisle. Kimi-VI VL-A 3B-T Teinkingg2225 2506: Okay, let's see. The user says there's a clear direct path to the end of the hallway wi without colliding with any object. A clear direct path exists because the shelving units are positioned symmetrically, leaving central aisle free of objects. There are visible hazards like hanging items, loose debris, or protruding elements the visible portion the hallway, allowing safe, unintermptedd movement to the end. The path clear because shelving units are symmetrically arranged, leaving the central walkway obstacle- free, with no visible hazards in the visible hallway.
central aisle. The aisle eppears
Figure 19: Some examples from the benchmark (Spatial Navigation: Obstacle Avoidance) and evaluation results.
64

---

Published as a conference paper at ICLR 2026
Spatial Navigation: Pathway Existence
Question: W Which walkways the safest cross according to the picture?
Options
1. Top and bottom
2. Top and left
Left and right
Bottom and right
Gold MCQ A nswer:
Gold O pen ended Answer: Top and bottom
MCQ nswers:
GPPT-5-mmn:
Gemini-2.55Flast
InternV L.3rmVL..5-72B.
Kimi-VL-A -L-VV-----TTT-A T----hBiiiing250000
pen-ended A nswers:
GPT-5-mini: The marked white crosswalks each comer of the intersection (the painted lines across the road) are the safest use. They are sigmalized and where pedestrians are waiting, unlike cutting across mid- block or diagonally through the intersection.
Gemini-2.5 2.5--Fashh The walkways s marked with white lines, commonly known as crosswalks, are the safest to cross. T These designated areas indicate where pedestrians have the right: of way expected to cross the street. the image, several people are visible using these marked crosswalks, suggesting they are the intended therefore safest routes for pedestrians.
InternVI L.3.5-VVL3..--22 1.3.--72B: The crosswalks ks aligned with the pedestrian signals at the comers are the safest places to cross, they are equipped with traffic signals ensuring vehicles are stopped. The designated pedestrian areas with painted lines across the streets clearly marked for safety. A Aditionally, sidewalks provide safe space for walking along the roads.
Kimi-VI Mini-LL-- L- 3B-T TerThiiniin-2255 2506: Okay, let's tackle this question. The user wants know which walkways are the safest to cross based on the image provided. The safest walkways to cross, based on the image, are the marked croswaaks equipped with pedestrian signals. Specifically. the cmsswalk at the bottom- left comer of the intersection safest because it features visible pedestrian signal indicating walk, reducing ambiguity
with traffic flow.
Figure 20: Some examples from the benchmark (Spatial Navigation: Pathway Existence) and evaluation results.
65

---

Published as a conference paper at ICLR 2026
Spatial Navigation: Viewpoint Visibility
Q Question: The human standing the closest (blue jacket) has clear view of the stadium, how?
O Options
1. Seeing directly
2. Watching the big screen the left side
There stadium
Unable to determine
Gold MCQ nswer:
Gold O pen ended Answer: Watching the big screen the left side
MCQ nswers:
GPT---mmii:
Gemini-2.55Flastt
InternV InternVL3.5-720 L3.5-72B:
Kimi-VL-A3BB T
pen-ended nswers:
GPT-5-mini: Because he's standing the raised walkway right behind the front-row barrier, his eye level is higher than the people immediately in front of him. That elevation (and the clear space by the railing) gives him unobstructed line of sight over crowd into stadium.
Cemini-2.5-Flann The person the blue jacket seated wheelchair, which places them at lower vantage point than the standing crowd. However, they have a clear view of the stadium because they are positioned at the very front of section, likely an accessible seating area. This front-row placement ensures an unobstructed line of sight over the heads of any standing individuals further back.
Intern' 3.B.--2B: The person the blue jacket is standing near the front the arena, which allows them clear, unobstructed view of the stadium. Their position close to the barrier separating the audience from performance area, providing a direct line of sight towards the stage and the large screens ov overhead. This proximity minimizes visual obstructions from other spectators structures.
Kimi-VL, L-A 3B-T hinking 2506: Okay, let's tackle this question. The user asking how the person in the blue jacket (who standing closest)
clear view the stadium. The person's elevated seating, minimized obstructions from nearby spectators, and potential alignment with stadium features (e.g., screens or lighting) enable their clear view despite being relatively close to the ground.
Figure 21: Some examples from the benchmark (Spatial Navigation: Viewpoint Visibility) and
evaluation results.
66

---

Published as a conference paper at ICLR 2026
Spatial Navigation:
Spatial Sequence
Question: W hich path is the easlest for the human sitting the breadire paper) follow to get on the train?
Options
1. Go straight to the first red colored door
2. To make turn and go to the second floor
Go through the door the end the train
No viable option
Gold MCQ nswer:
Gold pen ended Answer: Go straight to the first red colored door
MCQ nswers:
GPP--5mmni:
Gemini-2.5-Flash
InternVL..55 L3.5-72B:
Kimi-VL-A3B 3--.-Biniiigg25000
pen-ended nswwrs::
GPT-5-mini: Because he's standing the raised walkway right behind the front-row barrier, his eye level is higher than the people immediately in front of him. That elevation (and the clear space by the railing) gives him unobstructed line of sight over crowd into stadium. The simplest way for the person to
the train to rise from the bench, tum right, and walk a short: distance along the platform. They would then cross the yellow safety line and step through one of the train doors. T This route is straightforward and involves minimal effort.
Gemini-2.5-FFe
InternVI L3.5-72B: The person sitting on the bench can follow straightforward path to board the train by moving forward along the platform, parallel to the train tracks, while keeping the right of the yellow safety line. This path
them directly to the nearest open train door. They should avoid crossing the yellow line until they are ready enter the train to ensure safety.
Kimi-VL-A A 3B-T hinking- 2506: Okay, let's see. The question is about determining the casiest path for the person sitting on the bench to the train. First, need look at the image provided. The person sitting on the bench which is alongside the platform walls. The train on the opposite side. get the train, the person would..
Figure 22: Some examples from the benchmark (Spatial Navigation: Spatial Sequence) and evaluation results.
67

---

Published as a conference paper at ICLR 2026
Size and Scale: Scale Consistency
With the electric pole's capacity all this wires be redesigned to carry properly aligned structured design?
Question:
O Options
1. The size of wires are too big
2. The wires need to be move into another pole
Wires can set there following layout
No, the bulk size of wires are too much here one pole
Gold MCQ nswer:
Gold O pen ended Answer: No, the bulk size of the wires too much here for pole
MCQ nswers:
GPT--5mmin:
Gemini-2.5-Flas
InternVL.3.57220 L3.5-72B:
Kimi-VL-A LL- 3-B-hhining-25500
pen-ended A nswers:
GPT-5-mini: Not safely current state the pole clearly overlaaded and the wires are tangled, simply re- b-ndddlling them on the same pole would exceed mechanical, clearance and safety limits. A proper redesign is possible but requims removing redundant lines, separating circuits (high-voltage vs. telecom) onto organized crossams,, trays or underground ducts, and likely installing stronger or additional poles and anchors. G 2emini-2.5-Flashe W hile it's technically possible redesign the wiring for better alignment, the sheer volume entanglement wires the image suggest that the current electric pole's capacity severely overtaxed. Simply reorganizing the existing wines on same pole would not address the underfying capacity issues or the safety hazards presented by such dense and disorganized network.
InternVL. 3.5-72B: Redesigning the extensive web of wires around the electric pole into a property aligned and structured design feasible but would likely require significant effort and resources. Proper planning and coordination with multiple service providers would be essential. Implementing such changes would enhance safety and aesthetics, but involves logistical challenges and coordination to minimize service disruptions. Kimi-VL- LL 3B-T Teinhihii-igi555 2506: Okay, let's tackle this question. The user asking all the wires on the electric pole can be redesigned to be properly aligned in structured way. A redesigned system is feasible but dependent evaluating regulatory, technical, and safety constraints. Prioritzzigg organized routing for essential cables while removing redundant ones could create.
Figure 23: Some examples from the benchmark (Size and Scale: Scale Consistency) and evaluation results.
68

---

Published as a conference paper at ICLR 2026
Size and Scale: Shadow-Size Projection
Question: If one light is rotated so that they are perpendicular to each other, how will the shadow look over the wall?
Options
will cover almost all the wall
2. The shadow will only take the size of the light frame as it's actual size
3. There won't be any shadow
4. Shadow will smaller than the actual light body
Gold MCQ A nswer:
Gold O pen ended Answer: It will cover almost all of the wall
MCQ nswers:
GPT---mmni:
Gemimi-2.5-Flashh
InternVL..35 L3.5-72B:
Kimi-VL-A 3----Biniing-25500
pen-ended Answers:
GPT-5-minl: If one lamp is rotated 90a so the two beams are perpendicular, you'll get two elongated oval patches of light on the wall at right angles to each other. Where the ovals ovedap it will be brightest; outside them the wall will be darker, producing L-shaped darker region and perpendicular shadow lobes cast fixture.
G Gemini-2.5 2.5- If one light is rotated to be perpendicular to the other, the shadow cast. on the wall would change significantly. The current shadows are likely elongated and somewhat parallel due to the lights' current orientation. With one light rotated, the shadows would intersect at more acute angle, creating possibly darker overlapping shadow where the light beams cross.
complex
L.3.5 one light is rotated so that the two lights are perpendicular to each other, shadows on the wall will intersect at right angle, creating intersecting pattems. The light beams will cast shadows in perpendicular directions, resulting in a cross like effect w where the shadows overlap. The intensity and shape of the shadows will depend on the orientation and angle of the lights relative to the wall.
Kimi-VI. L- 3B-T 2B-Thinking-225 2506: Okay, let's try figure out this question. The is asking about rotating light so that they re parpassword to each other and how shadow on the wall would look. they 're further away, the shadow would be smaller.
Figure 24: Some examples from the benchmark (Size and Scale: Shadow-Size Projection) and
evaluation results.
69

---

Published as a conference paper at ICLR 2026
Size and Scale: Perspective Distortion
Question: If similar shaped pipes are stacked together separately will it take up less space?
0000 OU00OOO
POo 000008000
O Options
1. Yes
2. No
Take more
Similar
Gold MCQ A nswer:
Gold 0 pen ended Answer: Yes
MCQ nswers:
GPTT-5mmii:
Gemini-2.5-Flash
InternVL..35 L3.5-72B:
Kimi-VL-A T--33BBiiiiigg20551
pen-ended A Answers:
GPT-5-mink: Yes when like- shaped pipes are grouped you can pack them in their most efficient pattem ,which reduces wasted voids and the total footprint. The exact savings depend orientation and dimensions; some cases clever mixing nesting different shapes can beat separate stacks, but general rule same-
shape gmuping more space- space-efficient. effiiienttt..
Gemini-2.54 2.5-Flash: Yes, if similarly shaped pipes are stacked together separately, would likely take up less space. This is because organizing them by shape allows for more efficient packing and reduoes wasted space between different forms, For example, square pipes can be stacked flush against each other, and circular pipes hexagonal pattern for optimal density.
be nested or arranged
InternV L. L.35-72B: Yes, stacking similar shaped pipes together separately will generally take up less space. Organizing them by shape allows for more efficient packing, reducing gaps and empty spaces between different sizes and types. This mothod optimizes the use of available storage area and improves accessibility. Kimi-vL-A L-A 3B-T Thinking 2506: Okay, let's see. The question is asking if stacking similar shaped pipes separabely would take up less space. Stacking amilla-shaped pipes separately can potentially optimize space usage. W hen pipes the same diameter and shape (e.g., round or rectangular) are grouped tngether, they can be arranged more tightly, minimizing gaps between
Figure 25: Some examples from the benchmark (Size and Scale: Perspective Distortion) and evaluation results.
70

---

Published as a conference paper at ICLR 2026
Size and Scale: Relative Size Comparison
Question: The ash-colored, thicker coiled spring will be surpassed by which coiled springs if all are kept in the standing position?
Options
1. Right bottom corner
2. Right top corner
3. Left top corner
Bottom mid spring
Gold MCQ A nswer:
Gold O pen ended Answer: Right bottom corner
MCQ nswers:
GPT--5mmii:
Gemini-2.5-Flass
InternVL.355 L3.5-72B:
Kimi-VL-A3BB 1--3-BBiniiig-2D55
pen-ended A nswers:
GPT-5-mini: The ash-colored thicker spring in the center will be shorter than the thin, taller silver spring just behind it (top-center) and the long, dark- gray spring at the right edge when all are stood upright. Those two coils extend above the ash-colored spring.
G Demini-2.5-Flashee The ash- colored, thicker coiled spring, located near the center right of the image, would be surpassed by the longer, thinner coiled spring positioned the bottom right corner, dditionally, the smaller, conical springs the upper left quadrant would also taller fully extended and standing. These springs vary in height and coil density, indicating different, potential applications.
InternVI. L. 3.B-72B: The ash colored, thicker coiled spring in the image will be surpassed in height by the two taller springs its right. One of these taller springs also ash-colored while the other has slightly darker hue. Both of these springs have more coils and visibly longer when standing upright.
Kimi-VL- A-A 3B-Thinking 2506: Okay, let's try figure out this question. The image shows a bunch different coiled springs. The question is asking which coiled springs would be surpassed by the ash- colored, thicker one if all are kept in the standing position The thinner, tightly coiled springs (smaller in diameter or fewer coils) would be surpassed height by the ash-colored, thicker spring when all are kept upright.
Figure 26: Some examples from the benchmark (Size and Scale: Relative Size Comparison) and evaluation results.
71

---

Published as a conference paper at ICLR 2026
Size and Scale: Relative Size Comparison
Question: If the red bow taken out of it's place and kept far away the behind will look what will capacity comparing with it's other similar clips at front?
Options
Large, same
2. Same, less
3. Small, same
4. Small, large
Gold MCQ nswer:
Gold O pen ended Answer: Small, same
MCQ nswers:
GPT--5mmin:
Gemini-2.5-Flast
InternVL3.5-7200 L3.5-72B:
Kimi-VL-A Ta.33BBhiiiing255000
pen-endee A nswers:
GPT-5-mini: you move the red bow the back will appear smaller and more obscured by the bows front because of perspective and overlap, and its red color will look less vivid from that angle. Physically nothing about the bow changes - its size, shape and clasp strength remain the same as the other similar clips in front, so its holding capacity is unchanged.
G Gemini-2.5-1 2.5-Flash: If the red bow were taken out and placed far behind, it would appear smaller and less distinct due perspective. Its capacity, referring to its visual prominence ability hold hair, would remain the same as the other similar clips. However, its perceived capacity might diminish as it would no longer be in the foreground, drawing less attention.
InternVI L L3.5-72B: If the red bow is moved to the back, it will be partially obscured by the other bows in the foreground, making it less prominent. Its size and capacity hold hair remain the same the others, but visually, it won't stand out much
does currentyy when placed the front.
Kimi-VI. Limi-VL-A A 3B-T 28--Tinkiin-255 2506: Okay, let's tackle this question. The user asking about removing the red bow and placing it behind, then comparing its appearance and capacity to the other similar clips in front Removing the red bow would reduce its central visual impact, making the remaining front- factnffcigg bows appear denser but less balanced. Its capacity (physical or visual space) in the front area would decrease, potentially limiting the number similar bows that can be displayed
Figure 27: Some examples from the benchmark (Size and Scale: Relative Size Comparison) and evaluation results.
72

---

Published as a conference paper at ICLR 2026
Depth Occlusion Complete Occlusion Inference
Question: W is creating the striped shadow on the wall?
Options
1. The sunflower
2. Blinds
Window
Not applicable
Gold MCQ nswer:
Gold Oen ended Answer: Blinds
MCQ nswers:
GPT--5-mmn:
Gemini-2.5-Flast
InternVL33.57200 L3.5-72B:
Kimi-VL-A LL--B----BTTTT Wwt.V-------Thiiiiingeg 1BBThinkinn-25066
pen-ennde A nswers:
GPT-5-min Sunlight streaming through window blinds (slatted shades) is casting the diagonal striped shadow on the wall. The slats break the light into parallel bands, producing the striped pattern. Seemini-25--Faae The striped shadow on wall is created by sunlight passing through window with horizontal blinds. The blinds block portions of the light, casting distinct parallel lines onto the wall. This effect is common when direct sunlight enters a room through slatted window coverings.
into
InternVL 3.5--72B The striped shadow the wall is created by blinds window slats. The sunlight is passing through the gaps between the horzontal blinds, casting parallel lines light and shadow. This pattem often seen when light filtered through window coverings with narrow openings. The alignment and spacing of the lines suggest that the blinds are partially open, allowing the sun to project the shadow pattemn onto the wall.
Kimi-VL-A VL- 3--33B-hheee 3B-Thinking 2506: The striped shadow on the wall is created by sunlight passing through set of window blinds or shutters with narrow, evenly spaced slats. The parallel lines of the shadow result from the angular interaction between light source (likely the sun)
the rigid, horizontal slats the blinds, casting consistent geometric pattemss onto the wall. The uniformity and sharpness of the shadows suggest the light is direct and possibly filtered through the gaps between the slats.
Figure 28: Some examples from the benchmark (Depth and Occlusion: Complete Occlusion Infer- ence) and evaluation results.
73

---

Published as a conference paper at ICLR 2026
Depth & Occlusion Layering Order
Question: W the frontmost object in the picture?
Options
1. Trolley
2. Box
Man
Woman
Gold MCQ nswer:
Gold Oen ended Answer: Trolley
MCQ nswers:
GPP--5mmin:
Gemini-2.5-Flast
InternVL.355 L3.5-72B:
Kimi-VL-A 3B-BBBB-TT- 3B-TTinkinn--2500
pen-ended A nswers:
GPT-5-mini: The frontmost object small orange street-cleaing cart with a blue plastic bin mounted on it. A broom is attached to the cart and it sits on the pavement in the foreground.
Demini-2.5-Flaa The frontmost object in the picture the yellow cart with blue crate top, which man
a yellow and black suit sitting on. Its wheels are closest the viewer, indicating its position in the foreground. The people sitting on the bench behind the cart are further back in the scene. M.ternVVL3.5-22 The frontmost object in the image attached to it and broom standing upright. There's also noticeable yellow brush or shovel handle next to the container: The cart appears to be positioned on sidewalk in urban environment.
cleaning cart with whoels. It has blue container
Kimi-VL-A 3B-T Thinkinggiinnnn 2506: The perspective from the ground level, looking along the sidewalk. The cart positioned the foreground, closest viewer. frontmost object is high- high-visibility work cart.
Figure 29: Some examples from the benchmark (Depth and Occlusion: Layering Order) and evalua- tion results.
74

---

Published as a conference paper at ICLR 2026
Depth & Occlusion Partial Occlusion
Question: W What is the color of the ball behind the white colored square shaped device?
Options
1. Cyan
2. Yellow
Red
Not applicable
Gold MCQ A nswer:
Gold O pen ended Answer: Cyan
MCQ nswers:
GPTT-5mmii:
Gemimi-2.5-Flash
InternVL..55 L3.5-72B:
Kimi-VL-A 3---ABB-hhinkeee TB-Thinningg25500
open-nnded A nswers:
GPT-5-mini: The directly behind white, square- square-shaped device blue (a cyan/teal colored ball). It sits
slightly to the left and behind the device.
Gemini-2.5-FF 2.5-llah: The ball directly behind the white- colored square shaped device (GameBoy) is teal. There
also yellow green ball further to the right of the GameBoy.
InternV L3.5-72B: The ball behind the white, square shaped device blue.
Kimi-VL-A 3B-T B---Thnkiin---- 2506: The color of ball behind the white square-shaped device is blue.
Figure 30: Some examples from the benchmark (Depth and Occlusion: Partial Occlusion) and evaluation results.
75

---

Published as a conference paper at ICLR 2026
Depth & Occlusion Reflective Surfaces
Question: W hich object is the shiniest among all the objects in the image?
Options
1. Kettle
2. Tea glass
3. Table clock
Lamp chimney
Gold MCQ nswer:
Gold O pen ended Answer: L Lamp chimney
MCQ nswers:
GPTT-5mmii:
Gemini-2.5-Flash
InternVL..35 L3.5-72B:
Kimi-VL-A 3--BB---hhit T---hiiiing-25566
pen-ended A nswers:
GPT-5-mini: The frontmost object small orange street-cleaing cart with a blue plastic bin mounted on it. A broom is attached to the cart and it sits on the pavement in the foreground.
Demini-2.5-Flaa The frontmost object in the picture the yellow cart with blue crate top, which man
a yellow and black suit sitting on. Its wheels are closest the viewer, indicating its position in the foreground. The people sitting on the bench behind the cart are further back in the scene. M.ternVVL3.5-22 The frontmost object in the image attached to it and broom standing upright. There's also noticeable yellow brush or shovel handle next to the container: The cart appears to be positioned on sidewalk in urban environment.
cleaning cart with whoels. It has blue container
Kimi-VL-A Thinkinggiinnnnn 2506: The perspective from the ground level, looking along the sidewalk. The cart positioned the foreground, closest viewer. frontmost object is high- high-visibility work cart.
Figure 31: Some examples from the benchmark (Depth and Occlusion: Reflective Surfaces) and evaluation results.
76

---

Published as a conference paper at ICLR 2026
Depth Occlusion Transparency Effects
Question: How many obj ects has filaments inside them in the image?
O Options
1. 7
2. 6
5
4. 4
Gold MCQ nswer:
Gold pen ended Answer: 4
MCQ nswers:
GPPT-5-min::
Gemmmi-22.5Flaash
InternVL.3.57220 L3.5-72B:
Kimi-VL-A 3B- 3----BB-hhikkee 1BB-Thinking-2006
pen-ended A nswers:
GPT-5-mini: Three. The glowing bulb being held clearly shows filament, and two more filament bulbs
visible inside the round box on the left.
Gemini-2.5- 2.5-lash:: There are three objects with filaments inside them. One the illuminated light bulb being held, and the other two are unlit light bulbs inside the "Sunny Day Kit" box. All three are incandescent light. bulbs, which contain a filament that glows when heated.
InternVL. L. 3.5--7BB In the image, there are four obj ects with visible filaments. These include the lit light bulb being held, and three unlit bulbs inside the container labeled "Sunny Day Kit." Kimi-VL-s 3B-TBTTT Se--Thinniig--2 the 'Sunny Day Kit" container.
There are objects with filaments in image one held in hand and two in
Figure 32: Some examples from the benchmark (Depth and Occlusion: Transparency Effects) and evaluation results.
77

---

Published as a conference paper at ICLR 2026
Orientation Cardinal Direction
NORTH
Question: If you straight following route 6, which direction are you traveling?
11
EAST
SOUTH 11
6
Options
1. North
2. South
East
West
Gold MCQ nsser::
Gold 0 pen ended Answer: East
MCQ nswers:
GPT--5mmin:
Gemini-2.55Flas
InternVL33.57200 L3.5-72B:
Kimi-VL-A LL--B--TTTTTT 3BB-hiiking-22566
pen-ended A nswers:
GPT-5-min You would be traveling east. The Route shield is labeled "EAST" and the srright-aheal arrow
beneath shows that going straight follows eastbound Route 6.
traveling East. The sign for Route is directly below the EA ST sign, and there is an arow pointing straight up beneath the Route 6 shield, indicating that going straight leads to Route East
Gemini-2.8 2.5-Flas If you go straight. following route 6, you
InternVL33 L. 3.5--22B 1f you go straight following Route 6, are traveling east. This is indicated by the sign with the number 66' and the arrow pointing forward.
Kimi-VL-A -----hiiiing-25555 2506: Standard highway signs usually have the path the route. it's route 11 East, the arrow would indicate the diroction. But in the image, the left anro is under the NORTH EAST sign. Maybe that's indicating that route 11 East goes left? But the sign says East. A Iternatively, maybe be the arrows are indicating the direction of the route numbers. Like, route 11 North going straight, route 11 East is burning left, But then the route sign has upward arrow. So if you take route 6 straight, which direction that? The arrow on route shield is pointing straight up. the Question: is about direction- North, South, East, West. Wait, the signs here are for routes, not cardinal directions. So may the answer is that going straight on route 6 would be in the direction indicated by the upward ard arrow. But the upward arrow might be part of the route 6 sign.
Figure 33: Some examples from the benchmark (Orientation: Cardinal Direction) and evaluation results.
78

---

Published as a conference paper at ICLR 2026
Orientation Facing Direction
Suppose, Rabindranath Tagore's portrait is hung the left wall and faces directly into the room, what the direction the portrait with reference to the wall?
Question:
Options
1. Up
2. Down
Away
Toward
Gold MCQ nnswer:
Gold Oen ended Answer: Away
MCQ nswers:
GPP--5-mmn::
Gemini-2.5-Flaan
InternVL.3.57220 L3.5-72B:
Kimi-VL-A 3B- 3---hh-iiigg-5500066
pen-ended A nswers:
GPT-5-min the portrait the left wall and faces directly into the room, is oriented perpendicular to that wall pointing inward. other words, it faces away from the wall toward the center of the room. Deemini-2.5-Flase The portrait Rabindranath Tagore is located on the wall the left, as seen in image.
it faces directly into the room, its direction with reference to the wall would be perpendicular to the wall. This means the portrait is facing straight out from the wall into the space of the room.
InternVL3.. 3.55-72.. If Rabindranath Tagore's portrait is hung on the left wall and faces directly into the it
would be positioned perpendicular the wall. This means the portrait oriented so that viewers inside the room can face it directly when looking toward the wall on which it is hung.
Kimi-VL- -L 3B-T Thinking-2253 2506: The portrait is hung on the left wall and faces dimeclly into the room, meaning
is oriented inward toward the interior space.
Figure 34: Some examples from the benchmark (Orientation: Facing Direction) and evaluation results.
79

---

Published as a conference paper at ICLR 2026
Orientation Object Rotation
Question: W hat is the approximate angle the cases of the pocket watch?
Options
1. 90
2. 45
3. 180
120
Gold MCQ nswer:
Gold pen ended Answer: 180
MCQ Answers:
G.PT---min::
Gemini-2.5-Flast
InternVL.3.5-72B.
pen-ended nswers:
GPT--5mmni: bout 180*. The two covers are opened in opposite directions and lie nearly flat, forming an
almost straight line.
Gemini-2.55 2.5-Flash: The cases of the pocket watch open, revealing the intricate mechanism inside. The 220-degrae angle relative the watch face. The back case also open, but its angle is less clear, appearing to be around 90 degrees or slightly more.
front cover is open approximately
InternV MaternVL3..5 L..5-72B. 72B: The pocket watch in the image appears to be partially open, with the angle between its two cascs estimated to be around 60 degrees. The case opens hinge-wist wise, and the visible suggests that it is made to display the internal mechanical components of the watch. This angle provides good view of the watch face and some of its intricate gears.. hinking-2225 3-A3-Tinkiigg--
covers are both open to 180 degrees relative each other, that would be a straight line. But in this image, the back cover isn't fully open. looks like it's at around 90 degrees from the front cover. So the angle between the cases would be approximately 90 degrees.
Kimi-VL-A 3B-
W When the watch is open, cases form an angle.
Figure 35: Some examples from the benchmark (Orientation: Object Rotation) and evaluation results.
80

---

Published as a conference paper at ICLR 2026
Orientation Stacking Orientation
Question: W hat is the relationship between the road and the other objects in the image?
Options
1. Behind the building and plants
2. In front of the plants and leads towards the building
The road beside the building, with clear direction The road slacked above the building and plants
Gold MCQ nswer:
Gold 0 pen ended Answer: In front of the plants and leads towards the building
MCQ nswers:
GPP--5-mmn:
Gemini-2.5-Flas
InternVL335572200 L3.5-72B:
Kimi-VL-A T---A-B-Thiniig-22555
open-ended Answers:
GPT-5-mini: The road runs in front of the large white building, curving around a central planted island and seating area. separates and encircles the entrance and decorative planters, forming ttivewayymouddabut in the courtyard.
Demini-225-Flaas The road image serves access path, leading towards the grand white building.
curves around central island feature, which contains decorative fountain some potted plants. This suggests the road driveway or entrance loop designed to facilitate v vehicle access to the buil ding.
InternVI L.n.VLL.... L3.5-72B: The road in the image forms semi- circiiir driveway in front of a large white building with columns. It encircles landscaped garden area with plants, stone sculpture, and fountain the center, leading up the building's 's entrance. The road bordered pedestrian space. Lamp posts are situated alongside the road, enhancing Kimi-VI LA 118-Thnnking-2500 2506: The road curves around the front of the grand white building, acting as transitional pathway that guides movement and attention toward the entrance while dividing space between the main structure and other elements like the pergola. It establishes scale and perspective while integrating with the surrounding landscape.
curb that separates from the garden and
building's stately appearance.
Figure 36: Some examples from the benchmark (Orientation: Stacking Orientation) and evaluation results.
81

---

Published as a conference paper at ICLR 2026
Orientation Tool Handedness
Considering the likely process of creating the mural, which tool's handedness would be most evident the smooth, curved lines the bengali script?
Question:
Options
1. Paint roller
2. Paint brush
Tape measure Stencil
Gold MCQ nswer:
Gold pen ended Answer: Paint brush
MCQ nswers:
GPP--5-mmn:: Gemini-2.5-Flash:
InternV InternVL33.57220 L3.5-72B:
Kimi-VL-A335 T
pen-endee Answers:
GPT-5-mini: Those flowing, tapered curves are most characteristic paintbrush broad paint marker)
rather than spray can. The handedness of the brush user most likely right: handed would most evident, since right-handed handed strokes tend to pull and taper predictable left- lo-right sweeps visible in the Bengali script.
Gemini-2.5- 2.5--ii--.---ashh The smooth, curved lines of the Bengali script suggest the use of brush, likely fine- tipped one, to achieve such fluidity. The artist's handedness would be most evident the consistent direction and pressure applied to brushstrokes.
InternV L. 3.5-72B: In creating the mural, the smooth, curved lines of the B engaali script would most likely show the artist's handedness, particularty if the lines consistently curve more naturally in one direction. If the curves are more fluid and precise one side, it could suggest that the artist is either right-handes or left- handed, depending on the direction of the dominant curves.
Kimi-VL-A 3B-TBTTT Tee-Thinking-255 2506: If the artist left- handed, they might use brush their left hand, leading to strokes that curve more naturally. A Atematiivlly the tool itself (like specific brush tip) might be designed for left or right handed use.
Figure 37: Some examples from the benchmark (Orientation: Tool Handedness) and evaluation results.
82

---

Published as a conference paper at ICLR 2026
Relative Positioning A Aignmment Patterns
Question: W hich of the objects is the closest to the centerline of the image?
Options
1. Rice
2. Nut
Avocado
Berries
Gold MCQ nswer:
Gold O pen ended Answer: Berries
MCQ nswers:
GPP--5mmii:
Gemini-2.5-Flaat
InternVL3.55 L3.5-72B:
Kimi-VL-A 3L-VBB--BBTTTT Se--Thinninn-25000
pen-ended nswers:
GPT-5-mini: The red bowl of berries (raspberries and blueberries) is closest to the image's centerline. It sits almost directly in the middle of the composition, slightly left of the exact center.
2ommnii-25--Faaa The red bowl containing raspbenries and blueberries is the closest object the centertine of the image. position almost perfectly centered horizontally, and it located in the upper- middle section of the image vertically. W hile other objects like avocado halves are also quite central, the red bowl's is overall placement makes the most aligned with the image's central axis.
InternVI L.35-72B: In the image, the avocado halves are the closest to the centerline. One avocado is placed near the top center, and the otheris slightly below it the right, straddling the vertical centerline of the image. Kimi-VL-A 3B-T 3B-Thinking-222 2506; The bowl of mixed berries is closest the centerline.
Figure 38: Some examples from the benchmark (Relative Positioning: Alignment Patterns) and evaluation results.
83

---

Published as a conference paper at ICLR 2026
Relative Positioning: Betweenness Relationships
Q Qustion: W hat is the object between the building and the street lamp?
Options
1. A monument of various statues
2. Statue
Tree
Humans
Gold MCQ A nswer:
Gold O pen ended Answer: A monument of various statues
MCQ nswers:
GPPT-5mmin:
Gemini-2.5-Flas
InternVL335572200 L3.5-72B:
Kimi-VL-A T
pen-ended nswers:
GPT-5-mini: Between the building and street lamp stands a stone statue a person. It appears one of
the sculpted figures that are part of the fountain in the forgground
Gemini-2.55 2.5-Flash: The abject located between the building and the street lamp is a large, omatt monument.
This monument features multiple levels adomed with various statues and intricate carvings. its very top, there prominent statue person.
InternVL. L. The object between the building and the street lamp detailed stone monument or fountain. It features several statues, including a prominent figure standing on top and additional figures around its structure. The monument is omate, with decorative elements and possibly bas-rolie pas-roliefs on its sides, and it appears to be situated in open, possibly public area.
Kimi-VV-A 3B-T HB--Tinkiing22555 2506: The object between the building and the street lamp detailed stone monument or fountain. The monument omaat,, with decorative elements and possibly bas-reliefs on sides, and it appears to be situated in an open, possibly public area.
Figure 39: Some examples from the benchmark (Relative Positioning: Betweenness Relationships) and evaluation results.
84

---

Published as a conference paper at ICLR 2026
Relative Positioning Corner/Angia Positioning
Question: W hat is in the top left corner of the plant?
O Options
1. A lobster
2. A crab
An octopus
Nothing
Gold MCQ A nswer:
Gold Opn ended Answer: A crab
MCQ nswers:
GPP--5-mmn:
Gemini-2.5-Flas
InternVL33.57220 L3.5-72B:
Kimi-VL-A ---i-V---AB----- T-L-ABB-Thikking Se--Thiininn-2500 B
pen-ended Asswers:
GPT-5-mini:
top left coreer the plant, perched on rim the fishbowl. hanging over the water with its claws visible.
Gemini-2.5 2.5-Flahh: The plant in the image is type of seaweed or coral, and is contained within glass bowl. In the top left coreer of this plant, there is a branch of the plant extending upwards and slightly to the dark, greenish brown color
left. This branch
(.eerVVL3..-2222 In the top left comer the plant, there small crab that perched on edge of the bowl, outside the water. is gripping the rim with its legs, and its body is partially inside the bowl. The background shows the open water, providing contrast to the scene.
Kimi-VL-A -A-L---BB-TTT 1B--Thinking-2000 The answer is the crab
Figure 40: Some examples from the benchmark (Relative Positioning: Corner/Angle Positioning) and evaluation results.
85

---

Published as a conference paper at ICLR 2026
Relative Positioning Directional Relations
Q Question: W What located directly behind the cheetah?
O Options
1. Person
2. Tree
Saffari vehicle
Cheetah
Gold MCQ nswer:
Gold Oen ended Answer: Saffari vehicle
MCQ nswers:
GPP--5-min:
Gemimi-2.5-Flashh
InternVL...0 L3.5-72B:
Kimi-VL-A T
pen-ended nswers:
GPT-5-mini: Directly behind the cheetah a white open-sided tour vehicle or tram, Several people are seated inside the tram, watching and taking photos.
Cemmii-2..-Fllan Directly behind the cheetah, there is white open-sided vehicle, likely safari or tour cart. Inside this vehicle, several people are seated, observing the cheetah. Some these individuals holding up cameras or phones, seemingly taking pictures or videos.
InternVL.3. L3.5-72B: Directly behind the cheetah, there is group of people sitting and standing around, appearing to watch or interact with the cheotah. They are positioned near small covered vehicle, which looks like a safari cart shuttle. The area grassy patch, and there trees and hill in the further background. Kimi-VL-A 3B-T Teikkinnnggn2255 2506; People seated benches under canopy located directly behind the cheetah.
Figure 41: Some examples from the benchmark (Relative Positioning: Directional Relations) and evaluation results.
86

---

Published as a conference paper at ICLR 2026
Relative Positioning Proximity Gradients
Question: W hich book is closer to the book 'american marriage". "atonement" or cloud atlas"?
Wolf Hall PANTE
SEL FEEL BRIDE
SKITH ON SEATT
StEn Atoemee
AVI BEEICCA
Options
1. Cloud A tlas
2. Atonement
Crossed
nan
MEN WE REAPED
Gold MCQ nswer:
Gold pen ended Answer: A tonement
MCQ nswers:
GPP--5-mmn::
Gemini-2.5-Flast
InternVL33.57200 L3.5-72B:
Kimi-VL-A Lini-VL-AAB-Thaee T
pen-ended Answers:
GPT-5-mini: "A tonement" closer 'american marriage" than "Cloud tlas." "A tonement" sits just to the
right with only one small book between them, while "Cloud tlas" is further the right on the shelf.
Gemini-2.5- 2.5-Flah:: "A tonement" closer to "A A merican Marriage" than "Cloud Ulas." "A tonement" is directly adjacent to "An A merican Marriage" on its right side. "Cloud A tlas" is further to the right, with "eerianah"" soparating it from 'A tonement."
InternV L..5-72B: The book "atonement" closer to "american marriage" than "cloud atlas." " In the image, "atonement" directly next to "american marriage"" while "cloud atlas" is located further to the left..
Kimi-VL-A 3--ABBBhhikae T1B-Thinking-222 2506: Cloud A tlas is closer to "A merican Marriage" than "A tonement."
Figure 42: Some examples from the benchmark (Relative Positioning: Proximity Gradients) and evaluation results.
87

---

Published as a conference paper at ICLR 2026
K STATISTICAL ROBUSTNESS AND DATASET STABILITY
To ensure that the benchmark results are not artifacts of random initialization or dataset noise, we evaluate both model robustness across runs and dataset stability across repeated trials, Such analyses are essential for small to medium scale benchmarks, where minor fluctuations can lead to misleading fine-grained conclusions (Drummond] 2009. Drummond & Japkowicz 2010).
Let N denote the number of items questions in the benchmark and R 3 the number of inde- pendent runs of the same model. Each run uses a distinct random seed while preserving all other hyperparameters.
K.1 MODEL. ROBUSTNESS
For checking model evaluation robustness, perform several statistical studies, described below.
K.1.1 MULTIPLE RUN AVERAGES AND DEVIATIONS
Let 0i, 0, 1 denote the correctness of item 1,..., N run 1,2,3. The per-item average accuracy is:
* Σ 1 a
(1)
and its corresponding standard deviation is:
0;
ai,r a.)²
(2)
R
The overall mean accuracy per run is:
1
A,
0t
(3)
N
and its standard deviation across runs is:
R (A,
A)2
σ
R
where A= 1.2 ER A, is the overall mean accuracy across all runs.
K.1.2 INTRA-CLAS CORRELATION (ICC)
To quantify agreement between runs, we compute the Intra-Class Correlation coefficient (Sroutt & Fleiss 1979), using a two-way mixed-effects model:
hetween
ICC(3, k)
(5)
hetween + Guini
where 2 oetween is the variance between items and o2 within residual variance across runs. ICC values above 0.75 indicate good reliability, while values above 0.9 imply excellent consistency (Koo) Li 2016).
K.2 DATASET STABILITY AND INTERNAL CONSISTENCY
We also analyze the stability of the dataset itself, whether question difficulty remains consistent across
88

---

Published as a conference paper at ICLR 2026
K.2.1 RESAMPLING STUDY
To further assess robustness, we perform resampling analysis. From the total of C 30 subcate- gories, we randomly sample subsets of size S E 20, 25 for each sub-category from each of the R = 3 runs, across both models and evaluation types (MCQ and open-ended).
For each run r and subset size S, the accuracy is computed as:
1 S dir
A
(6)
where ai,r, denotes the accuracy for subcategory i in run under subset size S. The mean accuracy across runs for each subset size is:
A Σ = A
(7)
and the corresponding standard deviation:
H
1
0g
Σ
(Ars - As)
(8)
R
As complementary check, the Wilcoxon signed-rank test assesses median differences between paired samples. Let d, Ar.20 - A, the difference accuracies for run r, excluding ties (d, + 0). The test statistic is computed as:
W = min(W+, W-
(9)
where W+ and W- are the sums of signed ranks of positive and negative differences, respectively. The standardized z-score given by:
W mn++1)
(10)
1(n+1)(2n+1)
where n is the number nonzero pairs.
If the resulting valuue satisfies 0.05, again fail to reject Ho, confirming that the accuracies from 20- and 25-sample subsets are statistically indistigguishabl.
K.2.2 ITEM- CONSISTENCY
For each item i, its difficulty can be defined as d, = - * If item difficulty consistent, the correlation between should be high:
Cov(a..,, a..)))
p*r1,72-
(11)
Or, Tr,
A high mean pairwise correlation (p > 0.8) suggests stable item difficulty and reliable dataset behavior.
K.2.3 CRONBACH'S ALPHA
To measure overall internal consistency, we compute Cronbach's 's alpha (Cronbach 1951:
R R R-1
Σ
a
1
(12)
ST
where 52 is the variance of run and 2 is the variance the total scores aggregated across runs. Values of a 0.9 indicate excellent internal consistency (Tavakol Dennick 2011).
89

---

Published as a conference paper at ICLR 2026
K.3 EMPIRICAL RESULTS
We perform the analysis on two models: one large proprietary model, Gemini-2.5-Flash and one small open-source model, Qween22.5vl--7--nsstuuus thereby covering both large- and small-scale models as well proprietary and open-source paradigms. We also conduct all tests in both multiple- choice (MCQ) and open-ended evaluation formats (treated as binary: correct or incorrect). For statistical analysis, we used the following Python packages and functions: scipy. stats (iitanne ct al. 2020) for will coxon test; pingouir Valllt 2018) for intraclass correlation
and cronbach alpha; and numpy (Harris et al. 2020 numpy corrcoef for correlations.
K.3.1 MODEL EVALUATION ROBUSTNESS
Multiple Run Averages and Deviations
We analyzed the stability of model accuracy across R = 3 independent runs. The standard deviation of accuracy (d across runs is negligible (< 0.6%), indicating that the reported scores are precise estimates unaffected by random seeding noise.
Table 26: Model Accuracy and Variance across 3 Independent Runs.
Model
Format
Mean Accuracy (A)
Std Dev (a
Qwen-2.5-vl-7b
MCQ
0,4079
0.0021
Gemini-2.5-Flast
MCQ
0,5193
0.0054
Qwen-2.5-vl-7b
Open
0.1936
0.0047
Gemini-2.5-Flash
Open
0.3124
0.0042
The consistently low standard deviations across all models and evaluation formats indicate that performance estimates are highly stable and reproducible, with minimal sensitivity to stochastic run-to-run variation. This stability suggests that observed accuracy differences meaningfully reflect underlying model capability rather than noise introduced by random initialization decoding. Notably, although open-ended evaluation exhibits slightly higher variance than MCQ, the absolute magnitude remains negligible, confirming that free-form generation is also robustly assessed under the current protocol. These findings imply that the dataset size and scoring methodology are sufficient to yield convergent accuracy estimates without requiring extensive repeated runs. Consequently, single-run evaluations are likely to be reliable for large-scale comparisons, while multi-run analyses primarily serve to validate robustness. Overall, this strengthens the empirical credibility of the reported results and supports confident cross-mit cross-model and cross-formal performance interpretation.
Intra-Class Correlation (ICC)
We computed ICC(3,)) to quantify the reliability of the scoring mechanism.
Table 27: Intra- Intra-Class Correlation (ICC) Reliability Scores.
Model
Format
ICC(3,k)
Interpretation
Qwen-2.5-vi
MCQ
0.988
Excellent
Gemini-2.5-Flash
MCQ
0.990
Excellent
Qwen-2.5-vl
Open
0.983
Excellent
Gemini-2.5-Flan
Open 0.981
Excellent
The consistently high ICC(3,k) scores demonstrate that the evaluation exhibits near-perfect reliability across independent runs, regardless of model size response format. This indicates that item-level difficulty ordering is preserved, and that performance fluctuations do not alter relative rankings among samples. Such strong agreement suggests that the scoring pipeline internally consistent and not sensitive to stochastic variation in model outputs. Importantly, this rules out random guessing as a dominant factor, even open-ended generation, where variability is typically higher. From
'https://scipy.orrg/
httpp///iiggui---stttaahhaaihhan https://numpyyrrg//
90

---

Published as a conference paper at ICLR 2026
an evaluation standpoint, these results imply that the benchmark provides dependable signal for assessing model capability. Consequently, longitudinal comparisons and fine-grained ablations can be conducted with high confidence in measurement stability.
Overall, the combined evidence from multiple-run variance analysis and ICC reliability demonstrates that the evaluation is highly robust to stochastic effects. Both absolute performance (low standard deviation) and relative performance (near-perfect ICC) remain stable across independent runs, con- firming that results are not driven by random seeding or decoding noise. This robustness ensures that observed performance gaps across models and formats reflect genuine capability differences. As result, the evaluation framework provides reliable and reproducible basis for comparative analysis and benchmarking.
K.3.2 DATASET STABILITY AND INTERNAL CONSISTENCY
Resampling Study To ensure statistical reliability, this resampling procedure is repeated over 1,000 independent trials, each time drawing new random subsets of size the available subcategories. Repeating the test at this scale stabilizes the distribution of accuracy differences and p-values, ensuring that conclusions are not driven by a particular random draw. The consistent failure to reject Hg across all trials therefore provides strong evidence that the observed robustness is systematic rather than incidental.
20 and S = 25 from
Table 28: Resampling Study (1,000 Trials): Wilcoxon Signed-Rank Test (S 20 vs. S 25).
Model
Format
Mean p-value
% Trials p > 0.05
Conclusion
-wwn-2255v1--bb
MCQ
0.290
100%
No Sig. Diff.
Demini-2.5-Flaa
MCQ
0.285
100%
No Sig. Diff.
Owen-2.5-v1-77
Open
0.294
100%
No Sig. Diff.
Gemini-2.5-Flase
Open
0.294
100%
No Sig. Diff.
The resampling results provide strong empirical evidence that accuracy estimates have already converged at relatively small subcategory sizes. The consistently high mean (p)-values and the absence of statistically significant differences across all 1,000 trials indicate that performance metries are insensitive to moderate reductions in sample size. This stability holds uniformly across models and evaluation formats, suggesting that neither model scale nor output structure introduces additional sampling uncertainty. Importantly, these findings imply that the variance in accuracy is dominated by model behavior rather than data sparsity at the subcategory level. a result, the current subcategory size ( 50) can be considered statistically redundant, offering substantial safety margin beyond the minimum required for reliable estimation. This supports the validity of the benchmark design and suggests that future extensions could reallocate samples across categories without compromising statistical robustness.
Item-Level Consistency and Cronbach's Alpha We measured the mean pairwise Pearson correlation (p) between runs and Cronbach's 's Alpha to assess internal consistency.
Table 29: Leve-Level Consistency Metrics.
Model
Format
Mean Pairwise `
Cronbach's
Dwen-2.5-v1-77
MCQ
0.966
0.988
Gemini-2.5 2.5-ii--
MCQ
0.972
0.990
Dwen-2.5-v1-7
Open
0.950
0.983
Gemini-2. 2.5-Flash
Open
0.946
0.981
The extremely high mean pairwise correlations (> 0.94) demonstrate strong item-level stability, indicating that the relative difficulty of individual questions is preserved across independent runs. This consistency shows that model responses are systematically structured rather than subject to random variation, even under stochastic decoding. The similarly high Cronbach's a values (> 0.9) further confirm that the dataset exhibits excellent internal coherence, with items contributing reliably to the overall accuracy measure. Notably, the slight reductions observed in open-ended formats are expected given their greater expressive flexibility, yet remain well within the range of excellent reliability. Together, these results indicate that the benchmark functions a unified and internaly
91

---

Published as a conference paper at ICLR 2026
consistent measurement instrument. Consequently, model performance differences be confidently attributed to true capability variation rather than instability at the item level.
K.3.3 OVERALL DISCUSSION
Our expanded statistical testing shows our evlauiton and datset is robust ICC and Cronbach's a scores consistently exceed 0.98, and the standard deviation of accuracy across runs is negligible. Furthermore, the resampling study confirms that even smaller subsets produce statistically indistin- guishable results from the full set, proving that our subcategory sizes (~ 50) are sufficient for robust conclusions. These results demonstrate that the granular analysis in the paper reflects genuine model capability differences, not statistical noise.
Our expanded statistical testing shows that the evaluation protocol and dataset are highly robust, with ICC and Cronbach's scores consistently exceeding 0.98, indicating near-perfect reliability and internal consistency. The standard deviation of accuracy across runs remains negligible (< 0.6%), confirming that reported model performance stable and not influenced by stochastic effects such as random seeding or decoding variability. In addition, the resampling study demonstrates that even substantially smaller subsets (NV 20 and N 25) yield statistically indistinguishable accuracy estimates compared to the full set, establishing that the chosen subcategory size (~ 25) is well beyond the threshold required for convergence. Collectively, these results show that both absolute scores and relative performance rankings are stable across runs and sampling regimes. This statistical robustness ensures that fine-grained comparisons across models, formats, and subcategories are meaningful. Consequently, the granular analyses reported in the paper reflect genuine differences in model capability rather than artifacts of sampling variance or evaluation noise.
92


Reviews:
[{"review_id": "WbJb8t08tc", "reviewer": "Reviewer_e5YW", "summary": "The authors introduce SpatiaLab, a new benchmark dataset of 1,400 visual question-answer pairs designed to evaluate VLM spatial reasoning in \"in-the-wild\" contexts. The benchmark spans six categories (30 subcategories) and supports both multiple-choice and open-ended evaluation formats. The authors conduct a large-scale evaluation of over 25 VLMs, revealing a substantial performance gap between SOTA models (e.g., InternVL3.5-72B at 54.93%) and a human baseline (87.57%). The analysis is extended to improvement strategies (SFT, CoT, Agents), which are shown to provide limited or inconsistent gains, suggesting current models lack fundamental spatial grounding.", "questions": "I am curious about the true value of the dataset, as I strongly suspect that it may merely overfit to its own format.\n\nIf a base model (e.g., Qwen-VL) could be fine-tuned on this dataset and subsequently demonstrate performance gains on other benchmarks (such as VSI-Bench, OmniSpatial and SPACE), I would be much more inclined to recognize the dataset’s contribution.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Well-Motivated Problem**: The work correctly identifies a critical flaw in existing benchmarks: an over-reliance on synthetic, \"puzzle-like\" setups that fail to capture real-world visual complexity. The focus on cluttered, \"in-the-wild\" imagery is a necessary contribution.\n\n2. **Dual-Format Analysis**: The direct comparison of MCQ and Open-ended formats is a key strength. It provides a quantitative basis for the intuition that MCQ overestimates model capabilities, highlighting a significant $\\approx$23% average performance drop.\n\n3. **Failure Analysis**: The demonstration that standard improvement techniques (CoT, SFT) provide marginal, inconsistent, or even negative gains is an important finding for the field, pointing to deeper representational deficits.", "weaknesses": "1. **Statistical Robustness of the Benchmark**: The primary methodological flaw is the dataset's scale. 1,400 items spread across 30 subcategories means each sub-task is evaluated with a small sample (as few as 25 items, averaging <50). This $n$ is insufficient to draw robust, fine-grained conclusions. The granular analysis in Tables 5-9, while interesting, risks being statistically noisy. \n\n2. **Perplexing SFT Dynamics**: The SFT analysis (Sec 5.4, Fig 4)  is central to the paper's insight, but the results are anomalous and underexplored. The sharp U-shaped curve in open-ended performance (dropping from 34.4% to 12.6% before recovering to 35.5%)  is highly non-trivial. The paper gestures at \"catastrophic forgetting\"  but provides no direct investigation. This dynamic must be rigorously explained (e.g., via representational analysis or multi-seed validation) to be a credible scientific finding rather than a training artifact.\n\n3. **Marginal Novelty in a Crowded Field**: The paper's own related work (Table 1) demonstrates this is an extremely crowded and concurrent field (e.g., SpatialMM, OmniSpatial, BLINK, VLMAD). The claim to novelty rests on \"real-world complexity\" , yet several competitors also use \"Internet\" or \"Mix\" data sources. The authors must provide a much sharper justification for why SPATIALAB's 1,400 \"manual\" items provide fundamentally different insights than the (often larger) concurrent datasets.\n\n4. The model set omits several state-of-the-art commercial VLMs (e.g., GPT o3/5, Google Gemini 2.5 pro), which weakens the headline claim about “current VLMs.” They have more robust and powerful ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIu5UHTcdV", "reviewer": "Reviewer_tmjh", "summary": "This paper introduces SPATIALAB, a new benchmark designed to evaluate the spatial reasoning abilities of vision–language models (VLMs) in real-world, unconstrained settings.\n\nThe dataset comprises 1,400 visual question–answer pairs across six high-level categories (e.g., Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry) and supports both multiple-choice and open-ended evaluations.\n\nThe authors benchmark over 25 state-of-the-art models (open- and closed-source) and provide quantitative comparisons against human baselines, together with error analyses and several reasoning interventions (e.g., Chain-of-Thought prompting, supervised fine-tuning, and multi-agent reasoning).\n\nThe work aims to reveal systematic weaknesses in current VLMs’ spatial reasoning and propose SPATIALAB as a comprehensive diagnostic framework.", "questions": "Could the authors please justify their contributions in the \"in-depth analysis\" and provide key takeaways from it?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "**Comprehensive empirical evaluation.** The authors test a wide range of modern VLMs under multiple evaluation formats, which provides useful diagnostic data and an updated empirical snapshot of model limitations in spatial reasoning.\n\n**Well-structured benchmark.** The dataset taxonomy (6 categories × 5 subcategories) is clearly defined and covers a broad set of spatial reasoning tasks beyond synthetic toy examples.\n\n**Clear presentation.** The paper is readable and systematically organized, with detailed tables and qualitative examples that make the results easy to interpret.\n\n**Reproducibility focus.** The authors discuss data collection, annotation, and quality control in detail and commit to open release, which is commendable for community benchmarking.", "weaknesses": "**Lack of methodological contribution.** The work’s novelty lies almost entirely in dataset construction and large-scale evaluation.\nThere is no new modeling approach, algorithm, or analytical framework proposed.\nWhile benchmarks can be valuable, ICLR typically expects either new learning methodology, representation insights, or deeper diagnostic mechanisms beyond dataset release.\n\n**Limited depth of analysis.** Despite extensive tables, the analysis remains descriptive rather than mechanistic.\nThe paper identifies “what fails” (e.g., navigation and occlusion tasks) but not “why” in terms of representation or model architecture.\nThere are no probing studies, attention analyses, or causal/intervention experiments that explain the underlying representational failure modes.\nMany claims (e.g., “models lack geometric grounding”) are plausible but unsupported by direct evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mSFuq1MnRX", "reviewer": "Reviewer_w6pQ", "summary": "This paper introduces SPATIALAB, a comprehensive benchmark for evaluating spatial reasoning in vision–language models (VLMs) under realistic, unconstrained visual conditions. It contains 1,400 visual question–answer pairs spanning six major categories and 30 subcategories, each testing aspects like depth, occlusion, orientation, and navigation.\nThe benchmark supports both multiple-choice and open-ended formats, enabling comparison between discriminative and generative reasoning.\nExtensive experiments on 25+ models (open-source, proprietary, reasoning-tuned, and spatially specialized) reveal a substantial gap between human and model performance (e.g., 54.9% vs. 87.6% on MCQ; 40.9% vs. 64.9% on open-ended).\nThe paper provides error analysis, fine-tuning experiments, and attempts at improvement via CoT prompting, self-reflection, SFT, and multi-agent systems.", "questions": "1. How exactly does SPATIALAB differ from OmniSpatial or Spatial-MM beyond taxonomy size and annotation detail?\n\n2. Are there reasoning types uniquely represented here?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Evaluation\n- Evaluates over 25 VLMs, including open-source and proprietary systems, and human baselines.\n- Dual-format testing (MCQ + open-ended) is valuable, revealing a 20–25% accuracy gap between the two modes\n\n\nDiagnostic and Actionable Insights\n- The benchmark reveals concrete gaps (e.g., geometry-aware supervision, spatial chaining, embodied data) that can guide future research. The diagnostic perspective makes it a useful community tool even without conceptual novelty.", "weaknesses": "Limited Conceptual Novelty\n- Many recent benchmarks (OmniSpatial, BLINK-Spatial, Spatial-MM, SpatialRGPT, EmbSpatial) already use real-world imagery, multiple spatial categories, and QA-based evaluation. SPATIALAB’s innovation lies mainly in breadth and integration, not in introducing new reasoning types or data modalities\n\nLimited Guidance on Model Improvement\n- Although weaknesses of current VLMs are carefully diagnosed, the paper offers little practical guidance or insight into how to overcome them. The discussion remains observational (what fails) rather than prescriptive (how to fix it), limiting its utility for researchers aiming to design better spatial reasoners.\n\nModerate Dataset Scale\n- Despite 1,400 QA pairs sounding large, it is relatively small compared to existing multimodal datasets (often tens or hundreds of thousands). The modest size restricts its usefulness for training or fine-tuning and confines SPATIALAB to evaluation only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6B4sEBC4gV", "reviewer": "Reviewer_rSCs", "summary": "The paper introduces SpatialLab, a benchmark designed to evaluate spatial reasoning in vision-language models (VLMs). It features real-world images and carefully annotated questions that span a diverse range of spatial reasoning tasks. Using this benchmark, the authors assess various off-the-shelf VLMs and further analyze performance improvements achieved through different enhancement strategies, including prompt-based, multi-agent-based, and supervised fine-tuning (SFT)-based approaches.", "questions": "- In L1132, the authors mention “we review prior benchmarks … analyze their limitations.” Could the authors elaborate on the specific limitations identified in each benchmark? A comparison table summarizing these would greatly aid readers and inform future benchmark design.\n- [Suggestion] While page limits are understandable, brief descriptions of each improvement approach in Section 5.4 would improve clarity. For instance:\n  - What self-reflection prompt was used?\n  - Which dataset and dataset size were used for SFT fine-tuning?\n  - What base VLM was used?\n  - Could the authors provide a one-sentence description of SpatialXolver?\n- [Suggestion] Adding a table of contents to the appendix would help readers navigate the paper more easily.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Carefully curated set of images and questions covering major spatial reasoning types.\n- Comprehensive analysis showing that even proprietary VLMs perform worse than humans across nearly all subtasks.\n- Additional image complexity analysis provides insights into which visual domains require more attention in future training datasets.\n- In-depth quantitative and qualitative error analysis.\n- Evaluation of multiple strategies for improving spatial reasoning in VLMs (prompt-based, multi-agent-based, SFT-based, etc.) — particularly valuable in Section 5.4.", "weaknesses": "- The evaluation section (5.4) could be further strengthened by including reinforcement learning (RL)-based approaches for comparison, though this is not strictly necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)