Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
Authors: Xinmiao Huang
Qisong He
Zhenglin Huang
Boxuan Wang
Zhuoyun Li
Guangliang Cheng
Yi Dong
Xiaowei Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Spatial Reasoning
Vision-Language Models
Benchmark
Primary area: datasets and benchmarks
TL;DR: We created Spatial-DISE, a new benchmark to test dynamic cognition spatial reasoning. We found top VLM models fail at cognitive skills like mental simulation, not perception, showing a universal gap to human performance.

Abstract: Spatial reasoning ability is crucial for Vision Language Models (VLMs) to support real-world applications in diverse domains including robotics, augmented reality, and autonomous navigation. Unfortunately, existing benchmarks are inadequate in assessing spatial reasoning ability, especially the \emph{intrinsic-dynamic} spatial reasoning which is a fundamental aspect of human spatial cognition. In this paper, we propose a unified benchmark, \textbf{Spatial-DISE}, based on a cognitively grounded taxonomy that categorizes tasks into four fundamental quadrants: \textbf{I}ntrinsic-\textbf{S}tatic, Intrinsic-\textbf{D}ynamic, \textbf{E}xtrinsic-Static, and Extrinsic-Dynamic spatial reasoning. Moreover, to address the issue of data scarcity, we develop a scalable and automated pipeline to generate diverse and verifiable spatial reasoning questions, resulting in a new \textbf{Spatial-DISE} dataset that includes Spatial-DISE Bench (559 evaluation VQA pairs) and Spatial-DISE-12K (12K+ training VQA pairs).
Our comprehensive evaluation across 32 state-of-the-art VLMs reveals that, current VLMs have a large and consistent gap to human competence, especially on multi-step multi-view spatial reasoning. Spatial-DISE offers a robust framework, valuable dataset, and clear direction for future research toward human-like spatial intelligence. Benchmark, dataset, and code are available at https://shinmohuang.github.io/spatialdise_page/Spatial-DISE .

Published as a conference paper at ICLR 2026
SPATIAL-DISE: A UNIFIED BENCHMARK FOR EVAL- UATING SPATIAL REASONING IN VISION-LANGUAGE MODELS
Xinmiao Huang', Qisong Hel, Zhenglin Huang', Boxuan Wang', Zhuoyun Li', Guangliang Cheng', Yi Dong",, Xiaowei Huang School of Computer Science & informatics, University of Liverpool
Comparison of Existing Benchmorks Existing Spatiel Bench
1)) Fromowork of Spatid--ISE
Synthatic Data Generation Pipeline
peoticed person
Creidiing the
Intrinsic Dynomic:
2D/30 Folding
Extrissic Extrrisic-Dyramicc
2D Combination
Initidiiattion Seeding
Core Asset Seneration
Distraction
Generstion
Genenation
Verifisble
Retation and Punch
Quection ond Cornect Anower
Distraction
Strictggy
the pardue baceted with
respct
Extraa
Static
57%
Rendering
Geometric Varaation
Retthem/Orient
Errors
Incornect
VieNs
Replocetment
Component
Controlled Rendering Formatting
Quality Control
Spatial-DISE
S Which camers?
Intrinsic Static:
16
Extrinsic Stotic: Projection
Lorge-Rocie Row
Synthetic Finding
Duplication
Antwers
Mizsing
2D Shape Finding Shope Finding
What' sUp
mut froit of
20
Folding
World Data
Synthetic
Dataa
mag the right cpp
Shope Fixding
3D
Com Linctio
Fell
20%
Spatisl- DISK
Punch 14%
Wark
Stope Peciig
Spatial-DISE Bench
from
30 Renotion 20%
Spatte-DDSE 12K
Faidin
36 12%
make the neu 10 figur.
Fige es the only loar
Gemes Flasi
Space Tooter
YEARR
Clastia
Feiding
35 Retotiin
Projection
Dynonic
20%
VLMs & Human Performences
Statistics of Spoticl-DI55 DISE Bench and Datoset
Figure 1: A Comprehensive Overview of the Opatial-DISEE Framework, Generation Pipeline, and Benchmark Statistics. Comparison of examples from existing benchmarks, which primarily test general static reasoning. with cognition intrinsic-dynan dynamic tasks from our Spatial-DDe DISE benchmark. b) introduces the core DISE taxonomy, showing the four quadrants of spatial reasoning and their distribution in the 559-pair evaluation bench. c) presents evaluation results, showing a significant gap between model and human performance. d) details the synthetic data generation pipeline implemented in Blender, and e) provides statistical breakdown of the task categories within both the Spatial-DISE Bench and the Spatial-DISS --2
ABSTRACT
Spatial reasoning ability is crucial for Vision Language Models (VLMs) to sup- port real-world applications in diverse domains including robotics, augmented reality, and autonomous navigation. Unfortunately, existing benchmarks are in- adequate in assessing spatial reasoning ability, especially the intrinsic-dynamic spatial reasoning which is a fundamental aspect of human spatial cognition. In this paper, we propose unified benchmark, Spatial-DISE. based on a cogni- tively grounded taxonomy that categorizes tasks into four fundamental quadrants: Intrinsic-Static, Intrinsic-Dynamee -ynamic, Extrinsic-Statie Statii, and Extrinsic-Dynamit spatial reasoning. Moreover, to address the issue of data scarcity, we develop scal- able and automated pipeline generate diverse and verifiable spatial reasoning questions, resulting in new Spatial-DISE dataset that includes Spatial-DISE Bench (559 evaluation VQA pairs) and Spatial-DISE- 12K (12K+ training VQA pairs). Our comprehensive evaluation across 32 state-of-the-an VLMs reveals that, current VLMs have a large and consistent gap to human competence, especially on multi-step multi-vien spatial reasoning. Spatial-DISE offers robust framework, valuable dataset, and clear direction for future research toward human-like spatial intelligence. Benchmark, dataset, and code are available at Spattia--DSEE
"Corresponding author; xi aiwee huang@1 iverpool .ac.uk. yi dong@1 veepool .ac.uk

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in vision language models (VL.Ms) have demonstrated impressive capabilities in
Peng ct al., [023]] Anil et al. 2025). scene 2023). and visual question answering (Wang) et al., 2023] Anil al. 2025 Alayrac et al. 2022). However, their capability for sophisticated
various tasks such as object detection (Li et al.,
2022
caption [Alayrac et al. 2022; Chen et al. 2023 Li et al.
dynamic spatial reasoning, a comerstone of human cognition and a critical requirement for appli-
cations in robotics, augmented reality, and autonomous navigation, remains a significant limitation (Ramakrishnan et al. 2024) and largely under-evaluated.
Existing benchmarks for evaluating the spatial reasoning of VL.Ms have three major limitations. Firstly, current benchmarks lack a systematic cognitive framework for categorizing and evaluating different types of spatial reasoning abilities, leading to fragmented. unbalanced tasks that typically focus only on basic skills(iiu et al. 2023 Chen et al., 2024 Cheng et al. 2024). Consequently, there is notable scarcity of benchmarks designed to evaluate deeper cognitive abilities. Secondly, current benchmarks are limited in scope, focusing predominantly on static spatial questions that do not require multi-siep dynamic reasoning (Wang et al. 2024 Han et al. 2020). Consequently, crucial cognitive abilities like mental rotation and folding are significantly under-tested. Thirdly, the few benchmarks that address dynamic tasks are insufficient in scale (Ray et al. 2024 Ramakrishnan et al.) 2024),, making them insufficient to robustly evaluate the capabilities of the model or to drive further model development.
To bridge these gaps, we propose Spat Unlike previous benchmarks that focus on isolated abilities or static scenes, Spatial-DiSE adopts a unified 2x2 cognitive taxonomy (Maier] 1996] Uttal et al. 2013), as illustrated in Figure (b), which covers both 2D and 3D aspects, and critically, places a strong emphasis on dynamic spatial reasoning tasks. The first dimension distinguishes between intrinsic information, which defines an object by its internal parts and their arrangement, and extrinsic information, which pertains to the spatial relations among different objects; the second dimension differentiates static tasks, which involve fixed and stationary information, from dynamic tasks, which require mental transformation. Figure I provides an overview of the Spatial-DISE framework, generation pipeline, and benchmark statistics.
Spatial-DISE contains more than 12K verified spatial reasoning Visual Question-Answer (VQA) pairs. is created through a combination of real-world data collection and synthetic generation using Blender Firstly, it has set of 559 real-world and synthetic VQA pairs split into 10 different spatial reasoning tasks, covering the four DISE quadrants. Secondly, it includes a set of over 12,000 verified 3D spatial reasoning VQA pairs that are generated through an automated pipeline. The synthetic VQA pairs spread across five 3D Spatial Reasoning tasks.
We conducted a comprehensive evaluation across 32 state-of-the-art (SOTA) VLMs on Spatial-DISE. These encompassed a range of advanced VLMs, featuring both proprietary and open-source models: 22 foundation models, 7 reasoning models, and 3 models post-trained with spatial-related datasets. Our findings reveal a profound and universal weakness in current VL.s.. Overall performance remains low, with most models scoring only slightly above random chance and far below the human baseline. Our in-depth error analysis further reveals that these failures stem not from simple visual perception, but from fundamental deficits in cognitive processes like rule-based reasoning and mental simulation. Our key contributions include:
A Cognitively Grounded Taxonomy: We adopt cognitively grounded framework that, unlike previous task-oriented benchmarks, provides a unified taxonomy to classify any spatial task, revealing specific weaknesses like dynamic reasoning that are otherwise obscured.
A Scalable and Verifiable Data Generation Pipeline: We design and implement a novel, automated pipeline using Blender to programmatically generate complex 3D spatial reason- ing tasks. This methodology is a key contribution, offering a reusable tool for the community to overcome the data scarcity that has limited previous dynamic reasoning research. The pipeline ensures verifiability through seeded randomization and reproducible distractor generation strategies.
A Unified & Verifiable Cognitive Benchmark: Leveraging our pipeline, we introduce the Spatial-DISE and the accompanying 2.000--QQA dataset, as a benchmark to systematically
httpps//wwww.heedderrrgg
2

---

Published as a conference paper at ICLR 2026
and extensively evaluate complex cognitive spatial reasoning tasks at scale sufficient for robust evaluation and future model training.
Exploring the Boundaries of Cognitive Spatial Reasoning in VLMs: By benchmarking 32 SOTA models, we define the current boundaries of VLM capabilities cognitive spatial reasoning. Our analysis reveals universal performance ceiling, especially for multi-step mental simulation, highlighting the significant gap between Al and human-level spatial intelligence.
2 RELATED WORK
Table 1: Comparison of Existing Benchmarks under DISE Taxonomy. Abbreviations 1-S: Intrinsic- Statie; I-D: Intrinsic-Dynamic, E-S: Extrinsic-Statie F-D: Extrinsic-Dynamic.
Benchmark
Data Scale
Domain
Source
E.
IDD
E-S
E-D
SpatialRGPT Chenaln et al
2024
Ik+
General
Real-World
X
BLINK
World
X
VSR Lu
10k
General
Keal-World
/
/
X
What's Up
Kamath et
2023
820
General
Real-World
X
X
V
X
CV.Bench
Tong
Real Real-World
X
LEGO Puzzles
2025
1100
Objects
Syn.
X
V
V
COMFORT
Zhang et al 21.5
1220
Objects
Syn.
V
X
/
X
3DSRBench
(Ma al 2025
2772
General
Real-World
X
X
VSI-Bench Yong
General
Real-World
V
Sparial457
Wang ct
2125
20k+
Objects
Syn.
V
X
/
Q- 2-SpatiaBenchh (Liao et
2024
271
General
Real World
X
X
SAT Ray al.
175k
General
Reul-World-Syn.
X
SPARE3D HHn ef
2020
10k+
Cognition
Syn.
V
X
X
X
SpatialEval Wang al..
13k+
Cognition
Real World
BSA &t
Cognition
Real-World
SPACE [Ramakrishnan al
2024
5k+
Cognition
Real-World
X
OmmiSpatal Uia ml. 2025
1.5k
General-Cognition
Real World
Spatial-DIEE Bench
559
Cognition
Syy.
Spatial-DISE-12K
12k+
Cognition
Scal-World+Syn.
The evaluation of spatial reasoning ability in VLMs has been an active area of research, but prior work suffers from critical gaps in scope, cognitive depth, and scale. Table compares existing benchmarks in coverage scope, number of instances, and data sources.
Previous benchmarks offer a fragmented evaluation, lacking a unified cognitive framework. Bench- marks such as LEGO-Puzzles (Tang et al.) 2025), SAT (Ray et al. 2024) and VSI-Bench (Yang et al. 2024) are confined to narrow, specific tasks, preventing a holistic assessment of a model's true spatial abilities. Spatial-DISE overcomes this by introducing unified 2x2 cognitive taxonomy. This framework, rooted in cognitive science, enables a comprehensive and balanced evaluation, allowing for the precise diagnosis of model weaknesses.
Furthermore, prior benchmarks have a disproportionate focus on static reasoning. A vast number of wwwchearks--aae -inimmaark--iilluiinng GPT (Cheng al. (2024), SPARE3D (Han et al. 2020). VSR (Liu et al. 2023). CV-Bench Tong et al. 2024). BLINK (Fu et al. 2024). and What's sUp Kamath al. (022). SpatialEval (Wang et al. 2024) primarily test model's ability perceive fixed scenes and relationships. They evaluate what models "see" but not how they can "reason" about potential changes. Spatial-DISE targets this gap by focusing on intricate dynamic reasoning to thoroughly assess cognitive tasks such as
3D rotation and folding.
Finally, while SAT (Ray et 2024), SPACE (Ramakrishnan et al. 2024). BSA (Xu et al., 2025) and OmniSpatial (Jia et al. 2025) have begun to explore the dynamic domain, Spatial-DISE's uniqueness lies in its integration of a cognitively unified framework and a verifiable generation process. Our work complements these existing efforts by providing structured and reproducible approach to understanding model failures. Although one might consider aggregating existing benchmarks into a bentaabbnchmark, such union would remain OlSE-imbalanced hhrmmae-eteroggeeouus and unable to provide verifiable training data for the underserved dynamic quadrants gaps that Spatial-DIST specifically designed address.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Drawing from cognitive science research (Maier 1996 Uttal et al. 2013), we organize spatial reasoning into two key dimensions: Intrinsic vs. Extrinsic and Statie vs. Dynamic. The first dimension differentiates between Intrinsic vs. Extrinsic information. Intrinsic information refers to the essential characteristics and relationships that define an object. Extrinsic information refers to the relation among objects in group, relative to another to overall framework. The second dimension, Static vs. Dynamic, centers on movement. Movement can alter intrinsic information, such through folding, cutting, rotation. It can also shift object's position relative other objects and the surrounding environment.
This framework comprehensively covers existing task classifications by placing them into four distinet quadrants. This creates a 2x2 taxonomy that categorizes spatial reasoning into four distinct quadrants: Intrinsic-Static (I-S) tasks involve analyzing the internal properties of a single, unchanged object; Extrinsic-Static (E-S) tasks assess the relationships between multiple objects in a fixed scene; Intrinsic-ynnant (I-D) tasks require mentally simulating transformations on a single object; and Extrinsic- Extrinsic-Dynamic (F-D) tasks involve reasoning about the changing spatial relationships between multiple objects.
3.1 TASKS DESIGN
We designed 10 cognitive science- based tasks to probe spatial reasoning. Figure 2 provides a visual guide this categorization, showing how various spatial tasks map onto our Spatial-DISE taxonomy, The 10 tasks we designed not only fully map to the four quadrants of the DISE framework, but their design inspiration also stems from classical psychometric tests. These tasks are specifically designed to assess core spatial abilities such as mental rotation and spatial visualization (see Appendix A. for detailed correspondence).
Foldiigg Which figure that shows how the left-hand figure will look when folded alona the dotted Inc?
3D Foldinng
Rafation
the can made
Work out which 3D figure in the grey box has been retated make the new 30 figure.
Fold and Punch
2D Shape Findling shape right. which figure that contains the shape?
folded hale punched Work option shows square when unfolded.
figures
2D Combination
neressary given figure?
3D Shape Findlig
2D Rotation
figures on rigure should reprace the blank toca?
Which the figures presented rotation
3D Combinaaiion
Projection
Work out which aption is the 3D figure viewed from the right.
Work out which set of blocks can be put
together to make the 3D the left.
@
Figure 2: 10 Tasks in Spatial-DISE Bench. Orange shows the Intrinsic- mnrrinii-DDynmmic Tasks, Green shows the Staaii Tasks, Pink shows the Extrinsic-Staie Static Tasks and Blue shows the Extrinsic Extrinsic-Dynanii Dynamic Tasks.
Intrinsic-Static Tasks. These tasks evaluate the understanding of an object's fixed, internal spatial properties without transformation. This assessed through 2D/3D Shape Finding, which requires identifying a hidden shape within a more complex figure determining cube's missing face from other views, thereby testing the static analysis of intrinsic part-whole relationships.
Intrinsic- Intrisic-Dynamic Tasks. These tasks test the ability to mentally manipulate the internal properties of an object, requiring pure mental simulation. This includes 2D/3D Rotation, a classic test of mental transformation that requires predicting object's 's appearance after rotation, and 2D/3D Folding & Folc&Punch, which tests the outcome of folding a 2D net into a 3D shape or unfolding a punched paper.
4

---

Published as a conference paper at ICLR 2026
Extrinsic-Static Tasks. These tasks investigate the understanding of fixed spatial relationships from an external viewpoint. This is probed using 3D Projection, which requires identifying the correct 2D orthographic projection of 3D object from specified external direction, task dependent on the extrinsic relationship between observer and object.
Extrinsic-Dynamic Tasks. These tasks assess the ability to reason about the changing relationships between multiple objects or parts. The primary tasks here are 2D/3D Combination, which require mentally assembling separate parts into coherent whole and thus tests the ability to simulate how components must move, orient, and connect.
3.2 BENCHMARK CURATION
To ensure the scientific rigor and validity of the dataset, a 3-stage curation pipeline was employed. This process integrates real-world data from open sources with a scalable synthetic data generation, followed by human quality control.
Stage 1: Real-world Data Collection. The initial phase aimed establish conceptual foundation and a repository of templates for subsequent data synthesis. We collected a corpus of existing, high-quality spatial reasoning problems from publicly available and validated sources, including academic psychometric tests and professional aptitude assessments. This phase yielded an initial corpus of 1180 VQA pairs, providing diverse set of concepts and structures that informed the automated generation process. Detailed real-world data collection is presented in Appendix A.3.
Stage 2: Scalable Synthetic Data Generation. As illustrated in Figure dd. this core stage was designed to overcome the scale limitations of existing benchmarks, particularly for dynamic and 3D tasks. Leveraging the Blender engine, we transformed the concepts from the initial corpus into scalable, automated pipeline. This pipeline follows a general paradigm, customized for each of the five 3D task types: 1) Initialization and Seeding: Each generation task begins with a unique question id, which is hashed to create reproducible random seed, ensuring the uniqueness and verifiability of every generated instance. 2) Core Asset Generation: We generate the core 3D object for given problem. This includes creating complex, irregular shapes for tasks like 3D Rotation or generating cubes with unique face textures for 3D Folding and 3D Shape Finding. 3) Question and Correct Answer Rendering: We render the question and correct answer images from optimal camera perspectives. 4) Systematic Distractor Generation: To ensure the diagnostic challenge of each item, the pipeline implements a suite of tiered strategies create plausible, near- miss distractors. These strategies include: Geometric Variations: Introducing subtle alterations to the core object's geometry. such as adding or removing components. -Paatem/Orientaati Errors: Generating incorrect texture layouts or orientations on the faces of an object. Incorrect Views: Rendering a correct object from an incorrect orthographic perspective for projection tasks. Component Replacement: Swapping correct part with a geometrically similar but incorrect one in assembly tasks. 5) Controlled Rendering and Formatting: All question, correct answer, and distractors are rendered in a controlled virtual environment with consistent lighting, materials, and camera parameters. The final output is a standardized VQA data pair. Detailed illustration and pseudocode of synthetic data generation is shown in Appendix A.4.
Rigorous Human Quality Control. Following generation, all benchmark synthetic instances underwent rigorous manual verification process guarantee the benchmark's 's integrity and reliability. The review protocol assessed each instance against three quality criteria. 1) Solution Uniqueness: Each problem must have single, unambiguous correct answer. 2) Accuracy and Clarity: All images must be free of rendering artifacts, and the corresponding questions must be clearly articulated. All options must be valid according the task's criteria. 3) Redundancy
Stage 3:
Elimination: The instance should not be logically or visually redundant with other items in the dataset. Instances failing to meet these standards were removed from the final dataset. Combined with real- world data, two sets of Spatial-DISF. are created:
Spatial-DISE Bench: An evaluation set of 559 carefully selected VQA pairs, covering all 10 task types and four DISE dimensions, designed for model benchmarking.
Spatial-DISE 12K: A large-scale dataset consisting of over 12,000 verifiable VQA pairs cover five 3D tasks, intended as a valuable resource for the future training and fine-tuning of spatial reasoning capabilities in VLMs.
5

---

Published as a conference paper at ICLR 2026
4 EVALUATION ON SPATIAL-DISE BENCH
4.1 EXPERIMENT SETTING
Benchmark Models. For the evaluation, we select a diverse set of 32 models across 10 model families. Our selection includes both proprietary and open-source models, spanning general foun- dation models, reasoning models, and spatial-speciifee models. For proprictary foundation models, we evaluated Claude3.7-Sonme Doubao I.LVVL (Guo et al. (2025b). Gemini-2.0-Fin
Gemini- 2.5-Flash (w/ and w/o thinking), 1-n-no, GPT--40, GPT--4omini, GPT-5, and 04-mini. For open-source foundation models, evaluated Intern LL-3-
(Zhu et al. 2025). Llama-
3,2-1 11-1BB-iiiion (Grattafiori et al. (024)). Kimi-VL-A3B (Du et al. 2025). Ovis2-(8B/ (12-188/6BB) 16B] (Lu et al.
2024). (SB (Tong al. 2024) and -L-
(Bai al. 2025
For reasoning models, we evaluated LLa VA-CoT (Xu et al. 2024). LMM-RI (Peng et al.
2025
VLM-RI (Shen al. 2025). "LAA-Thinker--P"
(Chen al. 2025). Kimi-Vis (LL-A3B-Thinkin
Du et al. 2025) and Doubao- 1.5 15-thinking (Guo et al. 2005b).. For spatial-specifice models, we
evaluate Thackehinker (Chen et al. 2024), SpaceOM (Chen et al. 2024) and SpaceR (Ouyang et al.,
2025
Baseline. For comparison, we include Random Guessing and Human Performance. To establish a robust, psychometrically sound human baseline, we recruited 54 diverse participants (ages 15-55) and employed a matrix-samplin design to collect 1,679 valid responses. This design ensured each question was answered by average of three unique participants. The final human performance is reported as the average accuracy across all responses and cross- validaaad with Item Response Theory (IRT). More details of human performance in Appendix B.1.
Implementation Details. We evaluate multiple-choice accuracy using exact match via the VLMEvalKit (Duan et al., (025).. Deepseek-RI (Guo et al.] (2025a) is used to parse answers from malformed model outputs. Additional implementation details are provided in the Appendix B.2.
4.2 MAIN RESULTS
Our comprehensive evaluation reveals that spatial reasoning remains a significant and universal challenge for current VLMs. Table 2 3 present the main results of our evaluation. More results are presented in Appendix B.3. We summarize the key findings as followed:
Spatial reasoning remains a universal challenge. The overall performance across all 32 tested models was low, with average accuracy of 28.4%, only marginally above random chance (25%) and falling drastically short of the human bascline (76.8%). Of all models evaluated, the reasoning- enhanced Doubaol.55 Masbaoo...VL--ttte 1.5-VL-thinkkng achieved the highest overall accuracy at 42.0%. This widespread underperformance indicates a critical weakness in tasks requiring genuine mental transformation, highlighting failure move beyond pattern recognition to true spatial cognition.
Multi-Step transformations overwhelm VLMs reasoning. Models demonstrate a particular vulner- ability to tasks requiring sequence of mental transformations. The Fold and Punch task, which requires simulating a fold, a punch, and then an unfold, serves as clear example of this failure. Even the top-performing model, 1oubao-I..-ttinkite
only achieved 30.8% accuracy, while the average of all models is only 25.4%, performed near random chance. This indicates that while a model might handle a single transformation, its ability to maintain a coherent mental state breaks down across multiple steps. This suggests a critical deficit in "spatial working memory," preventing models from reliably tracking an object through sequence of changes.
Post-training shows improvement but not enough. The results reveal that post-training with rein- forcement learning or fine-tuning on spatial datasets offers limited improvements. While models like Doubao-1.5 1.5-thinking and SpaceThinker showed performance gains, their absolute accuracies remain low and far from the human bascline.
Static comprehension is not solved precursor to dynamic reasoning. Counter-intuitivell, the results show that proficiency in static reasoning is not a prerequisite for dynamic reasoning. Several top models perform better dynamic tasks than static ones. For example, Gemini2.0-Flac scored significantly higher on dynamic tasks (38.3%) than on static tasks (23.6%). Doubao-1 -15 1.5-thinking even outperform human performance in Extrinsie-Dynamic questions. This suggests that models

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results of 32 SOTA models and 2 models SFT on Spatial-Dul DDStE Row colors: Base. A vs base. Reasoning. Spatial, SFT on Spatial-DISE 12k. . row shows the absolute change percent- age points (pp) relative to its base model and is placed between the parent and the derived model. Values are accuracy (%); brackets use |lower, upper] for the 95% CI. Bold indicates the highest accuracy; Underline indicates the second highest.
Model
Acc. 195% CI
E-5
E- 195%
I-D
1-51 CII
Proprictary Rasey Claude 3.7 Sonnet
Base
30.6% [2688 34.3]
22,6%
[14.3, 32.1]
31.4%
[21.4, 42.9]
32.45 (2714, 37.7]
31.0% [21.8. 41.4]
DoV
Base
(29.9,37.])
40.5]
37.1%
[25.7,
(28.6. 39.00
[25.3,44 44.8]
(*8.2)
30.99
[1577)
(17.3)
(111)
Douhaol.SV Biool..V---iiiiii
RLHF-RIVF
42.0% [37.9. 46.2)
61.9%
[51.2, 72.6
31%
(21.4. 42.9]
40.9%
[35.5, 46.2)
35.6%
[25.3, 46.0)
Gemini 2.0 Flash
Bare
34.2% [30.4, 37.9)
25.0%
(15.5, 34.5]
31 4%
121.4, 42.91
41.8%
[6.55
17.2%
19.2. 25.3]
Gemini Klash
Ruse
[27.7.
(3.19
29.9]
Gemini Flash weo thinking
Bare
32.0%% [28.3, 35.8)
15.5% [8.3, 23.8]
28.69 67
[18.6, 38.6]
39.6%
45.00
23.0%
[14.9, 32.2]
GPT-4.1 nano
Baze
29.3% [25 53.1]
29.8%
[20.2. 40.5
35. 7%
125.7,
47.1
31.1%
[66 36.2]
17.2%
[9.2, 25.31
GPT-4o
Base
28.1% [34.5, 31.88
26,25 [16.7. 35.7]
9% 11.99.
12.9
29.9%
[34% 34.9)
27.6%
[18.4 36.8]
GPT-4o-miii
[22.0. 29.2)
123.0. 33.0
28.7%
19.55.37 37.91
GPT-S
Bare
30.1% [36.3, 34.01
23.8% 115.5, 33.3]
7%
115.7, 35.7]
33.6% [2.66 39.0]
26.45
1 35.61
o4-mini
Ruze
33.3% [29 37.2)
16.7%
[9.5, 25.0]
25. 7%
115.7.
15.7]
36
8..%
[31 42.1
42.5%
[32.2, 52.9)
Proprictary Average
31.9% [29.0, 34.71
26.0%
(17.2, 34.81
28.9%
(25.6,
32.31
35.2%
(32.0, 38.41
27.7%
[22.3, 33.01
Open-source Bases L L1mma3V-118
Baxe
24.5% [20.9. 28.1]
29.8%
[20.2, 39.3)
14.3% 17.1, 22.91
25.5%
[20.8. 30.5]
24.1%
[14.9. 33.3)
(10.5)
(18.6)
(U 1.00
(3
CoT
CoI
24.0% 120.6, 27.5)
29.8%
120.2, 39.3
22.9%
[12.9, 32.91
24.5%
[19.8. 29.21
17.2%
[10.3, 25.3]
Cambrian- 13B
Baze
26.7% [23.1. 30.4]
25.0%%
116.7. 34.51
32.9% 121.4, 44.31
25.8%
[21.1. 30.8)
26.4% 35.66
Cambrian-88
Bare
22.9% [19.5, 26.3]
19.0%
110.7. 27.4]
15.7%
[7.1. 24.3]
23.9%
[192, 28.6
78.7% 119.5. 37.9]
V
Bare
[28.6. 36.3)
27.4%
117.9,36.9 36.91
30.0%
[30
26.4%
35..
Bare
31.1% 34,94
21.4%
113.1, 29.8]
31.4% [20.0. 42.91
37.1%
43.5
18.4% 110.3, 26.4]
V1.388
Buze
26.3% [22.7. 29.9]
23.8%
[15.5. 33.3]
78 6% (16.6 40.0]
30.8%
[25 35.8]
10.3% (4.6,1721
Kimi- YL-A3B
Baxe
24.3% 220.8,27.99
17.9%
[9.5, 26.21
27.1%
[17.1. 37.31
27.7%
622.6,
16.1% 24.2.2411
0.4)
(19.5)
1- Kimi- -L.Thiiinng
CaT+RI.
24.7% [21.1, 28.3]
27.4%
117.9, 36.9)
69%
(18.6, 38.6]
23.99
119.2,
21.8% [13.8. 31.0]
Ovis2-16B
Baze
26.3% [22.7, 29.9)
20.2%
[11.9, 28.6]
27.1%
117.1, 38.6]
31.4%
[26.1. 36.8)
12.6% [5.7, 19.5]
Ovis2-883
Bare
23.8% [20.4, 27.4
15.5%
1833 23.81
21
4% 112.9, 31:41
29.
[24.5. 34,6]
12.6% [5.7, 20.7]
Dwee2255VL-322B
Baze
323.4.30.91
21.4%
113.1,29.8 29.81
31.4%
32.71
27.6%
[18.4, 37.91
Qwen2,5-VL--7
Baxe
26.1% (22.5.
32.1%
[22.6. 42.91
21.3%
[14.3, 34.31
27.7%
[22.6. 32.7]
24.11 16.1%
(T 1.8)
(177)
(12.8)
0.9)
(" 10.3)
VLAA-Thinkes
GRPO
27.9% (1.73
27.4%
[17.9 36,91
27.1%
[17.1. 37.11
28.6%
(23.9.33.6
26.4% 35.62.35.6)
<1
(1.66)
(1
11.4)
<t 15:11
(1.66.)
Qwen2.-VVL-B--
(SD-122)
47.0%
[42.9,
66.7%
[56.0, 76.21
35.7%
124.3,
43.1%
[37.7, 48.71
51.7% [414, 62.1]
(7.).))
(223))
(172))
(.9)
( (,669)
SpaceR
SG-RLVR
27.0% [23.4, 30.8)
29.8%
[20.2. 39.31
17.1% 18.6, 27.11
29.6% 121.5, 31,6]
23.0% [4.9, 32.2]
Qwen2.5VVL33
[6.55,
34.5]
25.7]
(1.47,3144]
20.7,
(12:2)
(1 (188)
2
13.81
1 -LMM-R
PPO
26.1% [22.5, 29.91
29.8%
[20.2, 39.3)
20.0%
I11., 30.0]
26.1%
21.731111
1% [17.2,
(
(18.3)
1.5)
7.2)
(5.00)
VLM-RI
GRPO
34.71
[23.8, 41.0)
18.6%
110.0, 28.6|
128.6, 39.01
27.6%
[184,,6.1 36.8]
3.00
(1366
(1 12.9)
1.3)
(12)
VLAA-Thinkr-338
GRPO
25.9% [22.4, 29.5]
78 6% (19.0, 38.1J
30.0%
(20.0, 41.4]
27.7%
[23.0, 32.71
13.8% (6.9, 21.88
(7.67)
10.7)
(7.11)
(.2)2)
11.5)
Spwc/Thinker
SFT
32.91
[20.2. 40.3]
22.9%
35.71 (1577)
[27.7, 37.71
[10.3, 26.4] (15.7)
(1 12.4
1.14
SpuccMM
SFT
25.9% [22.4, 29.5)
31.0% [20.2,
24.3%
34.3] (1.33 (12:5)
26.7%
(22.0, 31.81
19.5% [11.5, 28.7)
<T 15:4)
21.4)
<t 11.00
(T (5.7)
SpaceOM-sit
SET (SD-12k)
[37.4.454] 45.4)
[41.7, 63.11
27.1%
(1..1,
[32.4. 43.11
55.2%
Open soeneewwtce..ee Average
26.2% (25.2. 27.3)
23.2%
[20.7, 25.81
25.7%
(22.2, 28.01
29.1%
(27.7, 30.61
19.3%(117.
21.71
Human Level
76.8% [74.8. 78.9)
61.1%
[56.6, 63.5]
81.1%
[76.7, 85.5)
80.2% 82.3]
76.8% 172.9, 80.7]
Random Guessing
25.4%
26.2%
24.3%
24.7%
not learning spatial reasoning in a human-like, scaffolded manner. Instead of building dynamic capa- bilities upon a solid foundation of static scene understanding, they appear be learning fragmented strategies, recognizing patterns of "change" without robust, underlying model of the static world.
Computational rigor can outperform fallible human simulation. This is evident in Doubao- thinking model, which surpassed the human bascline on E-D tasks. This superior performance can likely be attributed to the nature of 2D/3D combination. As confirmed by our human performance analysis (Appendix, Table 99) these tasks are particularly arduous and cognitively demanding for humans. 3D Combination commanding the longest mean response time (59.2s) among all tasks for humans, who must rely on fallible mental simulation. In contrast, we observed that the Doubao model transforms these challenges into computational problems by employing a more algorithmic strategy to compute and compare geometric features of components-suck as edges, angles, and connection points. Essentially, the model excels by converting a cognitively exhausting simulation task into a precise computational problem, domain where it holds distinct advantage over human intuition.
7

---

Published as a conference paper at ICLR 2026
4.3 FINE-TUNING ON SPATIAL-DISE-12K
To ascertain whether Spatial-DISF-12K effectively enhances the spatial reasoning capabilities of vision-language models, and to evaluate the cross-domain generalizability of this training, conducted supervised fine-tuning experiments. Specifically, two representative open-source VLMs- -wen2.5-VL--7 and SpaceOm wper fine-tuned using Low-Rank Adaptation (LoRA) applied to all linear layers utilizing the Spatial-DISE-12K training split (see Appendix B.4 for im- plementation details). Following optimization, we structure our comprehensive evaluation around four research questions (RQs), assessing the models the Spatial-DISE Bench alongside five established external benchmarks: CVBench, SAT, SPACE, OmniSpatial, and VSIBench_ _CQQ
Table 3: Accuracy for Qwen2.5-VL (Base SFT) and SpaceOm (Base SFT). A is SFT-Base in percentage points (pp).
Spatial-DISE
CVBench
SAT
SPACE
OmniSpatial
VSIBench_MMe MCQ
SpaceOm +DISE SFT
25.9%
68.8%
46.67%
27.22%
27.91%
31.05%
41.3%
70.33%
49.33%
32.6%
34.28%
33.7%
(1 15.4%)
(1 1.53%)
(1.2.66%)
5..8%)
61.337%%
(*2.65%%
Owen2.5-VL-7B
26.1%
75.9%
65.3%
28.7%
21.8%
19.3%
+DISE SFT SFT
47.0%
77.4%
69.3%
32.2%
34.0%
22.6%
(1 (0.9%)
(1 1.5%)
(..4%%%
(1 3.55%
(1 12.2%)
(1.33%
3D Comb (E-D)
+8.3%
+4.8%
+1.3%
Train setln
-10
-13.5%
(E:5)
-12.5%
-2.6%
-8.7%
-19.0%
+3.8%
3D Sh.Find (I-S)
-20.8%
-5.1%
-4.3%
-14.3%
+2.6%
4.3%
13.9%
118.3%
2.1%
Fold+Rot
-4.2%
-9.5%
-6.4%
-13.8%%
+4.8%
20 Camb.
20 Fold.
20 Rot.
20 Shape
Fold&&Punch
$0
2D test tasks
Test quadrent
(a) Cross Cross-quadrant transfer.
(b) 3D-2D transfer across DISE tasks.
Figure33 3: Transfer heatmap by fine-tuning tuning Qwen2.5- D0W-72.5-VL-BB on Spatial
Spatial-DISE DISE quadrant-win wise.
RQI: Does 3D spatial training cultivate a broad, structured set of in-domain skills? To address this, evaluated the fine-tuned models the Spatial-DISE: Bench. Empirical evaluation demonstrates that fine-tuning on Spatial-DISE: -itiial---SE-1122 yields substantial performance gains. Specifically, the overall accuracy of Qwen2.5-VL-
improved from 26.1% % 47.0% and SpaceOm from 25.9% 41.3%. The most pronounced enhancements were observed in Intrinsic-Dynamic and Extrinsic-Dyname tasks, alongside a remarkable surge in Qwen2.5-VL-7B3 Intrinsic-Static performance (scaling from 16.1% 5% to 51.7%).
Do models learn factorized spatial dimensions or entangled, quadrant-specific represen- tations? To elucidate the underlying mechanisms driving these gains, we isolated the training curriculum by fine-tuning Qwen2.5-VIL-7B exclusively on individual DISE quadrants and visualized the subsequent accuracy fluctuations (Figure Fa). The resulting heatmap reveals robust diagonal pattern, indicating profound quadrant-specific specialization: training on specific quadrant predomi- nantly amplifies performance within that corresponding domain. Conversely, off-diagonal transfer is largely minimal or even detrimental, failing to exhibit clean factorization along the Intrinsic/Extrinsic or Static/Dynamic axes. We observe distinct directional asymmetries (c.g., mild positive transfer of +3.9 pp from 3D E-D to I-S, which starkly contrasts with the strongly negative transfer from I-S to E-D). This implies that the four DISE quadrants encapsulate relatively distinct cohorts of spatial
RQ2:
8

---

Published as a conference paper at ICLR 2026
competencies; strengthening one family does not automatically confer a "universal" spatial skill, and overall benchmark gains necessitate exposure to multiple 3D quadrants during training.
RQ3: Can 3D spatial training induce representations that transfer to 2D reasoning tasks? To probe cross mooss-dimessional transfcrability, we maintained the single-quadrant fine-tuning paradigm but evaluated the resultant models across all 2D DISE tasks (Figure (3b). The findings expose qualitatively divergent reasoning regimes. Training on extrinsic-dyl extrinsic-dynamic 3D tasks catalyzed broadly positive transfer across varied 2D settings, suggesting that scene-centric dynamic reasoning fosters highly reusable representations applicable to 2D projection, combination, and occlusion problems. In stark contrast, training on intrinsic-static or extrinsic-static 3D tasks yielded narrowly constrained or even negative transfer. Furthermore, while intrinsic-dynamic training facilitated 2D rotation performance, concurrently degraded the model's capacity to solve simpler 2D problems. These asymmetric patterns underscore that object-centric static reasoning tends to form overly specialized schemas, which can actively interfere with tasks reliant on relative or dynamic reference frames.
RQ4: Do the acquired spatial competencies generalize to external benchmarks without catastrophic forgetting? To assess broader applicability, we evaluated the fine-tuned models on suite of external spatial benchmarks (Table Fine-tuning Spatial-DISE engendered consistent yet highly selective oft-of-domain gains, importantly demonstrating that this targeted spatial curriculum does not induce catastrophic forgetting of general visual capabilities (c.g.. -.ven2..-VL-BB--- sft improved marginally on CVBench). The most substantial improvements materialized on the SPACE and OmniSpatial benchmarks, which intrinsically emphasize viewpoint transformations and 3D-consistent reasoning. Conversely, benchmarks amalge aaalganating spatial reasoning with generalized language or diagrammatic comprehension exhibited more modest benefits. This corroborates our prior analyses: Spatial-DISE- 12K fortifies specific spatial reasoning architectures that subsequently transfer primarily to structurally homologous external tasks.
Ultimately, even following rigorous fine-tuning. the apex model (Qwen2.5-VEL ------B--sfl remains markedly inferior the human baseline on the Spatial-DiSS Bench, highlighting substantial avenues for future architectural refinement. Collectively, the responses to these research questions demonstrate that while contemporary VLMs do not yet possess robust, human-like spatial schemas, meticulously structured 3D training regimens like Spatial-DISE-128 can systematically cultivate distinct spatial skill cohorts and induce meaningful, albeit selective, cross-task and cross-benchmark transfer.
ERROR ANALYSIS
5.1 ERROR TAXONOMY AND MAIN DEFICITS
Given the persistent performance gap between fine-tuned VLMs and human baselines, we must move beyond simply measuring what models fail at to understanding why they fail. To provide this cognitive diagnosis, we employ Doubao-1.6 thinking as an automated judge and, combined with human analysis, analyze a sample of 200 in- correct responses from four representative models: GeminiFFlash200, Qwen2.5-VL-3B, Doubao- 1.5- thinking, and Space Thinkrr, with 50 samples drawn from each.
Table 4: Error Types and Their Frequencies.
Major Error
Sub-category
Num.
Failure Rule Ap- plication
65
Reasoning Err.
Failure in Mental 58 Simulation
Failure in Holistic-
22
Local Processing
Perceptual Err.
35
We established a high-level error taxonomy to sys- tematically diagnose failures by deconstructing the model mistakes into three errors: Perceptual Error, Comprehension Error and Reasoning Error. The analysis reveals that Reasoning errors are the predominant failure category, accounting for overwhelming 72.5% of all analyzed failure responses. Perceptual errors constituted 17.5% of the total, while comprehension errors were the least common at 10%. This distribution strongly suggests that the primary bottleneck for current VLMs is not in visual perception but in complex spatial-logice inference. The predominance of reasoning errors (145) prompted a deeper analysis, which identified three fundamental cognitive deficits.
Comprehension Err.
20

---

Published as a conference paper at ICLR 2026
The most significant issue Failure in Rule Application (44.8%), where models disregard basic geometric axioms, such as the spatial relationship between adjacent and opposite faces on a cube. This suggests inability link visual data with abstract principles. The second major deficit was a Failure in Mental Simulation (40.0%), indicating a lack of "spatial working memory" to track objects through transformations, miscalculated. Finally, Failure in Holistic-Loca Processing (15,2%) was observed, where models cannot appropriately shift attention between an object's overall structure and its local details, often being misled by superficial similarities while ignoring critical flaws. Detailed error analysis pipeline, definition of error categories and more discussion is presented in Appendix
seen in Fold and Punch where state changes are consistently
5.2
DISENTANGLING PERCEPTUAL CONFOUNDERS
Table 5: Layout Ablation Study on Qwen2.5-VI.-7B Disentangle Perceptual Confounders.
Input Setting
Accuracy
Standard Merged Layout
26. 16.1% 1%
Separate Images Layout
24.9%
Although the preceding analysis categorizes the overwhelming majority of failures as reasoning deficits, confirming the absolute rigor of this conclusion requires eliminating potential perceptual bottlenecks. To ensure that our eтrоr analysis accurately captures genuine spatial reasoning deficits rather than superficial perceptual confounders- visual layouts-we conducted a layout ablation study. our standard evaluation setting, the question and corresponding options are presented to the model as single merged image. It is plausible that model's s failure could stem from visual parsing issues rather than spatial reasoning limits.
as inability to parse complex, multi multi-panel
To test this, compared the standard merged-image setting against separate-images setting, where the question and each option are provided to the model as independent image inputs. Evaluating Qwen2.5-Vi. V-LBBB under these two conditions yielded near-identical accuracies: 26.1% 1% for the standard merged layout and 24.9% for the separate images. If layout parsing were the primary bottleneck, isolating the images should have resulted in clear performance improvement. The absence of such gain strongly supports our qualitative findings that the dominant difficulty for VI.Ms lies in complex spatial-logies inference-such as maintaining 3D adjacencies and global consistency-rattee than format parsing.
6 CONCLUSION AND LIMITATIONS
Conclusion and future direction. We introduced Spatial-DISE and Spatial-DISE-122 to com- prehensively evaluate spatial reasoning in VLMs. Our findings reveal universal cognitive deficits, particularly in applying geometric rules and performing mental simulations. This work provides a necessary framework to guide future VLM development.
To advance spatial intelligence, future research must shift from mere visual perception to active, human-like reasoning. Key directions include: (1) closing the sim-to-real gap by transferring abstract cognitive concepts derived from synthetic settings; (2) evolving evaluations from isolated puzzles to interactive tasks (e.g., navigation and robot manipulation) and (3) adopting process-oriented assessments that require textual justifications or action plans, distinguishing true mental simulation from fragile heuristic matching
Limitations. Our error analysis relies on hybrid L.M+hhman pipeline in which Doubao-1 1.6- thinking proposes an error type and explanation that human annotator then verifies. While this reduces manual effort, the LLM's initial label may bias the annotator, and analysing only 200 errors from four models means Table 4 should be read as qualitative trends rather than precise estimates. Additionally, Spatial-DISE- 12K consists entirely of synthetic 3D scenes; despite consistent gains on real-world benchmarks, a sim-to-real gap may persist for naturalistic settings. Human quality control covers a stratified sample of 1,000 items, with programmatic single-solution checks applied to the remainder, exhaustive manual verification of the full 12K is infeasible.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
LLM Usage Statement. We declare that large language models (LLMs) were used exclusively for language editing and stylistic improvements in this manuscript. They did not contribute to the conceptual, methodological, or experimental aspects of the work.
Ethics Statement. This work adheres to ethical research practices. Human performance data were collected from 54 participants. The study protocol was reviewed and approved by the Faculty of Science and Engineering Research Ethics Committee the University of Liverpool (Project ID:: 16623). Prior to participation, all individuals were informed of the study's 's purpose and procedure. They provided digital informed consent before accessing the tasks and were explicitly notified of their right to withdraw at any time. No sensitive personal data was collected. All response data are anonymized and will be destroyed three months after the paper is accepted.
Reproducibility Statement. We provide comprehensive details to ensure full reproducibility. The complete dataset curation process, including the synthetic data generation pipeline, detailed in Section 3.2 and Appendix A.4. This includes procedural algorithms (pseudocode) and specific implementation details for the five core 3D tasks. All evaluation settings, including benchmark models, baselines, and implementation details, are described in Section 4.1. Our human performance assessment methodology thoroughly documented in Appendix B. The benchmark, dataset, and code will be made publicly available to facilitate direct comparison with results.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, et al. Flamingo: A Visual Language Model for Few-Shot Learning. In NeurlPS 2022, 2022.
Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, et al. Qwen2.5-VL Technical Report. arXiv:2502. 13923, 2025.
G. K. Bennett, H. G. Seashore, and A. G. Wesman, Differential Aptitude Tests. Differential Aptitude Tests. 1947.
GEORGE M. BODNER and ROLAND B. GUAY. The Purdue Visualization of Rotations Test. The Chemical Educator, 2:1-17, 1997. ISSN 1430-4171.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVL.M Endowing Vision-Language Models with Spatial Reasoning Capabilities. In CVPR2024, 2024.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models. arXiv:2504. 1468, 2025.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, et al. PaLI-3 Vision Language Models: Smaller, Faster, Stronger. arXiv:2310.09119, 2023.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifci Liu. SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models. In NeurlPS 2024, 2024.
Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, et al. Kimi-VL, LL Technical Report. 251.:22504.04991, 2025.
Haodong Duan, Xinyu Fang, Junming Yang. Xiangyu Zhao, Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. ViMEvalKit: An Open-Source Toolkit for Evaluating Large Multi Madii-Modality Models. In MM '4: The 32nd ACM International Conference Multimedia, 2025.
11

---

Published as a conference paper at ICLR 2026
Ruth B. Ekstrom and Harry Horace Harman. Manual for Kit of Factor- Reftto--efeeeeeedd Cognitive Tests, 1976. 1976.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal Large Language Models Can See but Not Perceive. In ECCV 2024, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 Herd of Models. arXiv:240.221783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 251i::2501./9448, 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, et al. Seed1.5-VL Techiciaa Report. arXiv:2505.07062, 2025b.
Wenyu Han, Siyuan Xiang. Chenhui Liu, Ruoyu Wang, and Chen Feng. SPARE3D: A Dataset for SPAtial REasoning on Three- View Line Drawings. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models. arXiv:2506.03135, 2025.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? Investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pp. 9161-9175, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2xXi::230111299,, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded Language-Ima Image Pre-training. In CVPR 2022, 2022.
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models. In EMNLP 2024, 2024.
Marcia C. Linn and Anne C. Petersen. Emergence and Characterization of Sex Differences in Spatial Ability: Meta-Analysis. Child Development, 1479, 1985, ISSN 0009-3920.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual Spatial Reasoning. Transactions of the Association for Computational Linguistics, 11:635-651, 2023. ISSN 2307-387X.
Shiyin Lu, Yang Li, Ding-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural Embedding Alignment for Multimodal Large Language Model. arXiv:2405.20797, 2024.
Wufei Ma. Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Celso M de Melo, and Alan Yuille. 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark. In ICCV 2025, 2025.
P.H Maier. Spatial Geometry and Spatial Ability- How to Make Solid Geometry Solid. In Proceedings of the Annual Meeting of the GDM, 1996.
Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. SpaceR: Reinforcing MLLMs in Video Spatial Reasoning. arXiv:2504.01805, 2025.
George Pallrand and Fred Seeher. Spatial ability and achievement introductory physics. Journal of Research in Science Teaching, 21:507-516, 1984. ISSN 0022-4308.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. LMM-RI: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Buse-Based RI. a7X5v 2025.
12

---

Published as a conference paper at ICLR 2026
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv:2306. 148244 2023.
Saville Peter. Minnesota Paper Form Board Test Manual, Series AA and BB. British (ed.) / (by) peter saville (et al.). edition, 1974. ISBN 978-0-7005-0012-33
Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does Spatial Cognition Emerge in Frontier Models? In ICLR 2025, 2024.
Arijit Ray. Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna. Kuo-Hao Zeng, and Kate Saenko. SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models. arXiv:2412.07755. 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. VIM--R: A Stable and Generalizable R -syyle Large Vision-Language Model. arXiv:2504.06615, 2025.
Roger N. Shepard and Jacqueline Metzler. Mental Rotation of Three-Dimennioa Objects. Science, 171:701-703, 1971. ISSN 0036-8075.
Kexian Tang, Junyao Gao, Yanhong Zeng. Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning? arXiv:2503.1 19990, 2025.
Shengbang Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang. Adithya lyer, Xichen Pan, Ziteng Wang. Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-I: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. In NeurlPS 2024, 2024.
David H. Uttal, Nathaniel G. Meadow, Elizabeth Tipton, Linda L. Hand, Alison R. Alden, Christopher Warren, and Nora S. Newcombe. The malleability of spatial skills: meta-analysis of training studies. Psychological Bulletin, 139:352 402, 2013. ISSN 1939-1455.
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is A Picture Worth Thousand Words? Delving Into Spatial Reasoning for Vision Language Models. In NewrIPS 2024, 2024.
Junke Wang, Linghenn Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning. arXiv:2311.0574,.22 2023.
Xingrui Wang, Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models. In CVPR 2025, 2025.
H. A. Witkin. Individual differences in ease of perception of embedded figures. Journal of Personality, 19:1-15, 1950. ISSN 1467-6494.
Guowei Xu, Peng Jin, Hao Li. Yibing Song. Lichao Sun, and Li Yuan. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. arXiv:2411.104400 2024.
Wenrui Xu, Dalin Lyu, Weihang Wang. Jie Feng, Chen Gao. and Yong Li. Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometries. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers),
pp. 11571-11590, 2025. ISBN
Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. In CVPR 2025, 2024.
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang. Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiangmiao Pang. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence. arXiv:2505.23764 2025.
13

---

Published as a conference paper at ICLR 2026
Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities. In International Conference Learning Representations, 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, al. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. hutps:/aarivvorg/abs//555.4 10479v3, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendices
A DATASET DETAILS
A. TASKS DESIGN DETAILS
This subsection describes the task design details, aligning the original cognitive science psychometric test with the spatial abilities defined by Linn & Petersen (1985), and its classification within the Spatial-DISE taxonomy.
It is important to note that the Spatial-DISE taxonomy is intentionally a high-level, problem-centric abstraction. It categorizes spatial reasoning tasks based on their core structural properties and primary objectives- sereddvve--seeifiicllll.e whether the problem relies on intrinsic versus extrinsic information, and whether the goal state requires static analysis or dynamic transformation.
However, human solvers and models may employ variety of lower-level, atomic cognitive strategies to solve these tasks. For instance, while "3D Shape Finding" is structurally classified as an Intrinsic- Static task (as the objective is to determine fixed intrinsic property of single object), viable solution strategy might involve mentally rotating the object (an Intrinsie-Dynamic process) to build a complete mental model. The DISE categorization reflects the dominant structural requirement of the task itself, whereas the classic spatial abilities (such as Mental Rotation [MR] or Spatial Visualization [SV] detailed in Table 5) represent the atomic skills that can be invoked across multiple quadrants. Therefore, our evaluation uses DISE to provide a coarse-gres coarse-grained organizational framework over task families, diagnosing systematic weaknesses along structural dimensions rather than isolating single low-level cognitive process.
Table Each spatial task used our study and its canonical source test. Spatial Perception (SP), Spatial Relation (SR), Spatial Orientation (SO), Mental Rotation (MR), and Spatial Visualization (SV)
Task
Original Test
DISE Taxonomy
Spatial Ability
3D Combination
Differential Aptitude Tests Bennett
1947
Extrinsic- Dynamic
SV
2D Combination
Minnesota Paper Form Board Test (eteer
1974
Extrinsic Dynamic
SV
3D Projection
Purdue Spatial Visualization Test Views (BODNER GUAY
1997
Satrinsic-Statie
SP, SV
Fold and Punch
Paper Folding Test (VZ-2) Pallrand & Seeber 1981 EX3trom & Harman
1976
Intrinsie-ynanii Dynamic
SV. SR
3D Folding
Paper Folding TVZ31 (Fx2rom 11 man 1970
Intrinsic-Dynann
SV, SR. SO
2D Folding
Paper Folding (VZ-2) (Exstron)
murinsic-Dyuamie
SO
3D Rotation
Meutal Rotations Test (Sixpiid & Merzer
Intrinsic- -Dynmme
SV, MR, SO
2D Rotation
Card Rotations Test ()
Intrinsic- Dynnmic
SV, MR, SO
3D Shape Finding
Cuhe Comparisons Test
[Ksinom & Harman
1976
Intrinsic Static
SV, SR
2D Shape Finding
Embedded Figures
Test
Wilkin
Intrinsic Static
SR
A.2 DATASET SPL.TT DETAILS
Subset
QNA Pairs
Source Mix (RWD/SD)
Tasks
Spatial-DiSE- Sontia-DDSE-Bend Bench
559
53% 747%
2D 3D
Spatial-DiSE- 12K
12355
3% 793%
3D
-Train
8648
5.1% 94.9%
3D
Vol
1853
5.5% 994.5%
3D
Test
1854
4,5% / 95.5%
3D
Table 7: Description of Spatial-DISE Subsets. RWD: Real- World Data, SD: Synthetic Data. Note that Spatial- DISE Bench includes 2D questions absent from training splits, enabling zero-shot 2D evaluation.
A.3 REAL-WORLD DATA COLLECTION DETAILS
Real-world data are collected from open source online resources, mainly from the following resources:
1. Open- Spatial Reasoning Tests: Psychometric test materials published by academic research entities for evaluating spatial abilities.
2. CEM 11+ Non-verbal Reasoning Tests: Validated spatial reasoning items from authoritative aptitude tests used for secondary school admissions in the UK.
3. Online Employment Aptitude Tests: High-quality spatial and logical problems administered by corporations during recruitment.
15

---

Published as a conference paper at ICLR 2026
A.4 SYNTHETIC DATA GENERATION DETAILS
Synthetic Data Generation
Folding
3D Rototion
5D 3D Combitation
O Projection
30 Shape Findin
Generate
Gererating Random Shapes using Growth Algorittims
Gererating Random
Genereting
Generate
Shapes using Growth Algoritimms
Shopes using Growth Algorithma
Textures
Textured
Colculate: 2D Net und Relative Orientation
Collision Deticciion To Ensure Shope Rotiorolisction
Collision Detection To Ensure Shops Rctionglisction
Greating Direction Indicators
CalculaTe 2D Net and Relative Omentation
Render from Question
Assess Visible
3D Shopes
Render Fixed
Render Fixed
Select Distraction, Strotagy
Select Strotagy
Stratagy
Stritegy
Strategy
Hesroente 1.2
Paot Temona
Orthoainal
Pace Toxtur
Render Distroctiors
Renden Distroctions
Renden Distroctions
Render Distractions
Render Distractions
Quest Distractions
Quisi Distractions
Quick Distractions
Ques+Ang+ Distractions
Distractions
Quality Control
Duplication fom
Multi Answers
Mlisiing
Figure 4: Synthetic Data Generation and Quality Control.
This section provides detailed procedural algorithms (pseudocode) and visual examples for automated generation of our five core 3D spatial reasoning tasks. Each algorithm is designed for verifiability and incorporates sophisticated, task-specific strategies for generating plausible distractors.
Synthetic data generation employs Blender 4.4.0 on Apple Silicon M4. Some texture icons @ Icons8 under Universal Multimedia License. Task details, pseudocode and examples for synthetic data generation are outlined below:
Algorithm 1: G
1: Input: question id q_id, list of isometric camera presets iso_views,, number of distractors
2: seed +- Hash(q_ia
3: SetRandoomSed(ssedd): ClearScene()
4: orig < CreateCComiinatioSSaapelllllll E [3, 15), rectanallaPPriss--Trne,, seed)
5: qView -iinBesttiewworiig iso (so_riew)
6: SetCameralgV ienieratggieee.jjteee---- jitter=True))
7: Renderlmage(g_jid_00
8: ansV + ChooscDiiferett views, exclude=gVieeu)
9: SetCamera(ons Viw, SeCCamecclooa//eeeiiiiorT-eee jitter-True)
10: Renderlmage(q_id4
for + n do
12:
ficulty t i/n (Higher + harder)
13:
dShape + GeneraleDiistractottorii,,, di ficullyy seed + i)
14:
dView +- RandomChoice((ss ((vieieee((ss_eewss)))))) views)
15:
SetCamera(dV (eeneerrrddWWwwww,eeee int jitter-True)
16:
17:
end for
SaveMetadata iq_id, id, qView, ansVieu. seed)) 18:
All functions referenced in the code listings project-specific utility routines; their full implementations will be provided in the accompanying public code repository.
16

---

Published as a conference paper at ICLR 2026
Qurestion
Figure 5: Synthetic 3D Rotation Data Example.
3D Rotation three-dimensional object and recognize it from different angle.
The 3D rotation matching task designed to assess the ability to mentally rotate a
The process begins by generating a complex 3D shape composed of multiple cubes or rectangular prisms. This shape then rendered from optimal viewpoint create the "question" image. This viewpoint is chosen to maximize the number of visible parts, ensuring a clear presentation of the object.
Next, a set of "answer" options is generated:
The Correct Answer: This is created by rendering the original shape from new viewpoint, different from the one used for the question image. This requires the participant to recognize that it is the same object, despite the change in perspective.
These are generated by creating new shapes that are slightly different from the original one. Each distractor then rendered from different viewpoint. These are designed confuse the participant by presenting options that are visually similar but structurally incorrect. The final output consists of the question image, one correct answer image, and several distractor images, along with metadata file containing all the generation parameters to ensure reproducibility.
Distractors:
Questoon
Figure Synthetic 3D Projection Data Example.
3D Projection The 3D projection task evaluates a person's ability to interpret 3D object from an isometric perspective and then identify its correct 2D orthographic projection from set of options.
The process starts by generating a complex 3D shape. "question" image is then created by rendering this 3D shape from optimal isometric viewpoint. A visual cue, typically arrow, is included in the question image to indicate the direction from which the orthographic projection should be imagined (e.g., "top-down," "front," " or "side" view).
set of options then generated:
The Correct Answer: This is the true 2D orthographic projection of the 3D shape from the direction indicated by the arrow in the question image.
Distractors: These incorrect 2D projections. They generated in few ways:
17

---

Published as a conference paper at ICLR 2026
Algorithm 2: GENERATT3DDRROOHCTOOUUUUTTTT
1: Input: s_id, ortho_otoott views {top, front, right, left, bottom, back), _iv_s, number of
distractors n
2: seed Hash(q_ic)
3: SefRanddomSeed(seed):: ClearScene()
4: shape CCcuaCCCbbbiiaiionShpposseeeeeee
5: qV iew + FFindBestViewssfheeee sho_views)
targetView RandomChoice((rthho views)
7: indicator t Create Vecaareeeeiiidcccoodddddrttrtrrr
8: SetCamera(gVieuu))
9: Renderlmageeg__idd_0 Delete((ndicalor))
Se(CameenOOthoo((rrgee/// iew)
11: Renderlmage(qu A0)
12: for + to n do if Random() 0.7 then
13:
14:
dShape + GennerateDistracttrrssskee difficulty=0.3 0.7 i/n, seed+i)
15:
dView +- targetVient View
16:
else
d.Shape shape
18:
dView + Ch Veeeesssoooooooeesseee exclude=targetVien ottp
end if
20:
ApplyScene(((Shape)
SelCameraOrtholddieer
22:
(idde(maage(q_idAAAi))
23:
end for
24: SaveMetadata( id, seed, View, targetVient
Incorrect Projections: These are valid orthographic projections but from the wrong viewpoint (e.g., "side" view when the "top-down" view was asked for).
Slightly Altered Shapes: These 2D projections of shapes that subtly different from the original 3D shape, testing attention to detail. The participant must select the 2D image that accurately represents the specified orthographic projection of the 3D object shown in the question.
Algorithm 3: GENNRRRTEDDCOCMBRAATTOOVVVEESST
Input: i_dd, _ss_viees, number of distractors n
2: seed +- Hash(q_ _id
3: Se(RandomSSedd(seed)) ClearScenc()
4: master ChaateCCmmiiitiinnsppeeesssessee
complexit==medium)
5: qView + FindFesi Vinemasseeeeemmssterr iso _iveis; SetCamera(gVieu)
6: Renderlmage(q id
7: oppV iepView t- Opposite ViepossteeVeex(gVVeeee SetCamera(oppVien S
8: Renderlmageeq__id__)
9: c + DecossnnucSShhppeeeeteee
AranggeCoompoeennnnrGodommmmmmoe gap=2)
11: SetCamera GlobaleeveGl
Renderlmage(q id A0_
13: for + to do
comp <- RandonClooie((cmpppeent))))
15:
dComp + CheateDDisraatoCCCmpnnnttooooo varaaaoiooooiiiiiiiiiiiii variation=i/n, seed+i)
16:
ReplaceCompaneetccomp.. dC'omp)
17: Lenderlmage(q_iidAAl
18:
RestoreCommponen(coomp)
19: end for
20:
SaveMetadata( _ seed, main main View, vepieee:oppVee))
18

---

Published as a conference paper at ICLR 2026
Question
Oppostle
Figure 7: Synthetic 3D Combination Data Example.
3D Combination 3D object into its constituent parts and then identify which of those parts could be used to build different target shape.
The 3D combination task, evaluates the ability to mentally deconstruct a complex
The task generation proceeds follows: Shape Generation: A complex 3D shape is created, which serves as the "source" object. This source object is rendered from two opposite isometric viewpoints give the user complete understanding of its structure. Component Segmentation: The source object is programmatically broken down into a set of smaller, non-overlapping 3D components. These components the basic building blocks that could theoretically form the original shape. Question Formulation: The "question" is presented as a new, different "target" 3D shape. Option Generation: The options provided the user are the individual 3D components that were segmented from the original source object. These components are laid out individually for clear inspection.
Algorithm 4: GENDRAATEDFOOLDNNNUUUSSSOO
1: Input: q_id, difficulty tier list (easy, medium, hard), number of distractors n
2: seed f- Hash(q_ia id)
3: Se ClearScene()
4: cube, ap CrreueeCueeeiiteeeeerrrsseeeeee
5: layout + RaanoomChoicc( (cross, T})
6: net < Un.foldCubb (cubec layout)
7: SetCamera(Top), Kenderlmagel s_______ id_Q)
8: bestView +- Best3 (cube)
9: (eCCamero(bessiieww
10: Renderlmage(q_idd_..
11: for
to do
12: tier < SelectTier(i, n))
13: dCube CeceteClleDasrrtttroooodttteeeer seed+i)
14: SetCamera((estlVieu
Renderlmage(q_id__4())
16: end for SaveMetadata(1 iid seed, layout, faceMop))
3D Folding The 3D box folding task evaluates a person's spatial reasoning ability, specifically their capacity to visualize how 2D pattern (a "net") will fold into 3D cube. The process for generating a question as follows:
Cube and Texture Generation: A standard 3D cube is created. Each of its six faces is assigned unique texture or color. This is the "target" cube.
Unfolding: The textured 3D cube is computationally "unfolded" into 2D net. The net is flat pattern that shows all six faces of the cube connected a way that it could be folded back up into the cube. Common net patterns like "cross" or "T-shape" used. This 2D net serves the "question" image.
Option Generation: set of 3D cubes is then presented as the options.
19

---

Published as a conference paper at ICLR 2026
Qurestion
Figure 8: Synthetic 3D Folding Data Example.
The Correct Answer: This is 3D rendering of the original, correctly folded cube, showing how the face textures are oriented in relation to each other.
Distractors: These are 3D cubes that are almost correct but have one or more faces manipulated in way that makes the folded result incorrect. These manipulations can include:
Face Rotation: One more faces the cube are rotated from their correct orientation. Face Swapping: The positions of two or more faces are swapped. Texture/Color Replacement: The texture or color of one face replaced with that of another.
Algorithm 5: GENERATTSSHAPFFFNDNNQUUUSSS
1: Input: question id q_id, difficulty {easy, medium, hard), options m
2: seed Hoad<HHashq___a id)
3: SelRandomSSed((seed)) ClearScene()
4: cube < CreealrCube ealTTaaauuussseeee
5: (Vo, V1, V22) Se--hhooooDDsiineeee ietiiisiiiiniiisseeeeeeeee 120%)
6: for j 6- 0 to do
7: SetCamera( Rentelmagggg__id_Vi
8: end for
9: vis < FiisibleFaces(cui V2)
10: f + -Sample/ ace vis, fiattegyydffffiulle
matorig -GGelMatrrall(enn
f*)
12: SetMaterial(cubbe f",Blue), SetCamera V2), Renderlmage(a__ie _2)_2)
SetMaterial (cube, f" .maforrg)
14: opts + } } Sample
f*), m
opts <- fle(opt)
16: for k, in Enumerate optss) do
17: CeCCamennFaaceNoomaalvvoooooo
18:
Lenderlmage(q_iid_00 19: end for
20:
SaveMetadata (ilidddttaii:qqq__.. id, seed, views:[Vo. Vi, V2], replaced: coorectldx::ndder0fff opts)
3D Shape Finding The 3D Shape Finding task is a visual memory and attention task that tests the ability to track specific face of 3D object as the object is rotated in space. Here is how a typical question is generated: Cube Generation: A 3D cube created with unique, distinct texture applied to each of its six faces. View Sequence: The participant is shown sequence of images (typically two) of the cube from different viewpoints. This allows them to see the cube and the arrangement of its face textures from multiple angles. The "Change" Event: third image of the cube is then presented. In this view, one of the visible faces of the cube has its texture replaced with a solid color (e.g., blue). This is the key event in the task. The Question: The participant is implicitly asked: "Which of the original face textures was replaced by the solid color?" Option Generation: The answer options are set of images, each showing one of the original, individual face textures from the cube. The Correct Answer: This is the image of the face texture that was replaced by the solid color in the third view. Distractors: These are the other original face textures from the cube.
20

---

Published as a conference paper at ICLR 2026
108
MS
Figure 9: Synthetic 3D Shape Folding Data Example.
Task Name
Distractor Generation Logic
Specific Implementation Details
3D Rotation
A new 3D shape is created that structurally different original, yet visually similar, testing the ability to spot sub- te structural changes despite viewing angle differences.
shape is generated by altering the "growth his- tory" of the original object (e.g. adding or removing block a different location), producing plausible but incorrect alternative.
the
3D Projection
An incorrect 2D orthographic projection is generated, testing the ability accurately project planc.
Generating a projection from an incorrect viewpoint (e.g. providing side view when the top view was re- quested). Generating projection of slightly modified (distractor) 3D shape.
3D object
3D Combination
A valid component from the original shape structurally modified, testing detailed anal- ysis part geometry.
A single authentic component is duplicated and then altered typically adding or removing block-yielding visually similar part that would correctly into the complete assembly.
3D Folding
The 2D net is "folded" into in- correct 3D cube, testing the ahil- ity to track face orientation adjacency during folding.
Rotation: A face's texture is rotated by 90". 180". 270". Swapping: Textures between two faces swapped. Flipping: texture is flipped horizontally or vertically.
3D Shape Finding
The options presented are the other, non-target faces the cube, testing visual working memory and attention.
The task is to identify the original texture of face was replaced by solid colour: distractors are original textures of the other cube faces that were not the replacement target.
Table 8: Summary of Distractor Generation Logic
21

---

Published as a conference paper at ICLR 2026
B EVALUATION DETAILS
B.1 HUMAN PERFORMANCE ASSESSMENT DETAILS
To establish a robust human baseline, we recruited 54 participants through a custom online platform. The process yielded 1,684 valid responses, with each of the 559 benchmark items being answered by an average of participants.
The median response time across all human responses 26.9 seconds, with mean of 40.3 seconds. The difference suggests that subset of questions required substantially longer deliheration, skewing the mean. A detailed breakdown of performance by task category, presented in Table ! reveals clear inverse relationship between response time and accuracy. The analysis highlights that the two
Table 9; Human Performance Task: Accuracy and Response Time.
Task
DISE Category
Accuracy (%)
Mean Time (s)
Median Time (s)
3D Combination
E-D
56.4
59.2
34.8
2D Shape Finding
I-
61.5
58.5
46.4
2D Combination
E-D
75,2
36.8
32.0
2D Folding
I-D
76.5
25.6
17.0
Fold and Punch
HD
76.8
55.4
44.4
2D Rotation
I-D
78,1
40.4
31.5
3D Projection
E-S
81.1
28.0
20.8
3D Shape Finding
I-
81.8
31.4
23.3
3D Rotation
I-D
82.0
29.8
21.8
3D Folding
I-D
86.6
44.4
33.6
tasks with the lowest human accuracy, 3D Combination (56.4%) and 2D Shape Finding (61.5%), are also the tasks that commanded the longest mean response times (59.2s and 58.5s, respectively). This empirically confirms that these tasks impose the highest cognitive load. The mental simulation required to assemble complex parts in 3D Combination (Extrinsic-Dynamii and the demanding visual search needed disentangle embedded figures in 2DD Shape Finding (Intrinsic-Static) inherently time-consuming and error-prone for humans, providing quantitative justification for their difficulty. Conversely, tasks with high accuracy, such as 3D Folding and 3D Rotation, generally required less time, indicating lower cognitive barrier.
In order to obtain unbiased estimate of human baseline performance over the full item pool, employ a matrix- sampling design in which each participant completes only a single booklet of K items out of the total pool of I items. Adjacent booklets share small set of anchor items 1=10 We recruited 54 participants for the study. Before participation, all individuals provided their informed consent and all procedures were conducted in accordance with relevant ethical guidelines. The data collection process yielded total of 1679 valid responses across all items. Each item was answered by an average of participants. The main paper reports human performance with Classical Test Theory (CTT) results, while Item Response Theory (IRT) is used for vols-valii.tion.
Analysis Methodology works:
The collected response data were analyzed using two psychometric frame-
CLASSICAL TEST THRORY (CTT) For each item booklet and for the anchor- "overall" pool, the proportion-comrett statistic was computed as
T N
where is the number of correct responses and N is the total number of responses to that booklet or pool. Sampling variability was quantified with the Wald standard error
2( D)
SEcrT
(2)
N
22

---

Published as a conference paper at ICLR 2026
yielding two-sided 95% confidence interval (CI) I + 1.96 SECTT
ITEM RESPONSE THEORY (IRT) To cross-validate the CTT findings and place all items on common latent-ability scale, fitted two-parameter logistic (2PL) model to the entire response matrix,
Po = ofa (0; - b,)] 1 expl (0; b.)]
where P.j is the probability that participant j (ability 0,) answers item i (discrimination a, difficulty b;) correctly.
For a designated item subset (e.g., DISE category) containing / items, the model yields item-level expected probability of success P.. The category-level expected accuracy is then
1 + E
PIRT =
(5)
Between-item variability was captured via the sample variance
82
Σ (P - (IRT), 1=1
(6)
I
leading to the standard error
8 VT
SERT
(7)
and the 95% CI
PIRT + 1.96 SEIR
(8)
Results To provide comprehensive view, we compare the results from both CTT and IRT analyses. Figure I0 Table juxtaposes the observed accuracy from CTT with the model-based predictions and item parameters from IRT for each DISE category. This comparison highlights the synergy between the two methodologies. The CTT accuracy provides direct, empirical measure of performance, while the IRT parameters offer an explanation for these results.
Table 10: Parameters of the Human Assessment
Table 11: Human Accuracy by DISE Category
Parameters
Num.
Category
Accuracy
Accuracy
Extrinii Dynamic
61.05%
4.46%
57.22%
8.67%
Number of Participants
54
Extrisii-Sure Strric
81.12%
4.38%
82.09%
7.02%
Intrinsic Dynamic
80.25%
2.05%
81.09%
3.42%
Total Number of Items
559
Intrissic Static
76.80%
3.90%
77.29%
7.49%
Total Responses N
1679
Overall
76.84%
2.02%
3.79%
Number of Booklets
19
23

---

Published as a conference paper at ICLR 2026
bustonn of Question Accuracy Rates
Human Average Accuracy Rate by Category
300
Overall Mean: 76.8% (75.0%,78.44))
100.
250
62.05
Number of Questions
200
19..%
Categoy
78.1%
150
140
70.00
100
61.5%
Oedar
56
0.2
Question Accuracy Rato
Avorage Accuracy Rate
(a) Human Accuracy CTT.
(b) Human Accuracy in different tasks by CTT.
Comparison of CTT and Results DISE Category
51.32%
Extrinsic-Static
82 00%
DISE Category
Intrinsic-Dynamic
$1.99%
75 k0%
Intrinsic-Static
Extrinsic-Dynamic
Analysio 7ypc
51.22%
0.0
0.2
0.4
0.6
0.8
Accuracy
(c) Comparison of CTT and IRT results.
Figure 10: Human Performance Results by CTT and comparison of CTT and IRT results.
24

---

Published as a conference paper at ICLR 2026
B.2 EVALUATION IMPLEMENTATION DETAILS
All evaluations were implemented on 3 NVIDIA A 100-40G with VLMEvalKit v0.2. Following the idea Duan et al. (2025), all the models used very low temperatures or temperatures equal 0 and set do_sample = False to ensure reproducibility and certainty of the results. The API checkpoints for proprictary models are listed in Table 12
Table 12: Proprietary APIs evaluated in this paper
Proprictary Model provider
API endpoint
Claude 3.7 Sonnet (Anthropic)
(aaudee-37--smmeet-22555606
Doubao 1.5 VL (volcengine)
Doubao 1.5 VL-thinking (volcengine)
Gemini 2.0 Flash (Google)
gemini-2.0-flassh
Gemini 2.5 Flash (Google)
Gemini 2.5 Flash w/o thinking (Google)
GPT-4.1 nano (OpenAl)
3pt-4.11nnao-2225--0--111
GPT-4o (OpenAI)
gpt-40-2024-08-06
GPT-4o-mini (OpenAI)
GPT-5 (OpenAl)
gpt-5
04-mini (OpenAl)
04-minn-2025--44
The prompt templates used in the evaluation for different models shown below:
Listing I: Prompt Templates used for Proprictary Models in VLMEvalKit
PROMPT PROMPT_TEMPLATEE TEMPLATES
"SYSTEM"; ; "You are helpful assistant. "
"USER": """<inage>
Question: The two images above show a 3D structure from different angles. which one of the options below could be constructed appear the same both given views when observed from the corresponding perspectives without rotation and overlaps? Select the most likely one. Options:
A.
B.
C.
D. D
Answer with the option's letter from the given choices
di rireetlp 1 an"
Listing 2: Prompt Templates used for Llama Serie Models in VLMEvalKit
PROMPT TEMPLATES = "SYSTEM": "",
"USER";
<im_startt
wwn_sstar>xximgg>iiin end>>
Question: The two images abeve show 3D structure from different angles. Which constructed to appear the same as both given views when observed from the corresponding perapectives without rotation and overlaps? Select the most likely one. Options:
of the options below could be
25

---

Published as a conference paper at ICLR 2026
B.
C.
D. D
Answer with the option's letter from the given choices directly. <
Listing 3: Prompt Templates used for QwenVL, InternVL, Ovis2 Serie Models in VLMEvalKii
PROMPT_ TEMPLAEES
"USER": """<inage>
Question: The two images above show 30 structure from di fffrentt angles. whi ch one of the options below could be constructed appear the same both given views when observed from the correspording perspectives without rotation and overlaps? Select the most likely one. Options:
A.
B.
C.
D. D
Please select the correct answer from the options above. "**
Listing 4: Prompt Templates used for VLM-RI and LMM-RI in VLMEvalKit
PROMPT TEMPLATES =
"USER": """<image>
Question: The two images above show a 30 structure from different angles. which one of the options below could be constructed appear the from the correspording perspectives without rotation and overlaps? Select the most likely one. Opticns:
both given views when observed
A.
B.
C.
D. D
Please select the correct answer from the options above. Output the tainking process in <think> </think> and final answer in <answer> </answer> tags. an*
Listing 5: Prompt Templates used for VLAA Thinker Serie Models in VLMEvalKit
"SYSTEM": : "You are Vi-Thinking, helpful assistant with excellent reasoning ability. You should first think about the reasoning process and then provide the answer, Use <think>... ../think//>i>>>> and <anower>...<annseer tags."
"USER": """<inage>
Question: The images abeve show 3D structure from fifernett angles. Which one of the options below could be constructed to appear the same as both given views when observed from the corresponding perspectives without rotation and overlaps? Select the most likely one. Options:
A.
B.
C.
D.
Please select the correct answer from the options above. ***
26

---

Published as a conference paper at ICLR 2026
B.3 MORE EVALUATION RESULTS
Table 13: Different-task accuracies on Spatial DISE Bench. Abbreviations 2D Comb.: 2D Combination: 2D
Fold.; 2D Folding; 2D Rot.: 2D Rotation; 2D S.F.: 2D Shape Finding; 3D Comb.: 3D Combination; 3D Fold.: 3D Folding, 3D Proj.: 3D Projection; 3D Rot.: 3D Rotation; 3D S.F.: 3D Shape Finding; F&P: Fold and Punch.
Bold indicates the highest accuracy; Underline indicates the second highest.
30 Proj.
3D
Propridtry
Clande 3.7 Sunnet
30.61%
29.2%
25.0%
30.4%
38.1%
20.0%
50.7%
31.4%
31.4%
25.8%
21.4%
Doubaol 1,5 VL.
33.8%
41.7%
25.6%
43.5%
33.3/
26. 16.7%
44.9%
37.1%
37.
34.8%
25.6%
Gemini 2.0 Flash
34.2%
20.8%
41.0%
30.4%
23.8%
76.7%
56.5%
31.4%
51.4%
15.2%
24,4%
GPT
28.1%
29.25
26.9%
17.1%
33.3/%
25.0%%
30.15
22.9%
32.99
33.3%
4PT4e-mini
25.6%
30.85
28.29
3048%
28.65%
15.0%
37.7%%
14%%%
22.99
28.8%
23.1%
Geminn 2,5 Flash
31.5%
12.59%
33.35
17.4%
33.34%
18.3%
60.69
27.15
40.0%
16.7%
24.4%
Gemini Flash thinking
32.0%%
33.35
17.4%
33.34%
GPT-5
04-mini
33.39
33.3%
30.8%
$2.25
10.0%
47.8%%
25.75
38.45)
26.95
Propnetory Average
30.9%
23,0%%
21.1%
31.2%
27.0%
27.3%
47.3%%
28.7%
37.0%
26.7%
23.3%
Open-souree
3VB11B
Cambrian- 13b
26.7%
20.8%
30.8%
30.4%
23.8/%
26.7%
21.7%
32.9%
25.7%
27.3%%
23.1%
Cambrian-8t
22.9%
25.0%
26.9%
30.4%
33.3%
16.7%
33.39%
15.7%
15.79
37. 3%.
17.9%
BoremVL3-38 38B
32.4%
29.24%
28.2%
47.8%
23.5%
26.7%
42.0%
30.0%6
40.0%
27.3%
30.8%
Intern ML3mVLL311BB
31.15
15.7%
24.4%
21.7%
14.3/2
20.0%
53.6%
31.4%
42,9%%
19.75%
34.6%
IniemVLL--88
Kimi- VI-A3B
24.3%
26.1%
20.0%A
35.75
20.55
Ovi62-168
26.3%
20.899
16.7%
13,09%
19.098
20.0%
$2.258
27.1%
42.9%
10.65
23.1%
Ovis2-SB 2BB
23.8%
25.09%
28.246
17.4%
28.098
11.7%
30,258
21.4%
34..3%
7..6%
24.4%
Qwen25-VVL-32
27.2%
20.855
19.25
21.7%
23.85%
21.7%
34 8
31,49
35.7%
28.8%
24.4%
Qwen2.5-V1-77
Owen2.5.VL-33 LLB33B
22.9%
12.
Open-snurce Avernge
26
25.3%
26.0%
26.45%
23.4%
22.4%
35.69%
25.7%
32.2%
18.75
23.0%
Reatoning Spuslal-Spwciffnn Modely
LL
17.4%
LMM-RI
26.1%
79.
28.25
21.7%
30.0%
36.25
20.00%
24.3%
19.2%
VLM R1
30.8%
25.0%
26.9%
39.1%
35
36.7%
47.8%%
18.6%
30.0%%
24.2%
29.5%
Kimi-VL-AB EL-AAB-Thinkale
24.7%
16.7%
26.9%
26.175
42.9%
31.7%%
26. 1/8
28.6%
22.9%
15.27%
19.2%
Dombbol.5- 5-Vi-iiinknng
42.0%
62.3%
28.25
43.5%
23.8
61.7%
56.5%
31.4%
50.0%
39.4%
30.8%
VL.A-Thinke--33
36.258
YLAA-Thinke::77
31.9%
SpankThinkerr
32.6%
29.24
20.546
43.5% %
33.36E
43.3%
49.36%
22.9%
35.7%
22.7% 7%
33.3%
SquceOm
25.9%
25.09
14.1%
43.5%
33.39%
36.7%
49.3 398
34.
32.95L
24.2%
37.2%
SpaceR
27.0%
37.5%
32.19
34.8%
28.698
26.7%
20.09E
17.1%
37.1%
21.25 256
19.2%
Reurowing Sputial Ateruge
Overat Average
27.8%
27.2%
33795
Space Omn-lt
33.8%
23.07
25.8%%
26.1%
2335%
15.0%
16.1%
31.4%
50.0%9
30.3%
20.3%
Qwen2.5-VL-7B--1
47.0%
33.39
34.68
34.8%
9.5%
78.3%
69.67
41.4%
65.7%
69:
21.8%
Homon
B.4 SUPERVISED TNN-TUNING HYPERPARAMETERS
The Supervised Fine-Tuning Tuning (SFT) experiments were conducted using the Swift framework. We employed the Low-Rank Adaptation (LoRA) technique efficiently fine-tune both the Qwen2.5-VL. 7B and SpaceOm models on the Spatial-DISE-122 DIK training set. All linear layers of the models were targeted for LoRA adaptation. The key hyperparameters used for the fine-tuning process are detailed in Table 13.
Table 14: Hyperparameters for Training
Hyperparameter
Value
Framework
Swift
Fine-Tuning Method
LoRA
Target Modules
all-linear
LoRA Rank (lora rank)
8
LoRA Alpha (lora_alpha)
Batch Size
48
Precision (torch_dtypp)
b6loat16
Learning Rate
1.5e-4
Warmup Ratio
0.05
Number of Epochs
Max Sequence Length
4096
Deepspeed
zero3
27

---

Published as a conference paper at ICLR 2026
C ERROR ANALYSIS DETAILS
This section provides detailed quantitative and qualitative breakdown of the error analysis conducted to understand the failure modes of VLMs on Spatial-DISS -DSE Bench.
C.I DEFINITION OF HIGH-LEVEEL ERROR
We established a high-level error taxonomy to systematically diagnose failures by deconstructing the model mistakes into three errors:
Perceptual Error, which the model fails to accurately interpret basic visual information, such as the shape, count, or spatial relationship of objects.
Comprehension Error, which the model misinterprets the natural language prompt or objective of the task, indicating a failure to understand the question.
Reasoning Error, which the model correctly perceives the visual scene and understands the prompt but fails in the logical deduction required reach the correct answer. This includes errors in mental rotation, folding, or spatial manipulation.
C.2 VLM-AS-JUDGE IN ERROR ANALYSIS
adopted an automated error analysis pipeline. As shown in Figure we use Doubao- I. 6--thinking as judge, combined with human inspection. Table [5 lists the distribution of the wrong responses sampled in analysis.
Inspired by Yang al. 2025)
Question
Table 15: Error Distribution by DISE category
VLM-as: judge
Model Response
DISE Category
Count
1.54
Intrinsic-Staal Static (I-S)
34
Image
Intrinsie-Dynamic (I-D)
107
Human
Erron Cotegary Reasoning
Extrinsic-Static (E-S)
34
Correct
Answer
Extrinsic- Extrinsie-yynamic (E-D)
25
Total
200
Figure 11: Error Analysis Pipeline
The prompt template used for error analysis:
Listing 6: Prompt Templates used for Error Analysis
ERROR ANALYSIS PROMPTS
"detailed saaledd_aalssss""""
Please provide detailed analysis of the visual-language model's incorrect answer:
Question Category: (category)
Question: (question)
Options: (options)
Correct Answer: (correct_answer) Model's Predicted Answer: (predicted_answer)
Model's Full Response: [model_prediction)
Please analyze in depth from the following perspectives:
1. Error Type Classification:
Perception Error: The model failed correctly identify visual. elements the image.
Comprehension Error: The model recognized visual elements but misunderstood their meaning.
Reasoning Error: The model understood the content but made stake in reasoning.
28

---

Published as a conference paper at ICLR 2026
2. Specific Cause Error
3. Severity Assessment (Low / Medium High)
4. Fossible Directions for Improvement
5. Suggestions to Prevent Similar Errors
Please return the analysis in JSON format:
11
"Error Type": "Specific type of error", "Error Subtype": "More detailed category of the errer", "Cause of Error"; "betailed explanation of the cause", "Severity"; "low//Medium/Highh "Summary"; "Brief sunmary of the error"
11
.*"
"catogory_analysa"
Please analyzc the error patterns of the visual-language model in the following (category) category questions:
(error_examples)
Analyze from the following perspectives:
1. Most common error types in this category
2. Common features and patterns of errors
3. Category-specific challenges
Please provide structured respense.
.""
P Please compare the error performance of the following models on the same questien:
(model pondel_comparianni
Analyze:
Differences error types across models
2. Strengths and weaknesses of each model
3. Compariscn of error severity
Please provide detailed comparative analysis.
***
Our analysis reveals clear and consistent pattern: Reasoning Error is the predominant failure category, accounting for an overwhelming 72.5% (145 out of 200) of all analyzed mistakes. Perceptual errors constituted 17.5% of the total, while comprehension errors were the least common at 10%. This distribution strongly suggests that the primary bottleneck for current VLMs is not in visual perception but in complex spatial-logical inference. While this initial classification identifies where the models fail, a more granular analysis is required to understand why they fail.
C.3 A DEEP DIVE INTO REASONING FAILURES
To move from symptom cause, we performed deeper analysis of the 145 reasoning errors, re-categorizing them based on the underlying cognitive abilities that are deficient. This approach, inspired by cognitive science, reveals that the models' failures stem from a lack of fundamental cognitive mechanisms for spatial intelligence. We identified three primary root causes.
Failure in Rule Application (44.8%) This the most critical category of failure. Models
demonstrate an ignorance of the fundamental axioms, constraints, and invariances the geometric world. The errors not in complex derivations but in the application of basic, non-negotiable
29

---

Published as a conference paper at ICLR 2026
rules. The root appears be a failure to link visual percepts abstract library of geometric principles; the models see pixels, not entities governed by rules.
frequent failure was confusing adjacent and opposite faces in 3D cube problems. For instance, model might correctly identify the symbols on a cube's 's faces but fail to apply the simple rule that adjacent faces cannot be opposite one another.
Failure in Mental Simulation (40.0%) The second most significant failure was the inability to construct dynamic. operable internal representation to simulate a continuous spatial transformation. Models lack reliable "spatial working memory" to track an object's 's state through a sequence of operations. They cannot robustly answer the question, "what happens next?"
This was most evident in "Fold and Punch" tasks. Models consistently failed to track the number of layers created by folds and, consequently, could not predict the symmetric replication of holes upon unfolding. For example, after simulating two-fold process (creating four layers), a model might incorrectly predict only two holes in the unfolded paper, demonstrating a breakdown in state tracking.
Failure in Holistic-Local Processing (15.2%) Finally, models exhibited an imbalance in processing visual information, struggling to shift between holistic understanding and local detail analysis. Their attention mechanisms appear unable dynamically allocate cognitive resources to the most salient features required by the task.
Models were often misled by superficial similarity. In rotation tasks, model might identify an option as correct simply because it "looks similar" to the target, while ignoring fatal flaw in the local arrangement of its components, such as incorrect orientation of a key part.
In summary, the failures of current VLMs in spatial reasoning are systemic and deeply rooted in cognitive deficiencies. They lack an internal "world model" that is constrained by geometric rules, can be manipulated through dynamic simulation in a spatial working memory, and is guided by flexible attentional mechanism. This points to clear direction for future research: efforts must transcend simple pattern matching and focus on imbuing models with the foundational capabilities for genuine spatial cognition.
Table 16: Error Analysis Across Different Models
Err. Models
Qwen2.5-VL
GeminiFlash
Doubao-1.5
Space
Reasoming lirr
31
31
37
46
Perceptual Ert.
12
12
3
Comprehension Err.
7
30

---

Published as a conference paper at ICLR 2026
Failure in Rule Application
Camiee
3D Shape Finding
-
Gemini-2.0-
Flash
The correct answer is D. The top face of all the cubes has the sun symbol. The third cube has the sun symbol on the top face and blue face. Therefore, the blue face most likely to have the sun symbol.
Model Ans.
The model incorrectly assumed that "sun symbol" visible on the top face of the cube would also appear on side face. This violates the fundamental rule that each face of the cube is unique and that symbol on the top face cannot simultaneously be on adjacent side face. The model failed to apply this basic geometric constraint, leading to high-severity reasoning error.
Error Reason:
Figure 12: Error example of Failure in Rule Application
31

---

Published as a conference paper at ICLR 2026
Failure in Mental Simulation
3D Rotation
Doubao-1.5-
Thinking
Model Ans.: To solve this problem, we analyze the folding and
punching process:
1. Folding Steps: The square is folded twice. First, horizontal fold (reducing height), then a vertical fold (reducing width). This creates 4 layers of the square.
Punching Hole: When hole is punched through layers, the unfolded square will have 4 holes (one in each layer).
Position of Holes: The hole in the folded square is symmetric across both folds. When unfolded, the holes must evenly distributed in a symmetric pattern.
Among the options:
a and d have only 2 holes (insufficient, as 4 layers mean holes).
b has 4 holes but their placement is not symmetric with the folding logic.
c has 4 holes arranged symmetrically, matching the result of two folds and single punch. Thus, the correct answer
The model correctly identified that the process would result in four holes. However, failed accurately simulate the unfolding process. It incorrectly calculated the reflective positions of the holes, placing them the middle and bottom edges instead of in the middle. This demonstrates breakdown in its ability to track the state and symmetric transformations of object through multi- step process.
Error Reason:
Figure 13: Error example of Failure in Mental Simulation
32

---

Published as a conference paper at ICLR 2026
Question
Opporall
Failure in Holistic-
Local Processing
3D Combination
Space Thinker
A
Model Ans.:
<think
To determine the correct answer, we need to analyze the two perspectives and find combination that matches both views without rotation overlaps.
1. **Question View**: The structure in the question view shows a central column with smaller blocks around it.
**Opposite View**: The structure in the opposite view shows similar central column but with additional blocks on the sides.
Looking at the options:
** *Optioon A **; This arrangement does not match the central column
structure in both views.
**Option 8** This arrangement also does not match the central
column structure both views.
** **Option C**; This arrangement matches the central column structure in both views.
**Option D** This arrangement does not match the central column structure both views.
Given that Option matches the central column structure both views without any overlaps rotations, it is the most likely correct answer. </thinle
cansweroC</onswer>
Although the model correctly identified the "central pillar structure" both views, it failed to analyze the detailed spatial configuration of the surrounding blocks, such as their exact positions and quantities. As result, it could not accurately determine whether an option satisfied the requirements of "no rotation" and "no overlap." Specifically, Option Cs surrounding block layout contradicts the details in one of the views (e.g., the block distribution is inconsistent with the perspective). whereas the correct Option D matches the spatial details in both views. However, due to imprecise spatial reasoning, the model incorrectly judged C valid.
Error Reason:
Figure 14: Error example of Failure Holistic-Local Processing
33


Reviews:
[{"review_id": "u82gEBFACN", "reviewer": "Reviewer_v2Mn", "summary": "This paper introduces Spatial-DISE, a new benchmark for evaluating spatial reasoning in VLMs. The work leverages a $2\\times2$ cognitive taxonomy (Intrinsic/Extrinsic vs. Static/Dynamic) to structure the benchmark. A key contribution is a scalable Blender-based pipeline for generating a 12K-pair training dataset and a 559-pair evaluation bench. The authors perform a comprehensive study on 28 VLMs, finding that current models perform poorly, especially on dynamic tasks, and that failures are rooted in reasoning (e.g., rule application, mental simulation) rather than perception.", "questions": "I am curious about the true value of the dataset, as I strongly suspect that it may merely overfit to its own format.\n\nIf a base model (e.g., Qwen-VL) could be fine-tuned on this dataset and subsequently demonstrate performance gains on other benchmarks (such as VSI-Bench, OmniSpatial and SPACE), I would be much more inclined to recognize the dataset’s contribution.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The adoption of the DISE cognitive taxonomy provides a structured and theoretically-grounded framework for evaluating spatial reasoning, moving beyond ad-hoc task collections.\n\n2. The Blender pipeline is a valuable engineering contribution, enabling the verifiable and programmatic generation of complex 3D spatial reasoning tasks, which are difficult to source at scale.\n\n3. The empirical study is extensive, testing 28 SOTA models. The resulting error analysis (Section 5) provides a clear insight: the primary bottleneck for VLMs is cognitive reasoning, not visual perception.", "weaknesses": "1. **Marginal Novelty in a Crowded Field**: The paper's own related work (Table 1) demonstrates this is an extremely crowded and concurrent field (e.g., BSA, OmniSpatial, BLINK, SPACE). I believe it is necessary to demonstrate that this dataset is fundamentally distinct from existing ones and possesses intrinsic value.\n\n2. **Dataset Utility is Questionable**: The paper presents the Spatial-DISE-12K dataset as a major contribution for future training. However, the paper's own finetuning analysis (Section 4.3, Table 3)  demonstrates this dataset may be flawed. While SFT on Qwen2.5-VL-7B improves performance on the in-domain Spatial-DISE benchmark (+23.6pp), the SpaceOm model shows \"catastrophic forgetting\" on a general benchmark like CVBench (a -32.4pp drop) . This strongly suggests the 12K dataset lacks diversity and induces severe overfitting to the specific generative patterns of the pipeline.\n\n3. **Limited Evaluation**: The model set omits several state-of-the-art commercial VLMs (e.g., GPT o3/5, Google Gemini 2.5 pro), which weakens the headline claim about “current VLMs.” They have more robust and powerful ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VoXBrPhWA4", "reviewer": "Reviewer_o491", "summary": "The paper proposes two semi-synthetic datasets (Spatial-DISE Bench and Spatial-DISE-12K) to evaluate spatial reasoning in Vision-Language Models (VLMs), with a special emphasis on intrinsic and dynamic tasks that are missing from previous benchmarks. The datasets are created using a new data generation framework based on the graphics software Blender to generate complex 3D scenarios that require spatial reasoning to solve. The work also offers a comprehensive evaluation of 28 state-of-the-art (SotA) VLMs on the proposed data showing that current models perform much worse than human subjects.\n\nDespite some good contributions (i.e., evaluation study), I am really not sure that creating new datasets and the entire data generation pipeline (which are claimed to be key contributions) was at all needed in the first place. Given the way the problem is framed (see Table 1), one could combine pre-existing benchmarks to obtain all combinations of domains, sources, DISE tasks and large scale, completely bypassing the need to create a new data generation pipeline and new datasets.\n\nThe motivation behind the need for spatial reasoning in VLMs is also unconvincing in the current presentation. I am not saying it should not be done, but right now it is unclear why the community should be interested.\n\nBecause the above issues pertain to key contributions and the main motivation of the paper, I am leaning towards rejection. I am, however, open to clarifications from the authors in case I am misunderstanding those aforementioned aspects, in which case I would be willing to update my score.", "questions": "**Questions to authors**\n\n-\t[**critical**] Why was it necessary to create Spatial-DISE datasets? Can’t we combine all existing benchmarks referenced in Table 1 to get a large-scale meta-benchmark that covers all domains and DISE tasks?\n-\t[**critical**] Why VLMs capability for sophisticated dynamic spatial reasoning is important beyond the fact that humans can do it? Can you give some detailed examples and elaborate on their importance?\n-\t[**critical**] The last paragraph in Section 2 discusses latest work that also covered most or all DISE tasks, but mentions verifiability as an important distinctive factor. Can you elaborate on the importance of verifiability and why it was so critical that it warranted creating a new data generation pipeline?\n-\tWhy was there a need to create two datasets (Spatial-DISE Bench and Spatial-DISE-12K) and not just one? Do they serve different purposes?\n-\tSection 3.2 Stage 1 mentions 1180 VQA pairs, but then Spatial-DISE Bench (which is 53% “wild” data) has only 559 samples? Where does the 559 come from?\n-\tWere all 12,000 generated VQA pairs manually verified by humans?\n-\tCan the proposed data generation framework generate only 3D scenarios? Can it generate 2D examples?\n-\tIs the error analysis done with Doubao-1.6-thinking free from any potential mistakes? That is, can the model make mistakes (e.g., miscategorise other models’ errors) and mislead the error analysis?\n-\tIn Section 5, why only a subsample (200) of incorrect responses was used and not 100%?\n\n**Additional feedback**\n\n-\tI do not think that claiming the DISE taxonomy as novel is accurate as it was derived from (Maier, 1996; Uttal et al., 2013). Authors should make it clear that this taxonomy has been adopted from previous work. For example, first contribution in line 98 calls it novel.\n-\tCalling non-synthetic data “wild” seems very uncommon and quite confusing. Using a more common term “real data” or “real-world data” would resolve the confusion. If you keep using an uncommon term, I would suggest to explain it early in the text.\n-\tMost of the code in the algorithm listings (Algorithms 1-5) is very hard to interpret as the code mostly consists of function names that are not explained anywhere. Are those functions built-in procedures in Blender?\n-\tThe paper uses phrases like “mental rotation”, “mental transformation” and alike throughout. I am not sure the word “mental” is the right choice as it generally refers to the mind, which I do not think VLMs have. Words like “simulation” or “hypothesis” might be more accurate.\n-\tVerifiability is being mentioned very often throughout the text, but its importance to this work is never properly explained.\n-\tLine 68: “have limited scopes” -> “are limited in scope”.\n-\tLine 69: “multi-steps” -> “multi-step”.\n-\tLine 83: “The Spatial-DISE” -> “Spatial-DISE”.\n-\tLines 89-90: mentions “Spatial-DISE Bench” for the first time without explaining what it is.\n-\tI do not think the first paragraph of Section 3 is at all needed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tAdoption of taxonomy to classify spatial tasks from the cognitive sciences.\n-\tComprehensive evaluation of 28 SotA VLMs.\n-\tHuman quality control of created datasets.\n-\tHuman performances (54 participants) collected and included in the evaluation as a baseline.\n-\tNew data generation framework based on Blender that can generate complex spatial reasoning scenarios.", "weaknesses": "-\tSeveral critical parts motivating this work are unclear or questionable, which makes me seriously concerned as to whether this work addresses a genuine gap in the literature (more below).\n-\tMany important details in the methodology (dataset creation, error analysis) are unclear or missing (more below).\n-\tAny discussion about potential limitations of the work is completely missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DL3GqZMOMz", "reviewer": "Reviewer_oTiT", "summary": "The paper presents Spatial-DISE, a unified benchmark for evaluating spatial reasoning in vision-language models (VLMs). It introduces a cognitively inspired taxonomy, DISE (Intrinsic/Extrinsic × Static/Dynamic), to categorize spatial reasoning tasks. Using a three-stage pipeline combining real-world, synthetic, and human-validated samples, the authors construct a diverse dataset and evaluate 28 VLMs against human performance (76.8%). Results reveal large performance gaps, especially in dynamic and extrinsic reasoning. A fine-tuning study shows that models can improve on Spatial-DISE but suffer from catastrophic forgetting. The paper also offers a cognitive error taxonomy analyzing model failures.\n\nMain Contributions\n- A cognitive taxonomy (DISE) for structuring spatial reasoning evaluation.\n- A large, human-verified benchmark dataset combining real, synthetic, and controlled samples.\n- A comprehensive evaluation of 28 leading VLMs with human baselines.\n- Error taxonomy and analysis revealing systematic weaknesses in spatial reasoning.", "questions": "1. Beyond taxonomy and scale, what new reasoning capabilities does this benchmark test?\n2. Is there evidence of cross-domain transfer? For example, does training on Intrinsic tasks improve performance on Extrinsic reasoning, or are these dimensions fully independent?\n\nI think overall it's a good benchmark. I would consider raising my score if the questions and weaknesses are well-addressed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear Cognitive Taxonomy\n- The proposed DISE framework (Intrinsic/Extrinsic × Static/Dynamic) is cognitively grounded, offering interpretability which is good for benchmark design.\n- This taxonomy provides a systematic structure that unifies fragmented spatial reasoning benchmarks.\n\n2. Cognitive Error Analysis\nThe error taxonomy (Perceptual / Comprehension / Reasoning errors) and the subtypes (Rule Application, Mental Simulation, Holistic–Local Failure) are both novel and psychologically informed.\n\n3. Human Benchmarking\nThey include 54 human participants to establish a strong baseline.", "weaknesses": "1. Fine-tuning and Generalization Analysis Could Be Deeper\n- The fine-tuning experiments (Qwen2.5-VL, SpaceOm) show interesting trends but lack representation analysis — e.g., why forgetting occurs, or what cognitive dimension the gains are concentrated in.\n\n2. Visual Accessibility\n- Figures are a bit too dense (figure 1, 2, 3) and may not fully communicate the DISE framework or task intuitions clearly to non-specialists.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nTL8BL2N72", "reviewer": "Reviewer_YgFh", "summary": "The paper introduces Spatial-DISE, a new benchmark and dataset designed to evaluate the spatial reasoning capabilities of VLMs, along with the data generation pipeline. The authors argue that existing benchmarks are inadequate, often focusing on static scenarios and lacking a systematic, cognitively-grounded framework. To address this, they propose the DISE taxonomy, which categorizes spatial reasoning tasks into four quadrants based on two dimensions: Intrinsic vs. Extrinsic and Static vs. Dynamic. Experiment results show that current models struggle in mental simulation.", "questions": "See as above in weaknesses. I'm happy to adjust the scores if the author can address the three concerns in the weaknesses. \n\nTypo: missing . in L433", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Comprehensive empirical evaluation. The paper presents an extensive and rigorous evaluation across 28 distinct Vision-Language Models. The selection of models is commendable, covering a wide spectrum of architectures including proprietary APIs, open-source foundation models, and specialized models fine-tuned for reasoning. The experiments with fine-tuning in Section 4.3 are interesting, where performance gains on in-domain tasks come at the cost of generalization to other spatial benchmarks.\n2. Cognitively grounded and unified taxonomy, which provides a comprehensive analysis framework in analysing spatial reasoning performance of VLMs. \n3. The paper is well-written, with rich error analysis.", "weaknesses": "1. Fail to disentangle perceptual confounders from reasoning deficits. While the paper attributes model failures primarily to reasoning, as far as I'm concerned, the analysis may not fully account for more fundamental perceptual challenges that act as confounding variables. The evaluation format often presents a complex multi-panel image containing both the question pattern and several graphical options. A model's failure could stem from an inability to correctly understanding this layout which is a visual parsing issue, not a spatial reasoning deficit (which I believe is not really the perceptual error and comprehension error in L447. Please correct me if I'm wrong). This potential confounder weakens several key conclusions:\n* The claim about the relationship between static and dynamic reasoning could be affected.\n* The \"catastrophic forgetting\" observed in Section 4.3 could be reinterpreted. The performance drop on other benchmarks might not be a loss of reasoning ability, but rather a failure to generalize to different visual formats and layouts on the perception side. \n2. Concerns in task categorization. While the DISE taxonomy is well-founded, the mapping of specific tasks to its quadrants can be ambiguous. A task may not be \"cognitively pure\" and could be solvable via multiple reasoning strategies that cross quadrant boundaries. For example, 3D Shape Finding is classified as Intrinsic-Static, implying a logical deduction about a fixed object's properties. However, a viable and intuitive strategy to solve this task involves mentally rotating the cube (an Intrinsic-Dynamic process) to build a complete 3D mental model before identifying the missing face. This ambiguity raises concern regarding the diagnostic clarity of the benchmark. If a model fails this task, it is unclear whether the failure lies in static deduction or dynamic mental simulation, which in turn limits the precision of the paper's conclusions about specific cognitive weaknesses.\n3. Lack of deeper analysis on model architecture. The evaluation is comprehensive in its breadth of models but could be deeper in its analysis of architectural influences. The paper groups models by family or purpose (e.g., \"reasoning models\") but does not delve into how specific architectural choices (e.g., type of vision encoder, cross-attention mechanism, size of the vision model vs. language model) might correlate with performance on the different DISE quadrants. (minor comments that are not factored into my score fyi)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Conditions for acceptance have been satisfied.
Decision: Accept (Poster)