Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
Authors: Linjie Li
Mahtab Bigverdi
Jiawei Gu
Zixian Ma
Yinuo Yang
Ziang Li
Yejin Choi
Ranjay Krishna
Venue: ICLR 2026 Poster (2026)
Keywords: spatial reasoning; visual reasoning
Primary area: datasets and benchmarks
TL;DR: STARE: a benchmark designed to rigorously evaluate MLLMs on tasks better solved through multi-step visual simulation.

Abstract: Spatial cognition is essential for human intelligence, enabling problem-solving through visual simulations rather than relying solely on verbal reasoning. However, existing AI benchmarks primarily
   assess verbal reasoning, neglecting the complexities of non-verbal, multi-step visual simulation. We introduce STARE (Spatial Transformations and Reasoning Evaluation), a benchmark designed to
  evaluate multimodal large language models on tasks better solved through multi-step visual simulation. STARE features ~4K tasks spanning foundational geometric transformations (2D and 3D),
  integrated spatial reasoning (cube net folding and tangram puzzles), and real-world spatial reasoning (perspective and temporal reasoning), reflecting practical cognitive challenges like object
  assembly, mechanical diagram interpretation, and everyday spatial navigation. Our evaluations show that models excel at reasoning over simpler 2D transformations, but perform close to random
  chance on more complex tasks like 3D cube net folding and tangram puzzles that require multi-step visual simulations. Humans achieve near-perfect accuracy but take considerable time (up to 28.0s)
  on complex tasks, reducing response time by 7.5 seconds on average with intermediate visual simulations. In contrast, models exhibit inconsistent performance gains from visual simulations,
  improving on most tasks but declining in specific cases like tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0 Flash), indicating that models cannot consistently leverage
  intermediate visual information. Even o3, a strong reasoning model, lags significantly behind human performance across tasks. By evaluating non-verbal visual reasoning beyond conventional
  text-based benchmarks, STARE highlights critical gaps in current AI spatial capabilities and sets a new standard for assessing spatial intelligence in multimodal models.

Published as a conference paper at ICLR 2026
UNFOLDING SPATIAL COGNITION: EVALUATING MULTIMODAL MODELS ON VISUAL SIM- ULATIONS
Linjie Li', Mahtab Bigverdi', Jiawei Gu², Zixian Ma', Yinuo Yang', Ziang Li",
Yejin Choi Ranjay Krishna'
Uniiersiity Washington
National University of Singapore
Stanford University
Question
Step-by-Step Step Human Mental Simulation
Can the net be folded form
cube, yes
MELM Textual Simulation
Visualization Textual Reasoning:
(1) Face becomes the top of the cabe.
2 Face a folds down to the left of Face
]
right beroming (5) l'ace 5 folds down below Face becoming the back face.
< Face- folds down
Reasoning Analysis: and Face 3 on the right (3)), they would be opposite, but they ane adjacent in the net. The correct third step should follow the maintaain their
(6) folds to close the bottom of the cube.
All faces will perfectly align & forming six faced cabe with no overlaps gaps.
Figure 1: Visual simulations play crucial role in real-world tasks, from assembling complex structures to interpreting mechanical diagrams and predicting spatial interactions. Different from how humans would ap- proach a cube net folding problem, existing multimodal models rely heavily on textual simulation, which not sufficient for reaching human-level spatial cognition. The above example shows how textual simulations of GPT-4o make obvious errors when we simulate the steps in 3D space.
ABSTRACT
Spatial cognition essential for human intelligence, enabling problem-solving through visual simulations rather than relying solely on verbal reasoning. How- ever, existing AI benchmarks primarily assess verbal reasoning, neglecting the complexities of non-verbal, multi-step step visual simulation. We introduce STARE (Spatial Transformations and Reasoning Evaluation). a benchmark designed to evaluate multimodal large language models on tasks better solved through multi- step visual simulation. STARE features 44 tasks spanning foundational geomet- ric transformations (2D and 3D), integrated spatial reasoning (cube net folding and tangram puzzles), and real-world spatial reasoning (perspective and temporal rea- soning), reflecting practical cognitive challenges like object assembly, mechanical diagram interpretation, and everyday spatial navigation. Our evaluations show that models excel at reasoning over simpler 2D transformations, but perform close random chance on more complex tasks like 3D cube net folding and tangram puz- zles that require multi-step visual simulations. Humans achieve near- wear-perfect accu- racy but take considerable time (up to 28.0s) on complex tasks, reducing response time by 7.5 seconds on average with intermediate visual simulations. contrast, models exhibit inconsistent performance gains from visual simulations, improving on most tasks but declining specific cases like tangram puzzles (GPT-4o, ol) and cube net folding (Claude-3.5, Gemini-2.0 Flash), indicating that models can- not consistently leverage intermediate visual information. Even o3, a strong rea- soning model, lags significantly behind human performance across tasks. By eval- uating non-verbal visual reasoning beyond conventional text-based benchmarks, STARE highlights critical gaps in current Al spatial capabilities and new standard for assessing spatial intelligence in multimodal models.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Spatial reasoning is not merely a subset of human cognitive abilities but rather a fundamental un- derpinning of intellectual processes (Tversky & Suwa 2009). Reasoning with space enables indi- viduals to solve complex tasks through visually simulating transformations of objects in the mind, anticipating how their actions would physically manipulate other objects. Cognitive psychologists have found ample evidence that humans simulate 2D and 3D transformations to reason about spatial problems (Mitko Fischer 2020 Duan et al. 2022 Wai et al. 2009; Battaglia et al.). [Shepard & Metzler (1971) found that the time taken by a subject to recognize two perspective drawings as the same 3D shape increases linearly with their angular difference orientation, suggesting an analog mental rotation process. |Hegarty| (1992) found that humans employ mental animation, incremen- tally simulating the movement of parts to understand mechanical diagrams. Such abilities enable everyday tasks like assembling furniture, reading maps or instructional diagrams, and navigating new environments. They are also strongly correlated with success in STEM disciplines (Judd & Klingberg 2021] Christensen & Schunn 2009 Hegarty) 2004b).
Despite their prevalence in real-world applications frrmm arranging furniture in a house to molec- ular docking for drug discovery- uating multimodal large language models (MLLMss). Existing datasets largely target static recog- nition or problems that can be re-phrased linguistic reasoning (Johnson] 2017 [Zhang, 2019 Ti
visual simulations are still under-represented when eval-
2022 Duan et al. 2021 Chollet, 2019 Ramakrishnan et al. 2024). In contrast, humans frequently solve spatial challenges- such as folding 2D net into 3D object, assembling a tangram, or tak- ing another visual perspective- by running internal, step-wise visual simulations (Figure T1, which have a long pedigree in the cognitive-sciend literature on human spatial reasoning (Huttenlocher &
Presson 1973 Gunalp et al. 2019 Shheaad & Feng. 1972 Preuss et al. 2024 Ayaz et al. 2012))
To bridge this gap, we introduce STARE (Spatial Transformations and Reasoning Evaluation), a benchmark focused on spatial reasoning tasks that are better solved through multi-step visual simulations. STARE evaluates whether MLLMs can perform complex spatial reasoning akin to the visual simulations humans perform. It spans a spectrum of spatial cognition challenges (Figure structured in increasing complexity:
Foundational geometric transformations: Tasks involving basic planar (2D) and volu- metric (3D) transformations, such as rotations, translations, scaling, and reflections.
Integrated spatial reasoning: Cube net folding, requiring understanding of how 2D pat- terns fold into 3D objects, and tangram puzzles, which assess sequential assembly and spatial positioning of pieces.
Real-worid spatial reasoning: Tasks demanding reasoning about perspective changes and temporal frame sequences, simulating the spatial cognition challenges encountered in ev- eryday scenarios.
In the first two categories, each transformation or operation (e.g., folding face) can be explicitly visualized step by step, and indeed humans often draw or imagine intermediate states when solving them. The last category demands higher-level visual simulation skills without always having clear intermediate visual cues (e.g.. perspective reasoning) (Bass et al.] 2022] Chen et al.] 2023). We carefully curate 44K total instances across these categories, controlling difficulty via distractor similarity and number of simulation steps, to push models beyond superficial pattern-matching.
Our experiments show that models find reasoning over simple 2D transformations relatively easy but struggle with 3D cube net folding and tangram puzzles, performing at near random chance due the need for multi-step simulations. Humans, though nearly perfect in accuracy, took significantly longer-up up to 28.0 seconds to solve some tasks but sped up considerably (down by 7.5 seconds on average) when given intermediate steps. Meanwhile, when models receive intermediate visual steps, their performance varies: e.g., GPT-40, Gemini-2.0 Flash Thinking and ol improve while Gemini-2.0 Flash and Claude worsen on cube net folding, suggesting that not all models effectively utilize visual guidance. In general, even o3 lags significantly behind human performance. To better understand these gaps, we conduct detailed eгror analyses, pinpointing specific reasons for model failures, such as difficulties in accurately interpreting 3D spatial relationships, inability "imagine in space", and struggles with extended visual contexts even when given explicit visual simulations. Fundamentally, models cannot effectively perform visual simulation.
2

---

Published as a conference paper at ICLR 2026
"Implicit" Simulation
Real-workd Spatial Reasoning
Temporal Reasoning
Perspective Reasoning
down indicating agent's dieection
?
indccting
best agent's
image connecily fill
missing
STARE
Integrated Spatial Reasoning
Tangram Puzzle
Cube Net Folding
Can the Tangram puzzle be completed with the avaiiable plece, no?
Can the net folded form cube, yes or not
Foundational Geometric Transformations
Transformation
Transformation
Make smaller then move it left and slightly upwards.
Move it and than make biggs
"Esplicit" Simulation
Figure 2: Overview of STARE. STARE consists of levels of tasks, 2D Transformation and 3D Transfor- mation for foundational spatial reasoning skills, tangram puzzle and cube net folding integrated spatial reasoning, temporal frame inference and perspective reasoning to mimic real-world scenarios. The interme- diate steps for completing tasks in the first two levels can be explicitly simulated, while the more real-world spatial reasoning tasks require more abstract and implicit mental simulations.
Overall, STARE aims to comprehensively test MLLMs' ability to perform sequential visual simu- lations as opposed pure textual reasoning. By evaluating models on tasks grounded in cognitive phenomena like mental imagery, we aim to reveal whether current MLLMs can approach the flexible spatial problem-solving of humans.
THE STARE BENCHMARK
STARE is designed evaluate multimodal models' abilities in spatial cognition and visual reason- ing, focusing specifically on tasks that humans solve non-linguistically, through visual simulation.
Current perception- focused multimodal benchmarks still rely heavily on linguistic reasoning (Fu"
ct al. 2024 Lu et al. 202T: Li et al., 2024a) or static visual recognition (Tong et al. 2024 Wu &
Xie
2023 et al. 2024)), failing measure models' abilities in sequential visual problem-solving
Parallel work in spatial cognition Yiu et al., 2024] Zhhan 2019 Hu et al. 2021 Ramakrishnan
al. 2024, Rismanchian et af., 2024) probes analogy making and pattern induction, yet simulation optional and intermediate visual states are seldom provided because of annotation cost. VSI- Bench (Yang et al. 2024) underscores the role of mental imagery in spatial reasoning, but focuses on spatial memory and estimation from video rather than explicit step-by-step simulation. STARE closes the gap by testing multimodal models across diverse spatial tasks that require step-by-step visual simulations with without explicit linguistic guidance. We describe the overall design of STARE ()2.1) highlighting key differences compared to existing benchmarks. We then provide detailed descriptions of each task, discussing how the data curated ($2.2)
2.1 OVERVIEW OF STARE
STARE is structured to comprehensively cover spatial reasoning at multiple complexity levels, from basic geometric transformations (2D and 3D) to more integrated tasks (cube net folding and tangram puzzles) and real-world spatial reasoning scenarios (temporal frame and perspective reasoning). Each task is presented multiple-choice textual prompts. total, the dataset contains ~4K instances across different evaluation setups (Figure 3.. Detailed statistics of STARE are provided in Appendix Figure 12
yes/no question using carefully designed visual and
3

---

Published as a conference paper at ICLR 2026
Question
Evaluation Settings
Check outa Tangram puocle below: leff panel empptyy Dingrim puecle while right panel hhouse wasiable pieces complete puzzle.
Without Visual Simulations
Question
Question
Steps
Steps
Below the
complete the Tangram puzzle:
Stup Rotase piece by about degpees kockeiee place place B with its uppardali coener Ds. yl Step Place plece with upper-left left-ieft comerr (s. - (o.
Step Rorate piace C by about gn clockliss and place place C with its upper left comeral at (0, (6, Baasd on the sbove steps can the langrum purrile be complesed with the woillbbe pieces, arno?
Intermediate Visual Simulations
With Visuel Simulations
Newe
Question
Intermediato
Visual
Steps
Figure 3: The different variants in the Tangram Puzzle task. We provide visualizations of the complete interleaved inputs for all three types in Appendix G.2
STARE separates tasks that can be visually simulated, i.e., where each transformation step is visually observable, from tasks demanding more abstract and implicit mental simulations, such as perspec- tive reasoning. To support mentally picture or even explicitly draw the intermediate steps, including 2D transformations, 3D transformations, cube folding and tangram puzzle. Additionally, STARE tasks intentionally crafted to closely reflect real-world scenarios such as assembling objects (e.g.. tangram puzzles), interpreting mechanical diagrams (c.g.. cube net folding) and navigating environments (e.g., per- spective reasoning). These scenarios can potentially shed lights on models' abilities in practical, everyday spatial cognition, providing meaningful assessments aligned with common human chal- lenges. A detailed discussion of related works in human visual reasoning and multimodal LLM benchmarks is provided in Appendix D
tie-grained evaluation, synthesize the tasks that humans can
2.2 DATA CURATION
2D transformations: We design two types of tasks assessing spatial reasoning through two- dimensional shape transformations: visual analogy, and instruction-basel tasks. In visual analogy tasks, shape A is shown to transform visually into shape A', after which a shape is provided with candidate shapes for applying the same transformation sequence to B. Instruction- based tasks explicitly describe transformations (e.g.. "Rotate 90 degrees clockwise, then make it bigger") and require selecting the correctly transformed shape from 4 answer choices. Transformations include rotations, translations, uniform scaling, reflection and shearing, with clearly defined parameters. Each task is created with three difficulty levels: easy (with two distractors out of three clearly dif- ferent in appearance), medium (one obvious distractor). and hard (all distractors visually similar, forcing the model pay attention to the transformation itself). In addition, we synthesize samples with 1/2/3 transformation steps to facilitate evaluations in multi-turn visual transformations. We programmatically generate all shapes and their transformed versions using Matplotlib (Matplothib) 2012). Visualizations of different 2D transformation variants are provided in Appendix Figures and
We develop two experimental setups: (1) question + transformation steps, where the transfor- mation steps are shown either verbally (for instruction-base tasks) or visually (for visual analogy tasks); and (2) question + transformation steps intermediate visual simulations, showing all intermediate visualizations of shape B, excluding the final step. We synthesize a total of 1000 instances, ~600 of which are without intermediate visual simulations.
3D transformations: We extend the 2D transformation tasks to three dimensions, creating similar tasks using 3D shapes. Reflection is omitted in 3D because the mirror plane isn 't obviously recogniz- able to human evaluators. The transformations include rotations around arbitrary axes, translations in 3D space, scaling, and shearing. Tasks, difficulty levels, and experimental setups mirror those of the 2D tasks, with a total of 1000 instances. Following (Johnson et al. (017)), we create abstract 3D shapes detailed meshes and use Blender (Blender) render realistic and consistent visuals.
4

---

Published as a conference paper at ICLR 2026
Tangram puzzles: Tangram puzzles test spatial reasoning about how individual pieces fit together to form a complete shape. Each puzzle provides a target grid and pieces, and the task is to determine whether the pieces small grids (3x3 or 4x4) into rectangular or square shapes, then randomly rotated. Irregular variants were also created by merging adjacent rectangles. Invalid puzzles were constructed by adding or removing pieces, altering piece sizes, or giving incorrect placement instructions.
exactly fill the grid. Valid puzzles were generated by randomly dividing
We create three setups for evaluation: (1) question-only, which presents the initial puzzle configu- ration with a query about solvability; (2) question + assembly steps, adding descriptive instructions of each assembly step without visual aids; and (3) question assembly steps intermediate visual simulations, providing both descriptive annotations and intermediate visualizations of the assembly process, excluding the final visualization indicating success or failure. This task comprises ~800 puzzles, evenly divided into solvable and unsolvable instances.
This task evaluates the model's capacity to mentally fold flat 2D patterns into 3D cubes. We provide examples comprising both valid nets (correctly folding into cube) and invalid nets (leading to overlapping or disconnected faces). Each cube net has explicitly labeled faces. To generate these examples, we implement step-by-step algorithm that simulates the folding process by designating a stationary base face and sequentially folding the connected faces. During each folding step, we detect and annotate errors, such overlaps or disconnected faces, and generate corresponding visualizations using Matplotlib, clearly delineating face boundaries. Similar to tan- gram puzzles, evaluate models in three setups, including (1) question-only, (2) question folding steps, and (3) question + folding steps + intermediate visual simulations. The final cube net folding task contains ~320 samples, balanced between valid and invalid configurations.
Cube net folding:
Temporal frame reasoning: This task evaluates a model's ability to infer missing sequential visual information. Each example consists of four consecutive frames from a video, with one frame hid- den. The model must identify the missing frame from set of three options, relying on temporal consistency and logical scene progression.
We construct 471 examples from the Objectron (Ahmadyan et al. 2021 dataset, which contains short, object-centric videos with camera pose annotations. To create meaningful sequences, we extract the longest continuous segment where the camera moves only in one direction (left or right), divide it into four equal intervals, and select a frame from the central portion of each interval. One of these frames is hidden, and the model must identify it from three choices: the correct missing frame and two distractor frames sampled from different, non-overlapping parts of the video.
This task assesses model's ability to understand how scenes appear from different viewpoints. Each example consists of a top-down map that indicates an agent's 's position and orientation, represented by arrow showing the agent's 's viewing direction. The model must then select the correct first-person view from four choices, emphasizing spatial perspective reasoning and spatial relationships in various indoor environments.
Perspective reasoning:
We construct 250 samples using the HM3D dataset (Ramakrishnan et al. 2021), a large collection of 3D indoor spaces derived from real-world environments. To generate each example,
the Habitat simulator (Savva et al.] 2019: Szot et al.) [021]] Puig et al.] 2023) to place an agent at a random position on the floor while ensuring the surrounding scene contains enough visual cues, such as objects and structures, rather than just walls. A top-down view of the agent's position is then captured, and random viewing direction is assigned (forward, right, left, or backward). The four answer choices correspond to these fixed 90-degree viewpoints, ensuring clear distinctions between them. To improve dataset quality, we manually remove ambiguous cases and low-resolution images.
EXPERIMENTS
In this section, we describe our experimental setup in detail, present comprehensive results, and provide an in-depth analysis of common model errors and limitations.
3.1 EXPERIMENTAL SETUP
For synthetic tasks involving explicit simulations (2D transformations, 3D transformations, cube net folding, tangram puzzles), we explore evaluation settings:
5

---

Published as a conference paper at ICLR 2026
Model
2D Trans.
Trans.
Cube Net
Tangram
Temp-
Overall
XVSim
/Vmim
XVSmm
/VSim
XVSim
VVSim
XV
/VSim
oral
pective
Random
25.0
25.0
25.0
25.0
50.0
50.0
50.0
50.0
33.3
25.0
34.8
Closed-source Models
GPT-4o
71.2
82.7 11.5)
65.5
68.4 ( 2.9)
50.3
52.2 1.9)
52.5
51.5 1.0)
39.0
38.7
53.9
Claude-3,5 Soneet
65.9
(1.55
51.5
57.8 (1 6.3)
52.3
51.6 (1 0.7)
59.0
67.6 8.6)
54,0
26.1
53.1
Gemini-2.0 Flash
69.5
75.2 (.77)
56.1
59.3
1.6
37.7
35.6 (1 2.1)
65.0
65.5 0.5)
38.6
37.2
51.3
Gemini-2.0 Flash Think
60.6
62.8 (1:22)
49.5
56.1
6.6)
48.3
50.7 2.4)
39.8
62.8 01 23.0)
45.0
32.7
48.8
81.8
87.7 5:9)
67.9
71.6
3.7)
51.3
53.4 (1 2.1)
55.3
53.2 2.11)
45.0
36.8
57.2
87.5
89.3 (1 1.8)
78.4
(.2)
79.4 11.0)
68.6
82.1
(1 13.5)
51.4
Open-source
LLaVA One Vision
32,9
32.2 0.77)
30.6 3.6)
34.2 3.7)
30.3
39.8 9.5)
35.7
InternVI.22 21.2.57788
47.5
50.1 (.66)
38.1
36.5 (L
37.1
37.3 0.2)
60.7
48.2
12.5)
31,4
26.0
39.2
Qwen2,5-VL.22
16.6
20.0 (1 3.4)
29.1
31.4 2.3)
43.5
41.0 2.5)
50.1
42.7
(1 7.4)
33.3
23.3
32.3
Qwen2.5 VL.-B
35.4
(L.3))
28.8
31.7
2.9)
40.7
44.9 (1 4.2)
54.5
(L 166
36.5
23.2
36.7
Qwen2.5-V1.-228
45.2
48.5 (1 3.2)
49.1
(1 6.1)
53.4 ( 18.2)
61.2
56.9
(1.3))
Human Performance
Accuracy
96.8
98.6 1.8)
96.9 (1 2.3)
98.9 (1 0.6)
91.5
95.8
4.33
99.0
98.
Response Time
14.2
(L.0
17.1
(1.66
13.7
5.2
28.0
10.1
17.9)
16.2
18.4
ABBest Model, Human)
-9.3
-9.3
-19.4
-18.5
-2999
-19.5
-22.9
-13.7
-550
-55.3
-2.00
Table 1: Model Performance With or Without Visual Simulation (VSim) Across Tasks in STARE. Even the top performer, ol, achieves just under 60% accuracy. Humans, in contrast, get near perfect scores. Green (Red) arrows indicate performance improvements (degradations) with visual simulation.
Without Visual Simulations Models receive only an initial image with or without step-by-step tex- tual instructions and had to mentally infer the subsequent transformations without visual guidance, thereby testing their internal mental simulation capabilities.
With Visual Simulations: Models were provided with step-by-ster visualizations clearly illustrat- ing each transformation step before the final result, enabling explicit visual reasoning. Instead of collating the complex step-by-step visualizations into single image, we provide the model with interleaved image and text query for evaluation.
For real-world reasoning tasks, including temporal frame and perspective reasoning, we evaluate models under the standard single image setting without providing explicit intermediate visual steps.
Evaluation Metries. We report accuracy for multiple-choice questions in 2D/3I) transformations, temporal frame, and perspective reasoning tasks. For cube net folding and tangram puzzles, which involve binary yes/no questions, we report the Fl score. We report macro-average performance across tasks as the overall evaluation metric.
Models. We consider the following models: (1) Closed-source models: GPT-4o (OpenAl), Claude- 3.5 Sonnet (Anthropic), Gemini2.0 Flash (Deepmind a), and the reasoning focused
feasoning-focused Gemimi..0 Flash Thinking (Deepmind) b), ol (OpenAl et al.) (024) and o3 (OpenAl 2025). (2) Open-source mod- els: Intern VI.2.5-788 (Chen et al. 2024). LLaVA-One Oniionniii (Li et al. 2024b), Qwen2.5-VI. 3B, Qwen2.5-VEL VL- and Qwen2.5-VL-72
(Bai ct al. 2025)
Additionally, we invite 5 undergraduate students to complete the same tasks the models. The aver- aged performance and response time are recorded to benchmark model capabilities against human- level spatial reasoning, with additional human evaluation results provided in Appendix H
3.2
MAIN RESULTS
The results present in Table show notable variations model performance across different spa- tial reasoning tasks in the STÄRE benchmark. Models achieve the highest accuracy (up to 89.3%) on simpler 2D transformation tasks, significantly surpassing random chance (25%). Accuracy de- creases by roughly 5% average for more complex 3D transformations. Tasks involving intricate multi-step reasoning. such as cube net folding and tangram puzzles, resulted in even worse model performance without visual simulation. Additionally, temporal frame reasoning and perspective reasoning, which require interpreting sequential visual contexts and viewpoint changes, posed con- siderable difficulties, with most models performing similarly to random chance.

---

Published as a conference paper at ICLR 2026
The use of visual simulations (VisSim) enhances model performance in most cases, but not all. GPT-4o exhibits a notable improvement of 11.5% accuracy on 2D transformations with visual sim- ulations, and Claude-3.5 Sonnet shows significant gains (+8.6%) on tangram puzzles. However, visual simulations did not uniformly benefit model performance; certain models like Gemini-2.0 Flash experienced slight performance declines (e.g.. 2.1% 1% decrease on Fl for eube net tasks), in- dicating that models can not always effectively leverage intermediate visual information. The latest reasoning- focused o3 model outperforms all other models with visual simulations. Overall, it im- proves over GPT-4o by 14.2% on average, but still lag behind human performance. Notably, 03 seems to be better at leveraging visual simulations. However, humans show relatively small per- formance gaps between conditions with and without visual simulation, indicating they can mentally simulate transformations effectively even without explicit visual aids.
Open-source models generally exhibit lower accuracy compared to closed-source counterparts, high- lighting a significant performance gap. Larger models like Intern VL.2.5-788 and Qwen2.5-VLL performe relatively better, suggesting benefits from scale, but their results with visual simulations also varied. For instance, V performance decreases significantly in tangram tasks (-12.5%). whereas Qwen2.5-VLL72B improves notably (+18.2%) in eube net folding.
Human performance consistently surpasses that of models, achieving high accuracy across all STARE tasks, and further improved by intermediate visual simulations. However, these tasks were cognitively demanding even for humans, reflected by relatively long response times (e.g., 28.0 sec- onds tangram puzzles without visual simulations). Although intermediate visual simulations significantly reduces cognitive load and response time, humans still require more than 5 seconds to mentally manipulate and reason through these problems and complete the last step. Thus, STARE tasks clearly involve complex, multi-step spatial reasoning beyond simple recognition tasks solv- able at a glance (Fu et al., 2024). These findings underscore humans' superior spatial reasoning capabilities, particularly when aided by visual simulations.
Moreover, study whether gains on abstract, synthetic spatial tasks translate to real-world tasks, we computed model-level correlations between the two domains. Concretely, for each model, we aver- age performance across with or without visual simulation on the 4 synthetic tasks and contrast that with its mean accuracy on the two real-world tasks. This yields strong overall Pearson corre- lation (r 0.88, p Rs 5e -4) across all 11 models. Including human performance further increases the correlation to 0.97 (p le 1e="). )
3.3 DETAILED ANALYSIS
To gain deeper insights into model limitations and identify specific reasoning challenges, we struc- ture our detailed analysis around several targeted questions. We focus on GPT-4o, which achieves the best overall performance among non-reasoning models, with extended analysis on other models in Appendix
Q1: How well do models understand individual transformation types in 2D and 3D? We eval- uate model accuracy on individual transformation operations- roottion, translation, scaling, reflec- tion, and shearing-for both 2D and 3D tasks, comparing performance with and without visual sim- ulation (Figure (.. For 2D tasks, scaling achieves the highest accuracy (approximately 90% without visual simulation), improving further with visual simulation. Shearing was the most challenging in 2D (around 54%). showing minimal improvement from visual aids. Reflection, rotation, and trans- lation significantly benefits from visual simulation, improving roughly 10 percentage points each. In 3D tasks, translation had the highest accuracy (about 76% without visual simulation), although it slightly declines with visual simulation. However, shearing, scaling, and rotation notably improve with visual simulation by about 3-8 percentage points. Overall, visual simulation substantially en- hances performance for complex transformations, especially in 2D, though the added complexity of 3D transformations continues to present significant challenges.
Q2: How does model accuracy change as task complexity increases? (1) Performance vs. Difficulty-level: The left sub-figure in Figure shows model accuracy decreased as tasks became harder. For 2D tasks, GPT-4o performed best on easy tasks (~~86% with visual simulation), with accuracy declining notably for medium and hard tasks, especially without visual simulation (drop- ping to ~66% for hard tasks). For 3D tasks, overall accuracy was lower, decreasing from easy tasks (~72% without visual simulation) to hard tasks (~60% without). Visual simulation generally im-
7

---

Published as a conference paper at ICLR 2026
2D
2D VSim
VSim
Perception Error
100
$2.4
Question: Chock net with squuar faces. Con the net be folded to form cuuee yes no?
31.0
Accuracy (%)
80
12.2
75.6
76.7
76.4
$3.5
Looking at the image, I see not with s
Claude:
60
53.7
59.5
Escor Analysis:
labcled squars arringed a sppcilfic pattionn
Squaree [yclow)) on the left
Square (gnom))
Square (purple) is the cop
40
below Square 5.
Square 3 (pinkkse)) the mindle right
Square
Square (blue) is below Segurne gand the
20
right Square
Square (orannge)
right Squarc 4 Sqqurre (green) bellow Square de
... answer thie-peeifie
Translate
Reflection
Transformation Type Figure 4: GPT-4o performance on individual 2D/3D trans- Figure 5: A perception error from Claude-3.5
formation types, with and without Visual Simulation (VSim). Sonnet. See Appendix G.3 for more case study.
2D
3D
20
3D
20
Vilim
3D VSim
20
VSi 83.2
3D VSMM
34.4
$2.0
80
78.3
77.9
Accuracy (%6
60
$1.0
60.8
66.4
0..5
57.4
Accuracy (%6
75
76.2
40
00:2
20
0
Easy
Medium
Hard
2
Difficulty Level
Number Transformation Steps
Figure 6: GPT-40 performance task complexity (left: difficulty levels and right: number of transformation steps) with or without Visual Simulation (VSim).
proved accuracy but was less effective even slightly detrimental for the hardest 3D tasks (60.5% without, 57.4% with). (2) Performance vs. Number of Turns: The right sub-figure in Figure 6 shows that how model performance varies with the number of transformation steps (N 1, 2, 3). Without visual simulation, accuracy for both 2D and 3D tasks initially increases from N = to N 2. and then decreases at N 3. The observed peak at N 2 likely occurs because two-step transformations combine simpler transformations (c.g., scaling) with more challenging ones (c.g., shearing), allowing models to leverage the simpler transformations determine the correct answer. In contrast, one-step transformations are evenly distributed across all transformation types, while N = 3, the increased complexity from multiple transformations compounds cognitive demands, reducing overall model accuracy. With visual simulation, accuracy remains consistently high across 2 and 3 steps in 2D tasks and shows stable or slightly improved performance at N = 3 in 3D tasks (performance at N = 1 with visual simulation is omitted as there is intermediate step).
Q3: Do model failures originate from basic visual perception errors? To determine if model fail- ures originate from fundamental visual perception rather than higher-level reasoning limitations, we design straightforward probing experiment. Specifically, we simplify the task by directly present- ing the model with the final, fully simulated outcomes, redueing the problem to visually matching these outcomes to the correct candidate answers. Under these conditions, accuracy improves by 4.2% (from 82.7% 86.9% on 2D transformations and 2.8% (from 68.4% to 71.2%) on 3D trans- formations, indicating only a modest improvement when eliminating intermediate steps. However, for more structured tasks like cube net folding and tangram puzzles, providing the fully completed final form drastically raises accuracy to 100% and 91.6%, respectively, highlighting that models can solve these tasks when the perceptual complexity is minimized. To further isolate the nature of perceptual errors in cube net folding, we create targeted tasks to test both 2D perception (color recognition and face connectivity) and 3D perception (identifying if a face has been folded). Re- sults from these tasks (Table 2 reveal perfect color recognition but notable decrease in accuracy for face connectivity (94.1%) and particularly low accuracy in correctly identifying folded faces (57.4%). Figure 5 illustrates an example of perception error connectivity misalignments from Claude-3.5 Sonnet. Moreover, these specific perceptual errors in folding explain the limited benefits from visual simulations observed in Table for GPT-4o. Overall, while some errors indeed stem
8

---

Published as a conference paper at ICLR 2026
Model
2D Perception
Perception
Input
2D Trans.
3D Trans.
Cube Nets
Tangram
Color
Connectivity
Folded?
Text-only
87.5
64.7
57.0
72.6
Image-only
75.1
67.7
56.0
62.5
GPT-4o
100.0
94.1
57.4
Image+Text 90.8
70.0
62.1
Table 2: 2D) and 3D perception accuracy in cube-net folding.
Table 3: GPT-4o performance without visual simulation under
different input representations.
Input
Cube Nets
Tangram
Simulation State
2D Trans.
3D Trans.
Cube Nets
Tangram
Partial
86.8
72.1
51.3
43,5
Question-only
50.2
62.4
All
82.7
68.4
52.2
51.5
Question+Sieps
50.4
34.7
Last
89.4
68.4
35.2
43.4
Table 4: GPT-4o performance with
Table 5: GPT-4o performance with different intermediate visual-
question-only vs. explicit reasoning steps.
simulation states.
from basic visual perception deficits, particularly in more complex 3D scenarios, the results suggest higher-level reasoning also plays a large role in overall model failures. We further extend the per- ception probing to 2D/3D transformations and tangram puzzles (Appendix HH; models score well above chance on most basic perceptual subtasks, yet still fail on the full reasoning tasks, confirm- ing that multi-step spatial simulation, rather than low-level perception, is the core bottleneck. See Appendix F and G.5 for additional quantitative and qualitative error analysis.
Q4: How well do models reason spatially in text? To evaluate how well models reason spatially from text alone, we translate each visual task into clear, concise descriptions. For 2D and 3D transformation tasks, each object is described by stating its shape, color, position, size and etc.-for instance, "a red square at position (3.4) with size 2". In the cube-net folding task, the unfolded cube is represented by numbering each face and arranging these numbers in grid matching the cube net's visual layout. For example, "123456" represents all six faces in a single row. Lastly, for the langram puzzle lask, each piece is labeled (e.g., "Piece A") and represented by compact grid indicating occupied cells marked by "1". For instance, a square piece might be shown as two rows of "11". Examples of the text representations for each task are shown in Appendix G.4
As shown in Table 3 providing the model with text representation removes much of the perception challenge, yet accuracy remains well below human performance- 65% on 3D transformations, and roughly 73% on tangram puzzles, suggesting that the model still lacks the ability mentally simulate the steps to solve each task. Text helps most on 2D spatial reasoning: accuracy on 2D transformations rises from 75% with images alone to 87% with text, and tangram performance climb from 63% to 73%. For tasks involving 3D) spatial reasoning, however, text gives little benefit, partly because the simple text description about shape, color, material, center, and size, cannot capture all the depth and adjacency cues in 3D spatial reasoning. We observe similar trends with o3 (Appendix Table 22) confirming that while text helps language-friendl tasks, strong 3D spatial performance still depends critically on visual perception and cannot be substituted by textual descriptions alone.
57% on cube-net folding,
Q5: How well do models verbally simulate without visual simulation? We evaluate how effec- tively models verbally simulate spatial reasoning without intermediate visual simulations by compar- ing performance when provided only the question (Question-only versus explicit verbal reasoning steps (Question+Steps). Table minimal improvement in cube net folding (50.2% to 50.4%), indicating limited benefit from verbal reasoning alone. Conversely, tangram performance notably decreases (62.4% to 34.7%), suggesting models adopt shortcuts like summing piece areas rather than genuine spatial simulation. This result partially reflects a bias in our question-only set: models can achieve 75% accuracy by simply checking total areas of available pieces rather than reasoning about spatial arrangement.
Q6: How well do models integrate textual context with isolated visual simulations? We com- pared accuracy when presenting models with complete visual sequences versus only the final or most relevant visual state (Table(5). Easier tasks like 2D and 3D transformations showed improved or comparable accuracy when presented only the final state (e.g., 82.7% for complete for last), suggesting that for these tasks, the final visual state closely resembles the initial state, re- ducing cognitive load. However, in complex tasks such cube net folding (52.2% complete 35.2% last) and tangram puzzles (51.5% complete vs. 43.4% last), the final state becomes more dis- connected from the initial configuration, requiring deeper understanding of preceding verbal steps.
89,4%

---

Published as a conference paper at ICLR 2026
This disconnection introduces significant challenges for models, aligning with earlier findings (Q4) and underscoring their difficulties in integrating complex visual sequences during multi-step rea- soning. Beyond synthetic tasks, explored extending visual simulations real-world settings (Appendix E perspective reasoning benefits strongly from intermediate views (GPT-40: 38.7% - 76.7%) while temporal reasoning shows negligible improvement. A pilot study on ALFRED robot planning tasks (Appendix E further shows that visual simulations substantially improve task planning accuracy, suggesting practical value for embodied Al applications. See Appendix for additional analysis on the impact of visual simulation granularity.
CONCLUSION
We introduced STARE, a benchmark of ~4K instances that evaluates multimodal models on spatial cogmition tasks requiring step-by-step visual simulation. Our experiments across 11 models show that visual simulations substantially improve performance on structured tasks but yield inconsistent gains on complex multi-step reasoning. Even o3, the strong reasoning model, lags behind humans and relies on externlly provided simulations rather than internal spatial reasoning. Our analysis identifies multi-step spatial simulation as the core bottleneck: models handle isolated perceptual subtasks well but struggle to chain multiple visual transformations into coherent spatial reasoning.
Limitations and Future Work. STARE's synthetic tasks enable controlled evaluation but may not capture the full complexity of real-world spatial reasoning. Pilot studies robot planning and mechanical diagram interpretation (Appendix D suggest that visual simulation benefits extend to practical domains. We also find that current interleaved generation models show limited ability to self- seff-generate task-aligned visual simulations (Appendix E), though this may improve with more ca- pable unified models. Future work should extend the benchmark more diverse real-world domains and investigate whether explicitly training models to perform visual simulation can effectively close the gap with human-level spatial cognition.
5 ETHICS STATEMENT
STARE provides standardized way to measure AI capabilities in spatial reasoning tasks, poten- tially guiding research toward Al systems that can better support robotics, autonomous driving, augmented reality, and educational applications. However, improved spatial reasoning could also lead to negative societal impacts if misused, such as enhanced surveillance capabilities or military applications. Additionally, the synthetic nature of STARE may introduce biases toward simplified or artificial scenarios, limiting direct applicability to diverse real-world conditions. Future versions should aim include more realistic and diverse datasets and consider ethical guidelines to minimize potential risks and ensure fair, positive societal outcomes.
REPRODUCIBILITY STATEMENT
We have taken substantial steps to ensure the reproducibility of our results. All experimental set- tings are described in detail in Appendix G We provide complete documentation of the statistics of STARE and the design spaces for all synthetic tasks, including 2D transformations, 3D transforma- tions, cube net folding, and tangram puzzles in Appendix [F] Data curation and evaluation code is included in the supplementary material to facilitate verification and reuse. Together, these resources enable the community to reproduce our experiments and extend our findings.
REFERENCES
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Ob- jectron: A large scale dataset of object-centric videos in the wild with pose annotations. Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Anthropic.
Claude
3.5
sonnet.
c claude-2 3- 5- sonnet
10

---

Published as a conference paper at ICLR 2026
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015.
Hasan Ayaz, Patricia A Shewokis, Meltem Izzetoğlu, Murat P Çakır, and Banu Onaral. Tangram solved? prefrontal cortex activation analysis during geometric problem solving. In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4724- 4727. IEEE, 2012.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://azxivi org/ abtpabbb. 13923
L.W. Barsalou. Grounded cognition. Annual Review of Psychology, 59:617-645, 2008.
llona Bass, Kevin A. Smith, Elizabeth Bonawitz, and Tomer D. Ullman. Partial mental simulation explains fallacies in physical reasoning. Cognitive Neuropsychology. 2022.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation an engine of physical scene understanding. Proceedings of the National Academy of Sciences,
P.W. Battaglia, J.B. Hamrick, and J.B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110K45): 2013.
Daniel M. Bear, Elias Wang. Damian Mrowea, Felix J. Binder, Hsiao Yu F. Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan- Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2 106.08666, 2022.
Mahtab Bigverdi, Zelun Luo, Cheng- Yu Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. arXiv preprint arXiv:2412.03548. 2024.
Blender. Blender free software. https://wwwt www.bee.ee....... org/
P.A. Carpenter, M.A. Just, and P. Shell. What one intelligence test measures: A theoretical account of the processing in the raven progressive matrices test. Psychological Review, 97(3):404-431, 1990.
Tony Chen, Kelsey R. Allen, Samuel J. Cheyette, Joshua Tenenbaum, and Kevin Smith. Just in timetipresentationns for mental simulation in intuitive physics. In Proceedings of the 45th Annual Meeting of the Cognitive Science Society (CogSci), 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
François Chollet. On the measure of intelligence. arXiv preprint arXiv: 1911.00571.. 2019.
Bo T Christensen and Christian D Schunn. The role and impact of mental simulation in design. Ap- plied Cognitive Psychology: The Official Journal of the Society for Applied Research Memory and Cognition, 23(3):327-344, 2009.
Google
Deepmind. Introducing gemini 2.0:
our new ai model for the agen- https: /ttps://bbog. http:/oooooooee/cccoooooooo htt////hhoooo/ggoggg..... deepmind/
tic
era.
poilll--emnnii ai cal-uupatee-eee wpduuuptdaeeeeaaaeeeeee december 2024// a.
Google
Deepmind.
Gemini 2.0 flash thinking mode. b.
ptttps://a..oog/1 dev
gemi ni wwaanniiiiiiiiii http////osttttttigggc/ee
Jiafei Duan, Samson Yu, and Cheston Tan. Space: A simulator for physical interactions and causal learning in 3d environments. In Proceedings of the ieeelcvf international conference on computer vision, pp. 2058-200 2058-2063, 2021.
11

---

Published as a conference paper at ICLR 2026
Jiafei Duan, Samson Yu, Soujanya Poria, Bihan Wen, and Cheston Tan. Pip: Physical interaction prediction via mental simulation with span selection. In European Conference on Computer Vision, pp. 445-421. Springer, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. In arXiv preprint arXiv:2306. 13394, 2023.
Deqing Fu', Ghazal Khalighinejad", Ollie Liu", Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models isomorphic rep- resentations, 2024.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint 21X1v:2404.13390 2024.
Dedre Gentner. Structure- Stracture-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):
155-170, 1983.
Peri Gunalp, Tara Moossaian, and Mary Hegarty. Spatial perspective taking: Effects of social, directional, and interactive cues. Memory & cognition, 47:1031-1043, 2019.
Wenyu Han, Siyuan Xiang, Chenhui Liu, Ruoyu Wang, and Chen Feng. Spare3d: A dataset for spatial reasoning on three-view line drawings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14690-14699, 2020.
M. Hegarty. Mechanical reasoning by mental simulation. Trends in Cognitive Sciences, 8(6):280- 285, 2004a.
Mary Hegarty. Mental animation: Inferring motion from static displays of mechanical systems. Journal of Experimental Psychology: Learming, Memory, and Cognition, 1865):004-11102, 1992.
Mary Hegarty. Mechanical reasoning by mental simulation. Trends in Cogmitive Sciences, 8(6):280- 285, 2004b. ISSN 1364-6613. doi: https://oo..ag///..ooo.....s.220 URL https:
// // /www. www.sceiiiiccc..ce...m....c..ccs rect. com http:///c..eeeeeeer..eaeaaaa//a/
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence volume 35, pp. 1567-1574, 2021.
Janellen Huttenlocher and Clark C Presson. Mental rotation and the perspective problem.
Cognitive Psychology, 4(2):277-299 1973. ISSN 0010-0285. doi: https://do..or//10.. 10. 101660010--22547390001-.7 URL
N. Ichien, Q. Liu, Fu, K.J. Holyoak, A. Yuille, and H. Lu. Two computational approaches to visual analogy: Task-specific models versus domain-general mapping. Cognitive Science, 47():: e13347, 2023.
Nicholas Ichien, Qing Liu, Shuhao Fu, Keith J. Holyoak, Alan Yuille, and Hongjing Lu. Visual analogy: Deep learning versus compositional models. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society (CogSci), 2021.
et al. Ji, W. Abstract visual reasoning with tangram shapes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2350-23600 2022.
et al. Johnson, Justin. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2901-2910, 2017.
Justin Johnson, Bharath Hariharan. Laurens der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2026
Nicholas Judd and Torkel Klingberg. Training spatial cognition enhances mathematical learning in a randomized study of 17,000 children. Nature Human Behaviour, 5(11):1548- 1554, 2021.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Clarence Lee, M Ganesh Kumar, and Cheston Tan. Determinet: A large-scale diagnostic dataset for complex visually-grounde referencing using determiners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 200190200000 2023.
Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang. Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision- language models on natural adversarial samples. European Conference on Computer Vision, 2024a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint Stli:2400.033326 2024b.
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. Advances in Neural Information Processing Systems, 01144093--409388 2024.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
A. Lovett and K. Forbus. Modeling visual problem solving as analogical reasoning. Psychological Review, 124(1)660-90, 2017.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Shixian Luo. Zezhou Zhu, Yu Yuan, Yuncheng Yang, Lianlei Shan, and Yong Wu. Geogram- bench: Benchmarking the geometric program reasoning in modern llms. arXiv preprint arXiv:2505. 17653, 2025.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint 2rXiv:2210007774, 2022.
Matplotlib. Matplotlib:
Visualization with python. https://matplotlib.coo
2012.
Alex Mitko and Jason Fischer. When it all falls down:
The relationship between intuitive physics and spatial cognition. Cognitive research: principles and implications, 5:1-13, 2020.
OpenAl. Hello
lattps://openai: http:s//pppeaaa..coco/dee/////os
OpenAI. Openai o3 and o4-mini system card, 2025.
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang.
13

---

Published as a conference paper at ICLR 2026
Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li. Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song. Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman. Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte. Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lemnon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz. Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov. Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofrr Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL BRLLhttps:// :tttpp:://aaxiv... /tw.iv.iiivv/..a org/ abs 2212.4112111 16720
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Tung Nguyen, Daron Anderson, Imad Ali Shah, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Jaeho Lee, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang. Robert Gerbicz, John- Clark Levin, Serguei Popov, Fiona Feng, Steven Y. Feng. Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang. Mstyslav Kazakov, Geoff Galgon, Johannes Schmitt, Alvaro Sanchez, Yongki Lee, Will Yeadon, Scott Sauers, Mare Roth, Chidozie Agu, Søren Rüs, Fabian Giska, Saiteja Utpala, Antrell Cheatom, Zachary Giboney, Gashaw M. Goshu, Sarah-Jane Crow- son, Mohinder Maheshbhai Naiya, Noah Burns, Lennart Finke, Zerui Cheng, Hyunwoo Park, Francesco Fournier-Facio, Jennifer Zampese, John Wydallis, John B. Wydallis, Ryan G. Hoerr, Mark Nandor, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Jungbae Nam, Edwin Taylor, Jun Jin, Gautier Abou Loume, Hangrui Cao, Alexis C Garretson, Damien Sileo, Qiuyu Ren, Doru Cojoc, Pavel Arkhipov. Usman Qazi, Aras Bacho, Lianghui Li, Suneet Motwani, Christian Schroeder de Witt, Alexei Kopylov, Johannes Veith, Eric Singer, Paolo Rissone, Jachyeok Jin, Jack Wei Lun Shi, Chris G. Willcocks, Ameya Prabhu, Longke Tang, Kevin Zhou, Emily de Oliveira Santos, Andrey Pupasov Maksimov, Edward Vendrow, Kengo Zenitani, Joshua Robinson, Aleksandar Mikov, Julien Guillod, Yuqi Li, Ben Pageler, Joshua Vendrow, Vladyslav Kuchkin, Pierre Mar- ion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Andrew Gritsevskiy, Dakotah Martinez, Nick Crispino, Dimitri Zvonkine, Natanael Wildner Fraga, Saeed Soori, Ori Press, Henry Tang. Ju- lian Salazar, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, T. Ryan Rogers, Wenjin Zhang. Ross Finocchio, Bikun Li, Jinzhou Yang, Arun Rao, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Ariel Ghislain Ke-
14

---

Published as a conference paper at ICLR 2026
mogne Kamdoum, Tad Hogg, Alvin Jin, Carlo Bosio, Gongbo Sun, Brian P Coppola, Haline Hei- dinger, Rafael Sayous. Stefan Ivanov, Joseph M Cavanagh, Jiawei Shen, Joseph Marvin Imperial, Philippe Schwaller, Shaipranesh Senthilkuma, Andres M Bran, Andres Algaba, Brecht Verbeken, Kelsey Van den Houte, Lynn Van Der Sypt, David Noever, Lisa Schut, Ila Sucholutsky, Ev- genii Zheltonozhskii, Qiaochu Yuan, Derek Lim, Richard Stanley, Shankar Sivarajan, Tong Yang. John Maar, Julian Wykowski, Martí Oller, Jennifer Sandlin, Anmol Sahu, Cesare Giulio Ardito, Yuzheng Hu, Felipe Meneguitti Dias, Tobias Kreiman, Kaivalya Rawal, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Jeremy Nguyen, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao, Pierrot Arsene, Sergey Ivanov, Rafat Pošwiata, Chenguang Wang, Daofeng Li, Donato Crisostomi, Ali Dehghan, Andrea Achilleos, John Arold Ambay, Benjamin Myklebust, Archan Sen, David Perrella, Nurdin Kaparov, Mark Inlow, Allen Zang. Kalyan Ramakrish- nan, Daniil Orel, Vladislav Poritski, Shalev Ben-David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Dan Bar Hava, Alcksey Kuchkin, Robert Lauff, David Holmes, Frank Sommerhage. Anji Zhang, Richard Moat, Keith Schneider, Daniel Pyda, Zakayo Kazibwe, Mukhwinder Singh, Don Clarke, Dae Hyun Kim, Sara Fish, Veit Elser, Victor Efren Guadarrama Vilchis, Immo Klose, Christoph Demian, Ujjwala Anantheswaran, Adam Zweiger, Guglielmo Albani, Jeffery Li. Nicolas Daans, Maksim Radionov, Václav Rozhoñ, Vincent Ginis, Ziqiao Ma, Christian Stump, Jacob Platnick, Volodymyr Nevirkovets, Luke Basler, Marco Piccardo, Niv Cohen, Virendra Singh, Josef Tkadlec, Paul Rosu, Alan Goldfarb, Piotr Padlewski, Stanis- law Barzowski, Kyle Montgomery, Aline Menezes, Arkil Patel, Zixuan Wang, Jamie Tucker- Foltz, Jack Stade, Declan Grabb. Tom Goertzen, Fereshteh Kazemi, Jeremiah Milbauer, Ab- hishek Shukla, Hossam Elgnainy, Yan Carlos Leyva Labrador, Hao He, Ling Zhang, Alan Givré, Hew Wolff, Gözdenur Demir, Muhammad Fayez Aziz, Younesse Kaddar, Ivar Ängquist, Yanxu Chen, Elliott Thorneey, Robin Zhang, Jiayi Pan, Antonio Terpin, Niklas Muennighoff, Hailey Schoelkopf, Eric Zheng, Avishy Carmi, Jainam Shah, Ethan D. L. Brown, Kelin Zhu, Max Bar- tolo, Richard Wheeler, Andrew Ho, Shaul Barkan, Jiaqi Wang, Martin Stehberger, Egor Kretov, Peter Bradshaw, JP Heimonen, Kaustubh Sridhar, Zaki Hossain, Ido Akov, Yury Makarychev, Joanna Tam. Hieu Hoang, David M. Cunningham, Vladimir Goryachev, Demosthenes Patrama- nis, Michael Krause, Andrew Redenti, David Aldous, Jesyin Lai, Shannon Coleman, Jiangnan Xu, Sangwon Lee, Ilias Magoulas, Sandy Zhao, Ning Tang, Michael K. Cohen, Micah Carroll, Orr Paradise, Jan Hendrik Kirchner. Stefan Steinerberger. Maksym Ovchynnikov, Jason O. Matos, Adithya Shenoy, Michael Wang, Yuzhou Nie, Paolo Giordano, Philipp Petersen, Anna Sztyber- Betley, Paolo Faraboschi, Robin Riblet, Jonathan Crozier, Shiv Halasyamani, Antonella Pinto, Shreyas Verma, Prashant Joshi, Eli Meril, Zheng-Xin Yong, Allison Tee, Jérémy Andréoletti, Orion Weller, Raghav Singhal, Gang Zhang, Alexander Ivanov, Seri Khoury, Nils Gustafsson, Hamid Mostaghimi, Kunvar Thaman, Qijia Chen, Tran Quoc Khánh, Jacob Loader, Stefano Cavalleri, Hannah Szlyk, Zachary Brown, Himanshu Narayan, Jonathan Roberts, William Al- ley, Kunyang Sun, Ryan Stendall, Max Lamparth, Anka Reuel, Ting Wang. Hanmeng Xu. Pablo Hernández-Cāmare Freddie Martin, Thomas Preu, Tomek Korbak, Marcus Abramovitch, Do- minic Williamson, Ida Bosio, Ziye Chen, Biró Bálint, Eve J. Y. Lo, Maria Inês S. Nunes, Yibo Jiang, MM Saiful Bari, Peyman Kassani, Zihao Wang, Behzad Ansarinejad, Yewen Sun, Stephane Durand, Guillaume Douville, Daniel Tordera, George Balabanian, Earth Anderson, Lynna Kvis- tad, Alejandro José Moyano, Hsiaoyun Milliron, Ahmad Sakor, Murat Eron, Isaac C. McAlister, Andrew Favre D. O., Shailesh Shah, Xiaoxiang Zhou, Firuz Kamalov, Ronald Clark, Sherwin Abdoli, Tim Santens, Harrison K Wang, Evan Chen, Alessandro Tomasiello. G. Bruno De Luca, Shi-Zhuo Looi, Vinh-Kha Le, Noam Kolt, Niels Mündler, Avi Semler, Emma Rodman, Jacob Drori, Carl J Fossum, Luk Gloor, Milind Jagota, Ronak Pradeep, Honglu Fan, Tej Shah, Jonathan Eicher, Michael Chen, Kushal Thaman, William Merrill, Moritz Firsching, Carter Harris, Stefan Ciobācā. Jason Gross, Rohan Pandey, Ilya Gusev, Adam Jones, Shashank Agnihotri, Pavel Zhel- nov, Siramut Usawasutsakorn, Mohammadreza Mofayezi, Alexander Piperski, Marc Carauleanu, David K. Zhang, Kostiantyn Dobarskyi, Dylan Ler, Roman Leventov, Ignat Soroko, Thorben Jansen, Scott Creighton, Pascal Lauer, Joshua Duersch, Vage Taamazyan, Dario Bezzi, Wiktor Morak, Wenjie Ma, William Held, Tran ue Huy, Ruicheng Xian, Armel Randy Zebaze, Mo- hanad Mohamed, Julian Noah Leser, Michelle X Yuan, Laila Yacar, Johannes Lengler, Katarzyna Olszewska, Hossein Shahrtash, Edson Oliveira, Joseph W. Jackson, Daniel Espinosa Gonzalez, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Da- souqi, Alexander Shen, Emilien Duc, Bita Golshani, David Stap, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Lukas Lewark, Miguel Orbegozo Rodriguez. Mátýás Vincze, Dustin Wehr, Colin Tang, Shaun Phillips, Fortuna Samuele, Jiang Muzhen, Fredrik Ekström, Angela Hammon, Oam
15

---

Published as a conference paper at ICLR 2026
Patel, Faraz Farhidi, George Medley, Forough Mohammadzadch, Madellene Peñaforr, Haile Kas- sahun, Alena Friedrich, Claire Sparrow, Rayner Hernandez Perez, Taom Sakal. Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Kenchi Okutsu, Mike Battaglia, Mohammad Magh- soudimehrabani, Alon Amit, Dave Hulbert, Roherto Pereira, Simon Weber, Handoko, Anton Peri- styy, Stephen Malina, Samuel Albanie, Will Cai, Mustafa Mehkary, Rami Aly, Frank Reidegeld, Anna-Katharina Dick, Cary Friday, Jasdeep Sidhu, Hassan Shapourian, Wanyoung Kim, Mari- ana Costa, Hubeyb Gurdogan, Brian Weber, Harsh Kumar, Tong Jiang, Arunim Agarwal, Chiara Ceconello. Warren S. Vaz, Chao Zhuang, Haon Park, Andrew R. Tawfeek, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan Kim, Johan Ferret, Yuzhou Wang. Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Abram Jackson, Shreen Gul, Gunjan Chhablani, Zhehang Du, Adrian Cosma, Jesus Colino, Colin White, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Petr Spelda, Vit Stritecky, Syed M. Shahid, Chaistopite Mourrat, Lavr Vetoshkin, Koen Sponselce, Renas Bacho, Florencia de la Rosa, Xiuyu Li, Guillaume Malod, Leon Lang, Julien Laurendeau, Dmitry Kazakov, Fatimah Ade- sanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Julien Degorre, Yigit Yalm, Ghenga Daniel Obikoya, Luca Amaboldi, Rai, Filippo Bigi, M. C. Boscá, Oleg Shumar, Kaniuar Bacho, Pierre Clavier, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mildred Tanwie, Denis Peskoff, Thomas C. H. Lux, Ben Rank, Colin Ni, Matthew Brooks, Alesia Yakim- chyk, Huanxu, Liu, Olle Häggström, Emil Verkama, Hans Gundlach. Leonor Brito-Santana, Brian Amaro, Vivek Vajipey, Rynaa Grover, Yiyang Fan, Gabriel Poesia Reis Silva, Linwei Xin, Yosi Kratish, Jakub Lucki, Wen-Ding Li, Sivakanth Gopi, Andrea Caciolai, Justin Xu, Kevin Joseph Scaria, Freddie Vargus, Farzad Habibi, Long, Lian, Emanuele Rodolà, Jules Robins, Vincent Cheng, Tony Fruhauff, Brad Raynor, Hao Qi, Xi Jiang, Ben Segev, Jingxuan Fan, Sarah Martin- son, Erik Y. Wang. Kaylie Hausknecht, Michael P. Brenner, Mao Mao, Xinyu Zhang, David Ava- gian, Eshawn Jessica Scipio, Alon Ragoler, Justin Tan, Blake Sims, Rebeka Plecnik, Aaron Kirt- land, Omer Faruk Bodur, D. P. Shinde, Zahra Adoul, Mohamed Zekry, Ali Karakoe, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina, Juan Carlos Gonzalez, Gabe Maayan, Sarah Hoback, Rodrigo De Oliveira Pena, Glen Sherman, Elizabeth Kelley, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu, Sandra Mendoza, Ismail Alarab, Joshua Cole, Danyelle Ferreira, Bryan Johnson, Mohammad Safdari, Liangti Dai, Siriphan Arthornthurasuk, Alexey Pronin, Jing Fan, Angel Ramirez-Trinidad, Ashley Cartwright, Daphiny Pottmaier, Omid Taheri, David Outevsky, Stanley Stepanic, Samuel Perry, Luke Askew, Raúl Adrián Huerta Rodríguez, Ali M. R. Minissi, Sam Ali, Ricardo Lorena, Krishnamurthy Iyer, Arshad Anil Fasiludeen, Sk Md Salauddin, Murat Islam, Juan Gonzalez, Josh Ducey, Maja Som- rak, Vasilios Mavroudis, Eric Vergo, Juehang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Anil Radhakrishnan, Antoine Jallon, I. M. J. Mclnnis, Pawan Kumar, Laxman Prasad Goswami, Daniel Bugas, Nasser Heydari, Ferenc Jeanplong, Archimedes Apronti, Abdallah Galal, Ng Ze- An, Ankit Singh, Joan of Are Xavier, Kanu Priya Agarwal, Mohammed Berkani, Benedito Alves de Oliveira Junior, Dmitry Malishev, Nicolas Remy, Taylor D. Hartman, Tim Tarver. Stephen Mensah, Javier Gimenez, Roselynn Grace Montecillo, Russell Campbell, Asankhaya Sharma, Khalida Meer, Xavier Alapont, Deepakkumar Patil, Rajat Maheshwari, Abdelkader Dendane, Priti Shukla, Sergei Bogdanov, Sören Möller, Muhammad Rehan Siddiqi. Prajvi Saxena, Himan- shu Gupta, Innocent Enyekwe, Ragavendran P V, Zienab EL-Wasif, Aleksandr Maksapetyan, Vivien Rossbach, Chris Harjadi, Mohsen Bahaloohoreh. Song Bian, John Lai, Justine Leon Uro, Greg Bateman, Mohamed Sayed. Ahmed Menshawy, Darling Duclosel, Yashaswini Jain, Ashley Aaron, Murat Tiryakioglu, Sheeshram Siddh, Keith Krenek, Alex Hoover, Joseph McGowan, Te- jal Patwardhan, Summer Yue, Alexandr Wang, and Dan Hendrycks. Humanity's 's last exam, 2025. URL
14249
Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in deep-learning model inspired by developmental psychology. Nature Human Behaviour, 6(9): 1257-1267, 2022.
Kai Preuss, Christopher Hilton, Klaus Gramann, and Nele Russwinkel. Identifying cognitive pro- cesses and neural substrates of spatial transformation in mental folding task with cognitive modeling. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024.
Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang, Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimír Von-
16

---

Published as a conference paper at ICLR 2026
druš, Vincent-Pierre Berges. John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakr- ishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots, 2023.
S. K. Ramakrishnan, E. Wijmans, P. Krahenbuhl, and V. Koltun. Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.06468, 2024.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d); 1000 large- scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2021. URL https://axxiv..ot [2IC9.08238
Sina Rismanchian, Yasaman Razeghi, Sameer Singh, and Shayan Doroudi. Turtlebench: visual programming benchmark in turtle geometry. arXiv preprint arXiv:241 2024.
William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, and Ritambhara Singh. Forgotten polygons: Multimodal large language models are shape-blind. arXiv preprint arXiv:2502. 25..2259999999 /5969, 2025.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Roger N. Shepard and Christine Feng. A chronometric study of mental paper folding. Cognitive Psychology, 3(2):228-24) 1972. ISSN 0010-0285. doi: https//doi.org/10
1016/00100025((299000--9 URL
commscciece//
atticlee/p1100010225577990
Roger N Shepard and Jacqueline Metzler. Mental rotation three-dimensional objects. Science,
171(3972)701-703, 1971.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10740-10749, 2020.
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Ro- bospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15768-15780 2025.
Andrew Szot, Alex Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Von- drus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang. Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems (NeurIPS). 2021.
J.B. Tenenbaum, T.L.. Griffiths, and C. Kemp. bhsery- bayesian models of inductive learning and reasoning. Trends in Cognitive Sciences, 10(7):309-318, 2006.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209, 2024.
Barbara Tversky and Masaki Suwa. Thinking with sketches. 2009.
T.D. Ullman, E.S. Spelke, P. Battaglia, and J.B. Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in Cognitive Sciences, 21(9):64--665 2017.
17

---

Published as a conference paper at ICLR 2026
Jonathan Wai, David Lubinski, and Camilla P Benbow. Spatial ability for stem domains: Align- ing over 50 years of cumulative psychological knowledge solidifies its importance. Journal of Educational Psychology. 101(4)).17, 2009.
Taylor W. Webb, Shuhao Fu, Trevor Bihl, Keith Holyoak, and Hongjing Lu. Zero-shot visual reasoning through probabilistic analogical mapping. arXiv preprini arXiv:2209. 15087, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL htths:::: /www. a acl www.wwe..b.... wwwprr////aoooooooooooo
emn 2020 emn pp- 11-.00mmmlpppeeeeee demos.
Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal
llms. ArXiv, abs/2312.14135, 2023, URL https: /tt:://app. /api. semanticscholestet org/
Cor CorpusID:: DouuID::22 2D666333DD66611111111
Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang. Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15273 2025.
Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in
Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412.14171, 2024.
Lingxiao Yang, Hongzhi You, Zonglei Zhen, Dahui Wang, Xiaohong Wan, Xiaohua Xie, and Ru- Yuan Zhang. Neural prediction errors enable analogical visual reasoning in human standard intel- ligence tests. Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
Eunice Yiu, Maan Qraitem, Charlie Wong, Anisa Noor Majhi, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal mod- els. arXiv preprint arXiv:2407.1777 2024.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint 200.:2200900883,, 2024.
Xiyao Yue, Yifan Ni, Kai Zhang, Tao Zheng, Ruixuan Liu, Wen Chen, et al. Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311. 2023.
C. et al. Zhang. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5317-5327, 2019.
Wenxuan Zhang, Sharifah M. Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Ad- vances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2023.
18

---

Published as a conference paper at ICLR 2026
A
OVERVIEW OF THE APPENDIX
This Appendix organized as follows:
Section
discusses the use LLMs.
Section discusses the limitations of STARE.
Section D
presents extended discussion about related works.
Section details the statistics of STARE and the design spaces for all synthetic tasks, including 2D transformations, 3D transformations, cube net folding, and tangram puzzles.
S provides additional analysis complementary to Section including detailed error analysis, impact of # visual simulations and reasoning efforts, additional results on human evaluation, and new experiments on open-ended answering, interleaved image generation, visual simulation for real-world tasks, downstream robot planning, and mechanical diagram interpretation.
Section G describes the experimental setup, covering the prompts used, model configu- rations, hyperparameter settings, and presents full visualizations of different experimental settings (e.g., evaluation settings with or without visual simulations, perception probing questions).
Section H provides results on additional models for analysis conducted in Section 3 in- cluding extended perception probing results and results on text representations.
B THE USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) as auxiliary tools during manuscript preparation, but only for surface-level editing such as grammar correction, minor rephrasing, and stylistic refinements to improve readability. Al-assisted coding was employed curating synthetic data, but under strict human supervision and review. In addition, LLMs served as judges in detailed error analysis; however, we manually reviewed a subset of their outputs to verify accuracy. All research ideas, methodologies, experiments, and conclusions were conceived and executed exclusively by the au- thors.
C LIMITATIONS
Although STARE provides valuable insights, still has several limitations. First, it uses simplified synthetic images that do not fully represent real-world complexity; future versions could include alistic or dynamic scenes with clutter and occlusion. Second, it focuses only on rigid shape transfor- mations; adding tasks involving flexible shapes, articulated objects, additional sensory cues (such as audio or depth) would cover wider range of spatial reasoning skills. Lastly, multiple-choice scoring hides intermediate reasoning steps; extending evaluations with explanations, step-by-step checks, or open- open-ended responses would give more detailed insights, which we briefly explore in Appendix E
Still, STARE 's current design has clear strengths. The simplified images isolate spatial reasoning from general object recognition tasks. Its structured variety of tasks helps pinpoint specific model difficulties. Automatic scoring ensures consistent and easy-to-scale evaluations. Modular task pre- sentations (image-only, text-only, image+text prompts) let researchers analyze individual modality contributions. Additionally, synthetic data makes STARE easily reproducible, accessible, and ex- tensible. Overall, STARE is a strong first step toward measuring multimodal spatial reasoning, with clear pathways toward more realistic and comprehensive future benchmarks.
D RELATED WORK
Human visual reasoning. Human visual reasoning relies two complementary faculties: lational analogy mapping abstract structures across scenes-are and mental simulation predicting future states through incremental transformations. Stapeture-Mapping Theory (Gentner 1983) and
19

---

Published as a conference paper at ICLR 2026
analyses of Raven's Progressive Matrices (Carpenter et al., 1990) first showed that success in visual problem-solving hinges on aligning relations rather than surface features. Computational accounts echo this claim: explicit relational models reproduce human-like performance (Lovett & Forbus 2017], whereas modern deep networks still struggle with visual analogy tasks (Ichien et al. 2021
Webb et al. 2022 Ichien et al. (023).
Mental simulation complements analogy-making Classic work on mental rotation (Shepard & Met- z 1 and mechanical reasoning (Hegarty) (2044) demonstrates that people mentally "run" transformations, consistent with grounded-cognition theories (Barsalou 2008). Intuitive-physics studies cast the mind as noisy physics engine that combines object-centric structure with proba- bilistic dynamics (Battaglia et al. [013]] Tenenbaum et al.) 2006) [Ullman et al.) (217). Object-based
predictivecoodng models such as PLATO extend these ideas, achieving human-like physical pre- diction and developmental trajectories (Yang et al. 2023 Piloto et al. 2022). Simulations are also selective: people allocate attention "just in time," focusing on the most diagnostic elements instead
of exhaustively modeling the entire scene (Bass et al. 2022: Bear et al. 2022 Chen et al. 2023).
Together, these findings suggest that effective problem-solving hinges on the ability to carry out step-by-step visual simulations; our benchmark therefore probes whether multimodal models can effectively leverage or even produce such simulations and exhibit human-like visual reasoning on sequential, compositional tasks.
Multimodal evaluation benchmarks. Recent advances in evaluating multimodal large language models have led to the development of benchmarks targeting diverse aspects of visual reasoning. Early benchmarks such as VQA (Antol et al.) (015) and CLEVR [Johnson] 2017) focus on compo- sitional reasoning and general visual question answering. However, more challenging benchmarks, such as MMMU (Yue et al. 2023) and Humanity's 's Last Exam (HLE) (Phan et al. 2022). assess expert-level, domain-specttn sppciffcc reasoning using complex multimodal inputs, where state-of-the-ar models achieve only around 60% on MMMU-pr (Yue et al. 2024) and below 20% on HLE.
In response to the growing demand for robust evaluation, several new benchmarks (Fu" et al. 2024
Lu et al. 202T] Li et al. 2024a; Tong et al. 2024 Wu & Xie 2023) have been introduced. For exam-
questions (Zhang et al. 2023). MME (Fu et al. 2023) and MMBench (Liu et al. (024)) separate low-level perception from higher-level cogni- tion. BLINK (Fu et al. 2024) departs from pure linguistic reasoning tasks to include tasks grounded in core computer vision capabilities, including relative depth estimation, semantic correspondence, visual similarity assessment, inpainting, etc. Improvements on BLINK require the use of perception tokens (Bigverdi et al. 2024). which generate latent intrinsic images to reason, demonstrating for the first time, that reasoning doesn't 't have to be linguistic. In this work, we build upon this finding, targeting primarily visual reasoning that can be better solved with visual cues.
ple, M3Exam repurposes multilingual professional-licene
The most relevant benchmarks to ours are KiVA (Yiu et al.] 2024). RAVEN/I-RAEN (Zhang 2019 Hu et al.) 2021), SPACE (Ramakrishnan al. 2024), and TurtleBench (Rismanchian al. 2024). which primarily evaluate static analogy or pattern induction, where intermediate visual simu- lations are optional and often infeasible to curate. VSI-Bench (Yang al. 2024) emphasizes mental imagery in spatial reasoning but centers on spatial memory and distance estimation from video. Other recent efforts-such as Forgotten Polygons (Rudman al., 2025). GeoGramBench (Luo) et al., 2025). and VisuLogic (Xu et al. 2025)-target target more isolated failures in visual reasoning, including shape recognition, symbolic geometry, or visual logic puzzles. contrast, STARE intro- duces programmatically generated puzzles-2D/31 2D/3D transformations. cube-net folding, and tangram assembly-that isolate a model's 's capacity to benefit from explicit visual simulations, and further extends to perspective-taking and temporal reasoning tasks that mirror real-world scenarios.
Table compares STARE with other spatial reasoning datasets: RoboSpatial (Song et al. 2025) provides large-scale real 2D/3D images with annotated spatial relations; MSR3D (MSQA) (Linghu ct al. 2024) and SQA3D (Ma et al. 2022) support situated QA in 3D scenes but focus on single-step queries; the Visual Spatial Reasoning benchmark (Liu al.) 2023) probes basic positional relations; SPARE3D (Han et al. 2020) presents synthetic 2D to 3D consistency puzzles; and DetermiNet (Lee) jet al. 2023) emphasizes logical spatial tasks without multi-step simulation. As shown, STARE is the only benchmark that offers diverse suite of multi-step visual simulation tasks across both 2D and 3D domains, uniquely combining procedural puzzles, geometric transformations, and realistic inference (perspective and temporal reasoning). Its synthetic design allows fine-grained
20

---

Published as a conference paper at ICLR 2026
Dataset
VSim
2D/3D
Synth/Real Multi-step
Train/Eval
Size
Focus
STARE
2D & 3D
Both
Eval
<4K
Multi-step spatial simulations
VSI-Bench
x
3D (viden)
Real
x
Eval
5K
Spatial memory & layout recall from egocentric videos
KIVA
X
2D
Synthetic
Eval
4.3K
Visual analogical reasoning spired by child cognition
TurtleBench
x
Symhetic
Eval
260
Reproduce geometric programs via turtle graphics
SPAREAD
x
2D
3D
Syntheric
Both
220K+
Spatial reasoning with 3-view CAD drawings
VSR
x
2D
Real
Eval
10K
Spatial relation understanding in caption-image pairs
DetermiNet
x
2D
Symthttic
x
Both
250K
Referring expression under- standing with quantifiers and determiners
Furgollen Polygons
x
2D
Symthttic
x
Eval
2K Shape identification and count- ing under visual ambiguity
GeoGramBench
x
2D
Synthetic
Eval
500
Symbolic geometric reasoning from procedural programs
VisuLogic
x
Synthetic
Eval
IK General visual logic across di- verse reasoning types
Table 6: Comparison of existing visual/spatial reasoning datasets versus STARE.
control step difficulty and granularity, enabling analyses of visual reasoning beyond what isting datasets support.
E ADDITIONAL EXPERIMENTAL RESULTS
Fine-grained Reasoning Evaluation Because each synthetic task in STARE includes ground- truth metadata for every intermediate simulation step, scrutinize model's entire reasoning chain somethig impossible on benchmarks that provide only final answers.
We have conducted deeper error analysis of GPT-4o and Claude-3.5 predictions on all synthetic tasks.
We first examined representative case studies (Appendix G.5 and identified four recurring categories of reasoning failure:
Error Category
Description
Representative Example
Misperception
The model misreads shapes, color, adjacency, or face layout.
Misidentifies cube-net face po- sitions (Figure 25
Flawed Spatial Simulation
model forms an incorrect mental model of how shapes transform.
Claims rotated hexagon un- changed after 30 rotation (Fig- ure rrihht
Heuristic Over-Use
The model falls back on shal- low heuristics (e.g., area count- ing).
Sums tangram piece areas and misjudges solvability (Fig- ure 26
D
Logic Inconsistency
The reasoning process contra-
Correctly identifies two valid answers but chooses the wrong arbitrarily (Figure
dicts itself or the final answer.
Table 7: Representative error categories (A-D) observed in model predictions, with descriptions and examples.
We implemented an automated LLM- judgment pipeline. Given a model's chain-of- thought, its prediction, and full ground-truth metadata (e.g., initial shape, transformation sequence, correct answer. and options), the judge model categorizes each incorrect predic- tion into one of the four failure types above. Here, use o3 the judge model.
21

---

Published as a conference paper at ICLR 2026
The table below summarizes the distribution of error types across 100 randomly sampled incorrect predictions per task, per model:
Model
Task
A
B
C
2D
75.8%
21.2%
0.0%
3.0%
3D
31.3%
67.7%
0.0%
1.0%
GPT-4o
Cube Net
12.7%
87.3%
0.0%
0.0%
Tangram
68.1%
12.1%
6.6%
13.2%
2D Transform
60.0%
35.8%
3.3%
3D
34.0%
64.1%
0.0%
2.0%
Claude-3.5 Sonnet
Cube Net
22.2%
74.7%
3.0%
0.0%
Tangram
65.9%
17.1%
9.8%
7.3%
Table Distribution error categories (A-D) across tasks GPT-4o and Claude-3.5 Sonnet.
Different tasks show distinct failure modes: 2D and tangram stem from mispercep- tion, 3D and cube nets from simulation gaps, with occasional heuristic over-use and logic inconsistency chain-of-thought.
We use LLM-as-judge with structured rubric to score how well model reasoning traces align with ground-truth metadata (e.g., shape interpretation, transformation accuracy), We then sorted model responses into quartiles based on alignment score and evaluated accuracy within each group.
Model
Task
Q1 (alignment)
Q2 (alignment)
Q3 (alignment)
Q4 (alignment)
2D
61.1% (0.664)
100.0% (0.917)
100.0% (0.950)
100.0% (0.952)
3D
33.3% (0.325)
100.0% (0.875)
100.0% (0.927)
100.0% (0.973)
GPT-4o
Cube Net
0.0% (0.004)
0.0% (0.096)
0.0% (0. (8)1
100.0% (0.606)
Tangram
0.0% (0.000)
50.0% (0.228)
100.0% (0.819)
100.0% (0.998)
2D
0.0% (0.214)
83.3% (0.768)
100.0% (0.931)
100.0% (0.959)
3D
0.0% (0.168)
90.9% (0.727)
100.0% (0.909)
100.0% (0.981)
Claude-3.5 Sonnet
Cube Net
0.0% (0.041)
0.0% (0. 100)
0.0% (0.150)
100.0% (0.714)
Tangram
0.0% (0.000)
42.9% (0.161)
100.0% (0.725)
100.0% (0.991)
Table 9; Task accuracy by alignment quartile for GPT-4o and Claudc-3.5 Sonnet.
This analysis reveals a strong correlation between reasoning quality and final task accuracy. The prompt we used to scoring model reasoning is shown below.
Reasoning Error Analysis Rubrics
You are an expert spatial-reasoning judge.
Given the information blocks below, score the model's 's reasoning using the rubric.
Question: (question.text)
Ground Truth
Start shape:
(transformation.se wttp:aa.eiootttttessessssee desc) (transformation
/cholcc.shape.des/)
Correct answer: (gt_answer]
Model Response: [model_
Model Final Answer: (model.pred
Rubric assign 0 to each item
1. Shape interpretation
2. Transformation comprehension
Spatial simulation accuracy
22

---

Published as a conference paper at ICLR 2026
4. Answer justification (choice evaluation)
Logical consistency
Return JSON
"shape_innterpreaatinn""" ("score": 011, "details"; ""),
httpafFrrmattioocoomreeeeeses ( (*score": 011, "details": ""), htw.iial_ssimllaiioaooooooe ( (*score"; 011, "details"; ""), "answer_ "anser__justificaiio"" ("score"; 011, "details"; ""), "Logical_consiste ("score": 011, "details"; ""), "prinary_errors". [],
""
Number of visual simulations vs. Performance, We investigated this question briefly in Table The key insight is that the benefit of visual simulations depends on task complexity and where in the sequence the decisive information appears. We also conducted stricter ablation that incrementally reveals simulation frames. The takeaway straightforward: visual simulations help only when the model can integrate them; otherwise they add noise.
Because examples have different total chain lengths, the same column mixes complete simulations for short chains (e.g., all 2-step tasks are already complete at # simulations = 1) and partial simula- tions for longer ones (e-g.. only one of two frames visible # simulations 1 for 3-step tasks). model could perfectly integrate every extra frame, scores would rise monotonically. However, the results above suggest otherwise, especially for more complex tasks. These results again highlight core limitation: models lack the capacity to mentally simulate and reason visual sequences, skill that humans perform reliably.
Model
Task
simulations
2D
71.2
78.0
85.6
3D
65.5
67.2
69.6
GPT-4o
Cube Net Tangram
50,3
50.5
49.2
49.1
52.5
51.7
46.6
54.7
65.9
67.7
75.3
3D
51.5
60.8
53.9
Claude-3.5
Cube Net
52.3
51.3
51.7
50.0
Tangram
59.0
60.1
62.2
69.0
Table 10: Ablation with 0-3 intermediate simulations. Bold indicates best performance per row.
Reasoning Efforts vs. Performance. In addition to the results of o3 reported in Table we further report o3 performance across different reasoning efforts (low, auto, high), it is specifically optimized to "think with images" and perform extended reasoning.
Task (w/o ys. w/ VSim)
03-Low
03-Auto
03-High
Human
2D
88.1 /92.3
87.5 89.3
89.7 / 89.5
96.8/ 98.6
3D
73.7 75.3
75.2 / 78.4
73.9 76.5
94.6 97.0
Cube Net
65.3 71.7
68.4 72.5
66.3 / 71.1
98.3 99.0
Tangram
68.6 / 76.5
68.6 82.1
66.4 / 82.8
91.5 / 95,8
Video-temporal
55.8
51.4
54.3
99.0
Perspective
42.8
44.0
98.1
Overall
67.5
67.5
68.8
97.1
Table 11: Performance of o3 model across different reasoning efforts (Low, Auto, High) compared human annotators on STARE tasks. For tasks with visual simulation (VSim), accuracy is reported as "w/o / VSim".
While o3 outperforms earlier models such ol and GPT-4o, its performance STARE remains significantly below buman-level. Notably, o3 leverages visual simulations more effectively than prior models. However, humans show only small performance gaps between conditions with and
23

---

Published as a conference paper at ICLR 2026
without visual simulation, reflecting their ability mentally simulate transformations. In contrast, o3 exhibits substantial drops without external visuals, highlighting a key limitation: the inability to perform internal, structured visual simulation- a component of human spatial reasoning.
In addition, while o3 has reported to benefit from extended reasoning, simply increasing the reason- ing efforts from low high does not guarantee better spatial reasoning.
Additional Results on Human Evaluation. Table I we reported the average human perfor- mance and average response time for each task across 5 participants. In Table[[2] wer further report the standard deviation. Mean accuracy across five annotators is 97.07% with a standard deviation of 0.47, indicating that the questions in STARE well-defined.
Task
Accuracy (%)
Time (8)
2D
96.75 + 1.30
14.23 + 1.99
2D VSim
98.56 + 1.10
10.95 1.51
3D
94.61 + 1.44
17.06 = 5.08
3D + VSim
96.98 + 0.80
12.53 = 0.65
Cube Net
98.29 + 1.65
13.67 = 3.66
Cube Net + VSim
98.86 + 1.46
5.16 + 0.49
Tangram
91.53 + 2.56
27.98 5.85
Tangram + VSim
95.78 + 1.07
10.08 + 4.20
Temporal
99.03 + 0.98
16.19 3.96
Perspective
98.10 + 0.17
18.04 + 4.46
Overall
97.07 + 0.47
Table 12: Task accuracy response across STARE benchmark tasks. with without visual simulation (VSim).
Correlation Analysis between Synthetic tasks and Real tasks. In Section 3.2 we briefly dis- cussed the correlation between averaged model performance on synthetic tasks (including 2D trans- formation, 3D transformation, cube net folding and tangram puzzle) and that real-world tasks (including temporal frame reasoning and perspective reasoning). FigureZZshows the averaged model performance synthetic and real-world tasks across 11 models and the fitted line with correlation coefficient 2s 0.88.
Claude 3.5 Sonnet
Closed-source
GPT-4o X
40
Open-source
X
Real-wond woold task average accuracy
Fit line (r 0.88) Canini-2.5
Genini-2.0 Flach
36
LLaVa-00eeWccon-122
32-
30
Qven-2.5-wen S-VL-7B
1wen-2.5-VL-723
InternVi-2.544 5-788
26
Qven-2. S-VL-BB
30
35
40
45
50
55
60
65
Synthetic task average accuracy (%)
Figure Correlation between model performance on synthetic tasks that on real-world tasks.
Note that for open-source models, the real-world task performance is close to random guessing (29%). Removing the open-source models, the correlation coefficient decreased to 0.58, still showing a weak but positive correlation between synthetic task performance and real-world task performance.
Visual Simulation for Real-World Tasks. While visual simulations improve performance on syn- thetic spatial tasks, investigate whether this benefit extends real-world tasks. We create three
24

---

Published as a conference paper at ICLR 2026
evaluation settings for perspective and temporal reasoning (see Figures and for examples): (1) w/o VSim: the standard setting; (2) w/ partial VSim: intermediate frame for perspective (sam- pled along the camera trajectory) and I extra context frame before and after the missing frame for temporal; (3) w/ all VSim: intermediate frames for perspective and 2 extra context frames on each side for temporal.
As shown in Tab the two tasks exhibit strikingly different patterns. For perspective reasoning. accuracy increases sharply more intermediate views are provided: GPT-4o improves from 58.7% to 76.7%, and o3 from 42.8% to 84.8%. This confirms that perspective tasks support a meaningful with/without simulation framing, and models benefit substantially from seeing the viewpoint tran- sition. For temporal reasoning. however, adding partial or full context yields only modest or no gains (GPT-4o: 39.0% - 37,5%; o3: 51.4% 50.3%), suggesting that current models struggle to infer coherent motion trajectories even when more evidence is available. Notably, humans already reach 99.0% (temporal) and 98. 1% (perspective) without any extra simulated frames, highlighting the large gap between human and model spatial reasoning.
Model
Task
w/o VSim
Partial VSim
All VSim
GPT-4o
Temporal
39.0
37.4
37.5
GPT-4o
Perspective
38.7
55.0
76.7
o3
Temporal
51.4
50.7
50.3
o3
Perspective
42.8
55.2
84.8
Table 13: Effect of visual simulations on real-world tasks. Perspective reasoning benefits strongly from inter- mediate views, while temporal reasoning shows negligible improvement.
weo Visual Simulation
Partial Vieual Simulation (+ before context)
All Visual Simulation (+ before after context)
Framo
Defore
Figure 8: Example from the temporal frame reasoning task with visual simulation. Top: without visual simula- tion, the model sees 4 frames with one missing and candidate options. Middle: partial visual simulation adds 2 context frames before the missing frame. Bottom: full visual simulation adds context frames both before and after. Green border indicates the correct answer.
25

---

Published as a conference paper at ICLR 2026
we Visual Simulation
Option
Opton
Optin
Option D
Top-doan Map
Visuel Simulation (+ trajectory views)
Tra: 2
TRal
Opt
Opt.
Opt
Gpt.D
Top-down Map
Figure 9: Example from the perspective reasoning task with visual simulation. Top: without visual simulation, the model top-down map with the agent's position/orientation and selects first-person view from options. Bottom: with visual simulation, intermediate trajectory views along the agent's path are provided as additional context. Green border indicates the correct answer.
Downstream Application: Robot Planning. To explore the practical value of visual simulation beyond benchmark evaluation, we conduct a pilot study on the ALFRED dataset (Shridhar et al., 2020), which involves household robot plamning tasks. The model sees current state image, goal state image, and several candidate plans (sequences of high-level actions), and must choose which plan reaches the goal (see Figure 10 for examples). For each sample, we also have intermediate state images along the execution of the correct plan (average 5.73 steps). We evaluate models with incremental visual simulations (25%, 50%, 75%, and 100% of the intermediate frames) 200 multiple-choice samples. As shown in Table 14] task planning accuracy improves substantially as more simulation frames provided. GPT-4o improves from 39.5% without simulation 79.5% with all frames, and o3 improves from 73.5% to 93.0%. This strong benefit from visual simulations procedural, embodied setting complements our findings on synthetic tasks, though we note that verbal reasoning alone can partially solve these tasks (e.g., matching goal descriptions to plan text), which is why we keep STARE focused on tightly controlled, nen-verbal spatial reasoning.
Model
w/o VSim
25%
50%
75%
All VSim
GPT-4o
39,5
60.5
66.5
68.0
79.5
o3
73.5
71.0
78.5
87.0
93.0
Table 14: Pilot study ALFRED robot planning tasks (200 samples), Accuracy improves visual simulation frames provided.
26

---

Published as a conference paper at ICLR 2026
wr Visual Simulation
Visual Simulation (+ intermedinte steps)
Figure Examples from ALFRED robot planning task. Top: without visual simulation, the model sees only the initial state and goal state. Bottom: with visual simulation, intermediate state images showing the progression of the correct plan are provided. The model must select the correct plan from multiple candidates.
Mechanical Diagram Interpretation. Motivated by the potential applicability of spatial reasoning to engineering domains, we conduct a pilot study on mechanical diagram interpretation. We curate 28 questions involving gear trains and pulley systems (see Figure 11 for examples), requiring models to reason about rotational direction, mechanical advantage, and motion transmission.
As shown in Table 15 o3 (67.88 (67.86%) significantly outperforms GPT-4o (39.29%) consistent with its stronger spatial reasoning observed on synthetic tasks. The large gap suggests that mechanical diagram interpretation requires the same kind of structured spatial reasoning that o3 excels at. This pilot demonstrates that our benchmark s spatial reasoning constructs extend to practical engineering domains, and we leave a comprehensive evaluation to future work.
Model
Accuracy (%)
GPT-4o
39.29
o3
67.86
Table 15: Pilot study mechanical diagram interpretation (28 questions gear/pulley systems).
Example 1: Pulley System
Question: How many pulleys tumiing clockwis??
B
G
Answer Cholces:
E
G
Example 2: Gear Train
Question:
Three cogwheels mesh in tram. which direction should gear Y rotate have gear rotate wul.erllckkk bee?
Answer Choloes:
Figure 11: Examples from the mechanical diagram interpretation pilot. Top: pulley system question requir- ing counting and spatial reasoning. Bottom: gear train question requiring models to reason about rotational direction propagation.
27

---

o3 as a judge model to extract Published as a conference paper at ICLR 2026
Open-Ended Answering. To verify that the multiple-choice (MC) format does not artificially inflate model performance, we evaluate GPT-4o on cube net folding and tangram puzzle tasks using free- form, open-ended answering where the model must generate the answer without seeing candidate options.
As shown in Table 16 MC and open-ended scores comparable: cube net folding achieves 50.3% with MC and 50.9% % with open-ended answering, while tangram puzzles show 52.5% with MC and 48.3% with open-ended answering. The small differences (<5%) in both directions confirm that the MC format does not systematically inflate or deflate performance, validating our evaluation protocol.
Task
MC
Open-Ended
Cube Net
50.3
50.9
Tangram
52.5
48.3
Table 16: Comparison of multiple-choiee (MC) vs. open-ended answering for GPT-4o. Scores comparable, confirming the MC format does inflate performance.
Interleaved Image Generation. investigate whether models capable of interleaved text and image generation can leverage self-generated visual intermediates for spatial reasoning, we evaluate Bagel-7B, an open-source model with interleaved generation capabilities. We modified the decoding logic to force interlcaved generation: after the model produces text "thought," we switch to image generation mode, then back to text generation with all previous context (both image and text), Since the model often does not strictly follow the requested format, answer choices.
As shown in Table 17 interleaved generation helps 50.8% > 76.5%; tangram: 48.5% - 64.8%), but performance remains low overall and even drops on temporal (22.5% 16.0%) and perspective (6.0% 4.0%). These results show that simply allowing interleaved image-text generation does not automatically produce meaningful, task-aligned visual simulations. The bottleneck lies in performing the correct spatial transformations and using visual evidence to reach the right answer.
tasks (3D: 24.0% - 48.0%; cube nets:
Setting
Cube Nets
Tangram
Temporal
Perspective
Text-only
44.9
24.0
50.8
22.5
6.0
Interleaved Gen.
46.0
48.0
76.5
64.8
16.0
4.0
Bagel-7B performance (w/o VSim) with text-only generation vs. interleaved image generation. In- terleaved generation helps on some synthetic tasks hurts on real-world tasks.
Table 17:
F DATA CURATION DETAILS
Figure 12 presents the overall composition of STARE. Table details the number of instances for each task in STARE, further broken down by whether the input contains an explicit intermediate visual simulations.
28

---

Published as a conference paper at ICLR 2026
Perspective
Tongram Puzzee Vaim
Temporal
6.4%
7.3%
Tangram Puzzle
18.3%
28.8%
20 Text Instruct
8.5%
RSR
ISR
3.0%
Folding Nets Vism
Folding Nets
3.6%
FGT
Text Instruct
5.2%
52.9%
30 Vo Vsim
7.8%
7.8%
20 Va
5.2%
5.2%
7.8%
20 Va Vaim
RSR: Resi-worid Spetial Reasoning FGT. Foundational Geometric maanioomations Integrated Spatial Reasoning
Figure 12: Data Statistics of STARE.
Task category
Without visual simulation
With visual simulation
Total
Foundational Geometric Transformations
2D transformations
639
423
1,062
3D transformations
612
408
1,020
Integrated Spatial Reasoning
Cube net folding Tangram puzzle
120
532
289
821
Real-world Spatial Reasoning
Perspective reasoning Temporal frame reasoning
250
471
471
Total
2,697
1,240
3,937
Table 18:
Dataset statistics grouped by task category and by the presence full intermediate visual simulation.
Below, we summarize the design space of data curation for synthetic tasks, including (1) 2D Trans- formations (FII)); (2) 3D Transformations FF2); (3) Cube Net Folding ($F3); and (4) Tangram Puzzles
F.1 2D TRANSFORMATIONS
Shape generation. Shapes are selected from fixed and assigned properties as follows:
Types: Circle, Square, Rectangle, Triangle, Ellipse, Hexagon, Pentagon.
Colors: Face color a random RGB tuple (r, g, E [0, 1]): edge color is fixed (black).
Center & Size: All shapes are centered (0, 0). For circles, squares, triangles, hexagons, and pentagons, size is a scalar drawn from [30, 35]; for rectangles and ellipses, size is tuple (width in [30, 35), height in [20, 25)).
Transformations. A sequence of randomly sampled operations is applied to the shapes:
Rotate:
Squares: +330, + (avoiding 90%).
29

---

Published as conference paper at ICLR 2026
Hexagons: =300, -
Others: =30", =60", or =9900
Rotation is applied w.r. the shape's center.
Flip: Horizontal (about y = 0) or vertical (about æ 0); not applied when the shape is centered at (0, 0) for symmetric shapes such square, circle and etc.
Translate: (dz, dy) with dx, dy {-30, - 10, 0, 10, 30} with constraints to ensure nonzero translation.
Scale: Factors chosen from {0.5, 2.0), ensuring the resultant size is within roughly [10, 40].
Shear: Parameters (shear,, shear,) are drawn from approximately [-1, 1]. with constraints to ensure a perceptible skew. Shear is excluded for 2D text instructed transformation tasks, human participants find it hard describe the degree of shear such that they differ- entiate among the answer candidates.
Number of Transformation Steps. The final dataset contains instances with 1, 2, or transforma- tion steps.
Easy
Medium
Imagine trandfomiig pentagon step step.
Imagine transforning a telangle step by stop. sahapell Image:
Rostaee the pentagon by degreen counter- clodlvise around center
Follow chese changes:
Follow these changes: Redure the dine of the triangle, making smaller cahapeli stop After these transdormation mantrooatinn, which of the following shapes best represcuts the final for referenee, black each pancl figures
tranflrmation, shapes regeesenes
following
figures indicate
origin...
Hard
2-step Hard
Imagine transforming bexagon
Imagine transforming a cincle step step. <slupell jmages
echapell Image.
Fullow these changes. Rotatic the hesagon degrees clockuise around itx center.
changes:
lesugoo dinificcnely upward. which the fallowing trandoemations the
-alupcl stcy.
these trandformations which of the following shapes best represents the final result for referenee, the black dots in each panel of the figures indicate the origin.
<shupel
best final result? For reference. the black dots each panel of the figunos indicate the origin.
3-step Hard (w/ VisSim)
Easy (wl Analogy)
Obserse transfrmation Shape A thnough squestion Image-
Apply same trandormation sequence Shape stop3
Imaaiin transforming square step by step- eshapell Imagee Tollow these changes:: Shifk the square to the right a signifiunt anount and slightly upward. schspell. anep oo- Reducee the dire square, making emaaller. colapell stop. othapell
blank each of the figures indicate the origin. Select eheeet ansver choice that matches the expected transfomaation resul, choicess
sqaare degrecs cleckvise reference, black dats
shapes
best the figures indicate
Figure 13: Design space of 2D Transformations (1).
30

---

Published as a conference paper at ICLR 2026
Medium (w/ Vis Analogy)
transfommaionn through Imago the transformation
Shape B and detemmine the final shape rimagee for B. For reference. the black does in each panel of figuree indicate the origin. Select the currect anevrr choice that matches the expected transformation result. sanswer choiceso
?
Hard
Hard
?
Hard
Figure 14: Design space of 2D Transformations (2).
F.2 3D TRANSFORMATIONS
Shape generation. 3D objects are loaded from external blend files and instantiated with random properties defined JSON file. Their attributes include:
Types: Various 3D models such as cube, sphere, cone, cylinder, torus, pyramid, etc.
Colors Materials: Colors are sampled from a predefined set, materials are selected from external files.
Objects assigned size scalar (from the JSON-specified values) and an initial 3D location (typically near the origin), with adjustments to ensure they remain above the ground plane.
Size & Location:
31

---

Published as a conference paper at ICLR 2026
Transformations. A sequence of randomly sampled operations is applied to the objects in 3D space:
Translate:
Axis selection: Randomly choose one or more axes from x, y, and 2 (e.g., "x", "xy", "xz", "yz").
Displacement: Translations are applied with discrete displacements: along x and by +2 units and along by +1 unit, with constraints to keep the object above the ground (2 2 0).
Rotate:
Axis: A single rotation axis is chosen randomly from or
Angle: The rotation angle is drawn from discrete (typically +300, 1600, or +900), with the range sometimes adjusted for specific shapes (e.g., cubes or pýra- mids).
Rotation applied about the object's center.
Shear:
Plane:
The shear operation is applied along of three directional pairs: Zy Zg. or
Yz-
Two shear factors are sampled uniformly from the interval [0.2, 1.0), with an enforced minimum difference (approximately 0.4) ensure a perceptible skew.
Factors:
Scale:
Factor: A uniform scaling factor is chosen from 0.5, 2.0, either reducing or enlarging the object while keeping its final size within acceptable bounds.
Flip:
Direction: The object is reflected along principal axis axis-flipped tion across the x-axis) or vertically (reflection across the y-axis).
flipped horizontally (reflec-
All transformation operations are applied sequentially, updating the object's 3D coordinates (includ- ing its bounding box and center) to reflect the cumulative effects.
Number Transformation Steps. Instances are generated with transformation sequences com- prising 1, 2, or 3 steps, where each step randomly selects one of the available operations. This multi-step approach enables a diverse design space of 3D transformations, as the operations can compound in various orders and combinations.
F.3
CUBE NET FOLDING
Net Representation. Cube nets are represented as collections faces, where each face is defined by its vertices in 3D space. Additional attributes include:
Face Geometry: Each face is a polygon (typically quadrilateral) with vertex coordinates stored as NumPy arrays.
Connectivity: A mapping of face connections identifies which faces share common edges, serving as potential hinges.
Visual Attributes: Faces are rendered with colors (sampled from a colormap) and labeled with their keys for easy identification.
Folding Operations. The folding process simulates converting a 2D cube net into 3D cube via sequence rotation operations:
Shared Edge Detection: The algorithm locates the common edge between candidate face and an already folded face. A tolerance used to robustly identify two shared vertices.
Rotation Calculation: Using the shared edge as hinge, a rotation is computed with fixed magnitude of 90º (i.e. # radians). The sign of the angle is chosen by comparing the candidate face's s center (projected onto the hinge's perpendicular plane) with the desired direction toward the cube's center, which is derived from the base face.
32

---

Published as a conference paper at ICLR 2026
Recursive Propagation: The rotation is applied not only to the candidate face but also recursively to all connected faces that have not been folded yet, ensuring that the entire net adjusts consistently.
Folding Sequence and Visualization. The design space supports iterative, step-by-step folding, with each step comprising:
Candidate Selection:
Among the faces not yet folded, the algorithm picks one that is connected to an already folded face.
Folding Parameters: It computes the rotation axis (the shared edge) and the appropriate 909 rotation (with correct sign) to fold the face into its 3D position.
Instruction Generation: Each fold is described in natural language (e.g.. "Fold face upwards towards face 3") based on changes in the face's center relative to the cube's base.
3D Rendering: After each step, the current state of the net is visualized using a 3D plot (with Poly3ICCollection) and saved image.
Perturbation and Validity. To enrich the design space and introduce challenge:
Perturbations:
Selected folding steps can be intentionally altered by inverting the rotation angle or modifying the rotation axis. This simulates errors or variations, yielding nets that might fold incorrectly.
Validity Checks: Functions provided to verify that folded faces do not overlap, that shared edges are consistently maintained, and that face connections remain intact. These checks ensure that the final folded cube is geometrically valid.
Dataset Generation and Perception Tasks. Beyond simulating the folding process, the design space incorporates mechanisms to create annotated datasets:
Instructional Sequences: Detailed, step-by-step folding instructions (with corresponding images) are generated, supporting tasks that require understanding the folding procedure.
Additional tasks query the observer's 's perception such verifying if a particular face has been folded or determining the connectivity between faces-using intermediate folding images.
Perception Variants:
Randomness and Parameter Control. Stochastic elements pervade the folding simulation:
Random seeds govern the selection of candidate faces, the decision to perturb folding step, and the choice of rotation adjustments.
This randomness ensures that a diverse range of cube nets and folding sequences are pro- duced, which is crucial for generating robust datasets and for studying perception and rea- soning 3D folding tasks.
F.4
TANGRAM PUZZLE
Segmentation. The puzzle begins with iterative segmentation algorithm that splits full rectan- gular board into smaller pieces. The process is governed by a minimum piece size and a maximum number of pieces. At each segmentation step, the algorithm:
Selects splittable rectangle based on its area.
Chooses split direction (horizontal if the height is greater or vertical otherwise) and split line ensuring both resulting pieces exceed the minimum size.
Records each split an action with details (original rectangle, split line, and direction) that form the basis for later textual instructions.
Piece Generation & Attributes. Each tangram piece is defined by its board coordinates (c.g., (rO, rl, cO, el)) ) and derived properties such as area and dimensions. Additionally:
Colors: Pieces assigned unique, randomly generated colors.
33

---

Published as a conference paper at ICLR 2026
Visualization: Grid lines and labels are overlaid each piece to indicate its boundaries and area, facilitating clear visualization during reassembly.
Scrambling and Transformation. Once segmented, pieces are scrambled to increase puzzle com- plexity. This involves applying series of random transformation operations:
Rotation: Each piece is rotated by discrete angle chosen from 0", 3,30,, ,0%,00 909
Translation: Pieces are repositioned into non-overlapping cells on a larger canvas.
Flip: In some reassembly variants, horizontal or vertical flips are applied to further ran-
domize the piece orientations.
G EXPERIMENTAL DETAILS
G.1 MODELS AND SETTINGS
To expedite response generation, we use the vLLM (Kwon et al. 2023 library, an open-source tool for fast LLM inference and serving. For all other cases, load models directly using the Transformers (Wolf et al. 2020) library. All model sources are official and listed in T9 When evaluating different models, use default hyperparameter values unless otherwise specified, with detailed parameter settings provided in Table 19, For all models, we explicitly prompt with Think step-by-step, and then put your final answer in \"A\boxed () ()).. to encourage chain-of of-thought reasoning and for easier answer parsing.
Model
Parameter Setting
Source
URL
GPT-4o
temperature 0.0
chatgpt-4o-latest
https://platt.om.ppenaa.
Claude 3,5 Sonnet
temperature temperature 0.0
claude-3 3-alle----sone-e
https///ww.attttrppis.coom
Gemini 2.0 Flash
gemini-2.0- 2.0--las
hatps://al.ggcogl..de
Gemini 2.0 Flash
gemini-2.0-flash-
temperature 0.0
httpe://all.gcogle..ee
Thinking
thinking-exp-1219
OpenAl o1
temperature 0.0
o1-2024-12-17
http:://pplttfomm.geeaa..
OpenAl
03-2025-04-16
httpst//platform.ooees
com
do sample=True, temperature 0.7
Qwen2.5-VL-3B
local checkpoint
https: / /tw.giggagg....... httpss//ugggirffacee
Qwen
30-ttrruct
jousamllc=True, temperature 0.7
Qwen2.5-V1--7B
local checkpoint
https://huggingfaa.
co/ wwee//e//00 wwwee//wwn2...
do Truele=True, temperature = 0.7
Qwen2.5-VL-72B
local checkpoint
https://huggingfaat
Co/ co/ONe//Meen
-- 728 Tnstruct
do sample=True, temperature 0.7
LLaVA-Onevision-22
local checkpoint
https://huggirofacee co/ 1allava-rt7
1lava -lva--oneiaaooo-geeen------ of
SouppllceTrr,,, temperature 0.7
VL.25711
local checkpoint
https://hugginggfaccc
Opend/WabbLLneenv22255555
Table 19: The sources of models used in the experiments and the hyperparameters configuratiooo....oo
G.2 VISUALIZATION OF EVALUATION SETTINGS
Figures 15 16 provide full visualizations of evaluation settings illustrated in Figure In addition,
we show an example how real- wolld spatial reasoning task temporal frame reasoning evalu- ated without visual simulation in Figure 17
34

---

Published as a conference paper at ICLR 2026
Cleck ouran Trngeun pezceeleoo Theleff isavengty Trrgonn pazele, while right avkiillllllee pecesto punci
Without Visual Simulations
Question
Check langnen punzk belee, pand empty panck panck, De
Without Visual Simulations
Question
Sleps
Belur are the stepo complene the Tamgnars pucele
Seep kotats piece about degrems cladoas and place piace wath uppen
let comer y) (a. Stop y Rosane pioce by about yo degres dlodwviee and place pince with spper lell theabovee
Skop Placepioc with
Tanguun
maillblee
Figure 15: Examples of Tangram Puzzle under "without Visual Simulations" Evaluation Setting (top: question- only, bottom: question+assembyy steps).
on Tanpura pand cxpty Tappan pacile right Showsandidle
With Visual Simulations
Question
Intermediate Visual simulations
Belovaae Lagram
Stops
ciechste plaeeppee with lokoomer al (. 2)
syper-kl
uxnnere (x.37- gl.
Soop Rotate picce byobou degrees place Ropper
staed on sebrr depa, cunche tangram puode compleve with the acailall place, yeear
Figure 16: Example of Tangram Puzzle under "with Visual Simulations" Evaluation Setting.
35

---

Published as a conference paper at ICLR 2026
Question
soe 4 axcuustaal of video masing (inadeed Rancaale, mages dircction cugat the
?
Without Visual Simulations
Question
Chocse the ounoa mining frane:
Figure 17: Examples Temporal Frame Reasoning under "without Visual Simulations" Evaluation Setting.
G.3 VISUALIZATIONS OF PERCEPTION PROBING QUESTIONS
In Figure Claude demonstrates perceptual error: while it correctly identifies all face colors, incorrectly perceives face to be positioned beneath face 4, when it is actually located beneath face 5. Such errors prompt an important question regarding task performance: for challenging tasks like cube net folding, to what extent does the low performance stem from perceptual inaccuracies rather than deficiencies in simulation capabilities outcomes? We design probing questions to evaluate model performance 2D and 3D perception on cube nets (Figure I8). which reveals that model fail substantially on 3D perception (Table 2 which may be the main bottleneck in understanding intermediate visualizations in cube net folding (Table T..
inability correctly interpret simulation
2D Porception (Color)
2D Perception (Connectivity)
Question: What color face 5 cube net shown in the image?
In the cube net shown the image, is face a directly connected to face 3. yes or no?
Question:
A. Light orange
Yellow
C. Light purple
D. Light
3D Perception (Folded?)
Question:
the cube folding process shown below, has face been folded towards yet, yes
Figure 18: Exemplary questions cube nets to probe model performance 2D and 3D perception.
G.4
VISUALIZATIONS OF STARE TASK IN DIFFERENT REPRESENTATIONS
Figures 19 22
provide concrete examples of the input modalities evaluated in STARE. For every task family visualize the image-only variant (the original format in STARE), the text-only variant (compact symbolic description that can be consumed without vision), and-where applicable-the combined image+text variant that concatenates the two.
2D and 3D transformations. In the text-only panels, each object is serialized as <shape>, <color>, y2, <si ze>, with attributes separated by commas (e.g.. "square, red, (3, 4) 2"). The image+text panels place the same textual de- scription beneath the image, that language and vision can be attended to jointly.
Cube-net folding. We flatten the cube into 2D grid and enumerate its faces from I to 6. The text-only representation thus becomes short digit string (e.g., "123456") or block array that mirrors the spatial arrangement of the net.
Tangram puzzle. Because rotations in the image cannot be expressed succinetly in the image+text setting, show only image-only and text-only variants. Each piece is labeled
36

---

Published as a conference paper at ICLR 2026
alphabetically and encoded by a binary occupancy grid grid-rows of "1" indicate filled cells, yielding a representation that is both human human-readabe readable and unambiguous for MLLMs.
Together, these examples clarify the correspondence between the natural visual stimuli and the stripped-down symbolic forms used in our text-only experiments, as introduced in Section 3.3
Image-only
mrnnfamnng
chargex:
Image+Text
Shit the rrtngies to the left by dig Hoaee smount and giily upward
tranaform trangle by #GBt 0.92)
Afte these trandormatoos, which of the following chapes
opprseets neul?? refermnos
nol
Follow these changes:
trangle by sign
Afte there tandoomations the osing duopes boxt repormenns firal
Text-only
Fel thas dhange::
The unooee chocks and
A trlangll located 53.35 1.2% 23 30. 30.6 RGBD01 033 599
by dgnittoe dightl spward
these trancfor mocione which lowinng chaped reprosenti
trlongle located 1633, 235L
RGB33.
The anaear cholon and
D stongga locaed (1.00 2.57, 5404 SURFU cccap 0107)
A. mntsggl lopoodd 13.2
wwkk RCAULCS, 832, BRY
mitangle Iocone (4.2
B. trange focesedan 2377 www.s RGRNCR5 (ww.e/9...27..000 C77
Figure 19: Visualizations of 2D transformations (w/ text instructions) in different representations (upper left: image-only, lower left: text-only, right: image+text).
37

---

Published as a conference paper at ICLR 2026
Image-only
tranforiing
shenges.
Image+
Move the object
tondornation,, which lowling
Im magiine tronsform pyramid step step.
best represens thal reulh? for miterincs the slade
panel figure cata orgin.
abe made MaMittal
pyrteiid
Fellow these changes:
okyecttug
Aftr these transformtions owing shapas represents
Text-only
Initally you sne berge pyramid at 9.00, 0.R0) arga,
The anseet chokes are
Follow these changes:
large pommii located 12.00, 0.00, 0.80 large, red made Miternll
Move the sbjecti
large
linge,
llowing shapes represents
a large tons located en 10.00, 0.00, 0,20, size lerge, red mede Meternal L.0
remul!
pranid locaed at 0000, 0.00, 1,80, lange, Manerial
a lange penamid losated at 09,00 0.00, 0.80 largs, madi Materiel.0)
Inder locmed et 0000, 1,09,
tonus
Du large param located at 3000 8.0, 1.80) large, made Materis .X.
Figure 20: Visualizations of 3D transformations (w/ text instructions) in different representations (upper left: image-only, lower left: text-only, right: image+text).
Image-only
Image+Tex Text
stepa
Step Fold face upwonds
Check out net with frcae below:
Shep inwards face
Step Fold fece 6 downwards towards fece
The ebove represented the fullowng text:
Resed the above steps, can net be
cuboe, you
Text-only
stepo
Sup Fold feee spwerds
12
raid face
53
Step
Step Fold face dowmerrds towards fece
Sup 1:
Fold lace spworch
Fold Invardk townrtk
raid fece upwarde
rispa can metbee cubs, orne?
Step 4:
foid face downwards towards fece
Fold Inwardk face-4
above riepa,
forn cubs, yeu
Figure 21: Visualizations of cube net folding in different representations (upper left: image-only, lower left: lext-only, right: image+text).
38

---

Published as a conference paper at ICLR 2026
Image-only
Text-only
Check puzzle
Text Purlle all lowing plecse-rotaring flpping.
Aablllio Places
Sham ss reeded--to snactly a 3xd squars without withoutoeerips paper
Degger
board: 2e3 grid
Arai pleces:
Place
Place
Pacce
For each place, the grid cads occupi are maried with the rect with
Instructins:
The left panall an empoy Sergonn parti, while the right panel showe avallable pieces complesa Keep
You must place eanry call each piece someafare in the grld.
orthp pieces.
flippdd hortoonallyyeeeeeer
Con the Tergiam puetle complesed with the evellable pieces you not
Question: is possible the 3a board exanly with thess pieces? (his hal)
Figure 22: Visualizations tangram different representations (left: image-only, right: text-only).
G.5
CASE STUDY
presents етrоr cases on 2D transformation tasks from ol and GPT-4o. The ol explanation correctly identifies that regular hexagon rotated 30 counter-clockwise will have vertex pointing straight up, so it chooses panel B, but its justification overlooks potential differences in size, color, or shape variants among the answer choices. GPT--o,, meanwhile, mistakenly claims the hexagon looks "exactly the same" after 30 rotation (confusing 60% symmetry with 30%) and vacillates between panels and revealing doesn't 't fully grasp the visual outcome of the specified rotation.
Figure 23
01 Error on 2D Transformation
GPT-4o Error on 2D Transformation
GPF-4::
Question: Imagine transforming besagon by stop. Follow these changes:
problem,
transformazion
hesagon through rotation:
Rotate the bexagon by 30 degrees counter clockwise around its center.
"Initil Hesagon Start with regular hesagon centered at the origin (blackdot).
Degrees
Clockwise"*;
After these transormationn, which of the following shapes represents final resuli?
regular hesagon
sides and notational symmetry throug
augles of ((6o")iie)) (each intensal augle Muao"iiee))..
Kotating the hesagon by ((re)) counter clockwise will resuls
the hesagon looking exactly the same due symmetry. position of neighboring
Issencially, vertex
vertex.
After considering these steps, the shape and size of the hexagon remain unchanged, jost orientation. Since it's regular hexagon, will (gonleirel)
Now referring to the images provided: ""(A"" This looks identical to regular besagon, which could march
Solution Sketch:
the rotated the hexagon due symmetry.
A regular besagon in the "flat 'flat-cop" orientation (as panel A) hasa
""(C)" **(D)* This looks like smaller version of (C).
**[B""" appears
regular hesagon in purple.
pentagon, not
horizental edge olong the top.
Recetting yo (einn))nnt) counterelocwwee around ihs center its vertices directly upward, without changing its size. Among the choices given, panel shows precisely hesagon reariented by leire)), matching both the new orsentation and the sams overall size.
Ascuming the only tronsformation was rotation, neither the size nor shope should change beyond orientation.
Besed these considevation, best option represents
regular hesagon after rotation both (A) and (C). Given that the
original hesagon most likeely the same size and color shown (A) based on typical transformation descriptions. the most direct match without
Therefore, the final shape after the specified rocation the shown panel
color changes mentioned would
Analysis:
Error Analysis: Model does not know what the shape would look like after 30 degree rotation.
Model only focuses the detail vertices directly upwards" while ignore differences in size, color shape.
Figure 23: Error cases of ol and GPT-4o on 2D transformation.
Figure 24 presents error cases on 3D transformation tasks from ol and GPT-4o. The ol response accurately places the torus up and left of the origin and selects panel A, but it overlooks that mate- rial/appearance differences could also rule out that choice. GPT-4o correctly sees that both panels A and D share the required displacement, yet it arbitrarily favors D, giving justification ('clearer displacement") unrelated to the stated transformation criteria, so its final selection is wrong despite partially sound reasoning.
39

---

Published as conference paper at ICLR 2026
01 Error on 3D Transformation
GPTT4o Etror on Transformation
Question: Imsgine transforming a tocus step by step. Follow these changes:
GPT-jo:
To solve problem, need to perforaa the given crnsioomation the torus and determine the final posihon.
object up
the
torus ocigin, as indicated by the
After these cransfomations, which the following shapes best represents the final result?
Transformaion::
Move the torus the left along the y-axis: standard coordinate systemis, moving leftwards
decreasing y-coordinate So, the is negatively along the axis. Move the torus up: involves increasing the --cordiinaee moving the torus upward.
Analysis Options:
positioned the left up,
transformation eequirements.
B: The torus is directly at the origin, which does not match any transformations
en:
C: Chehe torus the which does not match transformzions.
Step by step:
corus positioned
up,
"Left along y-axss" (as shown the panels) black
the
the transformation requirements.
Both and D match the transfoonaation described. However, option shows clearer displacement compared for both axes.
"Lip" raises above that black dot.
Among the choices, only panel shows the torus both leftward above the origin.
Thereforr, comsiderng movementt::
Analysis:
Analysis:
Model only focuses the detail "the corus both leftward and above the
Model correctly recognires and with trandfomation,
origin" while ignore the differences
for wrong
Figure 24: Error of ol and GPT-4o 3D transformation.
On cube net folding task, besides the perception error from Claude in Figure 5 and the text simulation error from GPT-4o in Figure when provided with intermediate visual simulation, models like GPT- 4o still struggles with understanding the intermediate visual cues. In Figure 25 GPT-4o asserts that face becomes the cube's ' top surface, whereas face is actually enclosed between faces and 3. This misrepresentation of face adjacency highlights GPT-4o's 's difficulty in maintaining accurate 3D spactial cognition, which led to the wrong final result. This observation aligns with the conclusion from the perception probing 3D perception than 2D perception.
cube nets in Section 3.3 where GPT-4o fails substantially on
Fail to Understand Visual Simulation
Hore are the qu Gold the not with face as the base: Step Fold face upwards
Question:
Step Fold foce 3 dovmards
Step y Fuld few inwards towards feee
Step Fold face dowoward. towards lace
Intermediate Visual
Step Fold fove dovmards towards fooe
Simudation:
Sased on the above steps, can the not be foided to forn cube, yes no?
GPT--p::
To denermine the nee fildled into cube, let's lewiify ench
Intial Strocture.
Faces toaching
stand
makes them upright connected
Faces
connected face
Step folld face upwands:
Face standl up face
Foces
connected face above 2.ce..
iold face
stands oppusite fare and fice
Fild face inwands touards face
Frror Amalysie:
the cube-lile structure.
Face musos invaad and covering of
Face not
Positon she
covering the top of the cube. nether cunnected eo both
fore isverical.
Face wetical opposdte frre
Face
cover the
fioe and facerg.
The final answer [[Dosse]]
Figure 25: A error case from GPT--4o on cube net folding, where the model fails to understand the intermediate visual simulation.
For tangram puzzles, the left example of Figure (auure26/sha shows how Claude takes a shortcut by conducting area-based feasibility check, correctly noting that the 4 board contains 16 unit squares, but
40

---

Published as a conference paper at ICLR 2026
erroneously counts the L-shaped piece C as 5 rather than its actual 7 squares. This miscaleulation duces the summed piece area to instead of the correct 16, leading to conclude mmnclude-incooreettlyy that the puzzle cannot be completed. The right example of Figure 26 shows how Claude fails to reason about the piece positions when given clear step-by-step instructions. Claude correctly interprets the coordinate instructions and board dimensions, yet its spatial reasoning falters when projecting how the rotated pieces occupy the grid. It underestimates the extents of pieces B and D, inventing over- laps and gaps that never arise, and therefore erroneously concludes the puzzle cannot be completed. This error stems from an inaccurate mental simulation of spatial relationships.
Perception
to Simmulate in
Pieguue Roard
Queetion:
Can the Targram pursle be complesed
The pe Dellneenke
Queetion: Beloe the steps to comomlltte the Tongrams purrlee
with the soulable preces you not
Seep Rooare piace Bi aboutt deggrve clocboctin. place
place with hss upper left corner at (t. -
Seep Retase piace by about degrees clockvise, and place pacce with upper-ln artr--ft
salve this
is a) daginess clockvine and place y) cladoniac.
Step Rotate piace
i Firs, obseree the purcle and pieces:
upper
"The board a ane
We have passuac A, and
in secrangle (coma color)
fixoed the abooe can Tangram pozzle cumplened with the available
Iubtble Faccs
Plece lica recrangle (pink;
Brew
shapod piece (purple)
Claude:
caledlate
let me solce this stop step.
"The board
Piace Arr
The
ii Fix, let' understard the dimersions.
Plooc 2 siquuaee
"The board is 3o3
"Trtal area of piecess 4 squares
Picce C: Crangg squares
needed
- uquare
al Stoce board squares but our pieces cover squares, imposeble complets
Analycis:
Piace
pleces
Frner Analysicc
Afker stop Fiace take up postion lua) -(33)
Piace
squares uncovered puzzle. aeringe pieces
Prece (ax) -xegument
1n aquans
TRoal of pieces squaeeniss
Prace D ((xz)
unite
Evos different and
This monches the boand zrea. which promising
only
311 folooe the stepe pip::
pleonsoweddp leave
(hoeed|No)
Step
positions (aaa)
Step a
up poskion fou)
protumn
Afterstop postition
Afherr
up
Choclang for over and
There oncilae between some piace
for esample, Plece it and Place a sorld rewelep
Alnn, some parts the gnd would mmaan unfilled
si Theretore, tolloiing these specific placement Instructions:
piecos evenlap
squares covcred
completdd
Figure 26: Left: A perception eиror case from Claude on tangram puzzle. Right: An error case from Claude on tangram puzzle, which failed to simulate the intermediate steps even when step-by step instructions given.
presents two error cases from Claude on temporal frame reasoning. the left exam- ple. Claude correctly inferred the camera's left-to-right movement across the given frames, it mis-evaluated the viewpoints depicted the answer choices and consequently selected the wrong completion frame. In the right example, the model erred even earlier, misconstruing the direction of camera motion itself: this foundational misinterpretation then led to an incorrect choice despite seemingly systematic rationale.
Figure 27
41

---

Published as a conference paper at ICLR 2026
Claude Error on Temporal Reasoning
Claude Error on Temporal Reasoning
Question: Which image correctly fills the missing frame?
Question: Which image correctly fills the missing frame?
?
?
Claude:
Claude:
stop by step:
Looking at step by step:
photographed hed from diffeernt angles.
Looking at elmage 1x can see bottle on wooden floor being
Frame u: Shows a table and chair from one angle
fiames Image i:
Frame Camera moves slightly to the right, showing similar view
Anelysing
-Fame Camera continnes moving right, showing more the chair
mooe Frame 2: Camera mooees more the right
Frane Bottle
missing fiae
Frame 3: Camera consinues moving right
sequence, continuing the moving motion.
Frame Missing frame (should show continued righeward motion)
Looking options in
Looking the options efmage zor
Option Shows initial view (samse first frame)
OOption A: Shows a view from the left side
Option Shows middle view the
Option B. Shows a middleicesiered view
Option C: Shows the chair from furthess right angle
Option Shows view from right side
Since the camera
right consistently: missing should the chair from
4) Since the camera motion the original sequence moves from to right, and we need to continue this motion after frame we need the view that shows the bottle from the furthest right angle.
rightward angle
This would complete the smooth motion from left to right Option C shows exactly this view
from rightmoat angle which maintain the consistent rightward camera movement of the sequence.
Option shows
Option C the only that would logically complete sequence, showing the rightmost view
Amalysis:
Frror Analysis:
Model correctly recognizes the camera mation, however fail interpret the view angle the choices.
Model fail to recognize the camera motion the video.
Figure 27: Error case on temporal frame reasoning.
Figure 28 presents error for perspective reasoning from GPT-4o. GPT-4o misprojects the agent's top-down pose into egocentric space: it assumes the arrowed direction is pointing towards a curved seating area and therefore selects option A. A correct geometric mapping should conclude that the agent would instead stand next to the curved seating area. This error underscores the model's diffi- culty in reason about perspective changing in 3DD space.
42

---

Published as a conference paper at ICLR 2026
GPT-4o Error on Perspective Reasoning
Question:
The image from the
down view with red
indicating agent red
indicating the agents direction
Which option represents the agent view??
GPT-ggo:
Let me solve this step by step:
top-down view of what appears be Looking image
incerior space, direction.
viewpoint
a) The arvons appears to be pointing towards w hat looks like curved or semi- circular seating area.
Looking
options Imsae
Image A shows curved seating area with view into another room Image B shows central area with
Image C shows curved seating with columns
straight seating
4) Trom the agent's position and direction Image they would be looking directly curved seating arrangement, with view into snother space beyand.
s) Comparing this expected view with the options, Image A most closely marches what we would expect see from that position and divection, showing curved scating with view through space.
Ervor Analysis: Model Model simply camnot understand the visuel detaila from the tap-down view image.
Figure 28: Error on perspeetive reasoning.
Figure 29 31
presents error cases from Saamini-..-Plast((-Thinnng. and GPT-4o. on temporal frame reasoning.
Figure 34 presents errors for perspective reasoning from Gemini-2.0 1.cimii-2.0PPP (-0O-Faahh-TThhe Thinking), ol and Claude.
43

---

Published as a conference paper at ICLR 2026
GPT-4a Error on Temporal Reasoning
GPT-40 Error on Temporal Reasoning
Question: Which image correctly fills the missing frame?
Question: Which image correctly fills the missing frame?
?
?
A
GPT-qo:
solve which image the movement the camera and orientation. the chair in images from <image to and match it with the options in <lmage zo.
missing sequence,
solve clis problem, need determine which image correctly missing fiame sequence given Imsage images and C to choose from in Image 2.
Orientation Position Anelysis:
Analysis of Imagee
clmage
the laptop desk
visible right.
TV stand.
ln the chird fiame, the chaic further to the right but still not in front the couch.
on the right side is visible. A
Frame: camera moved
more wall
Thind Frame: Missing frame.
the couch, with
Fourth The
have moved further left, with
couch visible,
lamp
sight.
Direction of Movemeent:
ver Choices in Image
Observing transition of chair from the first the fourth right.
Shows the laptup with partion of the right visible.
camera appears
Showss laptop
visible behind simillrr
identifying Conrer Frame:
position further
C: Shows the laptop with the lamp close to which suggests it's
frame.
For the accond frane (the missing one), the chairshooll should be positioned the locations the first third frames.
frame further left than all previous. A
position. between the starting position and the subsequent move.
Option <image
Direction:
movernett seems progressing from right to as more the left side (the lamp) becomes visible in subeqquet fremes.
Therefore, the corret image Fill in the missing frame [B]
Compatison:
Analysis:
Right
-st Trame: Most the right wall visible.
Model fails to identify that Option is really the same as the first frame. And Option the last frame.
and Frame: Wall is more the lelt, but space still visible on the right.
-4th Frame: Lamp visible, implynng maximum left position.
Missing Frame Analysis:
Given that the camera moves from right left:
A: Seems to show more of the right wall (finst frame position).
Fits between the first two frames between mostly right wall
visbiility
lamps
Shows lamp, indicating sting
position than
fiane..
The best fit the missing frame. which should Ingically be between the (moving left),
Error Analysis:
Model fail to recognize the camera motion of the viden, and the relative angle change between the choices the provided.
Figure 29: Error cases from GPT-4o temporal frame reasoning.
44

---

Published as a conference paper at ICLR 2026
Gemini Error on Temporal Reasoning
Gemini Error on Temporal Reasoning
Question: Which image correctly fills the missing frame?
Question: Which image correctly fills the missing frame?
?
?
A
A
Gemini-a Flash-Thinking: Step by step analysis:
Gemini-s o- Flashh
given image climage has four franss, with the third frame missing by a Y. possible options filling the frame are from <lmage 2>.
Amalyze the given frames (image i, frames 2, 4): Observe the
position of the chair in each frame.
Frame 2: Chair positioned slightly the right and viewed from the right-frentt
The frames show a bed with book lying on From <Image iv. the first the book. the cushion moved upwards.
frame shows book the with red cushion
centered viewed moore Frame 4: Chair is positioned slighlly the left and viewed from the
y: Chair
book, between second and
We need to images.
left front.
Deduce
right movement direction: The change chair position left zo
The fourth frame from along the bed the cushion visibble the the book.
shows the book has moved rightwards
Frame
Determine the expected view for the missing frame (Prame i): the camera moving left, Frame which precedes Hame should be taken from position further
suggrets a leftward camera movement.
A: Shows a book on the bed and cushion. The orientation is bit diffcrent to the first image the first image set.
From second imngge, we can see
should
than Fiame This Prome right perspective compared
Shows the baok the bed, with the cushion moved the right. and it appears book
missing third but apprars to be the fourth position. This does not consectly fill the gap.
position similar
Frame 2.
Examine the options (Image A, R, C): Compare each option with the expected view for Famee
-Option Shows the chaic from right front perspective, right.
So missing frome should the showing the book slightly the Therefore, the option fills gap.
similar
2, potentially slightly
right from
This consistent with being a frame taken from position further to the
right than Frame in leftward camera movement.
Option R: Shows vide of the chair This signifieantly different from views
Error Analysis:
milikely to
Model dors well on perception, convecly deseribes the given frames, and the provided answeer choices, but to which missing
sequence
resulting
simple left or right camera
movement.
Option C: Shows the chair from right front perspcctive but
appears slightly more frontal than Option and I'aam If right
moving expect progress
right
likely to fiame takea from posiiion funther to the right than Frame a. Compare Optiion A with trame Option A shows the chair from alightly right angle
view. Option being more frontal than Frame makes
lefoward
movement, the right of Frame 2.
(Option
position to
Conclusion: Option is the most plausible choice for the missing frame logically precedes Frame
frames taken with
leftward moving camern.
Error Analysis:
correctly recugnizes the camera motion, the camera view angle the answer choices.
interpret
Figure 30: Error cases from Gemini models on temporal frame reasoning.
45

---

Published as a conference paper at ICLR 2026
01 Beror Temporal
Temporal Reasoning
Question: Wiich image couneclly fills the missing frunc?
Question: Which iage comeetly file the miedin Samm?
?
?
on
fares and nonee bon, to viewpoint swoaps ansund
From the knows. hames first image set you can see thecomen starting on book ending on opposite
one from left-bar brronndd
of those chaic respectivcly (the teash and right
the
A
left looking frame the
looking
right bad krgound).)
The missing chird frame this must be the "straight behildd" view that lies betwom those two in cominuous laft right sweep.
Companng the candidase vieno A. and viee Bis the "middle"
penpective (roughly head-on to bool) this legiailly fils betwers the more lef--nglet and sight-ngged trumes
1. cafdhecandiebne, only the direct "sehind
(baclrrst ncarct camera). Henoe, missing
misting frrme
Urnur Analystc:
Analysis
isgeod percepcion
sequince fins inage
Figure cases from temporal frame reasoning.
ol Error on Perspoctive Reasoning
Error on Perspective Reasening
Question: The image from the
Question:
The image is from the
view of with square indicating agent
doow view scene with red
indicating agent red indicating ing agent's
indicating the agent's direction of view.
Which option best represents the agent's view?
Which option best represents the agent's view?
A
01:
Froms overhccdd view, arrow clearly placed the bedroom doos, pointing out toward the hallway (the turquoise floored space).
top down note agens square) positioned near the top side of the living space and facing "down" toward a large built-in cabinet or har on the opposite wall. A Arong the four candidlate vieus, D the only one that shows that straight ahead, which aligus the red.
Among the four candidate
shows the through open door toward bloe/tungaoiee floored corridor.
viewpoint
bedroom
large shelving/
direcion leence the agent's view best represented by D.
Therfore, Image C metches the agent's viewing dinection in the overhead plan.
Ervor Analysis:
Amalysis:
understands dirertion but points wrong object.
understand agent location.
Figure 32: Error from ol perspective reasoning.
46

---

Published as a conference paper at ICLR 2026
Error on Perspective Reasoning
Claude Perspoctive Reasoning
conernenceennn aecorneeeee image from top- down view of scene with a red aeuaare indicating agent and a red arrew indicating the agent's direction view
Question:
Question:
image co
top-
square indicating an agent and ced arror indic andiiarrng the agent's direvtion view.
doown view seene
option
represents
Which option best represents the agrat's
agent
Claude:
In the top-dlow view, the red (agrot) newr the hullway
Let me solve this step by step:
indicares agent facing along hallway,
Firse, understand what
Among four candidate photos (A, D), only option shows ballway view with door straight abead, matching A the agent's position and orientation in the top down image.
top-down what appears
space
square (cspersenting an agess's position)
There's red arrow showing
direction the agent is looking
Error Analysis:
2) The arrow in Image is pointing roughly towards what appears to be longer wall section the
Model understand the agent location but ballucinates about the objects.
Looking
images (A. images what appears an altic rooms
Image 2:
faamiture and storage
They all have white/lghtt colore slopel ceilings They taken from different anglen(psitionn
Analyzing perspective of each
Image
space
Imuge shows more cluttered storage
image shows similar view but from opposite direction
Image shonee what appearrs to be the end of the room
Comparing the direction of view in
with these perspectives:
perspective dirvetion Image aliggn
Thhe
Image Image shows straight view down the room, which matches the arrow's direction
Thersfov, the that bes reprosentts the a view is A.
Error Analysis::
Model conrectly describes decalls the top down view. the agent position and the answer checiees, but feil to mason (ardlass not messon) what the first person view the agent lonks like
Figure 33: Error case from ol and Claude on perspective reasoning.
47

---

Published as a conference paper at ICLR 2026
Gemini Ervor on P Perspective Reasoning
Gemini Perspoctive Reasoning
d
aonansenceenn mmag from top- dlown view of scene with a red aeuaare indicating agent and a red arnew indicating the agent's direction view
Question:
Question: image co
top
dowa view seene a square indicating an agent and aed arror indicating the agent's direvtion view.
option represents agent view?
Which option best represents the
Gemini z.0 Flasi: The ageent's is view elimage looking straight staircase poing down..
Gemini 2.0 Flash Thinking:
The user is zoking to identify the image from <Image that repeesents the view of the agent <limage
The agent also facing dlirection perpendicular another room opening on the right.
First, analyo clmagr understara agent's position vicoiing divection
clmagee to top-down view of sceme Thee red square indicates the postiion of the agent
Among the images and in cimage as, imsge depicts stalircase.
The red arrew indicates the direction the agent is facing and viewing
staircase image C appears to be going down perspective of the viewer.
From
observe:
agent
room appears have celling
(visible top right right-handd side <limage TTee agent positiooed near doorway, looking into another room.
The view in image C has wall the side.
Therefore,
C.
Thhe direction of the arrow suggests the is looking from room into adjacent
Error Analysis:
the agent looking appears contain solá and hand
Model fails to understand agent position from the top view.
hand <Image
Now, let's esamine each image elmage as to determine which one mate the agents varu deseribed abore.
Image A: This shows outdeor scene with door and This match location <image t>.
wetting implied agen'''
Image B: This image shows an indoor scene. We can idensily:
-A white ceiling fan the upper portion of the image.
-A sole and TV the main
perspective suggests looking
deorway
conteining sofs
some closely aligen the descristion desived climage
Ex
Image C: This image shows an outdoor scene with umbrella and outdoor furniture. This does match the indocr setting implied by the agent's location elmage
an outtoou sceree with fince plants, This does not match the indooe setting implied by the agents location in clmage to.
Imagee D: This image
Based this analysis, Image is the only depicts viewing direction <lmage io. room with a white ceiling fin. soft, and a TV, which matches the scene implied by the log-doww view.
agent's position
view doorway
Error Analysis:
conrectly describees details the cop-dlwn view, and ansver choices, but fill understand the agent position the agent view.
Figure 34: Error case from Gemini models on perspective reasoning.
H COMPLETE ANALYSIS RESULTS ON OTHER MODELS
Model Performance on 2D/3D Individual Transformation Types. Table 20 presents model ac- curacy across 2D visual analogy and text instruction tasks. Across the nine subtasks, adding visual simulation lifted accuracy for every model except in a few narrow cases, and the size of the gain correlates strongly with baseline capability. Closed-source leaders that were already solid on the raw pixel tasks-ol (~ +3 points overall) and GPT-4o (~ +8 points)-were pushed into the mid-80 s and low-90% s. effectively reaching ceiling on the rext-instruction variants, where gains were biggest (e.g., GPT-4o jumps +25 points "Reflection" and +18 points both "Rotation" and "Transla- tion"). Mid-tier proprietary models such as Gemini 2.0 Flash (~ +5 points) and its "Flash Thinking" mode (~ +5.5 points) benefited even more instructions than analogies, narrowing the gap
48

---

Published as a conference paper at ICLR 2026
GPT-4-class systems. Open-source vision-language models lag full generation behind- -te best of them (Inter 2.5-78B) still sits below 55% on average after simulation but they, too, record healthy boosts of 6-12 points, chiefly the analogy side. The lone regression is GPT-4o's -5 pt dip on "Reflection" analogies, suggesting that simulation may occasionally overwrite a correct latent heuristic. Overall, the pattern indicates that visual simulation chiefly helps models convert verhal transformation instructions into precise spatial operations, while stronger base perception/reasoning models harvest the largest absolute improvements and approach human-like proficiency.
Table 21 presents model accuracy across 3D visual analogy and text instruction tasks. Visual sim- ulation gives 3D spatial reasoning measurable but more uneven boost than 2D: averaged over all eight subtasks, every proprietary model gains between ~+2 points (GPT-4o, ol) and +6 points (Claude-3.5 Sonnet, Gemini-Flash Thinking). while the open-source field improves by +4-7 except InternVI., which slips a point. Gains concentrate in the conceptually harder operations: across models, Shearing (both analogy +6.6 points and instruction +6.6 points) and Rotation-instruccton ( (+6.4 points) see the largest lifts, whereas Translation under visual analogy actually falls slightly (-0.9 points), echoing a smaller 2D reflection dip. Even after simulation, closed-source leaders plateau in the high-60s to mid-70s most 3D subtasks subtasks-roughle roughly 15 points ceilings-indicatingg indicating that depth-aware transformations remain a major bottleneck. Open-source VL models still trail full generation (<45% average), but their sharper relative gains suggest they, too, leverage synthetic roll-outs to bridge language and geometry.
below their 2D ceilings
2D
Transformations w/ Visual Analogy
2D Transformations w/ Text Instruction
Model
Reflection
Rotation
Shearing
Scaling Translation
Reflection
Rotation
Scaling
Translation
Withour Visual Simulation
GPT-4o
$3.7
88.5
72.0
65.8
Claude-3.5 Sonnet
75.0
60.9
55.8
87.4
71.2
63.8
85.9
66.5
Gemini2.0 Flash
85.7
63.8
$1.0
84.4
71.4
65.8
62.3
88.4
70.3
Gemini2.0 Flash Thinking
52.4
48.9
46.9
71.9
55.1
63.2
60.6
83.0
67.8
ol
92.9
70.7
59.2
83.3
84.0
89.5
78.1
92.2
92.2
ULaVA-OOnVission
7.1
25.9
32.7
24.4
25.4
31.7
33.1
$1.0
34.6
Qwen2.5-ViL VLL-2B IntemVLL2.5-788
57.1
38.8
34.7
64.4
42.3
29.3
49.6
62.5
38.8
35.7
41.4
34.7
45.6
36.6
41.5
51.1
75.0
51.9
With Visual Simulation
GPT-4o
76.9
54.8
91.9
80.0
91.2
93.2
91.5
Claude-3. 3.5
50.0
73.9
Gemini2.0 Flash
73.1
70.9
59.5
85.5
74.5
79.4
90.5
78.2
Gemini2.0 Flash Thinking ol
61.5
40.5
71.0
56.4
70.6
68.2
89.2
73.9
80.8
80.6
54.8
87.1
84.2
100
93.5
94.6
97.6
LLaVA-Qne Vision
31.0
30.6
24.8
20.6
48.6
33.9
Qwen2.5-VL-72B V1temmVL.2.5778B
65.4
35.7
71.0
57.0
41.2
40.2
60.8
39.4
69.2
43.7
33.3
59.7
47.3
50.0
53.3
73.0
53.9
Table 20: Model Performance With Without Visual Simulation across 2D Transformation types in Visual
Analogy and Text Instruction Tasks.
49

---

Published as a conference paper at ICLR 2026
3D Transformations Visual Analogy
3D Transformations w/ Text Instruction
Model
Rotation
Shearing Scaling
Translation
Rotation
Shearing
Scaling
Translation
Without Visual Simulation
GPT-4o
60.7
55.7
76.0
80.1
60.1
46.9
71.1
71.2
Claudo-3.5 Sonnct
50.0
46.2
63.3
62.6
45.9
40.4
55.6
53.4
Gemini2.0 Flash
54.2
53.9
63.3
73.0
55.86
44.44
61.90
51.63
Gemini2.0 Flash Thinking
42.4
43.6
61.5
63.8
37.8
32.7
52.5
55.7
58.1
76.7
61.3
46.3
70.5
73.9
LLe Visiionn
18.8
29.1
28.9
25.3
27.0
19.4
41.0
30.7
Qwen2.5-Ve VL-72B I21ernVL2.5-78B
40.2
61.1
33.3
47.6
31.2
30.8
51.1
37.4
37.8
32.4
60.0
40.5
With Visual Simulation
GPT-4o
64.3
64.3
78.2
76.0
62.6
54.7
75.3
68.5
Claude-3.5 Sonnet
51.2
59.5
69.2
55.6
48.0
64.5
Gemini2.0 Flash
46.4
64.3
62.8
68.2
60.9
49.5
64.9
56.1
Gemini2.0 Flash Thinking
50.0
47.6
60.3
66.7
48.5
46.7
59.1
59,3
ol
63.1
63.1
76.9
79.8
69.7
50.7
79.6
74.1
LLaVA-OneVision
27.4
28.6
32.1
29.5
27.3
26.7
45.2
35.2
Qwen2.5-VI-722B
46.4
54.8
69.2
55.0
45.5
34.7
48.4
46.3
1.2.57V
31.0
28.6
43.6
32.6
43.4
37.3
57.0
40.7
Table 21: Model Performance With or Without Visual Simulation across 3D Transformation types in Visual Analogy and Text Instruction Tasks.
o3 Performance across Input Representations. In Table we reported GPT-4o performance un- der different input representations (text-only, image-only, image+text). Table 22 extends this analy- sis to o3. For 2D) transformations and cube nets, image+text combination achieves the best per- formance (98.5% and 69.0%, respectively). For tangram, text-only is strongest (87.7%), consistent with strong language priors about piece counts and simple configurations. For 3D transformations, image-only performs best (73.5%), and image+text does not recover the full image-only advantage (69.6%). These results verify Table 3/s trends with reasoning model: while text can help on more lunguage-friendly friendly tasks, strong performance 3D and other spatial tasks still critically depends visual perception and nerbverbal reasoning.
language
Input
2D Trans. 3D Trans.
Cube Nets
Tangram
Text-only
92.8
71.0
66.1
87.7
Image-only
95.2
73.5
64.3
78.2
Image+Text
98.5
69.6
69.0
Table 22: o3 performance without visual simulation under different input representations, extending Table (GPT-4o).
Task complexity performance. Table 24 presents model performance across different task difficulties for 2D and 3D transformations. Adding visual simulation helps most when tasks get tougher, but the effect differs by setting. For 2D text instructions tasks, observe big boost closed-source models jump about 10-20 points on medium and hard tasks, often hitting 90%++ For 2D visual analogy tasks, observe smaller lift-several points on easy, up IO points on medium/hard. For 3D tasks, only few-point gain, and some models slip on the hardest visual analogy tasks, showing 3D reasoning is still hard. Open-source MLLMs stay well behind; their scores move up and down unpredictably, meaning they haven't 't yet learned to use the simulated views well.
Table 23 presents model performance across different number of transformation steps for 2D and 3D transformations. Models struggle more as the number of transformation steps grows, and visual simulation mainly fixes that. Without simulation, accuracy often peaks one two steps and drops at saree-especially 3D visual-analogy, where GPT-4o falls from 73% (N = 2) to 49% (N = 3). When simulation is added, scores for the multi-step cases (N = 2-3) jump 10-15 points for the top proprietary systems and a few points for spen-source ones, erasing most of the earlier decline 2D tasks and cutting the 3D drop roughly in half. Single-step problems were already easy for the best
50

---

Published as a conference paper at ICLR 2026
models and see little change. Overall, simulation is most useful for longer, instruction-driver chains of transforms, while depth- heavy 3D sequences remain the hardest setting.
Visual Analogy
2D Text Instruction 3D Visual Analogy
3D Text Instruction
Model
N=1
N=2
N=3
N=1
N=2
N=3
N=1
N=
N=3
N=1
N=2
N=3
Without Visual Simulation
GPT-4o
60.46
74.84
73.86
67.27
77.56
73.55
62.75
73.37
48.69
63.07
63.40
60.78
Claude-3.5 3,5 Sonnet
63.73
75.82
65.69
65.17
65.02
60.61
45.10
57.35
57.35
50.98
55.23
45.75
Gemini2.0 Flash
64.71
73.53
68.53
63.96
76.24
70.25
61.76
60.78
63.73
46.08
56.86
56.86
Gemini2.0 Flash Thinking ol
54.58
52.94
55.56
61.71
67.33
71.07
47.71
53.92
57.19
45.59
50.00
20.59
66.7
81.4
82.4
82.0
89.1
89.3
66.67
72.55
77.45
61.76
66.67
62.75
LLaVA-One
30.39
26.47
24.51
49.57
33.70
31.53
25.49
28.43
22.55
30.39
30.39
24.51
Intern VL2.5788 Qwen2.5-VL-72B
43.14
34.31
42.16
61.74
52.17
50.45
40.2
29.41
36.27
34.31
48.04
40.2
50.00
45.10
41.18
55.65
36.96
40.54
48.04
42.16
45.10
38.24
43.14
41.18
With Visual Simulation
GPT-4o
78.43
73.53
88.04
90.57
70.59
72.55
61.76
68.63
Claude-3.5 Sonnet
70.59
70.59
71.74
72.64
56.86
57.84
65.69
50.98
Gemini2.0 Flash
73.5
80.43
77.40
61.76
61.76
53.92
Gemini2.0 Flash Thinking
46.08
58.82
79.35
67.92
55.88
60.78
53.92
53.92
ol
73.4
85.3
94.6
97.2
70.6
75.5
70.6
69.6
30.39
25.49
38.04
34.91
28.43
28.43
36.27
29.41
Intern 78.2.5-788 Qwen2.5-V1.-72B
39.22
51.96
56.52
52.83
25.49
35.29
46.08
39.22
51.96
58.82
43.48
41.51
49.02
58.82
47.06
43.14
Table 23: Model Performance With or Without Visual Simulation across number of transformation steps (N) in 2D/3D Visual Analogy and Text Instruction Tasks.
2D Visual Analogy 2D Text Instruction 3D Visual Analogy
3D Text Instruction
Model
easy
medium hard easy medium hard easy medium hard
easy
medium
hard
Without Visual Simulation
GPT-4o
80.4
61.4
76.2
70.4
71.3
74.2
69.0
63.1
55.2
Claude-3.5 Sonnet
76.5
66.7
62.1
68.7
61.8
59.4
54.9
54.4
50.5
55.6
52.0
44.4
Gemini 2.0 Flash
78.4
63.7
64.7
75.0
67.2
67.3
67.6
59.8
58.8
56.9
52.9
50.0
Gemini 2.0 Flash Think
66.3
52.3
44.4
65.5
69.4
65.4
54.6
53.9
50.3
48.0
44.7
46.1
ol
83.3
77.5
69.6
90.6
81.1
89.1
78.4
70.6
67.7
69.6
64.7
56.9
LLaVA-OneVisinn Vision
22.6
32.4
26.5
39.5
46.3
29.2
25.5
20.6
30.4
31.4
28.4
25.5
Intern'V1. 2.5-78B
45.1
40.2
34.3
63.2
50.9
50.0
32.4
34.3
39.2
48.0
37.3
37.3
Qwen 2.5- VL--2B
57.8
40.2
38.2
50.9
41.7
41.7
55.9
40.2
42.2
38.2
42.2
With Visual Simulation
GPT-4o
80.9
79.4
67.7
91.6
89.4
86.9
80.9
75.0
58.8
75.0
64.7
55.9
Claude-3.5 Sonnet
76.5
72.1
63.2
78.9
65.2
72.1
67.7
52.9
66.2
57.4
51.5
Gemini 2.0 Flash
79.4
72.1
63.2
81.7
86.4
67.2
64.7
58.8
58.8
57.4
55.9
60.3
Gemini 2.0 Flash Think
54.4
55.9
47.1
76.1
74.2
68.9
72.1
54.4
48.5
63.2
48.5
50.0
80.9
82.4
75.0
94.4
98.5
95.1
85.3
69.1
64.7
73.5
75.0
61.8
LLaVA-One Vision
36.8
19.1
27.9
39.4
34.9
34.4
26.5
20.6
45.6
25.0
27.9
Intern 11. 2,5-78B
57.4
44.1
35.3
64.8
48.5
49.2
23.5
27.9
39.7
55.9
27.9
44.1
Qwen 2.5-VL-72B
72.1
50.0
44.1
59.2
30.3
36.1
63.2
50.0
48.5
47.1
44.1
44.1
Table 24: Model Performance With or Without Visual Simulation across different difficulty levels 2D/3D Visual Analogy and Text Instruction Tasks.
2D and 3D Perception Probing with Cube Nets. Table 25 presents model performance 2D and 3D perception probing questions about cube nets, in comparison to the success rate on cube net folding task. The results show that success cube-net folding is driven by model's 3D perception, not its 2D eyesight. All closed-source systems (and several open- ones) already read colors and 2D face connectivity near ceiling, yet their cube-net scores diverge sharply.
51

---

Published as a conference paper at ICLR 2026
When compare cube accuracy XVSim column) with each perceptual measure, the strongest linear relationship is with the 3D "Folded?" test (Pearson r ze 0.89), while 2D connectivity (r = 0.68) and color (r 0.72) weaker. Gemini Flash illustrates the pattern: it pairs the top 3D perception score (69%) with the best cube-net performance (65%), whereas GPT-4o and Intern match its 21) vision but lag 10-20 points both 3D perception and cuhe folding. In short, being able to judge how faces come together in
than recognizing colors or flat adjacencies-largee determines how well model can reason about folded cubes.
Model
2D Perception
3D Perception
Cube Net Performance
Color
Connectivity
Folded?
XVSim
/Vsim
Random
25.0
50.0
50.0
50.5
50.5
Closed-source Models
GPT-4o
100.0
94.1
57.4
52.5
49.1
Gemini-2.0- 2.0--lshh
100.0
84.9
68.8
65.0
65.5
Gemini-22 1aalmi--2.0-1 2.0-Flash- Flash-Thinking
99.0
49.4
54.3
39.8
62.8
Open-source Models
LLaVA-One' Visiin
88.0
10.0
22.0
28.5
34.2
Intern VL 2,5-78B
92.0
86.0
40.2
43.5
Qwen 2.5-VL-72B
96.0
81.7
42.1
35.2
53.4
Table 25: 2D and 3D perception performance in cube net folding.
Extended Perception Probing. Beyond cube nets, we extend the perception probing to 2D/3D transformation tasks and tangram puzzles using GPT-4o and o3. For each task, we isolate individual perceptual attributes (c.g., color, shape, position, size for 2D/3D; counting, overlap, rotation for tangram) and test whether the model can correctly identify them. As shown in Table both models perform well above chance most basic perceptual subtasks (tangram count/position, 2D/3D color and shape), often near ceiling. However, some aspects remain challenging: 2D size identification (51.1% for GPT-4o, 55.6% for o3), 3D material recognition (55.6% and 56.3%), and fine-grained tangram rotation (53.0% and 67.5%). When contrasted with much lower accuracies on the full reasoning tasks, these perception scores indicate that many failures not due to basic perception alone. For most tasks, models can identify the right pieces, colors, and positions; they struggle to perform and track multi-step spatial transformations and composition.
52

---

Published as a conference paper at ICLR 2026
Task
Subtask
Random
GPT-4o
03
Tangram Puzzles
Count
25
90.0
100.0
Overlap (yes/no)
50
75.0
92.0
Position (which piece)
25
71.0
96.0
Rotation
25
53.0
67.5
Size+Count
25
74.1
88.9
Intermediate placement
25
71.0
96.0
2D Transformations
Color (same/different)
50
89.5
94.7
Shape (same/different)
50
94.8
93.5
Position (closest to origin)
25
71.3
83.0
Size (largest/smallest)
25
51.1
55.6
3D Transformations
Color (same/different)
50
96.2
96.2
Shape (same/different)
50
97.5
95.9
Position (closest to origin)
25
75.0
75.0
Material (same/different)
50
55.6
56.3
Table 26: Extended perception probing results (%) for GPT-4o and o3 across tangram, 2D, and 3D tasks. Each subtask isolates a specific perceptual attribute. Random baseline shown for reference.
Question-only vs. Question+Steps As shown in Table 27 adding explicit reasoning steps ("Q + Steps") has opposite effects cube-net tasks for the two model groups: open-source models gain, while closed-source ones do not. The three open-source VL models jump mean + 20 points on cube nets (driven by LLaVA's + 40 pts), whereas the five proprietary models average small decline (-1 pt, with mixed signs). On tangram puzzles, however, the pattern converges: every model-open or closed-drom sharply once reasoning steps are included, with average losses of about -24 pts for closed-soures and -19 pts for open-source models. Again, the trivial solution on tangram puzzles would be comparing the total areas of all available pieces and the grid area, which can easily lead to 75% performance. This result suggest that the models cannot leverage explicit text reasoning steps.
Model
Cube Nets
Tangram Puzzles
Q-only
Q+eees
A
Q-only
Qtetsss
A
Closed-source Models
GPT-4o Claude-3.5 Sonnet
50.2
50.4
+0.2
62.4
34.7
-27.7
51.5
46,4
-5.1
71.1
41.9
-29.2
Gemini-2.0 Flash
47.4
51.5
+4.1
72.8
59.8
-13.0
Gemini-2.0 Flash Thinking ol
47.2
49.6
+2.4
42.9
35.3
-7.6
56.0
47.0
-7.0
73.5
29.6
-43.9
Open-source Models
ILLVVA Visiin
0.0
40.5
+40.5
30.3
14.6
-15.7
InterVVI. 2,5-788 Qwen 2.5-V1-72BB
33.2
41.4
+8.2
69.5
51.7
-17.8
29.0
41.6
+12.6
72.3
47.7
-24.6
Table 27: Model performance on question-oni only prompts versus prompts that include explicit reasoning steps (Q+Steps). A values are Q+Steps performance Q-only performance.
Intermediate Visual Simulation States vs. Performance
Table 28 summarizes extended results varying the slice of intermediate visual simulation pre- sented to the model across different tasks. Across models, which slice of the simulation you show matters, and the "best slice" shifts with task type. For 2D transformations, most closed-source mod-
53

---

Published as a conference paper at ICLR 2026
els and the stronger open-source one (InterVVL) peak when they only the last intermediate state, gaining 2-6 points over the full roll-out; showing every intermediate frame ("all") often drags ac- curacy down few points. For 3D transformations, the pattern flips accuracy is usually highest with "all" states (RS +2-4 points over "partial"), while the last-only view tends to erase that gain, especially for GPT-4o, Gemini Flash, and ol. For cube nets, single view helps every model. Scores barely change with "all" frames, and last-only often hurts closed-source models (-8 points on average) yet uniquely rescues LLaVA (+11 points). For Tangram puzzles, seeing "all" steps is con- sistently best: every model but LLaVA jumps 7-24 points versus the partial view, whereas last-only falls back to-or below the partial baseline. Overall, for more complex tasks, models struggle to leverage intermediate visual states effectively.
Model
2D Transformation
3D Transformation
Cube Nets
Tangram Puzzles
Partial
All
Last
Partial
All
Last
Partial
All
Last
Partial
All
Last
Closed source Models
GPT-4o
86,8
82.8
89.4
72.1
68.4
68.4
51.3
52.2
35.2
43,5
51.5
43.4
Maude-3.5-Somme
67.8
71.4
70.7
54.9
57.8
55.9
58.7
51.6
46.8
43.5
67.6
43.3
Gemini-2.0-Flash
75.4
75.2
79.3
61.0
59.3
57.8
40.5
35.6
41.5
63.8
65.5
58.2
89.3
87.7
93.4
70.1
71.6
65.2
54.4
53.4
45.4
34.8
53.2
46.0
Open-sourve Models
LLa Vision
28.3
32.2
31.8
25.5
30.6
29.4
40.2
34.2
45.6
44.9
40.2
39.8
Intern VI. 2.5-78B
48.3
54.5
56.6
32.3
36.5
40.2
34.7
37.3
37.8
54.3
48.2
41.8
Qwen 2.5-VL-223
44.4
48.5
44.4
48.7
49.1
43.6
41.9
53.4
42.3
49.0
56.7
44.3
Table 28: Model performance with partial, all, and last intermediate visual simulations.
54


Reviews:
[{"review_id": "fT6YgSQgof", "reviewer": "Reviewer_uKSW", "summary": "This paper introduces the STARE benchmark, which evaluates multimodal large language models on vision-reasoning tasks involving spatial transformations and reasoning. It covers various contexts, reasoning types, and difficulty levels, showing that models excel at simple 2D transformations but struggle on more challenging tasks such as 3D transformations, real-world perspective taking, and temporal reasoning.", "questions": "I don't have any questions for the authors.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-motivated and well-written, with a clear structure and sufficient detail about the execution of both dataset construction and experiments, which makes it easy to replicate and build upon. The benchmark covers the spatial reasoning tasks the authors target, with multiple difficulty levels and checks for different types of reasoning in models. The evaluations are robust, including settings with and without visual simulations and step-by-step reasoning. The findings are presented clearly and are straightforward, making the takeaways easy to understand. Overall, the paper offers a valuable contribution in both the idea and its execution.", "weaknesses": "I didn't notice major issues with this paper, but the few minor points I saw are as follows:\n\nThere is a problem with Figure 1: the question image does not match the step images.\n\nThe text mentions \"Fig. 10\" on line 196, but I assume the correct reference is the figure references 9 and 10. And Figure 10 has missing pieces.\n\nLine 323: \"Notably, o3 seems to better at leveraging visual simulations\" has a grammatical error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUkELyhbgd", "reviewer": "Reviewer_Egcv", "summary": "The paper introduces a benchmark known as STARE (Spatial Transformations and Reasoning Evaluation) to rigorously evaluate MLLMs on 2D and 3D geometric transformation tasks ideally solved through multi-step visual simulations. The tasks span three major varieties : (i) Foundational Geometric Transformations, (ii) Integrated spatial reasoning and (iii) Real-world spatial reasoning. Through ~4000 instances of these tasks, they demonstrate that models perform relatively well on reasoning over 2D transformations, but demonstrate random chance responses with 3D cube net folding and tangram puzzles. The authors show that when models receive intermediate visual steps, their performance improves for GPT-4o, Gemini-2.0 Flash Thinking and o1 while Gemini-2.0 Flash and Claude worsen on cube net folding. The paper also includes a detailed analysis of (i) how models understand individual transformation in 2D and 3D, showing a 3% - 8% increase, (ii) how model performance erodes with task complexity, showing a ~20% reduction in 2D tasks and ~12% reduction in 3D tasks, (iii) how model failures originate, demonstrating failure reduction on complexity removal, (iv) how do models reason spatially through textual descriptions, showing that text mainly helps in 2D tasks and (v) how models utilize visual simulations and how well do model use disconnected final visual simulations.", "questions": "I have one suggestion for authors:\n\n1) The paper can include some more relevant work on geometric reasoning benchmarks. \"Forgotten Polygons: Multimodal Large Language Models are Shape-Blind\", \"GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs\", and \"VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models\" might be some works relevant to the current paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper presents the problem of composite reasoning through geometric transformation tasks clearly and concisely. I find these particular strengths of the paper interesting:\n\n1) Extensive task definition - The variety of tasks covering: (i) integrated spatial reasoning over 2D and 3D, (ii) Foundational geometric transformations, and (iii) real-world spatial reasoning encompasses a majority of reasoning behaviour abilities for MLLMs.\n2) Behavioural consistency with prior works - The paper demonstrates consistency of random memory associative behaviour with prior works in this domain, also extending to comprehensive real-world reasoning targeted towards helpful tasks such as cube folding and tangram shapes.\n3) Comprehensive error analysis and behaviour evaluation - The research questions posed and analysed cover the major behaviours responsible for reasoning. They cover the performance reduction for an increase in task complexity, failure modes for models in tasks and the effectiveness of textual descriptions in spatial reasoning.", "weaknesses": "I have only one major weakness/ criticism to identify for the authors, although this has been highlighted in the limitations:\n\n1) Comprehensive answering for models can be incorporated into the work, i.e. moving beyond just multiple-choice and binary question answering, towards one-word or two-word answering to also analyse the probabilistic tendency to actually give the true answer as a response and intermediate reasoning, and not just choose an option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yHcrkUAOcy", "reviewer": "Reviewer_Pqgq", "summary": "This paper introduces STARE, a new benchmark designed to evaluate the spatial reasoning capabilities of Multimodal Large Language Models (MLLMs). The authors argue that existing benchmarks neglect a crucial aspect of human intelligence: the ability to perform multi-step, non-verbal visual simulations. STARE is structured with a hierarchy of tasks, ranging from foundational 2D/3D geometric transformations to more integrated tasks like cube-net folding and tangram puzzles, and finally to real-world scenarios like perspective and temporal reasoning. Through extensive experiments on a suite of state-of-the-art closed and open-source models, the paper demonstrates a significant performance gap between current MLLMs and humans.", "questions": "Could the benchmark be strengthened by including real-world tasks that are more procedural and thus align better with the paper's core thesis on multi-step simulation? If sticking with the current tasks, is it feasible to frame them in a way that allows for a more consistent and meaningful evaluation under the with/without intermediate simulation settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Novel and Well-Motivated Benchmark: The paper introduces STARE, a new benchmark that addresses a critical area of multimodal AI: multi-step spatial reasoning. It provides a structured framework for diagnosing the spatial cognition capabilities of models.\n\n2. Thorough Experimental Evaluation: The study is grounded in a comprehensive evaluation of a wide range of contemporary models, including a crucial human performance baseline. The analysis offers some valuable insights into model failure modes.", "weaknesses": "1. **Potential Bias in the \"Simulation\" Paradigm**: The paper's core claim is about evaluating \"visual simulations.\" However, its primary method for this—providing intermediate visual steps—tests a model's ability to interpret a given sequence of images, not its ability to generate that sequence internally. A human performing mental rotation isn't shown snapshots; their mind creates the intermediate frames. This is a subtle but critical distinction. The benchmark is, therefore, a stronger test of sequential visual comprehension and context integration than of genuine, unprompted mental simulation. This could favor models that are good at processing image-text interleaved inputs over those with true non-verbal reasoning abilities.\n\n2. **Mismatch Between Core Tasks and \"Real-World\" Scenarios**: A potential weakness lies in the selection of the \"real-world\" tasks. While the paper's core thesis is to evaluate multi-step visual simulation, the chosen temporal and perspective reasoning tasks do not appear to measure this capability in the same way as the other tasks. \n\n3. **Limited Real-World Complexity**: The tasks categorized as \"Real-world Spatial Reasoning\" are highly constrained (selecting a missing frame or a rendered viewpoint). This raises questions about whether the observed model failures and successes would generalize to messier, embodied tasks like actual robot-based assembly or navigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DDJoB9vHPJ", "reviewer": "Reviewer_BDc7", "summary": "This paper introduces STARE, a novel multimodal benchmark evaluating a model's capability on tasks that requires visual simulation. STARE features 4k tasks spanning foundational geometric transformations, integrated spatial reasoning and real-world spatial reasoning. Through experiments, the authors show that state of the art MLLMs perform well in simple 2D transformation tasks, but struggle with complex tasks like 3D cube folding. Human performance is also detailedly evaluated: although humans can achieve near-perfect accuracy, the completion time is considerable (up to 28 seconds). In addition, when providing visual simulation intermediate hints, humans can speed up their completion time while models exhibit inconsistent performance gains.", "questions": "1.  I would like to hear the author's clarification regarding the weakness 1 above.\n2.  Have the authors tried reasoning models on table3’s experiment? It is expected that reasoning models can perform much better when text is provided.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper is trying to draw attention to a relatively new field for multimodal AI: visual simulation. Which is crucial since many studies indicate that this is the core of human intelligence, and there are many important applications requiring such ability, such as arranging furniture. In addition, the lack of such a dataset further emphasizes the importance of the developed dataset.\n2. The tasks introduced in the papers are diverse and well designed. The descriptions about each task are detailed and easy to follow. The inclusion of visual simulation hints is also useful in both model and human evaluation.\n3. The experiment is comprehensive, covering multiple state of the art models, with and without visual simulation hints. Particularly, human response time is an important aspect which is not covered by previous benchmarks, this is important especially in benchmarks that are extremely difficult for current MLLMs.", "weaknesses": "1.  The role of visual perception in this task is important and experiment in this aspect can be done better. Currently, only one task (cube folding) is further decomposed to basic perception questions, however, the difficulty of visual perception varies across different tasks, it is possible for another task that visual perception is the key bottleneck. This is crucial since it can inform model developers on how to improve their models. In addition, since the random chance is 50, 57.4 performance on 3D perception should indicate very poor performance, which is contradictory to the claim in line 422-425.\n2. Since most tasks are purely synthetic, it is questionable whether the performance in these tasks can reflect the model's performance on more real-world tasks (like object assembly, mechanical diagram interpretation as mentioned in the abstract).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)