SpaCE-Eval: A Benchmark for Real-World Multi-Modal Reasoning
Authors: Xuyou Yang
Yucheng Zhao
Wenxuan Zhang
Immanuel Koh
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Multi-modal Large Language Model
Visual Reasoning
Real World Environments
Evaluation
Primary area: datasets and benchmarks

Abstract: Multi-modal Large Language Models (MLLMs) represent a significant advancement in artificial intelligence. Among the growing capabilities exhibited by MLLMs, abilities to understand and reason in real-world environments stand out as particularly vital as a fundamental prerequisite for a wide array of real-world applications. The current methods for evaluating MLLMs often fall short in their ability to comprehensively assess these crucial capabilities. However, being able to reason on complex environment-scale spaces, for example, room spaces, building spaces, and even urban spaces, and to predict the future and plan actions, is essential for humans and various autonomous agents to survive in the real physical world. To address these gaps, we propose a visual-question-answering benchmark, **SpaCE-Eval** (**Spa**tial Reasoning, **C**ommonsense Knowledge and **E**nvironment Interaction) in the real world, designed to evaluate some of MLLM’s most important reasoning abilities in real-world environments. As the name suggests, it challenges the models to reason on complex spatial scenarios, invoke commonsense knowledge of the physical world, and interact with the environment. The dataset consists of all new diagrams purposefully produced by humans, where diagram-question pairs are meticulously refined and selected through a rigorous pipeline. Additionally, with the benchmark, we evaluate a selection of leading MLLMs, both proprietary and open source. The results suggest that a significant enhancement of MLLMs in reasoning in the real physical world is necessary to realise more advanced general artificial intelligence.

Published as a conference paper at ICLR 2026
SPACE-EVAL: A BENCHMARK FOR REAL-WORLD MULTI-MODAL REASONING
Xuyou Yang', Yucheng Zhao?, Wenxuan Zhang', Immanuel Koh
Sinnapore University of Technology and Design 2 Alibaba Group
nuyou__rang@nymail... sutd. edu.sg
yucheng, zhao@alibaba-inne com (wxzhang, immanuel_ @autd.
ABSTRACT
Multi-modal Large Language Models (MLLMs) represent a significant advance- ment in artificial intelligence. Among the growing capabilities exhibited by MLLMs, abilities to understand and reason in real-world environments stand out as particularly vital as a fundamental prerequisite for wide array of real-world applications. The current methods for evaluating MLLMs often fall short in their ability to comprehensively assess these crucial capabilities. However, being able reason on complex environment-seat spaces, for example, room spaces, building spaces, and even urban spaces, and predict the future and plan actions, is essential for humans and various autonomous agents to survive in the real physical world. To address these gaps, we propose disual-question-anse anwwerng benchmark. SpaCE- Eval (Spatial Reasoning, Commonsense Knowledge and Environment Interaction), designed to evaluate MLLM'ss reasoning abilities real-world environments. As the name suggests, it challenges the models to reason complex spatial scenar- ios, invoke commonsense knowledge of the physical world, and interact with the environment. The dataset consists of all new diagrams purposefully produced by humans, where diagram-question pairs are meticulously refined and selected through a rigorous pipeline. Additionally, with the benchmark, evaluate a selection of leading MLLMs, both proprietary and open source. The results suggest that significant enhancement of MI.MMs in reasoning the real physical world is necessary to realise more advanced general artificial intelligence. Code and dataset available htlhtps::// /tttp::://ithubb com/ xuyou SaangSSpaCE-a-
INTRODUCTION
Multi-modal Large Language Models (MLLMs) have advanced rapidly in recent years, showing growing capabilities in tasks that require joint visual and textual comprehension. Existing models, either commercial models such GPT-4o (Hurst et al. 2024), Grok (xAT 2024) and Gemini (Gemini
Team Google et al. 2023). or open-source ones such as Qwen2.5-Vi (Bai et al. 2025 Llama 4 (Meta AIl 2025) and LLa VA-One Vision (Li et al. 2024a), have achieved impressive performance on diverse visual question answering (VQA) tasks (Goyal et al. 2017 Masry et al. 2022 Singh et al.
2019 Mathew et
2021 Lu
2023)
As MLLMs are increasingly deployed in real-world applications, including robotics Driess et al.
2023 Li et al. [2024b] Yue al. (2024b) autonomous navigation (Shah et al.) [2023] Zhou al. 2024). and embodied agents (Szot et al. (024), it becomes critical to assess whether these mod- els can reason effectively dynamic physical environments, which require spatial awareness, commonsense understanding, and interaction with the environment.
number of benchmarks have been proposed to evaluate MLLMS' visual reasoning abilities, but existing efforts fall short in several important aspects. First, existing spatial understanding and
"Coreeponding author.

---

Published as a conference paper at ICLR 2026
Spatial Reasoning
Optinn
Opsion
POITT
JAY
TOM
Optin
Option
Which of the following perspectives cocrespond to how Jay will see Tom from his location?
A. Optica Option C. Uption
Option
Commonsense Knowledge
Environment Interaction
Buss
Which chair relatively safer for child in?
Cita cheer/ze
A Octasu A tn meus
you only can reach the enrliest from point
direction, poini you
B. Option B in the unaee
Camnot compare from the diagram
Point
Point
are ogaalyy
Point
Figure I: VQA examples of the three categories SpaCE-Eval. The ground truth is indicated with underline.
reasoning datasets (Wang et al., 2024 Chen et al. 2024 Liu et al., 2023 Cheng et al. 2024)
often focus on object scale understanding (c.g., household objects, table games), ignoring that the environment we live in has much more diverse scales, ranging from room, a building, community and even city. More importantly, the tasks are typically simple, such as object counting, position understanding, and relative relationships (e.g., left or right, close far etc.), whereas spatial relationships are more complicated in the real world, and require different types of reasoning abilities. Second, while some datasets probe physical or social commonsense, for example, PIQA (Bisk et al.) 2020), VisualCOMET (Park et al. 2020) and Cultural (Nayak et al.) 2024), they typically lack spatial grounding. making it difficult to evaluate how well MLLMs integrate such knowledge with physical context. Third, most existing datasets assess static understanding of given image, instead of reasoning about options or actions to interact with the environment. However, predicting and planning for what will happen next is essential for humans and autonomous agents to interact with complex environments to survive.
To address these limitations, introduce SpaCE-Eval (Spatial Reasoning, Commonsense Know]- edge and Environment Interaction), a new benchmark designed to evaluate MLLM's capability to reason in real- wolld environments. As shown Figure SpaCE-Eval consists three categories: (1) Spatial Reasoning assesses models' spatial reasoning abilities in environments scales. requires the MILMS to comprehensively reason on spaces which have complex spatial configurations and relationships in real-world scenarios.; (2) Commonsense Knowledge tests the MLLM's background knowledge necessary to conduct reasoning in the real-world spaces at the commonsense level. (2) Environment Interaction evaluates MILMs ability to compare options, make decisions and predict affordances in real-world conditions, as user or a decision maker, in order to better interact with the environment. The scale of spaces and reasoning subjects utilised in all categories ranges from small items, rooms, buildings, to urban contexts. Together, these three categories target core competencies required to deploy MILMS in real physical world.
multiple
2

---

Published as a conference paper at ICLR 2026
To construct SpaCE-Eval, employ rigorous pipeline. First, human experts with design back- grounds draw brand-new diagrams for every task category. Different from statistical charts, these new diagrams info-graphies, This approach offers two key benefits: (i) all diagrams are freshly created, eliminating data-contamination risks and preventing models from relying on prior exposure publicly available diagrams; and (ii) by encouraging contributors to follow professional standards while retaining their personal drawing styles, we achieve greater visual diversity. Next, two carefully crafted questions are written for every diagram, each with text or visual answer choices. Each item then undergoes a three-stage quality process: peer review for clarity and correctness, adversarial rewriting to remove linguistic shortcuts, and multiple rounds of verification by expert meta-annotators. The final release contains 1,139 high-quality image-question pairs across the benchmark's ' three main categories.
Based on the newly introduced SpaCH-Evvl benchmark, we evaluate a wide range of proprietary and open-source MI.LMs, covering different model families and sizes. SpaCE-Eval appears to be very challenging across all sorts of models, especially with the Spatial Reasoning category. For instance, the best overall result is only 56.37% across all tested models, while the highest accuracy in Spatial Reasoning is merely 42.25% both achieved by GPT-5 (OpenAI, (025). We also observe that models perform significantly worse as the spatial scale increases, and visual understanding consistently lags behind textual comprehension. Moreover, analysis reveals that many models rely heavily on surface-level patterns rather than engaging with deeper conceptual or spatial structures. These findings underscore the need for improved multi-modal reasoning capabilities and highlight SpaCE-Eval as robust benchmark for measuring progress in real-world MLLM reasoning.
RELATED WORK
MLLMS The remarkable progress large language models (LLMs) has driven widespread adop- tion of the Transformer architecture (Vaswani et al. 2017) in the computer vision domain, leading to the development of models like VIT (Dosovitskiy et al. 2020), CL.PP (Radford et al.] [202T). and MAE (He al.) 2022). Leveraging the foundational capabilities of LLMs, MLLMs such as GPT-4o (Hurst et al. 2024). Qwen2.5-VL (Bai et al. 2025). LLaVa (Liu et al. 2023) integrate infor- mation across multiple modalities and have shown strong generalisation across wide array of tasks. In particular, these models have demonstrated increasingly sophisticated reasoning capabilities in real-world environments, where understanding spatial relationships and physical constraints critical. In addition to excelling at tasks like different sorts of VQA (Yue et al. [024aa] [Singh et al., 2019 Mathew et al. [ Masry et al., 2022) and mathematical reasoning (Lu et al.) 2023), these models show growing competence in spatial (Cheng et al., 2024 Chen al. 2024h and commonsense (Park) cct al. 2020 Nayak et al. (2024) reasoning, marking a significant step toward grounded intelligence in complex physical contexts.
Benchmarks Our work mostly related to benchmarks and datasets that cover one or more aspects of the three categories in SpaCE-Eval. We compare some representative benchmarks related to our work in Table I Firstly, existing benchmarks on spatial reasoning (Wang al. 2024 Chen et al.)
2024
Liu et al. 2023
Cheng et al. 2024) focus on relative spatial relationships between objects, such
Table 1: Comparison of representative benchmarks in three categories.
Benchmark
Spatial Reasoning
Commonsense Knowledge
Ensironment Interaction
Spatial VQA Chhn et al.
2024
Simple relative location, dis- lance, height, etc,
Eyal (Wag el al
2024
Spatial map, maze, grid, etc.
GRASP Jassim et al.
Object grounding and intuitive physics
Cultural VQA Nayak et
Geo diverse cultural under. standing
VisnaCCMETT Park et
2020
Visual commonsense
Events before and alter, human intent
CLEVRER (Y al 2019
Object trajectorics, rolative po-
Physical commonsense (e.g..
sitions
collisions, feusibility)
Ours
Complex spatial relations
Intuitive science, engineer-
Interaction with environ- ment
multiple spatial scales
ing, physics, and culture
3

---

Published as a conference paper at ICLR 2026
relative position, adjacency and orientation. Benchmarks like SptialEval (Wang et al. 2024 test models' ability to perform table games, for example, navigate in mazes. Despite the challenging spatial reasoning process, these not the real physical world. Secondly, benchmarks (Jassim et al. 2023; Meng et al. 2024) that assess the commonsense knowledge of the models mainly concentrate understanding physical features and laws, and do not contain cultural context (Nayak al. 2024
that is also an important aspect of the real world. Lastly, datasets (Shridhar et al., 2020) Padmakumar al. 2022 Fan al., 2022) on interaction in various environments are mostly used to evaluate or enhance embodied agents given instructions. In addition to these
benchmarks, some
benchmarks (Park al. 2020 Yi al. 2019) cover two areas of the three.
THE SPACE-EVAL BENCHMARK
introduce the SpaCE-Eval, a novel and meticulously curated benchmark to assess the reasoning abilities of MLLMs in real physical world environments. The benchmark challenges the abilities of MLLMs to perform complex reasoning on multiple scales of spaces. It provides questions which require the integration of abstract visual perception, spatial simulation, commonsense knowledge and deliberate reasoning abilities to answer. These abilities are essential for humans and autonomous agents to navigate and survive in real physical world.
(4)
Durign Tune
Env.
Spentall
Intere
17.A7-
Mobibly in Space
Spatial Stoudiure
Urban Space 8.545
6
El 30.20%
6 6,28%
SR 35.12%
Form Trans. 9,31%
Abstract Geomenry
Usar-Ent Interaction 10.01%
SpaCE-Eval
Asahyii pars
Hui dunnggg Space 49 50%
Constr 1.4%
Specr Avand
CK 34 34.68%
& 3.2%%
Verasohar ahar Living 966%s
Others 1.57%
Caltunl
Contert
Marerials
Room
11.84%
12 555
Figure 2: Overview of the dataset. (a) The dataset comprises three main categories: Spatial Reasoning (SR), Commonsense Knowledge (CK), Environment Interaction (EI), and twelve subcategories, with each category assigned a corresponding data ratio. (b) The benchmark challenges models to reason on wild range of spatial scales.
3.1 DATASET DESIGN
Considering the abilitics and background knowledge required to understand and reason the real world, design three parallel main categories of the dataset: Spatial Reasoning, Commonsense Knowledge and Environment Interaction. To obtain more detailed insights, there are also four subcat- egories within each main category. Tigure@illustratee the composition of the dataset. Appendix A.0. displays examples of each category and subcategory.
Spatial Reasoning This category assesses models' fundamental spatial reasoning abilities real world spaces with complex spatial relationships and of various scales, from rooms, buildings, to urban contexts. It requires the MLLMs to comprehensively interpret spaces through various complicated spatial reasoning process, including (i) Spatial Interpretation: reasoning one correct perspective/view from a given angle or viewpoint, including indoor and outdoor, static and dynamic conditions; (ii) Space Association: associating spaces by linking different views (plan, section, elevation); (iii) View Analysis: analysing visibility of items or spaces in complex spatial setups; and (iv) Form Transformation:
predicting the form under explicitly implicitly given transformation rules.
4

---

Published as a conference paper at ICLR 2026
Commonsense Knowledge This category tests the MLLMs' background knowledge associated with spaces necessary to conduct reasoning in the real world at the commonsense or intuitive level. consists of a wide range of fields categorised into four groups: (i) Material and Science: assessing intuitive understanding of material science and structural stability; (ii) Construction and Fabrication: evaluating logic of construction fabrication of objects (e.g- joints, furniture) and small to large spatial structures; (iii) Vernacular Living: examining the knowledge of unique living style of local societies; and (iv) Cultural Context: testing model's understanding of regional, historic, and religious cultural representation.
Environment Interaction The environment interaction category evaluates MILLMs' ability to compare options, make decisions and predict affordances in real-world environments, as a user or a decision maker of different spaces. Envermnnirnnmnnt Interaction asks questions such as how to choose spaces under different weather conditions from space user's perspective. Design- Environment Interaction asks questions from a space decision maker's view, for example, how to set up space to achieve desired goals. Mobility in Space challenges the models to select or plan the navigation in various environments through different means of transportation or for different subjects (e.g. humans, vehicles, other animals). Finally, Environment Sustainability tests the understanding of models of environmental sustainability strategies and systems in the real world.
3.2 DATASET CONSTRUCTION
The dataset collection consists of two steps. First, 51 university students representing multiple nationalities, cultural backgrounds and design traditions are asked to produce brand new diagrams for every subcategory. Students with design (mostly architecture-rclate intentionally chosen because they are trained to possess strong spatial abilities, including the ability to create high-quality visual representations of the physical world from scratch. This approach yields two key benefits: (i) all diagrams are freshly created, eliminating Vata-contamination risks and preventing models from relying on prior exposure to publicly available diagrams; and (ii) by encouraging contributors follow professional standards while retaining their personal drawing styles, we achieve greater visual diversity of the diagrams. Second, the contributors are asked to carefully craft two questions for every diagram, each with text or visual answer choices. Specific requirements are given to the contributors during the data generation process: (i) the diagrams must accurately represent information aligned with specific categories; (ii) the questions should be closely related to the diagram and the categories; (iii) reasoning process must be involved to answer the questions to avoid simple pattern match; (iv) linguistic or positional shortcuts should be avoided.
Data curation
backgrounds
Data format The type of question in the benchmark is visual question answering. All the questions are single-answer multiple-chool choice questions with four choices. However, while some questions have four text options, others have purely visual options, where question text contains only the labels of the visual options in the image. To mitigate the potential model bias on the answer's locations and labels, the choices of all questions are randomly shuffled so that the probability of each position (A, B. C and D) being the correct answer is approximately evenly distributed (25.46%, 25.37%, 25.46%, and 23.71% %, respectively).
Data quality control To further control the quality of the dataset, all 742 diagrams and 1484 questions produced then undergo a multi-stage refinement and screening process. Figure illustrates the pipeline of data quality control.
(i) During the data creation phase, contributors met weekly with the meta-annotators to review subset of their Inagram-question pairs. In these meetings, ambiguous cases were discussed in detail, and concrete examples were illustrated. This iterative feedback loop ensured that contributors gradually converged to shared understanding of the rubric, leading to de facto annotator agreement over time rather than idiosyncratic interpretations by different students.
(ii) Volunteers from various backgrounds representing the general population invited to review all image-question pairs and point out clarity issues, logic flaws, and any other errors for the contributors to refine the diagrams and texts accordingly.
(iii) A few dedicated reviewing sessions were conducted with external reviewers who were not involved in the initial data creation. Their independent perspective helped to surface hidden biases or
5

---

Published as a conference paper at ICLR 2026
External Reviewers
surfice lidden blases of systemitic issues
Weekly aclieve:
sbsted understaning of the rubric annotaler agreement
Creators
Meta-Anmotators
Authors
Represcutation Accuracy
Dota fibering:
Calegocy sliemment
To point out elenty logic
Clanty
Must invllve
Avoul shoricuts
Multple of veriiicaation
Accimey Relevance Diversity
General Public
Figure The pipeline of data quality control.
systematic issues that might not be obvious to the original creators, further increasing the soundness and robustness of the dataset.
(iv) Multiple rounds of verification are conducted by meta-annotators. At this stage, 50 questions are adversarially rewritten to eliminate linguistic shortcuts where applicable. For example, options with features such as special length and sentiment that make them appear more likely to be the correct answer are modified. This involves small proportion of questions. Each data entry is examined to ensure that referring to the visual input is necessary to answer the questions. During this process, 1468 questions remain unchanged.
(v) Finally, all the data is filtered again by the authors, where the key selection criteria include clarity, accuracy, relevance, and diversity of the image-question pairs. In this process, 41 diagrams and 345 questions are excluded. As result, 701 diagrams and 133 questions that meet all requirements are kept.
3.3 DATA ANALYSIS
The dataset consists of 1139 image-question pairs in total, where Spatial Reasoning, Commonsense Knowledge and Environment Interaction contribute 400, 395 and 344 questions respectively. The proportion of each subcategories is displayed in Figure
In addition to the categories and subcategories, the dataset can be divided into nine groups according the scale of space each question requires be reasoned on, namely: object (e.g. furniture, columns, decorative tiles), room space (e.g. a bedroom, a hall, unit/apartment), building space (an individual building building cluster, interior exterior), spatial structure (e.g. shell, bridge, an arch), urban space (c.g. street blocks, road networks, a park, waterfront, village), abstract geometry and others. The distribution and example of each scale is illustrated in Figure (b).
To further enhance models' visual reasoning capabilities and reduce reliance on purely linguistic cues, 40% of the questions are provided with purely visual options. To correctly answer these questions. the models need to not only perceive visual information of the question but also interpret the visual descriptions of the options or simulate corresponding views. Figure T Spatial Reasoning part and Figure (b) demonstrate examples of questions with purely visual options. The rest 60% of questions provide text options.
To check the similarity of the diagrams to existing images, Google Cloud (2025) Vision is deployed to search for similar images of each diagram. An average cosine similarity is calculated between the diagrams and their top most similar images found in the search using CLIP (Radford et al.) 2021 embeddings. The comparison reveals that 75% of the diagrams achieve similarity scores under 0.723, with 50% scoring below 0.665, and overall mean of 0.654.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
Based on the SpaCE-Eval benchmark, we conduct extensive evaluations a selection of both proprietary and open-source MLLMs and analyse the results.
4.1 MODELS
We consider a wide range of proprietary and competitive open-source MLLMs to perform com-
prehensive evaluation on SpaCE-Eval. For proprietary models, we consider GPT-5 OOpenAI 2025).
GPT-S-mini (OpenAT] 2025), GPT-4o (Hurst et al., 2024). GPT-4o-mini (Hurst et al. (024), GPT-04
mini (Hurst et al. 2024). grok-2-vision- 1212 (xAl 2024) (grok-2-vision), claude-3.7-sonme (An-
thropic 2024), claude- Mave--rriiiicc-----288--
(Anthropic 2025 and
(Gemini Team
Google et al. 2023) (gemini-2. (ommini-2.5-faasse 2.5-flshh.. For open-source models, we evaluate the Llama-4 (Meta Al
2025) family, including (Llama-4-Scout); the gemma-3 (Gimmaa Team et al. 2025) family, including
(Llama-4-Maveric)) and Llama-4-
gemma-3-27b-it, gemma-3- 12b-it and gemma-3-4b-it; the Qwen2.3-VL. (Bai et al. (025) family,
including Qwen2.5-VL-72BB-B
(Qwen2.5-VL-72) and D0we22.5-VL-7B-Insstte
(Qwen2.5-
VL-7B); and other representative MLLMS including Pixtral-12B (Agrawal al. 2024), glm-4. 1v-9b- 1v-9--
thinking GGLM-V Team et al. 2025) (glm-4.1v-99) Idefics3-8B. Llamma [Laurencon et al. 2024).
i multimodal) and smolvim-26 (Marafioti et al.] 2024). In addition, human volunteers of various backgrounds are invited to manually solve the questions, through which average human performance
Juva-onevision-76 (Li et al.) 2024a). Phi-4-multinoal-- instruct (Abouelenin et al. 2025) (Phi-4-
(Human Avg.) is obtained.
We evaluate most models using the OpenRouter API (OpenRouter) 2025) for efficiency, for models that are not supported by OpenRouter, we deploy the models using VLLM (Kwon al. 2023) and use their default hyperparameter for inference. The detailed prompt structure and model API or links provided in Appendix [A.0.]] [A.022 When the model prediction is not exactly the same expression the ground truth (e.g. model prediction is not answer "A", but may have the same linguistic meaning as option A), use GPT-4o-mini (Hurst et al. 2024) classify whether the prediction is correct.
4.2
MAIN RESULTS
In this section, we present comprehensive comparison of different MLLMS based on SpaCE-Eval, the details are shown in Table For each model, the accuracy is represented by the percentage of correct predictions out of the total predictions in each category. We summarise our key finding as follows.
Overall result SpaCE-Fval a very challenging benchmark. The best result across all the tested models is only 56.37% achieved by GPT-5 (OpenAl, 2025). The proprietary models achieve an overall accuracy between 39.68% and 56.37%, 3,, and the majority of open-source models can only reach overall accuracy of less than 40%, except for Llaeri--Maverick (Meta Al 22225), glm-4.1 Iv- 9b (GLM-V Team et al. 2025 and Ilavaonevission7bb (Li et al. 2024a). whose accuracy is 45.92%, 43.37% and 42.41% respectively.
Cross-category result Trained on large-scale data, the models achieve as highest as 66.08% and 61.63% in the Commonsense Knowledge and Environment Interaction categories. However, their performance in the Spatial Reasoning is dramatically lower, with scores of only 42.25%, with the majority of which only slightly higher than random guesses, highlighting a significant capability gap between Spatial Reasoning and the other two categories.
Cross-model-scale result According to the results, generally, the models with larger sizes perform better than those with smaller sizes. For example, in the Gemma (Gemma Team al. 2025) series, the largest model (gemma-3-27bb (gemma-3- 12b-it) has the medium accuracy (37.05%), and the smallest model (gemma-3-4b-. has the least accuracy (36.00%). Similarly, Dwwen..5-VL-72B-. VL-72B-Instruuc (Bai et al. (025) has higher overall accuracy than D0ven2.5-VL--B-Innse
has the best performance (39.77%), the middle size version
7

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation results of various MLLMS on SpaCE-Eval. We evaluate the three main categories: Spatial Reasoning, Commonsense Knowledge and Environment Interaction. For each main category, we evaluate the four subcategories. SI, SA, VA and FT represent Spatial Interpretation, Space Association, View Analysis and Form Transformation, respectively. MaS, CF, CC and VL represent Materials and Structures, Construction and Fabrication, Cultural Context and Vernacular Living, respectively. UEI, DEI, MiS and ES represent User-Enn Seer-Ennironment Interaction, Design-Environnen Interaction, Mobility in Space and Environment Sustainability, respectively. Bold indicates the best performance of each category, while underlined denote the second-best performance in each category.
Overall Mean
Spatial Reasoning
Commonsense Knowledge
Environment Interaction
Model Name
S1
SA
VA
FT
Mean
MaS
CF
CC
VI.
Mean
UEI
DET
MiS
ES
Mean
Human Avg.
79.00
84.92
89.19
92.24
76.24 88
81.48
65,32
58.2 81.86 71.83
72.64
80.82
91.84
80.17
81.34
Proprictary MLLMs
GPT-5
56.37
39.70
45.95
46.55
43.4
42.25 69.44 61.29 62,60
69.61
66.08
65.79
60.27
50.0
74.58
61.63
GPT.5 miiii
52.15
33.67
45.95
39.66
66
38.68
37.00 70.37
58.06
53.66
62.75
61.27
58.77
54.79
53.06
76.27
59.3
moiiii-mini
39.68
16.08
27.03
32.76
29.25
23.00
62.96
45.16
36.59
47.06
47.85
55.26
45.21
36.73
66.10
49.71
GPT-o4-mini
45.04
25.63
29.73
37.93
25.47
27.75
61.11
56.45
47,51
56.86
54.94
56.14
42,47
47.96
72.88
53.78
grok-2-vision
42.32
30.15
24.32
32,76
30.19
30.00
59.26
45.16
35.77
54.90
48.61
55.26
35.62
41.84
67.80
49.42
claude-sounce-4
48.64
29.65
37.84
41.38
28.30
31.75
64.81
53.23
53.66
63.73
59.24
57.89
49.32
50.00
71.19
56.10
clmude-3.-ssomet
47.41
26.63
16.22
41.38
33.96
29.75
65.74 53.23 46.34
63.73
57.22
54.39
47.95
56.12
72.88
56.69
gemini-2.5-flssh
47.50
25.13
24.32
43.10
28.30
28.50 64.81 62.90 47.97
58.82
57.72
61.40
49.32
51.02
72.88
57.85
Open-source MLLMs
Qwen2.5-VI.722
37.84
19:10
35.14
27.59
33.96
25.75 57.41 38.71 26.83
59.80
45.57
53.51
35.62
31.63
50.85
43.02
gemma-3-27b-it
39.77
23.12
21.62
39.66
21.70
25.00 57.41
58.06
32.52
55.88
49.37
46.49
52.05
34.69
55.93
45.93
Llama-4-Maveric
45.92
22.11
37.84
37,93
29.25
27.75
67.59
58.06
47.97
49.02
55.19
$7.89
49.32
44.90
81.36
56.40
Llama-4-Scout
39.42
18.09
21.62
34.48
29.25
23.75
56.48
48.39
36.59
46.08
46.33
52.63
42,47
41.84
66.
10 49.71
22mma-3-122-it
37.05
19.10
16.22
36.21
22.64
22.25
52.78
45.16
33.33
50.00
44.81
42.11
39.73
35.71
74.58
45.35
Pixtrl-11B-22009
39.77
25.13
27.03
37,93
33.96
29.50
50.93
45.16
34.96
48.04
44.30
49.12
39.73
30.61
76.27
46.51
glm--.1v-9b
43.37
37.84
36.21
29.25
28.50
60.19
41.94
47.97
50.00
50.89
50.88
50.68
42.86
71.19
52.03
Idefics3-SBB SB-LLam33
38.72
21.11
18.92
44.83
34.91
28.00
56.48
41.94
35.77
49.02
45.82
54.39
35.62
29.59
52.54
43.02
Qwen2.5-VL-7B
37.14
19.60
21.62
41.38
31.13
26.00
48.15
45.16
34.96
41.18
41.77
47.37
41.10
34.69
61.02
44.77
Ilava onevisionn 7h
42.41
26.63
35.14
37,93
33.96
31.00
58.33
56.45
37.40
50.98
49.62
56.14
41.10
31.63
64.41
47.38
Phi-4- muliinolll
38.72
27.14
29.73
31.03
35.85 30.25
49.07
30.65
36.59
41.18
40.25
45.61
47.95
37.76
62.71
46.80
gemma-3-4b-it
36.00
22.11
24.32
32.76
24.53
24.50
55.56
35.48
30.89
45.
42.03
43.86
31.51
39.80
57.63
42.44
smolvlm-2b
36.00
28.14
32.43
27.59
33.96
30.00
58.33
41.94
34.96
35.29
42.53
38.60
27.40
33.67
42,37
35.47
Human vs. model In contrast model performances, the average human results in Table demonstrate a different pattern. Humans achieve the highest results in Spatial Reasoning and the second highest in Environmental Interaction. Both categories require stronger reasoning abilities than the Commonsense Knowledge category, especially for Spatial Reasoning. Noticeably, GPT- (OpenAll 2025) has surpassed human average results in the Cultural Context subcategory, under Commonsense Knowledge. Despite such differences, human performance in spatial reasoning (84.18%) and environment interaction (81.34%) is significantly higher than all the models (at best 42.25% and 61.63% respectively). This result suggests that humans are comparatively stronger in reasoning than in memorisation, and maintain a clear advantage over current models in reasoning ability.
The evaluation reveals notable imbalance between textual and visual reasoning. Figure a)) compares model performance questions with textual options and those with purely visual options. Consistently, throughout all categories, the models perform significantly lower in visual-only options compared to options with text. This disparity indicates that MLLMs possess sub- stantially weaker capabilities in visual comprehension and reasoning compared to textual reasoning. Additionally, Figure b)) further compares performance with and without image input. When images
Visual ys. textual
8

---

Published as a conference paper at ICLR 2026
70
$5.11
60.21
50.97
80
29.37 37
30.37
A2:19
42.-3
25.04
32.67
22.50
31.07
21:35
21.0
10
CJmn---Mirerik
(4)
Option
Vitaa Optine
Moaa
45.22
15.57
33.99
20
26
19
Acuiii-2.5.0 5-dahh Iluas4-Meriid
Object
Room Space
Spatal Simitrre
Buildnng Space
Udenn Spaces
Aberact Coumetty
With trange bejea
Whiduast trrgg
10-10
1gi- Mon
gemini-7 tolach Mron
- Lenoat--tsvickk Moon
Figure 4: (a) Model performances on questions with textual options and those with purely visual options. (b) Model performances questions with and without image input. (c) Models' accuracy across different scales. The dashed lines indicate models' mean accuracy over all scales.
are absent, model accuracy declines significantly, underscoring the strong dependency of MLLMs on visual inputs to answer questions in our benchmark.
4.3 ANALYSIS OF CURRENT LIMITATIONS
A particularly prominent limitation revealed by the evaluation is the models' incapacity to bridge spatial scales, especially when reasoning across local-to-globes -oobbblll and global-to-local relations. This may stem from challenges in reasoning over larger spatial extents and in performing spatial simulation. In addition, the models exhibit difficulties in abstract interpretation and reasoning. We briefly analyse these failure patterns in this section. More in-depth diagnostic analysis provided in AA.0.4] where we detail how models arrive at incorrect answers and discuss the likely underlying causes these failures.
Difficulty in reasoning on larger spatial scales As shown in Figure consistent trend across different spatial scales: they achieve relatively strong performance in object-scale reasoning, but performance declines the scale expands to rooms, buildings, and urban spaces, with abstract geometry reasoning posing the greatest challenge. In other words, models demonstrate weaker understanding and reasoning capabilities for larger-scale spaces compared to object-level reasoning, and their performance in these cases falls below the overall mean across all scales. This trend can be resulted from the fact that many existing datasets primarily focus on object-level reasoning, noted in Section [ highlighting the need for datasets and benchmarks that target larger spatial scales. By contrast, performance on spatial-structure tasks slightly higher, as these tasks emphasise intuitive understanding of structural patterns rather than the more demanding reasoning processes required in the other scale groups
4 cc. the models exhibit
Difficulty in spatial simulation Models demonstrate critical shortcomings when spatial simulation required the tasks. This limitation often leads to failures in questions involving global-local relationships. The evaluation shows that models frequently struggle to identify the view corresponding to a given location, or to localise the position associated with given view. Typical questions of this type include: (i) given global view, what would be observed when standing position A and looking in the direction of the arrow? or (ii) given a local view, at which position in the room, building, village would this perspective be obtained? Figure 5(a) presents example of such

---

Published as a conference paper at ICLR 2026
location-view association tasks, which are usually posed with pure visual options. Across models, the accuracy on these questions remains low, with fewer than 22.73% of responses being correct. Such failures commonly arise when models are unable to mentally simulate infer the consequences of spatial transformations, changes in perspective, or dynamic interactions within the environment. This highlights fundamental weakness of MLLMs in the spatial reasoning abilities essential for real-world understanding.
(3) Optioo D. Opiion Option D. Option
Q:Whint is the perspctiive of the luman in the image?? Whot does be see?
Q: Which black nearest to Civic Building?
Block C: Block
Block
GPT-40: Hozed on the morge, the black nonrest Civic Building is "Block D"
GPT-4o: The perapestive "Option
image matches
Figure 5: (a) An example of location-view association. (b) An example of abstract representation of
distance. The ground truth is underlined.
Difficulty in abstract interpretation and reasoning When challenged with questions involving spatial measurements, models often resort to comparing pixel-level distances rather than reasoning with the abstract spatial relations encoded in the diagrams. As shown in Figure GPT--4o (Hurst et al. 2024) bases its answer on the shorter pixel distance between the circles representing the Civic Building and Block D, while disregarding the abstract distance indicated by dashed lines, which reflect the actual spatial relationship in the real world. This illustrates a broader limitation: the models tend privilege surface-level visual patterns over the abstract principles that govern spatial and physical phenomena. Such behavior raises concerns about their reliability in tasks where correct interpretation requires abstraction beyond visual similarity, underscoring the gap between current multimodal reasoning and human-like spatial understanding.
5 CONCLUSION
As existing benchmarks do not comprehensively assess the ability of MLLMs to understand and reason in real-world contexts, we introduce SpaCE-Eval, a meticulously constructed benchmark designed to evaluate several crucial aspects of real-world understanding. The benchmark undergoes rigorous preparation and selection process, followed by extensive experiments on a range of state- of-the-art MLLMs. The results reveal that while current models approach human performance in knowledge-intensive categories, they fall considerably short in reasoning- in spatial reasoning. Key limitations include difficulties in bridging spatial scales, performing spatial simulation, and engaging in abstract reasoning, underscoring the need for continued advancements in both benchmark datasets and MLLM architectures.
iintonnin-ittesiieee intensive ones, particularly
ACKNOWLEDGMENTS
This research is supported by the Hokkien Foundation Early Career Professorship through the Artifccial-Acciteectie Lab. This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier grant, and funded through the SUTD Assistant Professorship Scheme (SAP 2025 001). We thank the students of 20.224 Artificial & Architectural Intelligence in Design class (2025) the Singapore University of Technology and Design for their contributions to the raw data.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. https: /twp///wwwcc anthropic. com/ hwtp//eebbbbbeeeee /tt.b//c333000bbbbbee18888333332 /2o/eeell
Card_ Claude_ 3, pdf
Anthropic. Introducing claude 4, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1432-7439, 2020.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024.
An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.015584 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2rX1v:2010.119299 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. In Proceedings of the 40 th International Conference Machine Learning, 2023.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended agents with -ctaee knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
ended embodied
Gemini Team Google, Rohan Anil, Sebastian Borgeaud, Lean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2rXiv:23/2.11803 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. et al. Gemma 3 technical report. arXiv preprint 1rXiv:25533119750 2025.
GLM-V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, et al. Glm- 4.5v and Iv-thikkin: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. URL httttttst///aattee org/ abs abs/2507.01000
Google Cloud. Google Cloud Vision AI, 2025, URL
Yash Goyal, Tejas Khot, Douglas Soammers-Stayy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9904-6913, 2017.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.2 1276, 2024.
Serwan Jassim, Mario Holubar, Annika Richter, Cornelius Wolff, Xenia Ohmer, and Elia Bruni. Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. arXiv preprint arXiv:231 122311.090 1.09048, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Pruceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understand- ing vision-language models: insights and future directions., 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03.0 033266 2024a.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1806 1-0010 2024b.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions the Association for Computational Linguistics, 1::35-651,, 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Andres Marafioti, Merve Noyan, Miquel Farré, Elie Bakouch, and Pedro Cuenca. Smolvim-small yet mighty vision language model, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation. arXiv preprint arXiv:2410.05363, 2024.
Meta AI. The Ilaaa 4 herd: The beginning of new era of natively multimodal ai innovation. httpss:/aa3 http://aaamaaaaaaaaaaammm meta. www.blog//ccccoccccocccccc.oooce
2025.
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Aishwarya Agrawal, al. Benchmarking vision language models for cultural understanding. arXiv preprint arXiv:2407.10992 2024.
OpenAl, Introducing gpt-5, http:ssr/pppaaaaccc///////tttttt
-
OpenRouter. The unified interface for Ilms, 2025. httips://openrouter..r
12

---

Published as a conference paper at ICLR 2026
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen,, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pp. 2017-2025, 2022.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of still image. In Computer Rsion-ECCY 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part V 16, pp. 508-524. Springer, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8760 PmLR, 2021.
Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. arXiv preprint M.XIv:2336.188444 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326. 2019.
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal Ilms to generalist embodied agents: Methods and lessons. arXiv preprint arXiv:2412.08842. 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jiay Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392-755573 2024.
XAI. Bringing grok to everyone, 2024.
1212
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01444 2019.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi -usti-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution. Advances in Neural Information Processing Systems, 37:56619-55 96649,,663,... 2024b.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-languaae navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7641-7649, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.0.1 DATASET CATEGORY DETAILS
This section demonstrates examples of each subcategory of the four main categories in the data set.
Spatial Reasoning
Subcategory: Spatial Interpretation
Meseraper
akgocraper
skyacoper2
digocraper
Question
From which skyseraper the perspective view shown on the right taken from?
A. Skyseraper
B. Skyscraper
C. Skyscraper
D. Skyscraper
Ground truth
A. Skyscraper
GPT-4o prediction
The perspective view the right was taken from Skyscraper 2. Correct answer: D. Skyscraper
x
Figure Example of Spatial Reasoning/Spatial Interpretation.
14

---

Published as a conference paper at ICLR 2026
Spatial Reasoning
Subcategory: Space Association
Question
Where is the highlighted roof (in the section diagram) located the axonometric drawing?
A.II
BIIII
C.IV
D.I
Ground truth
III
GPT-4o prediction
The highlighted roof the section diagram corresponds locution the axonometric drawing.
x
Figure 7: Example of Spatial Reasoning/Space Association.
15

---

Published as a conference paper at ICLR 2026
Spatial Reasoning
Subcategory: View Analysis
RESTAURANT
i8
FOVER
LABS
STAFFT
TICKETS
SHOPS
CAFE
Question
I standing front of foyer, likely which space outside my view?
A. none the spaces
B. cafe
shops
Ground truth
cafe
GPT-4o prediction
D. laby
X
Figure 8: Example of Spatial Reasoning/Viis View Analysis.
16

---

Published as a conference paper at ICLR 2026
Spatial Reasoning
Subcategory: Form Translormation
Option
Option
Option 3
Option
Question
Given folding plan drawing of an origami box, what this can produce?
A. Option
B. Option
Option
D. Option 2
Ground truth
Option
GPT-4o prediction
The
A. Option
X
Figure 9: Example of Spatial Reasoning/Forn Transformation.
17

---

Published as a conference paper at ICLR 2026
Commonsense Knowledge
Subcategory: Materials and Structures
Question
The wardrobe drawn above keeps toppling direction indicated by arrow, What the most likely reason?
wardrobe have different lengths B. The doors are heavier than the rest of the wardrobe
A. The legs
C. is not placed against a wall
D. Items inside are places tno close to the door
Ground truth
B. The doors are heavier than the rest of the wardrobe
GPT-4o prediction
Rased on image, the most likely reason the wardmoh toppling the dtrection indicated by the arrow is: A. The legs of the wardrobe have different lengths. This would cause an imbalance and lead to toppling in one direction.
x
Figure 10: Example Commonsense Knowledge/Materials and Structures.
18

---

Published as a conference paper at ICLR 2026
Commonsense Knowledge
Subcategory: Construction and Fabrication
A
B
C
D
E
Question
is the sequence install fastener and
A. EACBD
B. AECDB
C. EACDB
AFCCD
Ground truth
A. EACBB
GPT--o prediction
To determine the correct sequence, let analyze the sequence of steps visualired:: F Drilling a concrete. A Clearing the hole of debris. 3. C- C-Isserting the fastemer. D Setting the fastener with tool. 5. B 9-Paacing and tightening the mut or washer: The correct sequence is FACDB. Answer: C. EACDB
Figure 11: Example Commonsense Knowledge/Construction and Fabrication.
19

---

Published as a conference paper at ICLR 2026
Commonsense Knowledge
Subcategory: Cultural Context
Question
Which color palette best appropriate if the architecture diagram influenced by Classical Mediterranean architecture?
A.I
B. II
C.
III
D.IV
Ground truth
GPT-4o prediction
The color palette best appropriate for architecure influenced by Classicul Mediterrameon architecture This palette earthy typical of the Mediteranean region.
Figure 12: Example of Commonsense Knowledge/Cultural context.
20

---

Published as a conference paper at ICLR 2026
Commonsense Knowledge
Subcategory: Vernacular Living
Question
There objects hanging from the roof the Korean house. What the objects?
A. Laundry
B. Wind chimes
Fruit put out to dry Ornamcntal beads
Ground truth
Fruit pul out dry
GPT-4o prediction
The objects hanging from the roof of the Korean house appear be C. Fruit put dry.
Figure 13: Example of Commonsense Living.
21

---

Published as a conference paper at ICLR 2026
Environment Interaction
Subcategory: Faver-Faviromment Interaction
Deing
Casher
Question
From diagram, please select spaces least suitable for long-term comfort result of being stagnant-air zones.
A. Living and Dining Space
B. Kitchen
C. Flexible Space
D. Rath
Ground truth
C. Flexible Space
GPT--o prediction
The space least satahle for long- term comfort chee heing a stagmant air mon "Feexible Space"
Figure 14: Example Environment CeapriioonUse--Evionnnnnnnn Interaction.
22

---

Published as a conference paper at ICLR 2026
Environment Interaction
Subcategory: Design-Favirrommnt Interaction
Question
Which position would most ideal for photovoltuic panels to installed. Consider maintenance ease of access.
A. III
B.III
C.III
D.IV
Ground truth
C,
GPT-4o prediction
Based on image considering maintenance and ease of access, the most ideal position for photovoltnic panels would be:
Figure 15: Example of Environment AnterattionDeesinn-Eniiioo Interaction.
23

---

Published as a conference paper at ICLR 2026
Environment Interaction
Subcategory: Mobility Space
11:
11 11
11 141
VI
Question
Which point of roundabout has the highest volume flow entering roundabout?
A.
B.III
C.V
D.II
Ground truth
III
GPT-4o prediction
The point with the hrghhet volume flow entering the roundabout point "HI".
X
Figure 16: Example of Environment Interaction/Mobility in Space.
24

---

Published as a conference paper at ICLR 2026
Environment Interaction
Subcategory: Environment Sustainability
canpp
Question
Why is the building on stilts?
A. To allow eyclists to traverse freely on ground.
B. To miligate flooding.
C. To allow ground animals to roam freety with minimal obstruction.
D. To allow visitors to look at animals from ahove freely."
Ground truth
C. To allow ground animals to roam freely with minimal obstruction.
GPT-4o prediction
To mitigate flooding
Figure 17: Example of Environment Interaction/Envirommeen Sustainability.
25

---

Published as a conference paper at ICLR 2026
A.0.2 EXPERIMENT DETAILS
This section explains additional details of the experiment.
Prompt
For each data case, the prompt includes an image and the text of the question formatted as illustrated in Figure (a). For questions with visual options. the options and the main diagram are prompted as one image as shown in Figure T8 (b). The prompt for questions with text options is shown in Figure (c).
(a)
Options:
Question:
"Please select correct from the given options based the image."
(Quesrion test)
bb)
Prompt Image:
Quediou: Wheu shuding al poun looking an ducctiou wheze the stoo poining winch the coneet view? Options:
"Option in
"Ophon mmage".
"Opiion D in the imse""
Option the
Prompt Teat
Queslion. Wisuu slauding al peinl Jooking in the disection whatu the arow pointing, which is the correct view "uOpptions lud. Optisu A Option C the Image'uC. Option B in the image'uD. Option D in imane'nPPeesee select from the given based
the umgge'Bl
image.".
Prompt Imaage
Questio:: In the reciprocal semur uure shown alove, increase its height wilhout altering the pipes or layout, how this affect the sire of the highlighttd urea?
Options:
"Toceesse" Incresse them decretsee..
"Decreuse"
"Not Changing"
Prompt Text:
"Question: the reciprocal structure shown ahove, if increase its beight withou altering the pripes or layou, how will this affect the size of the
highlighted rearunOtionnnnn Incressetaii. based on image."
DecresseinD. Changing alaeese select
from given options
Figure 18: Prompt example used in the experiment, (a) structure of the prompt text. (b) prompt example of questions with visual options. (c) prompt example questions with text options.
26

---

Published as a conference paper at ICLR 2026
Model Links
We provide all links to the APIs or models used in our experiments in Table
Table 3: Links to the models in our experiment
Model
API/Model Link
GPT-5
htpp://ppammmicccaalopnnaagggggp
GPT-5-mini
GPT-4o
GPT-4o-mini
GPT-o4-mini
grok-2-vision
claudc-sonnet-4
claude-3.7-sonnct
gemini-2.5-flash
Qwen2.5-VL-72B
gemma-3-27b-it
Llama-4-Maverick
Llama-4-Scout
gemma-3-122-it
Hixtral-1B-2444
glm-4.1v-9h-hhnnking
Ideficx3-9B--Jlmaal
Qwen2.5-VI-7B
llava-onevision-00
Phi-4-multimdaa
gemma-3-4b-it
smolvim-2b
http://hwggggghacooooriigi//oiia Instrucct
27

---

Published as a conference paper at ICLR 2026
A.0.3 NEGATIVE DATA EXAMPLES
This section shows examples of diagram-question pairs removed for failing quality control with explanations.
Visual Independence
Which of the following pyramids known for its significant role in shaping the ancient Egyptian view of the afterlife, what period did its construction refiect terms of architectural and religious evolution?
A. The Pyramid of Khufu, around 2600 BCE
The Pyramid Khafre, around
BCE
C. The Step Pyramid of Djoser, around 2630 BCE
D. The Bent Pyramid, around 2570 BCE
Explonation: The question can answerdd withrud visual input. The question can he fully answerad based on historical knowledge of Egyptian monmments and does not rely on interpreting the accompanying image.
Which ancient civilization most closely associated with the construction of monumental stepped pyramids, and what was the primary purpose of these structures?
A. Ancient Egypt B. Ancient Mesopotamia
C. Mesoamerican Civilizations
D. Ancient Greeze
Explanation: The question can be unswered withond visuul input. The origios and purpose of stepped pyramicss can be identified from general historical knowledge, independent of the visual provided.
Figure 19: Examples of questions that do not require visual input to answer.
28

---

Published as a conference paper at ICLR 2026
Question Cannot be Answered
B
C
D
Which combination of quaddrants are the suitable for living spaces and oflices?
A. and
B. and
C. and
D. A and
The diagram does not provide enough information to onswer the question. The diagram only shows compass directions and simple division into four quadrants. There is no information about climate, sunlight or orientation, prevailing winds, surruunding context, or functional requirements, without which it is impossible to determine which quadrants would be mosi suituble.
Explanation:
What do the arrows in the diagram represent?
A. Wind direction in relation to the huilding
Viewpoints lowards building
Pedestrian movement through the site
D. Access routes and from the building
The diagram does not provide enough information to answer the question. The diagram shows arrows originating from undefined area and pointing toward the blue area, but ut does not include labels. a legend, or contextual cues. The grophic alone insufficient to determinne their specific meaning.
Explanation:
Figure 20: Examples diagrams that do not provide sufficient information to answer the questions.
29

---

Published as a conference paper at ICLR 2026
Lack of Clarity
Option
Orten2
Optin
Ozten
When the two forms on the left are assembled, which final shape will they create?
A. Option
B. Option 2.
C. Option 3.
D. Option 4.
The diagram does not provide enough visual information to determine w whether the upper element is flat circle or sphere. Withou clurity on its dimensionulity how it interacts spatially with the cube, is imponsible to identify which of the four configurations correct.
Explonation:
Main Image
Option
Option 3
Option 2
Option 4
Which view will you when standing the point marked in the Main Dingram? A. Option B. Option Option Option
Esplanation: The messy lines and curves make surfoces and volumes unclear Without clarity in spatiol relationships, is hard to determine which of the
the four provided views corresponds to the position marked X.
Figure 21: Examples of diagrams with insufficient visual clarity.
30

---

Published as a conference paper at ICLR 2026
A.0.4 DETAILED FAILURE CASE ANALYSIS
This section further demonstrates some typical failure cases, explaining how the model failed and what maybe the reasons causing the failure in detail.
POINT2
Option
Opton 2
POINT
POINTI
TOM
Option
Option
Which of the following perspectives correspond to how Jay will see Tom from his location?
A. Option in the image
B. Option 2 in the image [
C. Option 3 the image
D. Option 4 the image
GPT-5 Prediction
Jay's h location and viewing diraction are clearly shown in the left diagraw:
Jay positoned ussil cluster tell block-lie wells.
farther de-m sioghtly Jay's lef, Jay' vew (owardi snongly sodtpple
Only small pup allows Jay to see Tom, and the view wouid be:
Framed sightty walls, abmast like looking through slor.
appears
walhs very
Now compare the options:
Opoion
Too open. would
open ground; hes
both
Option 2
Shill spoo the right. Juy surrawshed by highor and chover walls than shann harn. Incoerect
Option
Tom is seen nirough warrow oponteg, with:
lange wall overbead.
Tell walls boak A restricted folld of viese
Manchas Jay's actual obstructed sightline
Too clear and too open, Jay wonldh have wide clear view fits given wall cheseer ground hom. Incorrect
Analysis
This is a failure of scale-aware spatial reasoning und egocentric viewpoint transformation. The model did not recognise that the white blocks are in building scale compared to the size of Jay and Tom, and the openings are actually large enough to permit visibility. It focused overly the 2D size of the opening to eliminate options, and never proceeds the actual decisive factor: precisely simulating Jay's first-person perspective determine which elements truly visible occluded along the sightliee to Tomn
Possible reasons fallure:
Misinterpretationn of schematic scale: The models are intentionally trained on large spatial scale (building or above) reasoning data. They fail infer that a gap spanning dozens human-widths image effectively a wide opening
Weak mental rotation and simulation: Accurately answering requires constructing 3D mental model, placing a virtual camera at Jay's position and them tracing rays toward Tom. Such capabilities remain weak in frontier models.
Failure of focal-to-globe perspective mapping: The lask demands bidirectional reusoning: using the globul perspective to predict local first-person visibility and viee versa. Most VLMss handle one direction poorly and completely fail when both directions required simultameously.
Figure 22: Failure Case Analysis.
31

---

Published as a conference paper at ICLR 2026
B
Which is the perspective drawing framing?
A. Optioon the image
Option the image
C. Option in the image
None of labelled area
GPT-5 Prediction
We vomporee the florr plue (lef)) with thes perapectivn drowing (right)
What the perpeette slonss:
large opentng
doorway opentog)
Stope loading op into u space
lnnide that
dusing
unltppie
pendont light honging from the cailing
aahewordd i cobinat the left
clearly duning
Mutch with the plan:
Area
corvidor-tlee lrstig spare
Courains seating sptral satrs
Not dining duoers' mutele
Baskroom area (sink bathtub)
Difoninll not shown parsuective dosan't match
Courams duning table with muinple chatss pendons light abows the whble Adpacent the lorge opening This marehes perspective exactly Thus, the perspetivv dravinng fromes Avea C.
of
Analysis
This is a failure of line-grained visual comparison and precise spatial layout verilication. The model made some coarse observations presented in both Arca C and the perspective view. However, key differences in the perspective such as the mumber and location of chairs, wardrobes on the left, steps in front of the table are ignored by the model.
Possible reasons for failure:
of spatial precision: Models excel global layout but rapidly lose fine positicnal and counting accuracy. systematic verification loop: Humans subconsciously run mental thefffhhedifffeeecee" process (count chairs, behind table, look closet, notice steps, etc.). Models almost never iterate or explicitly check for absences; they accept the first plausible match.
Figure 23: Failure Case Analysis.
32

---

Published as a conference paper at ICLR 2026
Above 'gat count' diagram, quantifying pedestrian flows through specified "gates". Which route from start to end should pedestrians take to meet as many people as possible?
A. Option A in the image
B. Option B in the image
C. Option C in the image
D. Option D the image
GPT- Prediction
Why Romte A liggesl euconnters
You vant the palh from Start - End that parees through the lorgest toral podostrian comnus, te., the ronte that intersects the warmest colors (orangereal
1060 people per
ligh-flmn govez:
996 (dork vel)
790, (rad)
(ovange)
Several 250-100 gaiez
Which Intersects those?
(lang-doshl passies lhrough:
750 (rei))
753 (rei))
(arange)
520 (arange)
296, 256, 236, 294, 184,
connisttatly slays highusifflo vorridor, cutting through wormeshundd yoles.
Other moste
B. shrough wonty green and bines flow flows), Intersecting only sparse mil-yollow segmentss.
They avoid buy raalunange hotppt curridur.
Finel Answe
Route allewar pedestrtmns the wartoum mamber of people.
Analysis
The model's reasoning is flawed. It incorrectly assumes that route maximises pedestrian encounters simply hecause it "cuts through the warm-colered gates." while routes "B, C, and D pass mostly through green and blue" gates. In reality. the correct approach is to sum the actual number of pedestrians each gate along every route. When calculated properly, roule D yields the highest total.
Possible reason failure:
Over-reliance on high-level semantic priors: Once model notices color pattems,, it activates a learned bias that warm colors correlate with crowds, statistically valid prior from training data. This strong prior overrides the weaker, noisier signal from actually parsing the numbers.
Figure 24: Failure Case Analysis.
33

---

Published as a conference paper at ICLR 2026
When standing point looking in the direction where the arrow pointing, which is the correct view?
A. Option in the image Option in the image
B. Option in the image
Option in the image
GPT-5 Prediction
Yow ome standing at point A, looking trouand paint X (hen emall block at the front of long, stoggemal norridor of bloda)
From postiton:
you large distcchd very
Innodiinelly ofter that on the bef) you cwe tall mestogular blunk
The corrdor aload contams sequence of bloeks, each slightly shifted, progressmely maller apporent due perspective.
long L-shaped block begins close you The view is desp woriterr compuoord of many resading vartical ehoneetts.
right
Among the foar viewy.
incorrect
chona the rignwent black wuch wn furge and hoo cluse.
chenas the contral corvidor hiacked by huge block that doer wotch the plan
the
appoartng wrong sequence proportion.
way Uls correct Shons
the left (metciies big point A block also on left just bebind
large
long requence recedling blocks striggt aiven.. depth.
right hand block posttioned
Everyhing aiggu with the 3D layout.
Analysis
The model fails to detect critical details, The difference between options C and is extremely subtle: the only distinction lies the shape of the farthest building. From the bird 'sye view the left, is clearly L-shaped building, Option incorreetly renders striight I-shape, whereus option D accurutcly depicts L-shape.
Possible reasons failure:
Positional bias toward contral/foreground objects: The models often prioritise foreground or center elements.
effectively ignoring low-salience hackgroumd details.
Lack of explicit spatial reasoning training: The model may have "seen" the pixels but never learned to treat exact building match as decision-critical, so it defaulted to superficial foreground match that looked plausible.
Figure 25: Failure Case Analysis.
34

---

Published as a conference paper at ICLR 2026
Consider the diagram the left. Which color the white wall in the structure the image the right according it? A. green
puple
C. pink
D. blue
GPT-5 Prediction
Ressowing
In the right-sble perspective image, the white wall is vertioni wail locoted roughly in the muddle of the tower stack.
idounf: which colored wall corresponcs the AD dlagram, matel:
heigit
Its ortentation
secation volutive to colored frames
Iduatfy white
right image
In the right mmage, the while walf stands betwern:
blue frame below
pink frame
11 to clearly by the vertiaal wall at the third ther from the bottom.
2. Look the left diagram
from Butiom: pink
2nd: blue
grzen only versical
level.
har the exoet some orimitaiion and with the white wall in the right parquective Therefore, the white wall the porspective corveroonds the wall. green
Analysis
The model hallucinates the wrong neighboring colors for the white wall. The reasoming is sound, but it arrives the wrong answer due clear perceptual error: misidentifies the position of the white wall. In the actual image (right side), white is positioned between yellow frame below blue frame above.
Possible reason for failure:
Locul context confusion during patch-based processing: thin vertical features like walls can straddle patch boundaries or occupy only few pixels within patch, causing their adjacent colors to be incorreclly associated after tokenization and pooling.
Figure 26: Failure Case 5 Analysis.
35


Reviews:
[{"review_id": "G1fbGxUEUn", "reviewer": "Reviewer_NpK9", "summary": "This paper introduces SpaCE-Eval, a new visual question answering (VQA) benchmark designed to evaluate the reasoning capabilities of Multi-modal Large Language Models (MLLMs) in real-world environments. The authors argue that existing benchmarks often fall short by focusing on object-scale understanding and static scenes, neglecting the complex, multi-scale reasoning required for practical applications. SpaCE-Eval addresses this gap with three core categories: Spatial Reasoning (SR), Commonsense Knowledge (CK), and Environment Interaction (EI). The benchmark consists of 1,139 high-quality image-question pairs based on brand-new, human-created diagrams to prevent data contamination. A comprehensive evaluation of numerousSOTA MLLMs on SpaCE-Eval reveals significant limitations, particularly in spatial reasoning, where even the best models perform poorly. The analysis highlights models' difficulties with larger spatial scales, tasks requiring spatial simulation, and reasoning with visual-only options, underscoring the benchmark's value in guiding future MLLM development.", "questions": "1. Could you please clarify the inconsistency regarding the top performance in the Spatial Reasoning category? Which value is correct: 31.75% as stated in the abstract or 42.25% as presented in Table 2?\n2. Given the modest size of the dataset, do you have plans to expand SpaCE-Eval in the future? A larger number of examples would strengthen the statistical power of the results, especially for the more niche subcategories.\n3. The analysis shows a significant performance drop for questions with purely visual options (26.58% accuracy). Is this poor performance uniform across the three main categories (SR, CK, EI), or is it particularly higher in one area, such as Spatial Reasoning, which seems intuitively more visual?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's greatest strength is the meticulous construction of the SpaCE-Eval benchmark. By commissioning brand-new diagrams from individuals with design backgrounds, the authors successfully avoid data contamination and create visually diverse and conceptually rich problems. The four-stage quality control process ensures the reliability and difficulty of the questions.\n2. SpaCE-Eval is uniquely comprehensive. The three categories which are Spatial Reasoning, Commonsense Knowledge, and Environment Interaction, cover a wide and crucial set of abilities for real-world intelligence. The benchmark's focus on multiple spatial scales (from objects to urban environments) is a significant step beyond existing datasets.\n3. The benchmark is demonstrably difficult for even the most advanced MLLMs, with the best model scoring only 56.37%. The paper provides deep insights into specific model failures, such as the stark performance gap between questions with textual vs. visual options, the degradation of performance on larger spatial scales, and the inability to perform mental spatial simulations. These findings are crucial for understanding the current limitations of MLLMs.", "weaknesses": "1. There is a discrepancy between the highest Spatial Reasoning score reported in the abstract (31.75% for claude-sonnet-4) and the one in Table 2 (42.25% for GPT-5). This should be clarified and corrected for consistency.\n2. While the quality-over-quantity approach is commendable, the total dataset size of 1,139 examples is relatively small for an evaluation benchmark. When broken down into the twelve subcategories, the number of questions per subcategory can become quite small (e.g., Space Association is ~3.25%, which is only about 37 questions), which may limit the statistical significance of the fine-grained analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NUpVqhcGJP", "reviewer": "Reviewer_xj7A", "summary": "The paper introduces SpaCE-Eval, a new visual-question-answering benchmark designed to evaluate the real-world reasoning abilities of Multi-modal Large Language Models (MLLMs). The dataset consists of novel, human-drawn diagrams to prevent data contamination and is structured into three categories: Spatial Reasoning, Commonsense Knowledge, and Environment Interaction. Evaluations across various MLLMs reveal that the benchmark is highly challenging, especially in the spatial reasoning domain where models perform poorly. The results highlight a significant gap between current model capabilities and human-level spatial understanding, underscoring the need for improved reasoning in complex physical environments.", "questions": "1.  **Clarity of Category Definitions:** What are the precise, mutually exclusive definitions for the subcategories? For instance, how do you differentiate a task requiring \"Spatial Interpretation\" (reasoning about perspectives) from one requiring \"Space Association\" (associating spaces across different views)?\n\n\n2.  **Meaning of CLIP Similarity Score:** The paper reports a mean CLIP similarity of 0.654 to argue for the novelty of the diagrams. How should this value be interpreted? Without a baseline (e.g., similarity scores within other VQA datasets or between random web images), it is difficult to judge whether 0.654 indicates true novelty or moderate similarity.\n\n\n3.  **Reliability of the Automated Judge:** The paper uses GPT-4o-mini to verify model predictions that are not exact string matches. What was the measured accuracy of this automated judge? How was it validated to ensure it does not have its own biases or make systematic errors in judgment, which could affect the reported accuracy of the models being tested?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  **Strong and Clear Motivation:** The paper does an excellent job of identifying a critical and timely gap in existing MLLM evaluation. It correctly argues that reasoning in complex, environment-scale spaces is a fundamental prerequisite for real-world applications like robotics and autonomous agents, and that current benchmarks often fall short in this area.\n\n\n2.  **Proactive Mitigation of Data Contamination:** The core methodological decision to create the entire dataset from brand-new, human-drawn diagrams is a significant strength. This approach directly addresses the pervasive problem of data contamination, where models may have already seen test images during pre-training, ensuring a more honest evaluation of their reasoning abilities.\n\n\n3.  **Comprehensive and Well-Defined Task Taxonomy:** The benchmark is thoughtfully structured into three distinct and meaningful categories: **Spatial Reasoning (SR)**, **Commonsense Knowledge (CK)**, and **Environment Interaction (EI)**. Breaking these down further into twelve subcategories provides a fine-grained framework for analyzing model capabilities and pinpointing specific areas of weakness.\n\n\n4.  **Novel Emphasis on Multi-Scale Spatial Reasoning:** A key contribution is the benchmark's explicit focus on reasoning across multiple spatial scales—from objects and rooms to buildings and urban contexts. This is a crucial aspect of real-world intelligence that is often overlooked in object-centric datasets, and SpaCE-Eval makes it a central part of its evaluation.\n\n\n5.  **Focus on Abstract and Schematic Interpretation:** While the use of diagrams limits its \"real-world\" photographic realism, it is a strength in its own right. The ability to interpret abstract representations like floor plans, maps, assembly instructions, and infographics is a vital cognitive skill. This benchmark is specifically designed to test this form of abstract visual reasoning.", "weaknesses": "1.  **Superficial Analysis of Model Failures:** The paper correctly identifies that models fail at spatial simulation and abstract reasoning. However, the analysis in Section 4.3 is brief. It doesn't deeply investigate *why* these failures occur. Is it a limitation of the vision encoder, the language model's reasoning capabilities, the vision-language alignment, or something else?\n\n\n2.  **Dataset Balance and Composition:** Figure 2 shows the dataset composition. \"Building Space\" accounts for nearly 50% of the data by scale. This heavy imbalance could mean the overall performance scores are disproportionately influenced by model capabilities on this specific scale, potentially masking poorer performance at other scales like urban or object.\n\n\n3.  **In-Depth Bias Analysis:** Beyond a brief mention of linguistic shortcuts, the paper lacks a thorough analysis of potential biases in the dataset. This could include:\n    *   **Answer Distribution Bias:** Is the correct answer (A, B, C, D) uniformly distributed? The paper claims it is, but doesn't provide the final distribution.\n    *   **Content Bias:** Are certain objects, architectural styles, or cultural contexts overrepresented?\n    *   **Negative Examples (\"Probing\"):** Including questions where the answer is \"cannot be determined from the image\" to test if models are hallucinating or guessing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9aUKbuxmx", "reviewer": "Reviewer_8eWj", "summary": "This work proposes an expert-designed and peer-reviewed dataset aimed at evaluating the spatial reasoning abilities of various LLMs. This dataset comprises real-world VQA tasks across a range of contexts. Model outcomes show that for real-world spatial reasoning and use of LLMs in complex environment-scale spaces and creation of autonomous agents that can survive in the real world, these models require a great deal of enhancement and improvements to the reasoning process, and this work establishes this dataset as a human-verified benchmark to assess the same.", "questions": "Suggestions:\n1. This work would benefit from demonstrating the question creation, review, and modification process for examples in the appendix, including questions that did not pass quality control\n2. Fig 2 contains a typo in the diagram: SpeCE -> SpaCE\n3. An exploration of how the quality control and review process was kept consistent and sound, and how annotator agreement was measured to ensure no bias or high variance in quality decisions, would lend more credibility to the curated dataset\n\nQuestions:\n1. Given the small set of problem creators (51) all from the same background (university students), how do you ensure that problems are diverse not only in context and content, but also complexity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dataset is well-curated, with a clear and comprehensive process for design, review, and modification.\n2. The newly-designed nature of these problems eschews concerns about data leakage\n3. Table 1 is a comprehensive comparison against similar datasets\n4. The analysis section provides a useful look at model performance patterns and interpretations.", "weaknesses": "1. The data curation and review process are vital to the creation and success of this dataset. While the methodology is reasonable, there is no discussion of the soundness and reliability of this process - especially regarding the following;\n- 1.1. The instructions provided for the design process, any rubrics that were followed, etc\n- 1.2. Rubric for the review process, and how sounds this process was in terms of annotator agreement. This is especially relevant to the meta-annotators.\n- 1.3 Examples of questions that did not pass quality control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaJLbOPsdY", "reviewer": "Reviewer_Mgud", "summary": "SpaCE-Eval introduces a new visual question answering (VQA) benchmark designed to evaluate multi-modal large language models (MLLMs) on real-world spatial reasoning, commonsense knowledge, and environment interaction. The dataset features 1,139 high-quality, human-created diagram-question pairs spanning diverse spatial scales (from objects to urban spaces) and includes both textual and visual answer options. Extensive evaluation of leading MLLMs reveals significant gaps in spatial reasoning and simulation, especially as spatial scale increases, highlighting the need for more advanced real-world reasoning capabilities in AI models.", "questions": "Flags identified and covered above in weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Real-World Focus & Diversity: Covers a wide range of spatial scales and real-world scenarios, moving beyond object-level or synthetic tasks\n- Rigorous Data Curation: All diagrams are newly created by human experts, eliminating data contamination and ensuring visual diversity.", "weaknesses": "- Need for artifacts: The paper identifies gap in MLLMs, provides a benchmark to help evaluate but should also provide basic artifacts like reasoning traces and training corpus that can be used to do SFT/RL and mitigate this gap, and further confirm the paper's hypothesis.\n- Expand Task Formats: While broad in spatial scale, incorporating open-ended, multi-step, or interactive tasks (e.g., navigation, planning, or manipulation) to better reflect real-world reasoning demands would help the benchmark remain strong and relevant.\n- Diagnostic Error Analysis: Provide more granular, qualitative analysis of failure cases, especially for abstract and large-scale spatial reasoning, to guide model and dataset improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)