SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Authors: Wufei Ma
Yu-Cheng Chou
Qihao Liu
Xingrui Wang
Celso M de Melo
Jianwen Xie
Alan Yuille
Venue: NeurIPS 2025 poster ()
Keywords: 3D Spatial Reasoning
Vision-Language Models
Primary area: applications
TL;DR: We introduce SpatialReasoner, a novel large vision-language model that improves 3D spatial reasoning with explicit 3D representations and generalizable 3D thinking.

Abstract: Despite recent advances on multi-modal models, 3D spatial reasoning remains a challenging task for state-of-the-art open-source and proprietary models. Recent studies explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual question-answering data. However, these methods typically perform spatial reasoning in an implicit manner and often fail on questions that are trivial to humans, even with long chain-of-thought reasoning. In this work, we introduce SpatialReasoner, a novel large vision-language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages--3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability to novel question types. Furthermore, by analyzing the explicit 3D representations in multi-step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMs. Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2.0 by 9.2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning.

SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
Wufei Ma"
Yu-Cheng Chou" Qihao Liu" Alan Yuille
Xingrui Wang Celso M de Melo
Jianwen Xie
Johns Hopkins University, !DECCOM Army Research Laboratory. o "Lambda Inc
Abstract
Despite recent advances multi- modal models, 3D spatial reasoning remains challenging task for state-of of-the-art open-source and proprietary models. Recent studics explore data-driven approaches and achieve enhanced spatial reasoning performance by fine-tuning models on 3D-related visual guestion-answering data. However, these methods typically perform spatial reasoning in implicit manner and often fail on questions that are trivial to humans, even with long chain-of- thought reasoning. In this work, we introduce SpatialReasoner, novel large vision- language model (LVLM) that addresses 3D spatial reasoning with explicit 3D representations shared between multiple stages-3D perception, computation, and reasoning. Explicit 3D representations provide a coherent interface that supports advanced 3D spatial reasoning and improves the generalization ability novel question types. Furthermore, by analyzing the explicit 3D representations in multi- step reasoning traces of SpatialReasoner, we study the factual errors and identify key shortcomings of current LVLMss Results show that our SpatialReasoner achieves improved performance on a variety of spatial reasoning benchmarks, outperforming Gemini 2.0 by 9.2% on 3DSRBench, and generalizes better when evaluating on novel 3D spatial reasoning questions. Our study bridges the 3D parsing capabilities of prior visual foundation models with the powerful reasoning abilities of large language models, opening new directions for 3D spatial reasoning. Our project page is available here
Introduction
3D spatial reasoning studies how models perceive, understand, and reason about 3D object properties and spatial relationships. is not only a fundamental task for vision-language models to achieve human-level intelligence, but also crucial to range of downstream applications in robotics [8] and embodied AI [10]. Despite the recent advancements of large moottl models, such as GPT-40 and Qwen2.5-VI., their 3D spatial reasoning capabilities remain limited and fall far behind human- level performance ( [1 55. Recent 3D-aware large vision- language models (LVLMss) injected 3D knowledge by fine-tuning the model on synthetic 3D-related question-ansen achieved improved performance on spatial reasoning benchmarks [3] 45 51 On the other hand, large proprietary models such Gemini 2.0 (15] have advanced 3D parsing by directly predicting 3D object bounding boxes, enabling the development of powerful generalist robotics models 43
pairs O 33 and
We identify two key challenges in 3D spatial reasoning: (1) 3D thinking the ability decompose complex 3D spatial reasoning question into small, manageable steps, and (2) 3D computation ability to solve these thinking steps in consistent and accurate manner. As shown in Figure []] prior reasoning methods adopt long chain-of-thought reasoning to tackle the problem but do not
Equaal first author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

User
Dwen2.5-VI--22.
X 3D thinking X 3D computation
From the perspective of the man, is the bottled water on the left right side him?
From the perspective of the man, the bottled water on his left side.
X 3D thinking X 3D computation The man seems to be inside a food truck, and from the image, the water bottle is on the left side, But looking from his perspective: he's facing roughly to the image (our left). Since be's holding the foil package his right hand, the water bottle is probably on his left side. This makes sense because the foil package is in front the right side, so the water bottle would logically opposite, the left side.
CharGPT o3
side of the
Gemini 2.0 Flash Thinking
V 3D thinking X 3D computation (omitted) 3. Locate the man in the image: (omitted) 4. Locate the bottled water: The bottled water in the foreground, partially obscuring the man. 5. Determine the man's orientation: The man is facing towards the viewer. (omitted) 6. Project the man's nerspective: Imagine line extending striight out from the man's nose chest, The bottled water is located to the left of this imaginary line, from his viewpeint, (omitred) From the perspective of the man, the bottled water on his left side.
computation The 3D location the bottled water is (0.2, 0.4, 0.6). The 3D location of the person is (0.0, 0.5, 1.0). The vector from the person to the bottled water is hence (0.2, -0.0, 0.4). The left direction the person is (-0.9, -0.1, 0.5). The cosine similarity between the vector and the left direction is -0.55 corresponding to an angle of 147.89 degrees. The angle larger than 90 degrees, meaning that the bottled water is right the person. Therefore, the final answer on the right.
SpatialReasoner
thinking
Figure I: Comparing 3D spatial reasoning of our SpatialReasoner with previous state-of-the-art models. Our SpatialReasoner builds on explicit 3D representations, performs 3D computation, and reasons about the final answer. Although Gemini 2.0 can also break down complex 3D spatial reasoning questions into small and tractable steps, it lacks reliable 3D computation and the long chain-of-thought reasoning ultimately leads to wrong answer.
have explicit 3D computation. ChatGPT o3 [36] fails to adopt systematic approach solving the problem and instead relies on other visual cues to assist the reasoning (e.g.. location of the foil package). In contrast, the Gemini 2.0 thinking model [15] employs an organized strategy to tackle the problem but ultimately fails to arrive at the correct answer due to a lack of reliable 3D computation.
In this work, we present SpatialReasoner. novel large vision- language model (LVLM) built with (I) explicit 3D representations and (2) enhanced and generalizable 3D thinking. Specifically, our SpatialReasoner adopts explicit 3D representations, such as 3D locations and orientations, as an interface that enables coherent and reliable reasoning across multiple stages, i.e., 3D perception, computation, and reasoning. On the other hand, we would like to learn enhanced 3D thinking capabilities that generalize to novel question types not seen during training. Hence we explore a two-stage training strategy following prior works [17]. In Stage I, we apply supervised fine-tuning (SFT) to equip the LVLM with explicit 3D representations, enhancing 3D perception and computation capabilities of the model. Then in Stage II, we leverage reinforcement learning (RI.) to develop robust and generalizable 3D thinking built on explicit 3D representations.
Inspired by previous 3D pscudo-annotation pipelines in 9 T 33 we synthesize basic 3D perception, ie., detection and pose estimation, and 3D computation question-answering data interleaved with explicit 3D representations. We further generate standard spatial reasoning question-answer pairs with tf-thuggh reasoning that breaks down complex 3D spatial reasoning questions into multiple steps-33 3D perception, computation, and reasoning. Experimental results demonstrate that our SpatialReasonee with explicit 3D representations can significantly enhance 3D spatial reasoning abilities of LVLMs and generalize novel question types.
Besides enhancing 3D spatial reasoning capabilities of LVLMs, reasoning with explicit 3D represen- tations allows us to interpret the reasoning process and to study the failure modes of LVLMs. We find that the accuracy of 3D perception lags significantly behind that of 3D computation, suggesting that most errors in downstream VQA tasks still stem from failures in 3D perception. Moreover, by predicting key 3D information, such as 3D object locations and orientations, as intermediate results, SpatialRe enables compositional reasoning for 3D spatial tasks [46). This not only allows our method to generalize better to novel spatial reasoning questions, but also makes it easily extensible other tasks that build on our explicit 3D representations.
Besides improving spatial reasoning performance on a variety of benchmarks, we experiment on various LVLMs fine-tuned with different combinations of data and training methods to study the key

---

factors toward improved 3D spatial reasoning. Our empirical results lead to the following insights: (1) For 3D-aware VLMs, SFT offers a more scalable approach than RL that requires high-quality 3D-aware data, which is often difficult to obtain. Our recipe of RI. followed by SFT achieves the best overall performance: (2) 3D-aware LVL.Ms fine-tuned with RL generalize better than SFT when tested on novel 3D spatial reasoning questions, echoing prior findings [2]; (3) Standard LVLMs often exploit 2D reasoning as shortcut to tackle 3D spatial reasoning problems, whereas our SpatialReasoner avoids the spurious correlations and always reasons with explicit 3D representations, achieving improved and robust performance on challenging real-world 3D spatial reasoning datasets.
In summary, our contributions are as follows: (1) We introduce SpatialReasoner. novel LVLM that solves 3D spatial reasoning problems with a compositional approach based on explicit 3D representations. (2) Reasoning with 3D representations allows us to interpret the reasoning process and to study the failure modes of LVL.Ms. (3) Our SpatialReasoner effectively improves the 3D) spatial reasoning performance
range of benchmarks and extensive experimental results provide valuable findings on data and training strategy designs for future development of 3D-aware LVLMs.
Related Works
3D Spatial reasoning. 3D spatial reasoning explores how models perceive and reason about 3D object properties and relationships. Early works [46] 47 studied this in simulated environments or datasets with object-level annotations [20] 25
More recent efforts construct benchmarks based on real-world imagery B1 51 and improve model performance via synthetic QA data [9] [ 33 39). However, these models often rely on implicit reasoning, offering little interpretability or intermediate 3D computation. Moreover, it remains unclear whether the 3D spatial reasoning capabilities acquired from data-driven fine- fine-tuning can generalize to novel question types that require complex 3D computation over different combinations of 3D perception outputs.
45 33
Explicit 3D representations. Explicit 3D representations simplify reasoning and expose model failures. Simulation- based works [35] used neural-symbolic methods to parse object-level structures, while PO3D [46] and its extensions 47 48 showed that structured visual modules enable inter- pretable reasoning [2). These results in simulation systems reveal the limitations of current LVLMs and highlight the importance of the visual module for successful reasoning model.
Post-training. Post-training aligns pre-trained models with downstream objectives via SFT 44 54
and RL 6 37 41), improving formatting and reward-guided alignment [21] 44). Recent systems like GPT-4 M. Claude-3.5 [2]. and DeepSeek-R [17] showcase these techniques. Despite recent advances, post-trained models still struggle with generalization under distribution shifts and novel tasks. While SET stabilizes outputs but often overfits, RL improves adaptability [12, and combining
both shows promise [17], making post-training vital for robust, aligned LLMs.
Test-time scaling. Scaling inference-time compute improves performance without retraining. Ap- proaches include beam search 16), best-of-N sampling (42), and MCTS [13), alongside prompting methods like CoT [50] and ToT [53]. Strategically allocated test-time compute has proven effective for enhancing reasoning in both unimodal and multimodal models. Building this, we apply CoT reasoning to 3D spatial reasoning by tne-tuning models to generate step-by-step rationales, providing both accurate and interpretable results.
3 SpatialReasoner
3.1 Overview
In this section, we introduce SpatialReasoner for explicit and generalizable 3D spatial reasoning.
Our SpatialReasoner features two key designs: (1) explicit 3D representations that serves as interface to support multi-stage spatial reasoning, i.e., 3D parsing, computation, and reasoning (see Figure 2 and (2) generalizable spatial reasoning from multi-stage training (see Figure 4)
In Section 3.2 we present the explicit 3D representations and describe how our model is trained to predict and to interpret the 3D representations for spatial reasoning. Then in Section 3.3 we discuss our training strategies, exploring standard supervised fine-tuning, reinforcement learning, as well as
3

---

Visual Question Toput bottled water on the left right side of him?
"On-demand" Computation
The cosinee similanity vector the left direcsion -0.85, corresponding to an angle 147.89 degrees.
Thinking Process
"On-demand" Parsing
Reliable Spatial Reasoning
The 3D locacion of the (0.0, 0.5, 1.0).
The angle than degrees, meaning that the bortled is on right of the person. Therefore, the final answer on the right.
direction person (-0.9, -0.1, 0.5).
Figure Overview our SpatialReasoner design. Our SpatialReasoner adopts explicit 3D representations as interface to enable coherent and reliable multi-stage spatial reasoning, i.e., 3D parsing, computation, and reasoning.
3D-aware process rewards. Lastly we introduce our 3D-aware data generation pipeline and different variants of training data used at different stages in Section 3.4
3.2 Learning Explicit 3D Representations
Despite improved spatial reasoning abilities achieved by 3D-aware VLMs, such as SpatialRGPT [ and SpatialLLM [33], and advanced proprietary models like Gemini 2.0 (T5), these methods lack explicit 3D representations and rely on natural language to perform 3D spatial reasoning. For example, Gemini 2.0 describes object poses as "facing towards the viewer and slightly to its right" and estimates 3D distances with phrases such as "far behind some other object". Such natural language descriptions are inefficient and often not accurate enough for complex 3D spatial reasoning.
Therefore, we propose integrating LVLMs with explicit 3D representations, such as 3D locations and poses, serve as an accurate and reliable interface shared across stages of 3D spatial reasoning (see Figure 2 Our SpatialReasoner can predict explicit 3D representations as intermediate results or take them as inputs to perform basic 3D computations or complex spatial reasoning tasks.
Explicit 3D representations. We define explicit 3D representations a calibrated camera 3D space, which is standard camera 3D space calibrated with the extrinsics of the camera. As illustrated in Fig- ure 3 the calibrated camera 3D space has its 2-axis aligned with the z-axis the 3D world space, and the origin on the z-axis is positioned close to the ground plane. Although estimating 3D object loca- tions is easier in original camera 3D space, adopting explicit 3D representations in calibrated camera 3I) space offers many advantages for subsequent spatial reasoning: (1) the z-coordinates directly correspond object heights, (2) estimating 3D spatial relation- ships such "above" and "below" is largely sim- plified, and (3) objects often are on plane parallel to the ground plane, which reduces many 3D spatial relationships to simpler 2D problems.
Original Camera 3D Space
Calibratte 3D Space
Figure 3: Comparison between original and calibrated camera 3D space. Our explicit 3D representations are defined within calibrated camera 3D space that simplifies subsequent 3D computation and reasoning.
as A unified interface for explicit 3D spatial reasoning. Explicit 3D representations interface enabling coherent and accurate 3D spatial reasoning across stages (see Figure 2). For 3D perception, the model predicts object locations and orientations 3D vectors, then estimates explicit distances or angles based on these predictions. Finally, the model aggregates explicit 3D information from earlier stages reason about and answer 3D spatial questions.
3.3 Training Strategies
Supervised fine-tuning (SFT). We adopt two-stage post-training strategy to equip the model with explicit, generalizable 3D spatial reasoning capabilities. In the first stage, SFT serves as critical initialization step, aligning the pre-trained LVLM with curated 3D-annotated datasets. By

---

SFT for Explicit 3D Rep-
Reinforcement Learning
Qwen2.5-VL
3D Perception
3D Calculation
Reasoning
Rule-based Reward (correct solutions & formar)
SpatialReasoner
(30 lacation
(3D disancc
(exploration & exploitation)
orlentation)
angle)
Figure 4: Overview of our Seattile training, We adopt a multi-stage training strategy: In Stage I, we apply supervised fine-tuning (SFT) equip the LVLM with explicit 3D representations; then in Stage II, we leverage reinforcement learning (RL) to develop robust and generalizable 31> spatial reasoning built on explicit 3D representations.
optimizing a maximum likelihood objective over paired input-output sequences, the model learns to identify objects in 3D scenes and predict explicit 3D representations such as object locations and orientations. This structured supervision enables the model to expose interpretable intermediate reasoning traces during spatial computations -..... calculating relative distances and angles. However, due to its reliance on static demonstrations. SFT shows limited capacity to generalize beyond observed reasoning patterns. As shown in Section 4.3 SFT-trained models tend memorize spatial templates in the training set and struggle with novel compositions or combinatorial variations in 3D queries.
Reinforcement learning (RL). To overcome this limitation, we further post-train the SFT-initialized model using reinforcement learning (RL) with rule-based rewards. Treating the spatial reasoning task as sequential decision process, each reasoning step framed as an action within a Markov Decision Process (MDP), and policy gradients are optimized using GRPO [4]]. Importantly, we integrate reward scheme that provides structured reward signals reflecting both the correctness of final answers and the coherence of intermediate 3D computation steps. This enables multi-turn optimization where the model learns revise inaccurate inferences and explore alternative reasoning paths. Empirically, this RL. stage substantially improves generalization to diitr--disttibttion 3D spatial questions (Table B3). particularly in settings involving multi-object arrangements. The combined SFT+RL strategy therefore balances response formatting and perceptual grounding with adaptive reasoning and robustness, making it especially suited for complex 3D spatial understanding.
Reward and policy optimization design. To train SpatialReasoner via reinforcement learning, we design composite reward scheme capturing both answer correctness and reasoning quality. For outcome rewards, we use an accuracy reward-aligned with the multiple-choice evaluation metric from MMBench 28 which gives positive feedback only when the model selects the correct answer, and a format reward [7] that encourages structured, readable outputs. We also explore process rewards to assess whether models can learn structured reasoning without SFT. These include reasoning steps reward, which promotes use of structured indicators (e.g.. "First", "Next"), and 3D-aware process reward that checks for necessary spatial terms like distance or orientation. The reward signal is then calculated through the accuracy of the presence of each term. Though not used in the final model, these rewards serve as diagnostics for emergent reasoning behavior. Finally, we ablate the KL divergence term in GRPO and find that it harms training accuracy despite stabilizing output lengths (Section 4.3) We thus remove in our final setting.
3.4 Training Data
To enable LVLMs to predict and reason with explicit 3D representations and to post-train LVLMs to solve various challenging 3D spatial reasoning questions, we generate a series of 3D-aware training data. We extend the data generation pipeline in 13 Our process begins with generating 3D pseudo-annotations, followed by optional human verification, and ends with constructing various VQAs and chain-of-thought reasoning based on the 3D pseudo-annotations.
Pseudo 3D ground-truths. We extend the 3D pseudo-annotation pipeline proposed in 9 [333 and generate various 3D annotations, such object category, 3D location, and 3D pose, on unlabeled images from the OpenImages dataset [22 Based on the object-level 3D annotations, we then apply rule-based methods to derive ground-truth labels for range of 3D spatial relationships. Despite significant progress in visual foundation models for segmentation (38) metric depth estimation (52), and object pose estimation [34], we notice many factual errors in the generated 3D ground-truth.
5

---

Method
Mean
Height
Location
Orientation
Multi-Oul Multi-Object
Open-Sourced Generalist
LLaVA-v1.5-7B 26
38.1
39.1
46.9
28.7
34.7
LLaVA-Next-8B 24
48.4
50.6
59.9
36.1
43.4
Cambrian-1-808 45
42.2
23.2
53.9
35.9
41.9
Swen2.55VLL33B--nstrree
4
43.9
45,2
56.8
35.7
35.7
Qwen2.5-VL--7B-Instie
48.4
44.1
62.7
40.6
40.5
Qwen2.5-VL-72B
54.9
53.3
71.0
43.1
46.6
Open-Sourced Specialist
SpaceI.L.aVA 40
42.0
49.3
54.4
27.6
35.4
SpatialBot 8
41.0
40.4
54.4
31.9
33.5
SpatialLIMM 33
44.8
45.8
61.6
30.0
36.7
SpatialRGPT
I
32.7
55.9
39.0
27.8
20.0
SpatialRGPT w/ depth
48.4
55.9
60.0
34.2
42.3
Proprietary
GPT-40-mini
39.7
44.3
52.4
21.0
36.5
GPT-4o
44.2
53.2
59.6
21.6
39.0
Claude 3.5 V Sonnet
48.2
53.5
63.1
31.4
41.3
Gemini 2.0 Flash
49.8
49.7
68.9
32.2
41.5
Gemini 2.0 Flash (thinking)
51.1
53.0
67.1
35.8
43.6
Qwen
52.0
45.1
70.7
37.7
44.8
Ours
Spatial ReasoneReas
54.0
46.4
67.3
48.4
47.2
PpatialReasoner-S/ SpatialReasoner
58.3
51.9
73.5
50.7
50.3
60.3
52.5
75.2
55.2
51.8
Table 1: Comparison with previous state-of-the-art methods on 3DSRBench [31]. Our Spa- tialReasoner outperforms previous open-source and proprietary methods challenging 3D spatial reasoning problems in 3DSRBench.
particularly when mistakes (e.g., missing objects or inaccurate object poses) propagate through the data pipeline. Therefore, we adopt a series of aggressive filtering steps ensure the quality of our training data, including: (1) removing images with densely cluttered scenes, (2) excluding object categories that are difficult to segment or estimate pose for, and (3) discarding boundary cases that could lead to ambiguity.
Human verification. Despite leveraging state-of-the-art visual foundation models in our data generation pipeline and applying multiple filtering strategies, the 3D pseudo-annotations remain susceptible to factual errors. Specifically, small issues such as missing objects inaccurate 3D pose predictions propagate through later stages of the pipeline, leading to factual errors in the final spatial relationship pseudo-anmotations. To assess the impact of data quality, create smaller but higher-quality dataset by manually verifying the correctness of the 3D pscudo-annotations
Training data variants. Based on the obtained ground-truth, we can generate different variants of data for fine-tuning. Specifically, we consider the following: (1) Basic3n-QQ consists of basic 3D perception and 3D computation question-answeriig representations without training various 3D spatial relationships considered in downstream tasks. (2) SR-QA contains visual question-answerrng pairs about various 3D spatial relationships, following previous 3D- 3D-aware datasets [|]] 33). (3) SR-CoT extends SR-QA and comprises chain-of-thought reasoning with explicit 3D representations. Questions are answered in a step- hop--y-ttep manner.
data. This can be used learn explicit 3D
4 Results
4.1 Experimental Setup
Baselines. We compare our ReatialReasoner with the following three types of baseline models. (1) Open-sourced generalists; such as LLaVA (26). Cambrian-1 [55, and Qwen2.5 [4] that are trained on

---

CV-Bench-3D
GQA
Method
Depth
Distance
Mean
Choose
Compare Logical
Query
Verify
Qwen2.5-VL-7H
82.5
83.2
58.8
82.7
71.5
78.9
40.6
82.5
SpatialReasoner Zero
77.5
81.8
60.2
81.6
67.9
79.2
43.8
82.1
SpatialReasoner-SFT
85.2
71.5
62.0
82.8
72.2
81.4
45.5
83.2
SpatialReasoner
87.3
73.3
61.8
83.2
81.1
71.8
45.2
82.9
Table 2: Performance on CV-Bench-3D and GQA. Our SpatialReasoner also improves the spa-
tial reasoning performance on GQA [19] and depth-related questions in CVBench-3D (45]. For distance-related questions, unlike Qwen2.5 that exhibits excessive dependence on 2D shortcuts, our SpatialReasoner employs rigid 3D spatial reasoning and achieves compelling performance. Regarding the performance on distance-related questions, see Section 4.2 and Section C for detailed discussions.
In Distribution Location
Novel Multi- Multi-Object
Method
Mean
Height
Orientation
Swen2.5-V1-7B--nsteee
48.4
44.1
62.7
40.6
40.5
SpatialReaasnner-Z5
53.7
40.6
68.4
50.2
46.6
SpatialRecsoner-SFFT
52.2
44.9
69.5
48.9
40.0
SpatialReasoner
56.4
52.5
72.6
54.1
43.4
Table 3: Evaluation of generalization ability by finetuning on simpler (in-distribution) 3D spatial reasoning questions and evaluating on complex (novel) questions types unseen during training. Our SpatialReasoner Zero and SpatialReasoner demonstrates superior zero-shot generalization, indicating RL fosters more robust and flexible reasoning than SFT.
general vision-language data. (2) Open-sourced specialists: We evaluate Spacel.LaV 40 (a public re-implementation of SpatialVL.M [9). SpatialBot 8 that enhances fine-grained spatial reasoning and robot control with RGB and depth images, and SpatialLLMM [3 that fine-tunes LLLVV model with multi-stage 3D-informed training. Note for fair comparison, we evaluate SpatialBot with RGB that were trained on abundant web-scale data and for Gemini 2.0 15), additional 3D-aware post-training.
inputs only. (3) Proprietary models such GPT-4o [I] and Claude 3.5 3
Evaluation benchmarks. We evaluate spatial reasoning abilities of various models three spatial reasoning benchmarks. 3DSRBench [31] is a comprehensive 3D spatial reasoning benchmark with 2,100 questions and studies various 3D awareness and reasoning abilities with robust evaluation setup. CVBench [45] is vision-centric benchmark that assesses models at classic vision tasks with a range of 21D and 3D understanding VQAs. In this work, we focus exclusively 3D-related questions, ie., CVBench-3ID, as 2D left-right relationships can lead to ambiguity with 3D left-right questions considered 3DSRBench. GQA [19] a widely adopted benchmark that studies visual reasoning and compositional question answering on range of spatial relationships between objects.
4.2 Advancing 3D Spatial Reasoning
to assess 3D perception, computation, and reasoning. As shown in Table I ReatialRe achieves new state- of-the-art 60.3% mean accuracy on 3DSRBench, outperforming Gemini 2.0 Flash (49.8%) ) and Claude 3.5 Sonnet (48.2%). It shows notable improvements on Location (+6.3%) and Orientation (+ (+14.6%) questions, and achieves 51.8% (+8.2%) on Multi-Objec reasoning relative to the second-best model, indicating stronger 3D perception and complex spatial understanding.
We evaluate SpatialReasonee on 3DSRBench 331, CVBench-3D 45 and GQA 19
On CVBench-3D and GQA (Table 2). SpatialReasoner achieves 87.3% on depth-related questions in CVBench-3D, and improved performance on GQA's Compare category, further supporting its multi-object reasoning capability. These gains validate our multi-stage training: SFT equips the model with explicit 3D representations, while RL enhances adaptive reasoning and generalization.
Despite this, we observe performance drop on CVBench-3D distance questions, in contrast to sub- stantial gains similar 3DSRBench cases ('multi-object-close-to 34.3% 70.9%). We attribute

---

Method
Mean
Method
Mean
1ann2.5-VL-7B-Insttee
48.4 .4
Swen2.5-VL-7B--nstruee
48.4
SpatialReassoer-SFT
58.3
SR-SFT
58.3
SpatialReassoer-SFFT (+HQ SFT)
54.7
SR-SFT (w/o exp. 3D rep.)
51.9
SpatialReessner-Zeero
54.0
Table 4: Comparisons between SpatialRea- soner with and without explicit 3D repre- sentations and CoT reasoning. Results high- light the benefits of explicit 3D representa- tions as interface to support enhanced 3D spatial reasoning.
SpatialReasoner-ZZoro (w/ KL)
52.4
SpatialReasoner-ZZrro (w/ 3D Rwd)
54.6
Table 5: Ablation study on various design choices in RL and SFT. Notably with 3D-aware rewards. SpatialReusoner-
produces coherent chain-of of-tthtghtt reasoning on explicit 3D repre- sentations and improves benchmark performance.
this gap to shortcut-driven 2D biases in CVBench-3D. Unlike existing LVLMs, SpatialReasoner relies on explicit 3D reasoning, offering greater robustness to such spurious cues (Section Cp.
4.3 Analyses and Findings
Generalization abilities. While SFT stabilizes outputs, it often overfits training distributions, limiting generalization to out-of-distribution (OOD) variations [2]. In contrast, outcome-based RL promotes transferable reasoning strategies and perceptual adaptability. To examine this, we compare SpatialRcasoner-STi (SFT-only), SpatialReasoner-Zerr (RL-only), and SpatialReasoner (SFT+RL) on multi-object reasoning, with all multi-object data removed during training.
As shown Table 3, SpatialReassonr--Zer achieves 46.6%, outperforming SpatialReasoner- (40.0%) and SpatialReasoner (43.4%), confirming RL's superior zero-shot generalization ability. Although SFT+RL improves over SFT, it does not fully match RL-only performance, indicating that RL fosters more flexible reasoning than memorization-proot prroaa SFT.
When multi-object training data are available (Table SFT attains 50.3% via pattern matching, but drops 40.0% when data withheld, whereas RL-based models retain stable performance (46.6% vs. 47.2%) Mean accuracy further supports this: SpatialReasoner- -erroorrrrrZooooooo maintains performance (53.7% vs. 54.0%), while SpatialReasoner-SFT degrades significantly (58.3% to 52.2%), reinforcing that RL enables more robust generalization.
Compositional reasoning. We assess whether structured 3D reasoning can emerge from process re- wards alone, without curated COT. As shown in Figure SpatialReasoner-Zerr trained with 3D-aware rewards generates coherent multi-step reasoning grounded in 3D cues, improving interpretability over the baseline relying on shallow shortcuts. This demonstrates the model's ability to synthesize novel compositional reasoning strategies from basic 3D perception and computation skills. Quantitatively, achieves performance gain from 54.0% to 54.6% (Table 5), indicating that outcome-drine -dinen reward shaping can induce coherent reasoning without caratee COT. Nonetheless, for the final SpatialReasoner, we use SFT to guide reasoning traces, as it outperforms process rewards alone.
Scaling of training computation. We ablate on the scaling of training computation for SpatialReasoner-SFF and SpatialReasoner in Figure Results show that with more training steps, SpatialReasoner-SFT starts to overfit and exhibit decreased performance, while SpatialReasoner trained with RL retains a stable and competitive performance.
Scaling of training data. 3D-related tasks often lack high-quality data with 3D (pseudo-Jamnotations due to the cost and expertise required. While synthetic or pseudo-labeled data is more scalable, they introduce noise and domain gaps. To study data scaling, we mix 1.2K verified and 24K unverified samples and compare SpatialReasoner with and without RL. shown in Figure 6 RL performs best with only the 1.2K verified data, whereas DaatialReasonerSFFT benefits from more unverified data. This suggests that SFT is more tolerant to noisy pseudo-annotations while RL favors fewer but higher-quality samples.

---

60
60
3DSRBench (mean)
55
3DSRBench
55
50
SpatialReasoner (SFT)
50-
Spatial Reasoner (SFT)
SpatialReasoner (LFT+RL)
SpatialReasoner (SFT+RL)
1000
4000
10000
20000
1000
4000
10000
20000
Training Steps
Number Training Samples
Figure Scaling training computation. SpatialReasoner-SFT may overfit to training data while SpatialReasoner trained with RL retains a stable and competitive performance.
Figure Scaling of training data by mixing 1.2K human-verifiet with 24K unverified data. Results show SFT remains a compelling choice when scaling with potentially noisy training data.
Ablation study on explicit 3D representations. To study the importance of explicit 3D represen- tations and step-by-step 3D computations, we train a variant of SpatialReasoner on the same data but without explicit 3D representations. Results in Table 4 show that explicit 3D representations can enhance 3D spatial reasoning by a wide margin.
Ablation study on KL divergence. We evaluate KL divergence regularization in GRPO and find harms learning in our setting. As shown in Figure adding KL causes training to collapse, while removing yields stable improvements. The shorter completions without KL may reflect mitigated length bias (29), improving token efficiency. As shown in Table 55 SpatialReasoner- Zeroo reaches 54.0% accuracy without KL vs. 52.4% with it. We thus omit KL for better spatial reasoning.
Ablation Study on SFT+RL vs. SFT+SFT To verify the performance gains are not merely from additional data exposure, we compare sequential SFT+RL against SFT+SFT. From Table while SpatialRcasoner-SPF achieves 58.3% accuracy, applying second round of SFT (SFT+SFT) degrades performance to 54.7%. In contrast, RL after SFT (SFT+RL) improves accuracy to 60.3%. These results indicate repeated SFT exacerbates overfitting training patterns, whereas RI effectively builds on the structured outputs from SFT to promote adaptive and generalizable 3D spatial reasoning.
4.4 Interpreting Failure Modes
Explicit 3D representations enhance both spatial reasoning and interpretability [46). We categorize 3D spatial reasoning into two stages: 3D perception (parsing key 3D information from the image input) and 3D reasoning (computing 3D metrics and deriving final answer). To quantitatively analyze model behavior, we manually verify 300 questions with human-verified verified 3D answers. For 3D perception, we evaluate questions on object location and orientation; for 3D reasoning, models estimate distances, depths, and angles from 3D inputs, with accuracy (within /6) for angles and mean error for positions. As shown in Table 7 models reason more accurately than they perceive-indicating most VQA errors stem from flawed 3D perception, consistent with our qualitative visualizations. Additionally, RL slightly degrades perception and reasoning accuracy, likely due to rewards focusing only on final answers and reasoning format, not intermediate 3D outputs.
Conclusions
In this paper, we presented SpatialReasoner that performs explicit and generalizable 3D spatial rea- soning by predicting intermediate 3D representations across perception, computation, and reasoning stages. Our two-stage post-training pipeline with SFT followed by RL achieves state-of-the-art results on multiple benchmarks while generalizing well to novel tasks. By analyzing model behavior, we show that explicit 3D reasoning improves both accuracy and interpretability, and reveals key limita- tions in 3D perception. This work highlights the value of structured reasoning and adaptive training for robust 3D understanding, paving the way for future research in multimodal spatial reasoning.
Acknowledgements. W.M. and Y. acknowledges support from the Office of Naval Research with N00014-23-1 --004--2--1-241 and ARL award W91INF232008..

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku.
[3] Anthropic.
Claude 3.5 sonnet.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar 2rXi 13923, 2025.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0553 2022.
[6] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22/2.08073 2022.
[7] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: A large benchmark and model for 3D object detection in the wild. In CVPR, Vancouver, Canada, June 2023. IEEE.
[8] Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642, 2024.
[9] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[10] An-Chieh Cheng, Yandong Ji, Zhaojing Yang. Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Biyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. arXiv preprint M.Xiv:2412.04455: 2024.
[11] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint 2XXx:2406.00588. 2024.
[12] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint 17X1v:2501.17161, 2025.
[13] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
[14] Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision- language models for robotic manipulation. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.
[15] Google. Introducing gemini 2.0: our new ai model for the agentic era, 2024. Accessed: Dec 2024.
[16] Alex Graves. Sequence transduction with recurrent neural networks. arXiv:1211.371,, 2012.
arXiv preprint
10

---

[17] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 122448881..29,, 2025.
[18] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.0// 0/652, 2024.
[19] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[20] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
[21] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.213211 2025.
[22] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[24] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, May 2024.
[25] Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma. Benjamin Van Durme, and Alan L Yuille. Super-clevr: virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14963-14973, 2023.
[26] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction uning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[28] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all- player? In European conference computer vision, pages 216-233. Springer, 2024.
[29] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
[30] Bruce P Lowerre and B Raj Reddy. Harpy, a connected speech recognition system. The Journal of the Acoustical Society of America, 59(S1):597-S9) 1976.
[31] Wufei Ma, Haoyu Chen, Guofeng Zhang, Celso M de Melo, Alan Yuille, and Jieneng Chen. 3dsrbench: comprehensive spatial reasoning benchmark. arXiv preprint arXiv:2412.0777 2024.
11

---

[32] Wufei Ma, Angtian Wang, Alan Yuille, and Adam Kortylewski. Robust category- 6d pose estimation with coarse-to-fine rendering of neural features. European Conference on Computer Vision, pages 492-508. Springer, 2022.
[33] Wufei Ma, Luoxin Ye, Celso M de Melo, Alan Yuille, and Jieneng Chen. Spatialllm: A compound 3d-informed design towards spatially-intelligentt large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2025.
[34] Wufei Ma, Guofeng Zhang, Qihao Liu, Guanning Zeng, Adam Kontylessii, Yaoyao Liu, and Alan Yuille. Imagenet3d: Towards general-purpose object-level 3d understanding. Advances in Neural Information Processing Systems, 37:96127-966 19119999999, 2024.
[35] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations, 2019.
[36] OpenAI. Introducing openai and 04-mini, 2025, Accessed: Apr 2025.
[37] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01153328-537444 2023.
[38] Nikhila Ravi, Valentin Gabeur, Yuan- Tinn-TTingg Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything images and videos. arXiv preprint arXiv:2408.000 00784,00144 2024.
[39] Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2412.07755, 2024.
[40]
remyxai. Remyxai/spacellava, 2024.
[41] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint :xxx:2442.033300 2024.
[42] Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20229. 2024.
[43] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Bapt-tae Alayrac Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503. 20020, 2025.
[44] Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, al. A survey on post- post-training of large language models, arXiv preprint arXiv:2503.0607. 2025.
[45] Peter Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-l: A fully open, vision-centric exploration of multimodal lms.. Advances Neural Information Processing Systems, 57:87310-87356 2024.
[46] Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan Yuille. 3d-aware visual question answering about parts, poses and occlusions. Advances in Neural Information Processing Systems, 36:58717-58755 2023.
[47] Xingrui Wang, Wufei Ma, Angtian Wang, Shuo Chen, Adam Kortylewski, and Alan Yuille. Compositional 4d dynamic scenes understanding with physics priors for video question answer- ing, arXiv preprint arXiv:2406.000 246220022 2024.
12

---

[48] Xingrui Wang. Wufei Ma, Tiezheng Zhang, Celso M de Melo, Jieneng Chen, and Alan Yuille. Pulsecheck457: A diagnostic benchmark for comprehensive spatial reasoning large multi- modal models. arXiv preprint arXiv:2502.08633, 2025.
[49] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.001 09600............. 2021.
[50] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughu prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824-24837, 2022.
[51] Jihan Yang. Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14177, 2024.
[52] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875- 21911, 2024.
[53] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1809-11822, 2023.
[54] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu. Lili Yu, al. Lima: Less is more for alignment, Advances in Neural Information Processing Systems, 36:55066-55011, 2023.
13

---

A Limitations
Our system has a few limitations. First, despite the use of reinforcement learning (RL), supervised finetuning data with chain-of thain-of-thought reasoning and explicit 3D representations remains crucial as warm-up. Training with RL alone-i.e., SpatialReasoner-ZZ Merener---ro-falli feil-fails to produce high-quality reasoning processes. Second, SpatialReasoner consistently performs explicit multi-step 3D reasoning, regardless of the difficulty of the problems. Ideally the model should adopt a hybrid strategy- commen sense and visual cues for simpler questions while reserving explicit 3D) computation and reasoning for more challenging problems. This would retain many strong and generalizable knowledge from the base Qwen2.5-VL model while improving inference efficiency.
B Implementation Details
We train SpatialReasoner using different combinations curated datasets and training objectives. Starting from the Qwen2.5-VL-7B [4] base model, we first apply SFT with 24k curated SR-CoT data alongside 24k randomly sampled LLaVA [27] data, resulting in SpatialReasoner-SFT Next, further train SpatialReassoer-SFF with RL using 1.2k SR-QA examples, leading to final SpatialReasoner. For comparison, if we instead fine-tune SpatialRRasoneer SFT with SFT on the same 1.2k SR-QA set, we obtain SpatialReasoner SFT (+HQ SFT), serving an ablation study baseline. In parallel, we directly fine-tune the base model with RL. using the 1.2k SR-QA data without prior SFT, resulting in SpatialReasonnr-ZZeno Additionally, investigate if explicit 3D perception ability can enable the model to self-organize reasoning trajectories under process rewards, we train the base model with SFT using 12k Basic3D-QA data alongside 12k randomly sampled LLa data before applying RL training on the 1.2k SR-QA data, yielding SpatialReasoner-ZZero (w/ 3D Rwd).
We conduct all training experiments using 4xNVIDIA H100 80GB HBM3 GPUs. For SFT, we train the model for 10 epochs (approximately 20K steps with a batch size of 6) on the combined 24k SR-CoT and 24k LLaVA datasets. For RL training, train for 100 epochs (approximately 13K steps with a batch size of 12) on the 1.2k SR-QA dataset, using 1 GPU with vLLM [23] for efficient inference acceleration. For the experiments withholding multi-object training examples, we double the number of training epochs to compensate for the reduced training set size. We set the learning rate to 5e-6 for SFT and 5e-7 for RL, both following cosine learning rate scheduler with warm-up ratio of 0.1. In the KL divergence ablation study, we set the KL penalty weight to 0.04, We monitored the model every IK training steps and reported results based the best-performing checkpoint.
C 2D Reasoning as a Shorteut
As shown in Table D and Table 2 our SpatialReasoner outperforms previous open-source and proprictary models on 3DSRBench B1) and achieves notable improvements on GQA [19 (from 58.8% to 61.8%) and depth-related questions in CVBench-3D [45] (from 82.5% to 87.3%). However, if we focus multi-object 3D distance-related questions in CVBench-3D 45] and 3DSRBench BI. we observe contradictory results: SpatialReasoner achieves a substantial improvement of 21.5% on 3DSRBench, but exhibits notable performance drop of 9.9% on CVBench-3D (see Table
We attribute this discrepancy to the abundant shortcuts in distance-related questions in CVBenhh-DD. From the qualitative examples in Figure 7 the provided 2D bounding boxes in CVBench-3D can exploited as shorteuts to answer the 3D spatial reasoning question. Rather than reasoning about 3D distances between objects, we can easily derive the correct answer by comparing the 2D distances between the red and blue boxcs and between the red and green boxcs. Meanwhile, 3DSRBench is human-collected VQA dataset and manually avoid such spurious correlation, e.g., "objects closer in 3D space are also closer in 2D image plane". For the 3DSRBench example in Figure the bounding box of the dog is actually closer to the bounding box of the man in black that farther away in 3D space.
Given the 2D bounding box annotations in CVBench-3D and 3DSRBench, we derive a simple heuristic to answer distance-related spatial reasoning questions by simply comparing L.2 distances between 2D centers of object bounding boxes. We achieve an 80.2% accuracy on distance questions in CVBench-31 and 34.3% in Bench This demonstrates that baseline models such as Qwen2.5- VL are largely exploiting 2D spatial reasoning a shortcut to answer complex 3D spatial
14

---

CVBench-3D (Distance)
3DSRBench (Multi-Object Closer to) (bounding box added for visualization)
Question: Estimate the real-world distances between objects this image. Which object closer to the lamp (highlighted by red box), the sofa (highlighted by blue box) the television (highlighted green box)?
Question: Consider the real-world locations of the objects. Which is closer to the dog, the person in red or the person in black?
Answer: Person in red.
Answer: Sofa.
Figure 7: Comparison between (multi-object) distance-related questions in CVBench-33 45 and 3DSRBench B1..
CVBench3D Distance
3DSRBench
Method
multi-bbject-loser-tt
2D Heuristic
80.2
34.3
Qwen2.5-VL -L-1BB---V--BB---- -LL-7B-Instruct
[4]
83.2
49.4
SpatialReasoner
73.3
70.9
Table 6: Comparison between Qwen2.5-VL 4 and SpatialReasoner (multi-object) 3D distance-
related questions in CVBench-3D and 3DSRBench [31].
reasoning questions. Meanwhile, with our 3D-aware post-training, our SpatialReasoner adopts explicit 3D representation for 3D spatial reasoning questions. The trade-off between exploiting 2D shortcuts and adopting explicit 3D representations results in slightly lower performance of SpatialReasoner on test data with abundant spurious correlations, but more importantly, leads to robust and largely improved performance on challenging real-world dataset.
P Visual cues or explicit 3D reasoning? LVLMs that exploit 2D reasoning as shortcuts may achieve improved performance test data with abundant spurious correlations (e.g., distance questions in CVBench-3D). However, they cannot genuinely solve 3D spatial reasoning problems and fall far behind SeatialRe that builds explicit 3D spatial reasoning when tested on challenging real-world problems in 3DSRBench.
D Additional Results
We present the following additional results:
1. Table We study the failures modes of SpatialReasoner. Results show that 3D computation is more accurate than 3D perception, indicating that most VQA emrors stem from flawed 3D perception.
2. Figure 8 Qualitative comparisons between different variants of SpatialReasoner.
3. Figure Ablation study results on KL divergence.
15

---

User
Consider the real-world 30 locations and orientations of the abjects. Which side bus is facing green board? right, front, left, back.
SpatialReasoner-Ste w/o explicit representation
D. back.
SpatialReasoner-222 <think-The board on the front of the wusc://hinbo-anssesd////sssssss
SpatialReaonser-Zo -eero w/ 3D-aware process reward
<thinkofirst, will determine the frant direction of the bus, The front direction of the bus is 0.17, 0.25, -0,96). Next, will calculate the 30 location the bus the green board. The 30 location of the bus is (3.97, 2.35, 17.15), and the location of the green board is (1.75, 1.65, 13.95). Then, will calculate the vector from the bus to the green board, The vector is (-1.70, 0.99). that,
is (-0.92, 0.37, -0.29. Finally, will check the angle between the front direction and the green hoard and compare with the angle between the left direction and the green board, The angle the front direction and green direction and the green board is 0.33. Therefore, the correct answer is the front side of the bus facing green board.<//te board.x/thinlo httrk///tioooonnseees fronts/answer>
angle
left
SpatialResonser-SFT
The location bus (-4.3,
The location green board
22. 1). The vector from the bus to the green board is hence (9.2. 0.9, 4.4). The left direction of bus (-0.7, 0.0, -0.7). cosine similarity between
vector pointing and the left direction is -0.94, corresponding to angle 160.87 degrees. Thus the angle between the vector pointing to the grean board and the right direction is 19.13 degrees. The pointing
green
Green Board
bus [0.7, the green board and the front direction 0.34, correspanding an angle of 70.30 degrees. Thus angle between degrees. Among the four directions, the smallest degrees. Thus right side of the bus is facing the green board. Therefore, the final answer is A. right.
direction
similarity
vector pointing to the green board and the back direction 109.70 is right direction, with angle of 19.13
Left
SpatialReaonser
<thinkoThhe 3D location the bus (-3.7, 2.1, 14.7). The 3D location the green board is (2.2, 3.1, The vector from the of the bus is (-0.9, D.0, -0.4). The cosine of the angle between the vector pointing to the green board
Green Board)
the green board hence (5.3,
left direction
left direction is -0.99, corresponding to an angle of 172.09 degrees. Thus board and the right direction 7.91 degrees.
angle between the vector pointing the The front direction of the bus (0.1, 0.0, -1.0). The cosine of the angle between the vector pointing to green board degrees. Thus the angle between the vector pointing the green board and the back direction is 81.53 degrees. Among the four angles, the smallest is the right direction (7.91 degrees), which question.</thine wwt.://nwwaaswerriggineaaaaeee
front
-0.15, corresponding an angle
Foout
Figure 8: Qualitative comparisons. Our explicit 3D spatial reasoning improves interpretability over the baseline that relies on shallow shortcuts.
E Open Access
All code, data, and models will be available on our project page to support reproducibility and benefit the research community.
1. Codebase for our full 3D-aware data generation pipeline.
2. Codebase for our SFT and RL finetuning.
3. Synthesized 3D-aware training data.
4. Weights our SpatialReeaoneraad SpatialReasoner-SFTT.
16

---

3D Perception
3D Reasoning
Method
Orientation (t)
Location (1)
Angle (t)
Distance (L)
Depth ())
SpatialReasoner-SFF 35.5
0,91
55.0
0.17
0.13
SpatialReasoner
31.0
1.05
52.5
0.19
0,25
Table 7: Studying failure modes of SpatialReasoner. We observe that 3D reasoning can estimate angles, distances, and depths lot more accurately than 3D perception that predicts orientations and locations from visual features.
1.0
Completion Length
100
Accuracy
0.6
SR Zero
60
SR-Zero w/ KL
0.4
SR-Zero SR-Zero
2000
4000 6000
12000
2000
4000
8000
12000
Training Steps
Training Steps
(a) Accuracy of SpatialReasoner Zero w/ and w/o KL divergence.
(b) Completion length of SpatialReasoner Zero w/ and w/o KL divergence.
Figure 9: Training curve between SpatialReessoner-Zero and Spattal/eeasone--Zeroo w/ KL.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introduction supported by quantitative results Section and qualitative results Figure
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
18

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all information needed to reproduce the main results Section B including dataset, hyperparameters, and training setup.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
All our code and data will be made public detailed in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpg//udde/ooeeSubiiiiiiie Stt////ooeSSeiaaaeeeeeeee SooddeSuumiissinoooiioo.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We present the training and test details in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to limited computational resources, we were unable perform multiple runs for each experimental settings that are necessary for statistically robust error estimation. However, we re-run our main method with different random seeds and obtain standard deviation of 0.20, which shows that our experimental results are sound and significant. Lastly we will open- our code and data to support public reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are detailed in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: Our work builds on public academic datasets and codebases and we carefully follow the NeurlPS Code of Ethics in each stage of our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work is a foundational research on 3D spatial reasoning, enabling large vision language models reason in 3D space. Our work does not have direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper studies the 3D spatial reasoning of large vision-language models. Our data and models do not have risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For all the codebase and data used in this work, properly cite the creators in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
The paper introduced new code, data, and models that will be released to public upon paper acceptance. assets are well documented and will actively maintained in the future.
Justification:
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hettps///eerrrpp.ccc//oofrrrerre cc///ooferreeee///22////... for what should or should not be described.
24


Reviews:
[{"review_id": "y17TnKmgKM", "reviewer": "Reviewer_4UW2", "summary": "This paper proposes a LVLM called SpatialReasoner especially designed to solve 3D spatial reasoning problem based on explicit 3D representations. Besides, this work constructs the corresponding training data for SFT and RL training. The paper comprehensively analyzes the proposed model, including the effectiveness of explicit 3D representations and the multi-stage training strategy.", "questions": "1. I'm curious about the comparison between Qwen2.5 VL and SpatialReasoner when both are given ground truth 3D locations for reasoning to answer questions, thereby verifying whether SpatialReasoner actually learns something better after SFT and RL training.", "limitations": "1. As the authors state, SpatialReasoner consistently performs explicit multi-step 3D reasoning, which is not efficient.\n2. In more complex scenarios like multi-image inputs, it may be difficult for the model to generate such explicit 3D representations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is well-written.\n2. The idea of reasoning with explicit 3D representations is reasonable, and the comparison with Gemini 2.0's reasoning is especially convincing regarding its necessity.\n3. The experiments on RL training are interesting and reveal some intriguing findings.\n4. The open access commitment of this work, especially the RL training code and data, will be beneficial to the community.\n\n\nWeakness:\n1. The performance drops on CVBench-3D distance questions remains confusing, though the authors claim this can be attributed to shortcut-driven 2D biases in CVBench-3D. In my opinion, reasoning in 3D should not perform worse in such simpler scenarios. I think this can be attributed to errors in predicted 3D positions. I would like to see whether the model can outperform Qwen2.5 VL given ground truth 3D positions.\n2. The reward design for RL training is ambiguous, especially for the 3D-aware process reward. It would be better to express it with a clearer formula.", "was_revised": true, "final_justification": "Thanks for the detailed response which has addressed most of my concerns. Based the submission and the response, I am happy to raise my score to 5.", "rebuttal": "We thank the reviewer for the time and effort to review our paper. We appreciate the positive feedback, mentioning that our paper is \"well-written\", our method of involving explicit 3D representations \"reasonable and convincing\", and our experiments \"interesting and revealing intriguing findings\". We discuss new results and address detailed concerns below. We will incorporate all valuable feedback and suggestions in our future revision.\n\n**Inference with ground-truth 3D locations.** We agree with the reviewer that inferencing Qwen2.5-VL and our SpatialReasoner with groundtruth 3D information provides valuable insights into the models. Therefore, we obtain the ground-truth 3D object locations from the Omni3D dataset and append the information to the end of the question as part of the prompt.\n* Results show that with 3D ground-truths, our SpatialReasoner achieves an almost perfect performance of 97.9% while Qwen2.5VL performs similarly to the standard evaluation. This demonstrates that Qwen2.5VL does not rely on genuine 3D spatial reasoning to answer depth- and distance-related questions. It is likely making predictions by exploiting 2D shortcuts. In contrast, our SpatialReasoner learns to ground objects in 3D space and perform reliable spatial reasoning on 3D information.\n* Moreover, these results suggest that shortcuts in CV-Bench-3D can lead to deceptively strong performance, undermining meaningful evaluation. Truly solving the spatial reasoning challenges in CV-Bench-3D requires both a robust visual module for accurate 3D parsing and a powerful reasoning model capable of genuine spatial understanding.\n\n| evaluation | model | CV-Bench-3D | depth | distance |\n|:-|:-|:-:|:-:|:-:|\n| standard | Qwen2.5VL | 82.8 | 82.5 | 83.2 |\n| standard | SpatialReasoner | 80.3 | 87.3 | 73.3 |\n| with 3D GT | Qwen2.5VL | 82.5 | 83.3 | 82.8 |\n| with 3D GT | SpatialReasoner | 97.9 | 99.5 | 96.3 |\n\n**Clarification on the reward designs of reinforcement learning.** We thank the reviewer for raising this concern. We introduce the motivation and design of our reward functions as follows, and will rewrite this section with detailed formulas and examples in our revision.\n\nIn this work, we consider three types of reward functions: accuracy reward, format reward, and 3D-aware process reward.\n* The accuracy and format rewards follow the DeepSeek-R1 design that checks if the predicted final answer matches the ground truth and if the format of the answer follows a reasoning step \\<think\\>...\\</think\\> and an answering step \\<answer\\>...\\</answer\\>.\n* Process rewards have been considered in [R1,R2,R3]. We propose novel 3D-aware process rewards to encourage the model to predict necessary 3D representations in the reasoning step. This is realized by implementing a range of regex patterns that recognize spatial representations --- 3D locations, distances, front/left direction, angles, etc. Then, given the question types, we reward the model if the reasoning trajectory contains relevant spatial representations. As an example, for a multi-object distance question, we give reward=1 if the reasoning trajectory contains 3D locations and distances, and reward=0 if the reasoning trajectory contains 3D orientations and directions.\n\n| reward | value | example |\n|:-|:-:|:-|\n| accuracy | 1 | \\<answer\\>red sofa\\</answer\\> (*matches the ground truth*) |\n| | 0 | \\<answer\\>blue sofa\\</answer\\> (*does not match the ground truth*) |\n| process | 1 | \\<think\\>The 3D location of the red sofa is…\\</think\\> \\<answer\\>red sofa\\</answer\\> |\n| | 0 | \\<think\\>The 3D location of the red sofa is… red sofa\\</answer\\> |\n| 3D-aware process | 1 | \\<think\\>The 3D location of the red sofa is…\\</think\\> |\n| | 0 | \\<think\\>The red sofa is next to the dining table…\\</think\\> |\n\n**Extension to multi-image inputs.** We agree with the reviewer that extending our SpatialReasoner to multi-image or video inputs presents nontrivial challenges. However, as motivated by the comparison between Gemini 2.0 and SpatialReasoner in Figure 1, we argue that adopting explicit 3D representations remains essential for MLLMs to achieve reliable and accurate spatial reasoning. While multi-view inputs introduce complexities such as varying camera viewpoints, recent feed-forward transformer, such as DUSt3R [R4] and VGGT [R5], have shown promising results in predicting grounded geometries (e.g., point maps) within the 3D coordinates defined by the first frame. Although some technical issues remain unexplored, we are optimistic about the potential of our explicit 3D representations to generalize to these more complex settings.\n\n[R1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n\n[R2] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations\n\n[R3] VisualPRM: An Effective Process Reward Model for Multimodal Reasoning\n\n[R4] DUSt3R: Geometric 3D Vision Made Easy\n\n[R5] VGGT: Visual Geometry Grounded Transformer"}, {"review_id": "eEPOaq3GEQ", "reviewer": "Reviewer_Tsq1", "summary": "The paper aims to impart explicit 3D reasoning to VLMs. Specifically, the paper collects and pseudo-annotates images with 3D annotations and chains of thoughts, and uses this to fine-tune a model to make it 3D-aware. The paper explores doing this via, RL-only, SFT-only, SFT+RL and achieves good improvements over prior SOTA. The paper also ablates several design choices and studies failure modes of their models.", "questions": "I think this paper has good contributions and detailed results, and most of the weaknesses I listed are fixable. Hence, I am recommending acceptance for now, and looking forward to author rebuttal. The main actionable experiments in my opinion are:\n-  SFT basleine with joint data from the proposed dataset instead of two-stage fine-tuning\n- (can skip if this is painful / non-standard to setup) specialized models fine-tuned on the collected dataset\n\nBesides these, fixing up the wtiting concerns that I pointed out in my review would help quite a lot!", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow\n- Making VLMs have explicit 3D reasoning is an interesting direction, and the paper achieves good results by doing so\n- I liked the ablations in the papers, very thorough and cover a lot of important baselines. I also like the failure analysis on the CVD-bench datasets as well as analysis on which among the 3D perception and computation contributes to the most errors\n\n\nWeaknesses:\n- I strongly recommend moving the implementation details section to the main paper. In my first reading, I was lost on a) what is the base VLM the paper starts with b) what is the training data? What exactly is the SFT and RL model are trained on. These are crucial details to understand the experiments, and I think they need to be in the main paper\n-  Spatialreasoner-zero is not defined and was misleading for me initially. I think this is following the deepseek’s terminology; however “-zero” to mean only RL is non-standard in the literature and I initially confused it with “zero-shot”. Please either introduce this term or instead use spatialreasoner-RL-only which is perhaps much more clear\n- Another crucial detail from the implementation detail section was that the SFT model is trained on one set of dataset, and then an additional portion of dataset is used for RL fine-tuning. Naturally, one thinks of an SFT baseline which uses both the datasets and authors include it as SpatialReasoner-SFT (+ HQ SFT) [which was unclear to me before reading implementation details]. This baseline underperforms the model without the additional fine-tuning. \n    - I think a more stable baseline would be to concatenate both datasets together and train an SFT baseline on them jointly instead of sequential training\n    - In the sequential training, what checkpoint did the authors take? Since the second-stage SFT data is tiny, it is likely that a very early checkpoint would lead to the best performance, while the latter checkpoints would be quite overfitted [that’s why joint training would actually be more stable for getting a stronger SFT baseline. Unlike RL training, SFT training does not need two stages, and hence it should be allowed to see all the data together]\n- The paper does not contain the details on how the pseudo-annotations were created. I would highly recommend the authors to add those details at least to the supplementary. \n- Additional Baselines: Did the authors consider fine-tuning the “specialist” baselines like spatial-rgpt on their additional collected data? \n\n\nI think this paper has good contributions and detailed results, and most of the weaknesses I listed are fixable. Hence, I am recommending acceptance for now, and looking forward to author rebuttal.", "was_revised": true, "final_justification": "The rebuttal addresses all my concerns. I will be recommending this paper for acceptance", "rebuttal": "We thank the reviewer for the time and effort to review our paper. We appreciate the positive feedback, mentioning that our paper is \"well-written and easy to follow\", our method of involving explicit 3D representations \"interesting\", and our experimental results \"good and thorough\". We discuss new results and address detailed concerns below. We will incorporate all valuable feedback and suggestions in our future revision.\n\n**Regarding implementation details of our SpatialReasoner and some writing concerns.** We thank the reviewer for raising the questions and will improve the clarity of the writing in our revision. We provide detailed responses as follows:\n1. Our SpatialReasoner model is based on the Qwen2.5VL-7B model and post-trained on spatial VQA data generated with 3D pseudo-labels. We will introduce key implementation details at the beginning of the experimental section.\n2. The training data of our SpatialReasoner in Table 1 and 2 includes: (1) 24k spatial VQA with pseudo 3D labels generated by our data synthesis pipeline, (2) another 1.2k spatial VQA synthesized and filtered by human verifiers, and (3) 24k standard VQA randomly sampled from the mix 665k SFT data used in LLaVA v1.5. For SFT, we jointly finetune SpatialReasoner on a combination of (1) + (2) + (3). For RL, we finetune the model on (2) only. **We apologize for the confusion, but for the main models in Table 1 and 2, there was only one SFT stage on combined data, and for ablation studies in L295-301 and Table 5, we consider a special SFT+HQ-SFT setting, which is trained on (1) + (2) + (3) and (2) sequentially, for fair comparison with the two-stage SFT+RL model.** Lastly we generate several variants of spatial-aware training data for ablation study, including spatial VQA data with only basic 3D perception and 3D computation QAs and spatial VQA data without reasoning trajectories or 3D representations.\n\nWe appreciate the reviewer’s valuable feedback and will revise the paper to improve its clarity and presentation. In our revision, we will include key implementation details and will also open-source all code, data, and models to support reproducibility and benefit future research.\n\n**Regarding SpatialReasoner-zero.** The naming convention of our SpatialReasoner models follows the deepseek-series models. Specifically, SpatialReasoner-zero is finetuned with reinforcement learning without first-stage SFT. We agree with the reviewer that these naming conventions can be confusing and will introduce the design of each model clearly in the next revision.\n\n**Regarding the implementation details of our data generation pipeline.** Thank the reviewer for raising this issue. We will include more technical details in our revision and will open-source both our generated spatial VQA datasets and the code for the data synthesis pipeline to support reproducibility and benefit future research. We summarize our data generation pipeline below.\n\nTo post-train multi-modal LLMs for better spatial understanding, previous works explored various visual foundation models such as Segment Anything and Depth Anything, and generated pseudo-labels for 3D depths, distances, and orientations [9,11,33] (L187-194). However, we extend beyond the previous spatial VQA data synthesis pipeline from the following two aspects:\n* Spatial relationships and reasoning trajectory: Compared to [9,11], which only studies object 3D depths and distances or [33] that studies a limited number of orientation relationships, we study a broader range of spatial relationships that are both challenging and require multi-step spatial reasoning. Answers in our generated spatial VQA data also contain chain-of-thought reasoning trajectories interleaved with explicit 3D locations and orientations, which are found to benefit models’ spatial reasoning.\n* Data quality: as we extend to more complex spatial relationships, we notice an increasing number of factual errors due to the long multi-step synthesis pipeline. Hence, we adopt a series of aggressive filtering based on model predictions and heuristics (L197-202), which is followed by an optional human verification stage. This leads to a smaller (24k vs over 1M) but much higher quality spatial VQA dataset. We further study the influence of data quality on SFT and RL in L278-284.\n\n**Finetuning a specialist model.** We agree with the reviewer that finetuning specialist models offers valuable insights into the nature of our synthetic data. However, as SpatialRGPT requires object masks as input, we instead finetune SpaceLLaVA, an open reproduction of SpatialVLM. Results in the table below show that finetuning on SpatialReasoner data leads to clear improvements on 3DSRBench for both LLaVA-v1.5 and SpaceLLaVA, highlighting the effectiveness of our data in enhancing the spatial reasoning abilities of multi-modal LLMs. Notably, the gap between the final models is small — this is because both models share similar architectures and the SpaceLLaVA training data contains some basic 2D spatial relationships (e.g., left and right in the 2D image plane) that are not very helpful for 3D spatial reasoning.\n\n| model | 3DSRBench |\n|:-:|:-:|\n| LLaVA-v1.5 | 38.1 |\n| LLaVA-v1.5 + our data | 44.2 |\n| SpaceLLaVA | 42.0 |\n| SpaceLLaVA + our data | 45.5 |"}, {"review_id": "ntWliWLvvx", "reviewer": "Reviewer_rCGV", "summary": "This paper explores the integration of explicit 3D representations into the outputs of large vision-language models (LVLMs). The motivation is that 3D representations offer interpretable and debuggable structures, enabling clearer reasoning and computation over visual content. The proposed approach demonstrates performance improvements on both in-domain objects and, to a lesser extent, on out-of-domain objects.", "questions": "- **Dataset**: What's the size of the SFT and RL dataset? Any numbers showing the distribution of different question types?\n\n- **Results**\n  - SpatialReasoner-Zero outperforms on novel objects. Could this suggest that explicit 3D representations may actually hurt performance in some cases? For instance, requiring the model to generate 3D outputs might increase hallucinations.\n  - In L248, what does “multi-object-closer-to” mean? \n\n- **Training**: Line 179: It’s unclear how the 3D-aware process reward is defined or explored. More detail would help.", "limitations": "The paper does not explicitly discuss its limitations. Given that its core contribution is the integration of explicit 3D representations into reasoning traces and the proposal of a data synthesis method, the lack of clarity around the data construction process is a significant issue. The writing should be improved to make the data synthesis pipeline and its characteristics more transparent.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- The use of explicit 3D representations in LVLM outputs is a novel and compelling idea\n- Comprehensive evaluation across both in-domain and out-of-domain objects\n\n\n# Weakness\n- Section 3.4 lacks sufficient detail on data synthesis. It's unclear how the dataset is constructed, its size, or the distribution of question types, making reproducibility difficult.\n- The method for computing the 3D coordinates in Figure 2 is not clearly explained. It's also unclear what units are used and whether the representation can handle distant objects, such as cars in street scenes.\n- The presentation of experimental results is difficult to follow. Key findings are scattered, and understanding the full picture often requires digging into the appendix. For example, the analysis in Table 6 on spurious correlations in CVBench-3D depth is insightful, but easy to overlook.", "was_revised": true, "final_justification": "See my response to the rebuttal.", "rebuttal": "We thank the reviewer for the time and effort to review our paper. We appreciate the positive feedback, mentioning that our method of using explicit 3D representations \"novel and compelling\" and our evaluation \"comprehensive\". We discuss new results and address detailed concerns below. We will incorporate all valuable feedback and suggestions in our future revision.\n\n**Implementation details on data synthesis.** We thank the reviewer for raising this concern. We will include more technical details of our data synthesis pipeline in our revision. For reproducibility, we will open-source code, data, and models to (1) reproduce our data generation process with 3D pseudo-labels and question-answering, and (2) reproduce key experimental results in our paper. Our code can be reused to benefit future research or be extended for other studies. We summarize key technical aspects as follows:\n\nTo post-train multi-modal LLMs for better spatial understanding, previous works explored various visual foundation models such as Segment Anything and Depth Anything, and generated pseudo-labels for 3D depths, distances, and orientations [9,11,33] (L187-194). However, we extend beyond previous spatial VQA data synthesis pipelines from the following two aspects:\n* Spatial relationships and reasoning trajectory: Compared to [9,11], which only studies object 3D depths and distances or [33] that studies a limited number of orientation relationships, we study a broader range of spatial relationships that are both challenging and require multi-step spatial reasoning. Moreover, answers in our generated spatial VQA data also contain chain-of-thought reasoning trajectories interleaved with explicit 3D locations and orientations, which are found to benefit models’ spatial reasoning.\n* Data quality: as we extend to more complex spatial relationships, we notice an increasing number of factual errors due to the long multi-step synthesis pipeline. Hence, we adopt a series of aggressive filtering based on model predictions and heuristics (L197-202), which is followed by an optional human verification stage. This leads to a smaller (24k vs over 1M) but much higher quality spatial VQA dataset. We further study the influence of data quality on SFT and RL in L278-284.\n\n**Unit of 3D coordinates in Figure 2 and its applicability.** Our SpatialReasoner learns to predict these 3D coordinates from supervised finetuning (SFT) on our generated data. The pseudo-labels for 3D locations are obtained with metric depth estimation (L196), so the units are in meters. Regarding method applicability, our training data is built on web images from the OpenImages dataset, which involves a wide range of scenes and objects. Illustrations in Figure 8 demonstrate that our approach with explicit 3D representations also applies to street scenes with distant objects such as buses and shop boards. We will involve more qualitative examples in our revision, as NeurIPS does not allow image uploads during the rebuttal stage.\n\n**Presentation of experimental results is difficult to follow.** We thank the reviewer for pointing out this concern. Our SpatialReasoner includes a range of design components, such as data generation (different size and quality), reasoning trajectories (with and without 3D representations), and training methods (SFT, RL, 3D-aware process rewards). We conducted experiments to study the influence of each component, aiming to demonstrate the strengths of our approach as well as identify the weaknesses. Some results in the appendix are referenced (e.g., spurious cues in Section C from L250), but we agree with the reviewer that the clarity and organization of the experimental section need to be improved. We will reorganize our experimental results in the following four sections and add a summary paragraph before going through all the results.\n1. Standard benchmark results: Table 1 and Table 2.\n2. Ablation study: Table 4 and Table 5.\n3. Failure case analyses: Table 7 and spurious cues in Section C.\n4. Scaling results and design considerations: Figure 5 and Figure 6.\n\n**Training data statistics.** The training data of our SpatialReasoner includes: (1) 24k spatial VQA with pseudo 3D labels generated by our data synthesis pipeline, (2) another 1.2k spatial VQA synthesized and filtered by human verifiers, and (3) 24k standard VQA randomly sampled from the mix 665k SFT data used in LLaVA v1.5. For SFT, we jointly finetune SpatialReasoner on a combination of (1) + (2) + (3). For RL, we finetune the model on (2) only. As the training data are synthesized by our data generation pipeline, our spatial VQA training data are uniformly distributed across different question types.\n\n**SpatialReasoner-Zero outperforms on novel objects.** For clarification, all SpatialReasoner models (SFT, RL, Zero) involve exploiting 3D representations, and the difference is how they obtain the reasoning trajectories (see Figure 8). The goal of Table 3 is to study how post-trained spatial MLLMs generalize to novel spatial questions, which is an important yet largely understudied problem.\n\nSpecifically, in Table 3, we report the generalization results of various MLLMs to novel question types. Models are finetuned on questions about height, location, and orientations, and we explore if the learned spatial knowledge generalizes to novel question types about spatial relationships between multiple objects ($\\geq$ 3). From the results in Table 3, we conclude the following:\n1. From the comparison between \"Qwen2.5-VL-7B-Instruct\" and \"SpatialReasoner\", we find that our SpatialReasoner learns explicit 3D representations, such as 3D locations and orientations, that transfer to novel question types and benefit spatial reasoning.\n2. From the comparison between \"SpatialReasoner\" and \"SpatialReasoner-Zero\", we find that while SFT on synthesized spatial VQA data can largely benefit the in-distribution spatial reasoning abilities, it falls behind our SpatialReasoner-Zero model trained with 3D-aware process rewards. Without reliance on SFT data with crafted chain-of-thought reasoning, SpatialReasoner-Zero learns to explore spatial reasoning trajectories from the process reward functions and generalizes better to novel question types compared to the SFT counterpart (L257-260).\n\nWe will include more discussions on these results in our revision.\n\n**Clarification on \"multi-object-closer-to\".** \"Multi-object-closer-to\" is one of the 12 question types defined in 3DSRBench. We discuss it in L248 as it is the question type that is most similar to the questions in CV-Bench distance-related questions. For example,\n* Multi-object-closer-to in 3DSRBench: Consider the real-world 3D locations of the objects. Which is closer to the desk, the bed or the fridge?\n* Distance-related questions in CVBench-3D: Estimate the real-world distances between objects in this image. Which object is closer to the chair (highlighted by a red box), the refrigerator (highlighted by a blue box) or the door (highlighted by a green box)?\n\n**Clarification on the 3D-aware process reward.** We apologize for the confusion and will introduce the motivation and detailed designs of our 3D-aware process rewards more clearly in our revision. We summarize the key technical aspects below.\n\nIn this work, we consider three types of reward functions: accuracy reward, format reward, and 3D-aware process reward.\n* The accuracy and format rewards follow the DeepSeek-R1 design that checks if the predicted final answer matches the ground truth and if the format of the answer follows a reasoning step \\<think\\>...\\</think\\> and an answering step \\<answer\\>...\\</answer\\>.\n* Process rewards have been considered in [R1,R2,R3]. We propose novel 3D-aware process rewards to encourage the model to predict necessary 3D representations in the reasoning step. This is realized by implementing a range of regex patterns that recognize spatial representations --- 3D locations, distances, front/left direction, angles, etc. Then, given the question types, we reward the model if the reasoning trajectory contains relevant spatial representations. As an example, for a multi-object distance question, we give reward=1 if the reasoning trajectory contains 3D locations and distances, and reward=0 if the reasoning trajectory contains 3D orientations and directions.\n\n| reward | value | example |\n|:-|:-:|:-|\n| accuracy | 1 | \\<answer\\>red sofa\\</answer\\> (*matches the ground truth*) |\n| | 0 | \\<answer\\>blue sofa\\</answer\\> (*does not match the ground truth*) |\n| process | 1 | \\<think\\>The 3D location of the red sofa is…\\</think\\> \\<answer\\>red sofa\\</answer\\> |\n| | 0 | \\<think\\>The 3D location of the red sofa is… red sofa\\</answer\\> |\n| 3D-aware process | 1 | \\<think\\>The 3D location of the red sofa is…\\</think\\> |\n| | 0 | \\<think\\>The red sofa is next to the dining table…\\</think\\> |\n\n**Limitations of our work.** We discussed limitations of our work in the NeurIPS submission checklist (L575) and in Section A of our supplementary materials — including the reliance on supervised finetuning data and the inference efficiency of predicting nuanced 3D representations. We apologize for the confusion regarding the data construction process and will clarify the writing in our revision. Moreover, we will open-source all code to reproduce training data generation and to support future research.\n\n[R1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters\n\n[R2] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations\n\n[R3] VisualPRM: An Effective Process Reward Model for Multimodal Reasoning"}, {"review_id": "tZ5TeDB4AH", "reviewer": "Reviewer_X79Q", "summary": "The paper introduces SpatialReasoner for 3D spatial reasoning. SpatialReasoner employs structured 3D representations. The model understands the space with transformations among different space coordinates. The model is trained in two stages: SFT and RL to enhance adaptability and generalization. Experimental results across multiple benchmarks show that the model achieves excellent performance.", "questions": "1. How would the model perform on robotics setting? Please consider some spatial understanding datasets on robots:\n\n[1] Song, Chan Hee, et al. \"Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n[2] Li, Yun, et al. \"Sti-bench: Are mllms ready for precise spatial-temporal world understanding?.\" arXiv preprint arXiv:2503.23765 (2025).\n\nI don't have further questions. The paper is good to be accepted to NeurIPS.", "limitations": "Yes, the authors have discussed the limilations of their method.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n1. The work is solid with a generated 3D-aware training data. Also, SFT and RL are both adopted and analyzed.\n\n2. Comprehensive experiments are done on various baseline models and benchmarks.\n\n3. The paper is well written and easy to follow\n\nWeakness:\n1. Please provide more visual examples of your dataset, and comparasion of model outputs.\n\n2. What is the accuracy of estimation 3D positions? (On-demand 3D parsing in Figure 2)\n\n3. The images in the article does not appear to be vector images.\n\n4. Please provide more metadata. e.g. the size of training dataset. For line 207~208, the authors may want to provide more details.", "was_revised": true, "final_justification": "I recommend the paper to be accepted.", "rebuttal": "We thank the reviewer for the time and efforts to review our paper. We appreciate the positive feedback, mentioning that our paper is \"well-written and easy to follow\", our work \"solid\", and our experiments \"comprehensive\". We discuss new results and address detailed concerns below. We will incorporate all valuable feedback and suggestions in our future revision.\n\n**More visual examples of training data and comparison of model outputs.** We thank the reviewer for the helpful suggestion. We show some qualitative examples in Figure 1 and Figure 8. However, due to NeurIPS policy restrictions, we cannot upload images or provide new qualitative results at this stage. We will include additional visual examples in the revised version of the paper.\n\n**Accuracy of estimation 3D positions.** As SpatialReasoner predicts 3D locations (3D center of the objects without height/width/length) in the learned 3D coordinate system (Figure 3), we cannot directly evaluate it on standard object detection datasets. To study how well our SpatialReasoner model predicts 3D locations and orientations, we (1) present qualiative examples for visual inspection (Figure 1 and Figure 8), and (2) evaluate our model on unseen testing data with human verified 3D groundtruths (Table 7). Results in Table 7 show that\n1. 3D reasoning (estimating distance and angle from location and direction) is more accurate than 3D perception (estimating location and estimation from image).\n2. After RL, we notice a small drop in both 3D perception and computation abilities. For future work, we will consider more training designs to ensure reliable 3D perception and computation, while improving the overall MLLM reasoning abilities.\n\nWhile we cannot directly compare with other expert models on 3D object detection, results in Table 3 provide valuable insights that guide the design of model, data, and training strategy.\n\n**Concern regarding image quality.** We appreciate the reviewer’s comment regarding image quality. Due to time constraints prior to the submission deadline, we included some JPEG images in the manuscript to meet the file size limitations. In our revision, we will replace these images with vector-format illustrations and ensure appropriate DPI settings to improve presentation quality.\n\n**More metadata on size of the training data.** The training data of our SpatialReasoner contains (1) 24k visual-question-answering data with chain-of-thought reasoning interleaved with 3D pseudo-labels (e.g., locations and orientations), and (2) an additional 1.2k visual-question-answering data where the answers are manually checked for better quality. We will include more dataset statistics in our revision.\n\n**Regarding human inspection on L207-208.** While our training data is built on predictions from state-of-the-art visual foundation models such as Segment Anything and Depth Anything, our training data is still error-prone. Specifically, we found that the estimated 3D orientations are often less accurate than 3D locations, and errors from one visual foundation model can be propagated and accumulated in later stages, leading to degraded data quality. Hence we adopt a human verification process to study the influence of data quality in SFT and RL post training. We will discuss more details on our human verification process in the supplementary materials of our revision, including the following: (1) web-based UI for human labeling, (2) statistics of human verification, and (3) percentage of raw VQA data filtered out by human annotators for each question type.\n\n**How would the model perform on robotics setting.** We thank the reviewer for the suggestion. As our SpatialReasoner is an image-based model, we cannot directly evaluate on STI-Bench. However, we evaluate our SpatialReasoner on RoboSpatial for reference.\n\nSince the training data of RoboSpatial is not released, we evaluate zero-shot performance of our SpatialReasoner on RobotSpatial-Home and compare with several baselines in the RoboSpatial paper. Results show that our SpatialReasoner performs on-par with prior works. Moreover, we note the following:\n1. Our SpatialReasoner attempts to solve various problems in RoboSpaital with explicit 3D representations (see example below). This demonstrates that our SpatialReasoner learns strong and transferable 3D knowledge.\n2. Although post-trained with various spatial reasoning VQA data, SpatialReasoner does not clearly outperforms previous methods on RoboSpatial. This is likely because RoboSpatial includes a range of 2D ego-centric left/right spatial relationships in the image plane, while our SpatialReasoner are only finetuned with 3D object-centric left/right spatial reasoning data. For future work we will consider various data mixing approaches and improve the general spatial reasoning abilities.\n\nLastly we will involve a paragraph in our revision that discusses the broader impact of our SpatialReasoner in the area of robotics, including the results on RoboSpatial, how spatial reasoning abilities of SpatialReasoner can benefit robotics tasks, and how explicit 3D representations can be extended in robotics settings.\n\n**Example output of SpatialReasoner on RoboSpatial.**\n* **Question:** Can the dish soap fit right of the bowl? Answer yes or no.\n* **Ground truth answer:** No.\n* **SpatialReasoner:** \\<think\\> The 3D location of the dish soap is (-0.6, 0.5, 1.1). The 3D location of the bowl is (-0.9, 0.8, 1.7). The vector from the bowl to the dish soap is hence (0.3, -0.3, -0.6). The length of this vector is 0.79. The size of the bowl is roughly 0.27. The size of the dish soap is roughly 0.44. The ratio of the distance between the two objects and the object sizes is 1.75. The answer is no. \\</think\\>\\<answer\\>no\\</answer\\>\n\n**Quantitative results on RoboSpatial.** Note that for \"context\" questions model predictions must follow certain formats so models without finetuning on the unreleased RoboSpaital training data performs very badly on this split.\n\n| model | Configuration | Context | Compatibility |\n|:-|:-:|:-:|:-:|\n| SpaceLLaVA | 61.0 | 2.5 | 61.0 |\n| 3D-LLM | 39.8 | 0.0 | 35.2 |\n| Molmo | 58.6 | 0.1 | 18.1 |\n| SpatialReasoner | 61.0 | 0.0 | 60.6 |"}]

Author rebuttal:

Meta-review: This paper presents SpatialReasoner, a large vision-language model designed for 3D spatial reasoning with explicit 3D representations and a multi-stage training framework combining supervised fine-tuning and reinforcement learning. 

Three reviewers (4UW2, Tsq1, and X79Q) rated the paper as "accept" and acknowledged its novelty, clear presentation, and thorough experiments. One reviewer (rCGV) provided a borderline reject score. While acknowledging the novelty, reviewer rCGV remained unconvinced that the models were genuinely learning accurate 3D representations and had concerns about the claim in L260. 

Overall, after considering all reviews, rebuttals, and discussions, the AC finds the paper is timely and relevant given growing interest in integrating spatial reasoning into multimodal foundation models. Given the clear majority of positive reviews and the author rebuttal's success in addressing most concerns, the AC recommends acceptance.  The authors are recommended to revise the statement in L260 and expand their discussion to provide a deeper analysis of the experimental results.
Decision: Accept (poster)