Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
Authors: Haoyu Zhang
Meng Liu
Zaijing Li
Haokun Wen
Weili Guan
Yaowei Wang
Liqiang Nie
Venue: NeurIPS 2025 spotlight ()
Keywords: Visual-Spatial Understanding
Vision-Language Model
Prompting
Primary area: applications
TL;DR: A framework that integrates spatial prompting and fine-tuning to enhance the 3D spatial understanding of VLMs from scanning videos

Abstract: Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision-language models (VLMs). To address these challenges, we present a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes through an automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding.

Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
Haoyu Zhang 22
Meng Liu Zaijing Li
Haokun Wen',
Weili Guan', Yaowei Wang Liqiang Nie
Harbin Institute of Technology (Shenzhen) PPeggchegg Laboratory Shandong Jianzhu University *Zhongguancun Academy
Abstract
Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision- language models (VLMs). To address these challenges, we present a unified frame- work for enhancing 3D spatial reasoning in pre- pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable anestion-answering dataset built from diverse 3D simulation scenes through automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding.
Introduction
Visual-spatial understanding, the ability to infer spatial relationships and the layout of objects from visual input, is core component of human perception [1 2 From single image, human observers can intuitively estimate distances, relative sizes, and even infer occluded structures. As intelligent systems become increasingly embedded in
applications such autonomous driving [4] 6) robotic navigation 7 8 [9. and augmented reality [10] [2), it becomes crucial to endow models with similar spatial reasoning capabilities for robust perception and interaction.
Unfortunately, a single image is inherently limited in capturing the complexity real-world 3D scenes, constraining its utility in practical scenarios [3]
14 15). To address this, point clouds have become a mainstream representation for 3D scene understanding due to their ability to encode rich geometric information [16] [7]. Yet, generating high-quality point clouds typically requires expensive sensors and incurs significant computational overhead, limiting scalability and accessibility.
These limitations motivate the pursuit of vision-only solutions that operate on scanning videas or multi-view images of scenes. Such approaches offer more human-like and scalable pathway to spatial understanding [18). However, performing 3D spatial reasoning from scanning videos presents two significant challenges: (1) Spatial Uncertainty. In the absence of explicit depth information. models must infer 3D structure from inherently limited 2D observations. This process is further complicated by occlusions, perspective distortions, and texture ambiguities, all of which introduce significant spatial uncertainty. Effectively addressing this challenge demands multi-step logical reasoning across frames to reconstruct coherent spatial layouts. (2) Data Scarcity. Existing datasets for this task are limited in both scale and diversity, restricting the ability of vision- vision-language models
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(VLMs) to acquire robust spatial knowledge and perceptual capabilities. Moreover, these datasets involve scans of real-world scenes, which leads to poor scalability. This highlights the need for scalable and extensible data sources to support effective spatial reasoning in VLMs.
To address these challenges, we propose dual approach for enhancing 3D spatial reasoning in pre-trained VLMs, without modifying their underlying architecture. First, we introduce SpatialMind, structured Chain-of-Though (CoT) prompting strategy that guides VLMs through step-by-step reasoning over spatial relationships. Second, we present ScanForgeQA, large-scale synthetic question- question-answerinn answering (QA) dataset constructed from diverse 3D simulation scenes using an automated generation pipeline. Fine-tume tuning VLMs on this dataset equips them with spatial commonsense knowledge, significantly improving their generalization to unseen spatial layouts. We have validated our approach through extensive experiments across multiple benchmarks. Results demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding.
Our contributions are summarized follows:
We introduce SpatialMind, spatial prompting strategy that decomposes spatial reasoning into structured steps, enabling pre-trained VLMs to perform multi-step inference over spatial relationships from visual input alone.
We develop a scalable dataset generation pipeline to construct ScanForgeQA, a synthetic spatial anestion-anssering dataset that enables VLMs to acquire spatial commonsense through fine-tuning.
Experimental results validate the effectiveness and generalizability of both SpatialMind and ScanForgeQA, with their combination achieving further gains and providing valuable insights for future research.
2 Related Work
2D Image Spatial Understanding focuses on modeling spatial relationships among objects within the 2D image. Most existing models are trained on 2D images paired with textual descriptions, which offer limited cues about 3D structure. Consequently, their capacity for spatial reasoning remains constrained. To mitigate this, several approaches, such SpatialVLMM [3]. SpatialRGPT [14), and SpatialBot [15), have been proposed. These methods enhance the spatial understanding by fine-tuning models on datasets specifically designed for spatially grounded QA tasks. To enable more comprehensive evaluation, recent studies [19 20 21 22 23 have introduced hierarchical benchmarks that assess models across varying levels of spatial reasoning complexity. Parallel efforts have explored more explicit forms of spatial interaction 24 25). For example, point-based
methods (26) (27 interpret spatial instructions by predicting specific target points. Building on this trend, SpatialCoT (28) proposes a two-stage strategy that aligns multimodal inputs with spatial coordinates and incorporates CoT reasoning better address complex embodied tasks. Despite these advancements, model performance often degrades in complex real-world 3D environments, highlighting the limitations of 2D-based approaches in representing complex 3D scenes.
3D Indoor Spatial Understanding focuses on enabling intelligent agents to identify object positions and infer their spatial relationships within enclosed environments, thereby supporting both object manipulation and interactive scene comprehension. Early 3D models are trained on standard indoor using point clouds facilitate downstream tasks like 3D object
datasets (29 30 32
33
34
35
39) and primarily focus on object-level geometry and More recent work extends this focus to complex indoor scenes, emphasizing inter-object spatial relationships and holistic scene-level understanding. To address challenges such as geometric complexity and annotation sparsity, many of these models employ cross-modal strategies that combine point cloud data with auxiliary multi-view 2D images [16] I7 44
detection and instance segmentation 36
(3
38
appearance features 40
42
43)
Inspired by the way humans perceive spatial layouts through vision alone, emerging research
18 has begun to explore purely lssion-base approaches to 3D spatial understanding. These methods rely solely on visual inputs, such as scanning videos, without requiring explicit 3D priors like point clouds. This line of work offers more practical and scalable alternative for real-world deployment. In this context, we further investigate whether purely rision-based inputs can provide a more effective solution for indoor scene understanding.
2

---

Scene Decomposition
Question Decomposition
Local Modeling
Coordinate Mapping
Cognition Generation
Description
Question
Select an appropriate object os global reference point,
Monitor: locale one meter
redefine positions of other objects relative...
to left the reference in front of the reference point
Router
Instance Estimation
Bed:
Step1: Identfy abjects of Step2: Estimale
Reference Point
VILO
poaition coordinates for these objects
Stop3: Calculate between
bed and the monitor
3D Map
2D Grid
Scanning Video
Figure 1: Illustration our SpatailMind prompting strategy.
3
SpatialMind Prompting Strategy
As shown in Figure [ our SpatialMind prompting strategy consists of two main components: 1) Scene Decomposition, where the 3D scene depicted in the video is transformed into multiple different representations; and 2) Question Decomposition, in which the question is broken down into sequence of fine-grate fine-grained reasoning steps. Further details can be found in Appendix D
3.1 Scene Decomposition
The scene decomposition process includes three sequential steps: local modeling, coordinate mapping, and cognition generation.
Local Modeling. The first step processes scanning video frames to extract object instances and their relative spatial configurations within localized coordinate systems. To handle scene complexity and reduce the search space, we leverage GPT-4q to identify all objects mentioned across the questions associated with given scene, using them as candidate targets. For each frame i, we prompt VLMs to detect a subset of objects {Cij) from the candidate targets and estimate their positions plycal e R These positions are defined relative a randomly selected reference object (i.e., origin) within the same frame, forming a local 3D map:
C; = {Gg local | I,=,,..,,n
(1)
where nı denotes the number of objects in frame i. Because each video frame captures only a limited field view, the same object may appear across multiple frames from different perspectives. Thus, this step focuses accurate per-fieee per-frame object detection and spatial localization, laying the foundation for subsequent alignment in a global coordinate system.
Coordinate Mapping. To integrate spatial information across video frames, this step transforms all locally detected object positions into a unified global coordinate system. The global origin is defined by selecting the reference object in the first frame. To estimate motion between frames, we prompt the VLM infer the relative rotation and translation between adjacent frames. These relative transformation are accumulated sequentially to compute each frame's transformation T; with respect to the global coordinate system:
II R
tR
T
where R. and f2.k-1 denote the relative rotation and translation from frame k - to frame k; respectively. This accumulated approach provides more stable and accurate alignment than directly estimating each frame's 's absolute pose. Using these transformations, each object's 's local coordinates converted into global coordinates via homogeneous transformation:
global
local
P
T
P

---

global denotes the global coordinates of the object in the frame i. This step ensures that all where Pig detected objects across frames are positioned consistently within the same 3D space. Since objects may appear in multiple frames under different perspectives, we merge duplicate detections based on spatial proximity and semantic consistency via prompting. The result a global 3D map of the scene:
G = [ct, Pk
global
k=1
where N is the total number of all object instances in the entire scene. This map serves as a unified spatial abstraction that captures the overall layout from egocentric scanning videos.
Cognition Generation. Beyond constructing a 3D map, explore two additional formats for representing scene structure: a 2D spatial grid and natural language descriptions. We define a regular 2D grid over the global scene, typically aligned with the XY -plane. Each grid cell corresponds to fixed real-world area (e.g., meter per cell, denoted by cell size 8 Each object ey. mapped to a discrete grid location (i):
(L= Tk
(ix. jk)
(5)
where (It, yk) are the horizontal components of the object's global position Pk plohal In parallel, we generate natural language descriptions of object locations relative to a designated reference point. Using prompting, the model produces statements such as ("monitor": "locate meter to the left of the reference point"). These descriptions serve as human-interpretable form of spatial cognition, bridging visual perception and symbolic reasoning.
3.2 Question Decomposition
Different types of spatial questions require distinct reasoning strategies 46]. To accommodate this diversity, we first categorize questions into several types (e.g., object size, relative distance, and relative direction). For each category, design dedicated reasoning procedure using GPT-4o, followed by human verification to ensure correctness and interpretability. For instance, consider question from the "relative distance" category: Among the refrigerator, window, and microwave, which object is closest to the door? The reasoning process for this type follows four structured steps: 1) Identify all mentioned objects, 2) Estimate the spatial coordinates of all relevant objects, 3) Compute the pairwise distances between the door and each candidate object, and 4) Select the object with the minimum distance the answer. During inference, the system correspondingly selects the appropriate reasoning procedure based on the identified question type.
To perform 3D spatial reasoning, feed the VLMS with the input scanning video, one form of scene representation (e.g., 3D map. 2D grid, or textual position descriptions), the question, and the corresponding step-by-step reasoning plan. To assess which scene representation format is most interpretable for VLMs, we have conducted comparative experiments, as shown in Figure 3
ScanForge Dataset Construction
The construction of the ScanForge dataset involves three-stage pipeline, illustrated in Figure These stages are: 1) Scene Construction, where single-room 3D environments are created; 2) Scan Creation, in which egocentric videos are simulated by scanning through the constructed scenes; and 3) QA Generation, where textual question-ansswriig pairs are automatically generated based on object annotations and the spatial layout of each scene.
4.1 Scene Construction
To ensure diversity and richness in single-room scene collection, we adopt two parallel strategies:
Separation. We modify existing scene datasets to leverage available resources effectively. Specifi- cally, we utilize the 3D-FRONT dataset [47], which contains 6,813 multi-room scenes furnished with diverse 3D objects and annotated with detailed layout semantics and high-quality textures. Since our focus is on single-room environments, we disassemble each multi-room scene into individual rooms. For each scene, we isolate and load one room at time, along with its corresponding ceiling and walls, and save as an independent instance. This disassembly process yields 44,427 single-room scenes. We further filter out uncommon room types (c.g., garage, auditorium) and those lacking sufficient

---

Scene Construction
Scan Creation
Scanning Video
Full Scene
Single-room Scene
Separate
Navigation
QA Generation
Scene Annotation Question Template
F
touching
What's
Distance
Synthesize
meteee?
Ground Truth
Furniture Model
Figure 2: The pipeline of ScanForgeQA data construction.
object content (e.g., aisle, stairwell). The final dataset consists of 34, 16 single-room scenes across six common categories: bedroom, kitchen, bathroom, living room, dining room, and storage room.
Synthesis. To introduce additional diversity and originality, we synthesize novel room layouts using LLM-guided generation approach. Specifically, adopt HoloDeck [48), 3D generation framework that leverages LLMs to parse natural language prompts, retrieve matching assets from large-scale 3D object repositories such Objaverse [49], and optimize their spatial arrangement to form semantically meaningful scenes. To drive the generation process, we first use GPT-4o to create diverse textual descriptions for various bedroom with a bed, window, armchair, and wardrobe", " We define eight room categories, including two additional types-office and store-and generate 20 distinct descriptions for each. These prompts are fed into HoloDeck to produce corresponding room layouts, with human verification to ensure spatial plausibility and realism. This synthesis process yields 160 additional single-room scenes.
types. For example, a bedroom may be described as: "A
4.2 Scan Creation
To simulate egocentric scanning videos from the constructed single-room scenes, we implement scanning procedure using the Unity engine" Each scene is scanned using two complementary strategies designed to emulate natural human visual exploration:
Orbit Scan. We define circular trajectory centered in the room height of approximately 1.5 meters, corresponding to typical adult eye level. The circle's diameter is set to two-thirds of the shorter side of the room. The camera is randomly initialized at a point on this path and moves along the circle either clockwise or counterclockwise. An image is captured every degrees rotation, resulting in 72 frames per orbit scan. This strategy provides comprehensive 360-degree panoramic view of the scene.
Navigation Scan. To simulate movement through the environment, we label navigable ground regions based on object categories and generate navigation mesh using the NavMesh Baking API. We randomly select two objects as the navigation start and end points and compute the shortest path between them on the mesh. Among the candidate paths, the two longest are chosen for scanning to achieve more complete coverage of the scene. For each path, the camera first performs a 360-degree rotation at the starting point, capturing an image every 12 degrees (30 images total). It then traverses the path toward the destination, during which 12 frames are uniformly sampled. Upon arrival, another 360-degree rotation is performed, again capturing 30 images. In total, 72 frames are recorded per path. Due to the limited size of indoor environments, rotational movement yields more visual variation than translation; hence, fewer frames are captured during motion.
https://unityy:
5

---

Table Comparison 3D QA Datasets
Dataset
Source
Scenes
Format
QAs
Scalability
SPARTUN3D
3RScan
478
Point Cloud
133K
Hard
MSQA
SeanNet, 3RScan, ARKitScenes
1.7K
Point Cloud
251K
Hard
3D-LLM
ScanNet. HM3D
1.2K
Point Cloud
300K
Hard
ScanForgeQA
Simulation/Synthesis
34K
Scan Video
925K
Easy
4.3 QA Generation
To generate diverse supervised fine-tuning (SFT) data and enhance the 3D spatial reasoning capa- bilities of existing VLMs, we define three categories of question types: attribute estimation, spatial reasoning, and hypothesis analysis. These categories encompass both quantitative and qualitative dimensions, and cover both open-set and closed-set scenarios. Below, we describe each category in detail, along with the methodology for deriving corresponding ground-truth answers.
Attribute Estimation. This type focuses on static properties of objects and scenes, such as object count ("How many chairs are in the room?"), object size ("What the length of the longest side the refrigerator in meters?"), room size ("What is the size of this room in square meters?"), and room rype ("Based on the object layout, what the most likely type of room (c.g.. kitchen)?")), Ground-truid answers for these questions are directly derived from 3D scene annotations and object metadata provided in the dataset.
Spatial Reasoning. This category targets inter- object spatial relationships, requiring models to infer positional and geometric properties such distance, orientation, and contact. Represen- tative question types include: relative distance ("Which of these objects (refrigerator, couch, ceiling light) is closest to the TVT"), absolute distance ("What is the distance between the couch and the table in meters?"), relative di- rection ("If am standing by sofa and facing the table, which side is the trash can on?"), and contact relationship ("Is there gap between the bed and the headboard?"). For distance- related questions, we compute Euclidean dis- tances between object centroids in the global 3D coordinate space. For contact relationships, ob- ject dimensions are also considered to determine physical adjacency. To resolve relative direction, define object's front the side oriented toward the room center. Angular sectors are divided clockwise into four directional categories: right (45%-1359, back (135--225%) left (2259-3159). and front 45%).. For example, object located at 80 relative the reference point is classified as being on the right.
Base
+Map
+Grid
+Des
44.0
40.8
39.2
39.2
37 2
34.0
GPT-40
Qwen2.5-VL-7B
Qwen2.5-VL-P27
Figure 3: Effects of different scene expression.
Hypothesis Analysis. This category introduces conditional reasoning under hypothetical scenarios, often requiring geometric and commonsense inference. A typical example is operation feasibility ("Considering only object dimensions, is it feasible place the television the table?"). Feasibility determined by comparing object dimensions. For stacking, the movable object's length and width must be smaller than those of the supporting surface. For embedding (e.g., fitting item into drawer), the object's 's height must also fall within the bounds of the specific container's 's volume.
A comparison with existing 3D QA datasets (e.g.. SPARTUN3D [O]. MSQA (5T], and 3D-LLM (17) presented in Table [ The full FoanForge dataset includes 34,276 single-room scenes, 103K simulated video scans, and 925K ronments allows scalable and controlled data generation across diverse spatial scenarios. Additional implementation details are provided in the Appendix
pairs for training. Leveraging synthetic envi-

---

Table Performance comparison on VSI-Bench. f indicates results on VSI-Bench (tiny) set.
Abs. Count
Obj.
Obj.
Room
Rel.
Rel.
Route Plan
Appr. Order
Method
Avg
A
Dist.
Size
Size
Dist.
Dir.
Close-source
Human Levelf
94.3
47.0
60.4
45.9
94.7
95.8
95.8
100.0
79.2
Gemini-1.5 Prof
49.6
28.8
58.6
49.4
46.0
48.1
42.0
68.0
48.8
Gemini-1.5 Pro
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
45,4
+SpatialMind
63.9
51,8
70.2
47.3
56.3
45.9
42.6
44.3
52.8
7.4%
GPT-4o
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
34.0
+SpatialMind
40.0
27.1
62.7
40.9
41.0
39.6
37.1
38.5
40.8
6.8%
Open-source
Struct2D
46.0
34.7
56.4
42.6
35.1
44.9
33.5
41.9
InternVL.2288 VL22-B
23.1
28.7
48.2
39.8
36.7
30.7
29.9
39.6
34.6
+SpatialMind
35.8
28.9
49.7
44.4
37.2
34.8
35.1
45.5
38.9
4.3%
+ScanForgeQQ
45.3
33.4
54.8
45.0
41.1
36.1
33.4
43.0
41.5
6.9%
+Both
47.0
32.8
53.2
46.6
39.8
36.8
37.9
47.5
42.7
8.1%
Intern VL22-0B
34.9
26.9
46.5
31.8
42.1
32.2
34.0
39.6
36.0
+SpatialMind
36.4
30.0
49.1
41.8
43.8
36.1
35.6
50.0
40.4
4.4%
+ScanForgeQAA
51.0
29.2
52.7
38.1
47.2
36.4
35.9
47.6
42.3
6.3%
+Both
52.2
30.5
54.4
41.0
50.5
37.0
40.2
50.3
44.5
8.5%
Qwen2.5-VL-77 VLL-B
40.3
22.2
50.1
38.9
38.0
40.7
31.4
35.9
37.2
+SpatialMind
45.1
25.2
52.1
41.4
38.7
41.6
34.7
34.5
39.2
2.0%
+ScanForgeeAA
53.2
30.5
56.8
44.9
42.3
44.0
37.3
37.7
43.3
6.1%
+Both
55.0
29.5
57.3
44.0
43.5
44.3
38.3
39.2
43.9
6.7%
Qwen2.5-VIL-722
37.9
28.6
57.4
49.8
45.5
38.4
20.6
35.4
39.2
+SpatialMind
42.3
32.0
61.7
53.8
48.2
43.9
30.4
39.3
44.0
4.8%
+ScanForgeQA
45.2
32.7
63.3
52.4
50.1
41.7
32.8
40.2
44.8
5.6%
+Both
48.6
34.4
68.9
54.7
53.4
43.9
30.1
42.7
47.1
7.9%
Table Performance comparison on the EM-EQA subset of Table 4:
Effects of different fine-
OpenEQA and the validation set of ScanQA and SQA3D.
tuning data and prompting strategy.
OpenEQA Acc/Score
ScanQA BLEU-1
SQA3D
Room Size
Method
Method
Avg
EM-1
Qwen2.5-VL-7B
50.1/3.1
32.5
17.2
Qwen2.5-VL-7B
38.9
37.2
+SpatialMind
53.7/3.2
33.1
19.8
+SQA3D
38.8
38.9
+ScanForgeeAA
56.2/3.3
34.8
23.3
+ScanQA
38.5
39.1
+Both
58.6/3.5
37.9
24.5
-ScanForgeeAA
44.9
43.3
Qwen2.5-VL-728
53.8/3.2
35.4
34.8
Qwen2.5-VL-72B
49.8
39.2
+SpatialMind
55.7/3.2
38.0
39.2
+CoT-Question
50.6
41.3
+ScanForgooAA
59.1/3.4
42.5
43.0
+CoTT-ccnee
52.1
42.7
+Both
60.4/3.4
44.1
46.3
+SpatialMind
53.8
44.0
Experiments
The experimental settings (including benchmarks, baselines, etc.) and more experimental results can
be found in the Appendix and B
5.1 Performace Comparison
We investigated the following five key questions assess approach:
QI: Which scene representation format is most interpretable by VLMs? Figure presents performance comparison across different representation formats: no additional spatial context (Base), inclusion of 3D map (+Map), 2D grid (+Grid), and object-centric textual descriptions (+Des). Across all models, consistent trend emerges: the +Des variant outperforms others, followed by

---

72.5
MVBench
( BoBthh Basel
72.0
Video-MME(W sub)
6.5
Accuracy (%)
71.5
6,0
71.0
5,5
5,0
70.0
512
Reselution
25g
69.0
Frame
24
Base
+ScanForgeQA
+5%
+10%
32
128
Figure 4: Performance of Qwen2.5-VL-7B on MVBench and Video-MME.
Figure 5: Ablation study of Qwen2.5-VL under varying numbers of frames and resolution.
+Grid, while +Map yields the least improvement. These results suggest that current VLMs are more adept at interpreting onc-dimensional textual descriptions than high-dimensional structured spatial formats. Consequently, we adopted the textual description format in subsequent experiments as the default scene representation.
How do SpatialMind and ScanForge impact VLM performance? As shown in Table 2 progressively applied the SpatialMind prompting strategy and the ScanForge fine-tuning data across range of VLMs, varying in architectures, parameter size, and openness (including both open- and closed-source models). The results reveal three key findings: 1) Both SpatialMind prompting and ScanForgeQA fine-tuning consistently improve visual-spatial understanding across models. This includes large-scale proprietary models such as Gemini-1.5 Pro Pro GPT-4o, demonstrating the effectiveness and generalizability of our approaches. 2) Model size affects the relative benefit of prompting versus fine-tuning. Larger models (c.g.. 72B) benefit more from prompting, which enhances their reasoning capabilities, while smaller models (e.g.. 7B) show greater improvements through fine-tuning For instance, Qwen2.5-VL. V-BBBB gains 6. from fine-tumin fine-tuning, compared to only 2.0% from prompting. 3) Humans and VLMs exhibit complementary strengths. Human participants excel in qualitative tasks (e.g., achieving 100% accuracy on the Appearance Order task) but perform poorly on precise quantitative estimations (e.g., Object Size). In contrast, VLMMs show strong quantitative reasoning ability and, in some cases, even surpass human-level performance. This contrast underscores the potential of VL.Ms to complement human perception spatial tasks.
Can combining prompting and fine-tuning yield further gains? To assess whether SpatialMind and ScanForgeQA provide complementary benefits, we applied the SpatialMind prompting strategy to models that have already been fine-tuned on the ScanForgeQA dataset. The results, reported in the "+Both" row of Table show consistent performance improvements across all evaluated models. These findings confirm that the two approaches are complementary.
Does the improvement generalize to other spatial benchmarks? To assess the generalizability of our framework, we conducted evaluations on multiple benchmarks, including OpenEQA [52). ScanQA (53] and SQA3D (54 As shown in Table 3, both SpatialMind prompting and ForgeO fine-tuning lead to consistent performance gains across all benchmarks. These results validate the robustness of approach and confirm its applicability across diverse spatial tasks and datasets.
Does fine-tuning affect performance on other tasks? To investigate whether enhancing visual- spatial capabilities via fine-tuning adversely impacts model's general performance, conducted evaluations on MVBench (53) and Video-MME [56], two broad multi-task video benchmarks, As shown in Figure 4 fine-tuning with ScanForgeQA slightly improves performance MVBench but leads to a marginal drop on Video-MME. This difference likely stems from MVBench containing event comprehension. To mitigate this trade-off, we further experimented with mixed fine-tuning, combining a small proportion (5% and 10%) of traditional data from ShareGPT4Video (57 with ScanForgeQA. Results show that
spatial reasoning tasks, while Video-MME: focuses

---

(a) Route Plan
You are a robot beginning at the chair and facing lamp. You want to navigate lamp an the cabinet. perform the following actions (Note: for each jplease fill choose either 'turn back," "turn left, or "tum right.): 1. [please fill in] 2. Go forward until the wardrobe. 3. [please fill In]. 4. Go foward until the lamp. You have reached the final destination.
Qwen2.5-VL-7B: Turn Back, Tum Left
(Ours): Left,
(b) Appearance Order
What will be the first-time appearance order of the following categories in the video: door, towel, refrigerator,
Qwen2.5-VL-7B door, towel, refrigerator, microwave
+Both (Ours) lowel, microwave, refrigerator, door
Figure 6: Two examples from VSI-Bench comparing predictions from Qwen2.5-VL-7B and Ours.
this strategy achieves improved performance, surpassing the original -L- baseline, suggesting that spatial fine-tuning be harmonized with broader capabilities through data balancing.
5.2
Ablation Study
In this section, we explored the impact of various design choices, including prompting strategies, fine-tuning datasets, frame sampling strategies, and input resolution, on the performance VLMs.
On prompting strategy. To isolate the contributions each component the SpatialMind prompting strategy, evaluated variants: one containing only the question component (CoT-Question) and another containing only the scene description (CoT-Scene).. As shown in Table 4 both variants independently improve spatial reasoning performance, but less effective than the full combined prompt. Notably, the scene description contributes more significantly to model performance than the reasoning steps, suggesting its central role in facilitating spatial understanding.
fine-tuning data. investigate the effectiveness our proposed ScanForgeQA against cxisting spatial datasets, fine-tuned Qwwen.5-VLL-7B on SQA3D [54] and ScanQA (53) As shown in Ta- 4 fine-tuning on either of these datasets results in lower performance compared to ScanForgeQA, and reduces accuracy is primarily due to the limited presence of fine-grate fine-grained spatial estimation samples in the existing datasets. Importantly, both datasets and the VSI-Bench benchmark originate from the same source (i.e., ScanNet (31), resulting in minimal data discrepancy. This contrast emphasizes the advantage of simulated data generation pipeline.
tasks involving precise spatial estimation (e.g., Room Size). This
On frames and resolution. To evaluate the robustness of our approach, we analyzed performance sensitivity to the number of imput frames and image resolution. Figure visualizes the performance of the +Both variant and the baseline Qwen2.5-VL-77. under various configurations. Our method consistently outperforms the baseline across all settings, with performance further improving as the number frames and resolution increase. This indicates that our approach remains stable and effective under varying visual input conditions.
5.3 Qualitative Analysis
we presented two illustrative examples from VSI-Bench, comparing predictions from the baseline Qwen2.5-V VL-7B and our enhanced variant (+Both). In Case (a), Qwen2.5-VLL-75 fails to produce the correct directional prediction, likely due to its limited capacity for 3D spatial reasoning. In contrast, our method successfully identifies the correct answer. Case (b) involves simpler spatial reasoning task, however, Qwen2.5-VL-77 still fails, potentially due to insufficient object localization.
In Figure 6

---

Our enhanced variant, benefiting from both structured prompting and spatially grounded fine-tunnn fine-tuning, demonstrates notable improvements in accuracy and reasoning robustness.
Conclusion
In this work, we present an effective framework for enhancing visual-spatial reasoning in VLMs without modifying their underlying architecture. This makes our approach readily adaptable across models of varying scales and types. By integrating the structured prompting strategy (SpatialMind) with automatically constructed dataset (o)a interpret and reason about 3D spatial relationships in complex visual scenes. Extensive evaluations across multiple spatial reasoning benchmarks demonstrate that our framework consistently improves accuracy, robustness, and generalization. Furthermore, our analysis reveals that prompting and fine-tuning play complementary roles in advancing visual-spatial understanding.
enable VLMMs more effectively
Acknowledgments and Disclosure of Funding
This work supported by Shenzhen Science and Technology Program, No:KKQTD20240729 (e2222990227700 102207002 the National Natural Science Foundation of China, No.:62366144 No.:6247607,, No.:625B2065, U24:022420328 62.:6244 and No::U23320015; the Major Key Project of Pengcheng Laboratory, No::PC22025A14 GuangDong Basic and Applied Basic Research Foundation, No.:2025A151501 173;; the Science and Technology Innovation Program for Distinguished Young Scholars of Shandong Province Higher Education Institutions, No.:2023KJ122 the Special Fund for Taishan Scholar Project of Shandong Province; the Fundamental Research Funds for the Central Universities, No.:HIT.DJJ..0. 20025004888 and No.:HIT.DZJJ..2 2002444..
References
[1] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint 21Xv:22412.14171, 2024.
[2] Yizhang Jin, Jian Li, Jiangning Zhang, Jianlong Hu, Zhenye Gan, Xin Tan, Yong Liu, Yabiao Wang, Chengjie Wang, and Lizhuang Ma. Llava-vsd: Large language-and-visite assistant for visual spatial description. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1 1 11425, 2024.
[3] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal minecraft agent with goal-observation-actiee conditioned policy. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9039-9049, 2025.
[4] Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, XianPeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. In 8th Annual Conference on Robot Learning.
[5] Kun Wang. Hao Liu, Lirong Jie, Zixu Li, Yupeng Hu, and Liqiang Nie. Explicit granularity and implicit scale correspondence learning for point-supervised video moment localization. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9214-9223, 2024.
[6] Haoyu Zhang, Meng Liu, Yuhong Li, Ming Yan, Zan Gao, Xiaojun Chang, and Liqiang Nie. Attribute- guided collaborative learning for partial person re- re-ideniificttin. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12): 1414)))::4444111 14160, 2023.
[7] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pages 8469-8488. 2023.
[8] Haoyu Zhang, Meng Liu, Zixin Liu, Xuemeng Song, Yaowei Wang, and Liqiang Nie. Multi- factor adaptive vision selection for egocentric video question answering. Proceedings of the 41st International Conference Machine Learning, volume 235, pages 593 10-59328, 2024.
10

---

[9] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-:: 1: Hybrid multimodal memory empowered agents excel in oon-hhoizzon tasks. In Advances in Neural Information Processing Systems, volume 37, pages 49881 19881-49913, 2024.
[10] Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. Advances in Neural Information Processing Systems, 37:53168-53199 2024.
[11] Haoyu Zhang. Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exolego: Exocentric knowledge guided milm for egocentric video understanding. arXiv preprint arXiv:2503.09143, 2025.
[12] Yisen Feng, Haoyu Zhang, Meng Liu, Weili Guan, and Liqiang Nie. Object-shot enhanced grounding network for egocentric video. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[13] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[14] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37: 37:135062-135099 2025.
[15] Wenxiao Cai, laroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642, 2024.
[16] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for unni-3d understanding reasoning and planning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428-26443 2024.
[17] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
[18) Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Un- derstand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428. 2025.
[19] Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. Embspatial- Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346-355 2024.
[20] Wenyu Zhang, Wei En Ng. Lixin Ma. Yuwen Wang, Jungqi Zhao, Allison Koenecke, Boyang Li, and Lu Wang. Sphere: Unveiling spatial blind spots in vision- -sioun-language models through hierarchical evaluation. arXiv preprint arXiv:2412. 12993, 2024.
[21] Jianlong Wu, Wei Liu, Ye Liu, Meng Liu, Liqiang Nie, Zhouchen Lin, and Chang Wen Chen. A survey video temporal grounding with multimodal large language model. arXiv preprint arXiv:2508. 10922, 2025.
[22] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. Attentive moment retrieval in videos. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 15-24, 2018
[23] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Weili Guan, Dongmei Jiang, and Liqiang Nie. Optimus-3: Towards generalist multimodal minecraft agents with scalable task experts. arXiv preprint 2025.

---

[24] Chenyang Ma. Kai Lu, Ta- Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision- vision-language models through prompting and interacting 3d priors. In The eiihth Annual Conference on Neural Information Processing Systems.
[25] Jun Yu, Yunxiang Zhang, Zerui Zhang, Zhao Yang, Gongpeng Zhao, Fengzhao Sun, Fanrui Zhang, Qingsong Liu, Jianqing Sun, Jiaen Liang, et al. Rag-guided large language models for visual spatial description with adaptive hallucination corrector. In Proceedings of the 32nd ACM International Conference Multimedia, pages 1 11407-11413, 2024.
[26] Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction in robotics. In 8th Annual Conference on Robot Learning.
[27] Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision- rision-language models for robotics. arXiv preprint arXiv:2411.16537, 2024.
[28] Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, et al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning. arXiv preprint arXiv:2501. 2025.
[29] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, et al. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
[30] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017.
[31] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828-5839, 2017.
[32] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-5994, 2022.
[33] Yongsen Mao, Yiming Zhang, Hanxiao Jiang, Angel Chang, and Manolis Savva. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. Advances in neural information processing systems, 35:9058-9077 2022.
[34] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets. Alexan- der Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel Chang, et al. Habitat- embodied ai. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
3d dataset (hm3dd): 1000 large-scale 3d environments for
[35] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12 22, 2023.
[36] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2708-2717, 2022.
[37] Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEFF/VV Conference on Computer Vision and Pattern Recognition, pages 4840 4851, 2024.
12

---

[38] Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis. Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4018-4028, 2024.
[39] David Rozenberszki, Or Litany, and Angela Dai. Unscene3d: Unsupervised 3d instance segmentation for indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19957-19967, 2024.
[40] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point- bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.006155 2023.
[41] Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge. Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pages 21 214-238. Springer, 2024.
[42] Zhe Liu, Jinghua Hou, Xinyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, and Xiang Bai. Lion: Linear group rnn for 3d object detection in point clouds. Advances Neural Information Processing Systems, 37:13601-13626, 2024.
[43] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm Empowering large language models understand point clouds. In European Conference on Computer Vision, pages 131 131- Springer, 2024.
[44] Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liangyan Gui, and Yu-Xiong Wang. Lexicon3d: Probing visual foundation models for complex 3d scene understanding. Advances in Neural Information Processing Systems, 37:76819-76847 2024.
[45] Zhenyi Liao, Qingsong Xie, Yanhao Zhang. Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via -l-zero-like training. arXiv preprint 2XXi:22504.00833 2025.
[46] Haoyu Zhang. Meng Liu, Zan Gao, Xiaoqiang Lei, Yinglong Wang, and Liqiang Nie. Mul- timodal dialog system: Relational graph-based context-aware question understanding. In Proceedings of the 29th ACM international conference on multimedia, pages 695-703, 2021.
[47] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiamning Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 10933- 10942, 2021.
[48] Yue Yang, Fan- Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16227-16237, 2024.
[49] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13 13142-13153, 2023.
[50] Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. Spartun3d: Situated spatial understanding of 3d world large language models. arXiv preprint arXiv:2410.03878. 2024.
[51] Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning 3d scenes. Advances Neural Information Processing Systems,
2024.
13

---

[52] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16488-16498, 2024.
[53] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11129-1119,, 2022.
[54] Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In The Eleventh International Conference on Learning Representations.
[55] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12222 2024.
[56] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal iims in video analysis. arXiv preprint arXi::2405.21075. 2024.
[57] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472- 19495, 2024.
[58] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of source multimodal models with model, data, and test-time scaling, arXiv preprint arXiv: 2rXiv:2412.05271, 2024.
[59] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03.2 03326, 2024.
[60] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 13100, 2025.
[61] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing iisionlanguage model's perception of the world at any resolution. arXiv preprint (4fi:2200912191,, 2024.
[62] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.. 13923, 2025.
[63] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXin:2410.21276, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state four main contributions made in the paper. Our experimental results also support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have dedicated section to discuss the limitations of our work. Please refer to Appendix F
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have included detailed information the methodology (Appendix D). data construction (Appendix C). and experimental settings (Appendix A).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided instructions data access and preparation. Code files for prompting can be found in the Supplementary Material. The data pipeline, data, and model weights will be publicly available upon paper publication
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc///pp http:/ccccccccguuddeeeseeeeeeeee huabbiiccguidee/odeeeuubiiis
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental settings in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Performing multiple repetitive experiments in order to compute error bars is labor-intensive and has significant overhead for the large language model development.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computational resources needed in order to reproduce the experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpus:/ cc wttpt////tt.ttttiititiiiiiiiiaii
Answer: [Yes]
Justification:
Our research conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We have dedicated section to discuss the societal impacts of our work. Please refer to Appendix
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research builds upon existing datasets rather than creating new ones from the internet. So our work does not pose such risks the best our knowledge.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our work are commonly used datasets with open access. We have adhered to their licenses and provided citations to give them credit.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The data pipeline, data, model weights, and benchmark will be publicly available upon paper publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppccc/ooooeereeee/ //t/////ooeeeeecc//22////1. for what should or should not be described.
21

---

A Experimental Settings
A.1 Benchmarks
To validate the effectiveness of our SpatialMind strategy and ScanForgeQA data in enhancing the sisual-spatial understanding capabilities of VLMs, we conducted evaluations on four comprehensive benchmarks: VSI-Bench [1, OpenEQA 52 ScanQA [53]. and SQA3D (54. The comparisons are
shown in Table 5
Table 5: Comparison of different benchmarks. Note that the statistics for the ScanQA and SQA3D datasets are reported on their respective validation sets.
Dataset
#Questions
#Scenes
Source
Question Types
ScanNet,
VSI- Bench
Configuration, Measurement
55,000
288
ScanNet++,
Estimation, Spatio-temporal Reasoning
ARKitScenes
Object Recognition, Attribute
ScanNet,
OpenEQA
1,899
>18
Reasoning, Spatial Understanding,
HM3D
Functional Reasoning
ScanQA
9,353
71
ScanNet
Object, Color, Quantity, Location
Spatial Relations, Commonsense, Navigation, Multi-hop Reasoning
SQA3D
3,261
65
ScanNet
A.1.1
VSI-Bench
VSI-Bench [ a comprehensive evaluation benchmark designed to assess the visual-spatial rea- soning capabilities of VLMs in dynamic 3D environments. It comprises over 5,000 high-quality question-anne answer pairs grounded in 288 diverse indoor video scenes, sourced from real-world 3D reconstruction datasets such as ScanNet [31], ScanNet++ [35), and ARKitScenes 229 The bench- mark defines eight distinct tasks organized into three major categories: configuration-base tasks (e.g., object counting, relative direction, and path planning), measurement estimation tasks (e.g.. object size, absolute distance, room size), and spatiotemporal reasoning tasks (e.g., object appear- ance order). These tasks collectively assess VLMs' abilities to reason about spatial relationships, estimate scale, and understand temporal information. Evaluation is primarily based on accuracy, with tolerance-based error metrics applied numerical estimation tasks to account for minor deviations. All evaluations follow the official protocols defined VSI-Bench
A.1.2 OpenEQA
OpenBBQA 5 introduced by Meta AI in 2024, is benchmark designed to evaluate the spatial understanding and reasoning capabilities of VLMS in real-world indoor environments. The dataset comprises over 1,600 numan-annotater question-anne answer pairs spanning 180+ diverse indoor scenes and supports two primary evaluation tasks. In our experiment, we focused on the Episodic Memory (EM-EQA) setting, where models must answer questions based solely on a previously observed egocentric video trajectory, without access to external spatial priors. Questions cover wide range of reasoning types, including object attributes, spatial relationships, object locations, and functional reasoning. To evaluate model perfromance on open- ended responses, we employed GPT--4o as an automatic evaluator assess semantic alignment between predicted and ground-truie answers. This enables consistent and reliable scoring across diverse question formats.
A.1.3 ScanQA
ScanQA [53] is large-scale benchmark designed to evaluate spatial question answering in world indoor environments, grounded in richly annotated 3D scans. Built top of the ScanNet [31] dataset, contains over 41,00 41,000 human-curated question-answe -nnswer pairs across 800 RGB-D scenes, with each question grounded in the 3D geometry and semantics of the scene. The dataset covers broad range of open-ended questions involving object attributes, spatial relations, and scene-level understanding,
22

---

Table 6: Performance comparison VSI-Bench. indicates results VSI-Bench (tiny) set and LLaVA-NV denotes the LLaVA-NeXT-Video model.
Obj.
Abs.
Obj.
Room
Rel.
Rel.
Route Plan
Appr. Order
Method
Avg A
Count
Dist.
Size
Size
Dist.
Dir.
Close- source
Human Leveli
94.3
47.0
60.4
45.9
94.7
95.8
95.8
100.0
79.2
Gemini-2.0 Flash
52.4
30.6
66.7
31.8
56.0
46.3
24.5
55.1
45.4
Gemini-1.5 Prof
49.6
28.8
58.6
49.4
46.0
48.1
42.0
68.0
48.8
Gemini-1.5 1.5 Pro
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
45.4
+SpatialMind
63.9
51.8
70.2
47.3
56.3
45.9
42.6
44.3
52.8
7.4%
GPT-4o
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
34.0
+SpatialMind
40.0
27.1
62.7
40.9
41.0
39.6
37.1
38.5
40.8
1 6.8%
Open-source
Intern VL2-8B
23.1
28.7
48.2
39.8
36.7
30.7
29.9
39.6
34.6
+SpatialMind
35.8
28.9
49.7
44.4
37.2
34.8
35.1
45.5
38.9
1 4.3%
+ScanForgeQA
45.3
33.4
54.8
45.0
41.1
36.1
33.4
43.0
41.5
1 6.9%
+Both
47.0
32.8
53.2
46.6
39.8
36.8
37.9
47.5
42.7
1 8.1%
Intern
34.9
26.9
46.5
31.8
42.1
32.2
34.0
39.6
36.0
+ MSpatialMind
36.4
30.0
49.1
41.8
43.8
36.1
35.6
50.0
40.4
1 4.4%
+ScanForgeQA
51.0
29.2
52.7
38.1
47.2
36.4
35.9
47.6
42.3
1 6.3%
+Both
52.2
30.5
54.4
41.0
50.5
37.0
40.2
50.3
44.5
1 8.5%
InterVV2..5-81
7.0
34.1
43.0
42.4
38.0
40.1
23.2
35.9
33.0
+ Mpnd
18.7
29.6
46.5
45.2
40.3
40.9
27.8
46.6
37.0
1 4.0% %
+ScanForgeQA
56.5
33.2
50.7
43.4
39.0
33.1
28.4
34.3
39.8
1 6.8%
+Both
52.8
30.7
53.4
44.8
41.1
38.7
29.6
47.0
42.3
1 9.3%
InterVLL..53382
43.6
33.0
53.0
48.8
53.5
35.7
34.5
34.0
42.0
+SpatialMind
56.1
38.3
59.3
52.3
59.7
43.6
39.0
32.9
47.7
1 5.7 %
LLaVA-NV-7B
48.5
14.0
47.8
24.2
43.5
42.4
34.0
30.6
35.6
+SpatialMind
49.0
22.6
47.9
24.6
41.3
43.0
37.1
29.8
36.9
1 1.3 %
LLAVA-NV-72B
48.9
22.8
57.4
35.3
42.4
36.7
35.0
48.6
40.9
+SpatialMind
52.5
24.5
60.2
37.7
42.7
39.5
37.3
51.0
43.2
1 2.3 %
IideoLLaMA3-1
36.8
22.2
34.7
24.9
44.6
41.7
36.1
28.8
33.7
SpatialMind
38.6
23.9
36.2
36.6
42.3
40.9
35.9
33.2
36.0
1 2.3 %
Qwen2-VL. VL-7B
39.4
25.0
25.8
43.2
32.6
30.9
27.8
32.6
32.2
MSpa
43.7
29.0
29.4
46.5
33.9
32.8
32.0
31.9
34.9
t 2.7
Qwen2.5-VL-7B
40.3
22.2
50.1
38.9
38.0
40.7
3..4
35.9
37.2
+SpatialMind
45.1
25.2
52.1
41.4
38.7
41.6
34.7
34.5
39.2
1 2.0% %
+ScanForgeQA
53.2
30.5
56.8
44.9
42.3
44.0
37.3
37.7
43.3
1 66.1% %
+ Both
55.0
29.5
57.3
44.0
43.5
44.3
38.3
39.2
43.9
t 6.7%
Qwen2.5-VL-722
37.9
28.6
57.4
49.8
45.5
38.4
20.6
35.4
39.2
+SpatialMind
42.3
32.0
61.7
53.8
48.2
43.9
30.4
39.3
44.0
1 4.8%
+ScanForgeQA
45.2
32.7
63.3
52,4
50.1
41.7
32.8
40.2
44.8
1 5.6%
+Both
48.6
34.4
68.9
54.7
53.4
43.9
30.1
42.7
47.1
1 7.9%
23

---

Qwen2.5-VL-77
trash bin
Messuring from the closest point each objoct, which of these objects (chaii, radilator, table, trash bin) the closect the deor?
Scene Decomposition
Question Decomposition
"bed": "global reference point"
Object Relative Distance
"table": 'The table is located approximately meters right
global
point."
Step1: Identily reference target objects the question.
"chair" "The cheir is located and about 2 moters to the right of the global referance point""
Step2: Estimate spalial posilions for these objects.
"wardrobe": "The wardrobe located 1.5 global refarence point." "door": door
Step3: Calculite the Euclidean distance between each target object and reference object
moters to the right of the wardrabe."
approximataly
Stap4: Sort the calcurated distances and choose the target with the smallest distance
"radiator": "The radiator located approximately
1.5 meters the
the globel reference
"trash bin" "There are trash
located approximately 1 meters to the right and slighll in front of the global reference point, while the other
it",
Fine-tuned Qwan2.5-VL-78
table
"window": window located approsimataly maters right the global referance point and above the raciator."
Figure 7: A complete example illustrating the visual prompting process with intermediate outputs.
such as "What is between the sofa and the table?" or "Where is the lamp located?". To evaluate model performance, we adopted BLEU-1 as the primary metric. Given that the answers are typically short and factual, unigram precision offers reliable measure of response quality.
A.1.4 SQA3D
SQA3D 54 is a reasoning-centicc benchmark designed to evaluate situational understanding within 3D indoor scenes. Built 650 scenes from ScanNet (31] 1 it comprises approximately 6,800 annotated situations and over 33,000 questions that span a diverse range of reasoning types, including spatial relations, commonsense reasoning, and multi-hop logical reasoning. Each question is grounded in specific 3D context, requiring precise and spatially informed answers. To evaluate model performance, we used the Exact Match (EM-1) metric. This metric is well-suited for the benchmark, as answers are typically concise, specific, and context-dependent
A.2 Baselines
As shown in Tables our evaluation includes a diverse set of VLM baselines spanning range of architectures and scales. These include open-source models such as Intern VL2 (58) (8B, 40B), Intern VL2.5 588 (8B, 38B), -eLLVA---eT--iiee
59 (7B, 72B), VideoLLaMA3 60 (7B), Qwen2-
VL [6] (7B). and Qwen2.5-VL 662 (7B, 72B). We also evaluated closed-source models, including Gemini-2.0 Flash Gemini-1.5 Pro and GPT-4o [3 For open-source models, we adopted each model's s default parameter settings, including learning rate, number of frames, and input resolution. For closed-source models, GPT-4o processes 16 frames per video, while all Gemini models operate at fixed sampling rate of frame per second (1 FPS). All experiments are conducted on 8 NVIDIA H20 GPUs.
B More Experimental Results
B.1 Performance Comparison
In Table 6 we extended our evaluation to additional VLMs with varying architectures and param- eter scales, applying both the SpatialMind prompting and ScanForgeQA fine-tuning. The results
24

---

bathroom
11179
bedroom
10737
kitchen
3935
dining room
3051
storage 760
store 20
2000
4000
6000
8000
10000
Count
Figure Distribution room types in the ScanForge dataset.
are consistent with those reported our main analysis, further reinforcing the effectiveness and generalizability of proposed methods. Moreover, the combined use of prompting and fine-tuning continues to yield superior performance across models, highlighting their complementary strengths and demonstrating the robustness of our framework.
B.2
Case Study
Figure 7 presents detailed example that illustrates the full prompting process along with intermedi- ate outputs. The scene decomposition module produces approximate spatial descriptions of object locations, effectively capturing the overall layout of the 3D scene. Meanwhile, the question decompo- sition module identifies the question type as "object relative distance" and selects the corresponding reasoning steps to guide inference. By combining this structured information and feeding into the Qwen2.5-V VL-7B model fine-tumed with ScanForgeQA, the correct target object is successfully identified. This example demonstrates the interpretability and effectiveness of our framework in performing multi-step spatial reasoning grounded in visual context.
C More details for ScanForge
C.1 Scene Distribution
Figure presents the distribution of the eight room categories included in ScanForgeQA. The dataset exhibits a clear long-tailed distribution in which categories such as bathroom and bedroom are heavily represented, whereas store and office types comparatively rare. This imbalance is primarily due to the inherent distribution room types in the 3D-FRONT dataset
C.2 Scan Example
We provided two frames captured by the camera during the creation of the scanning video, as shown in Figure visually illustrate this process. Additionally, further provided scanning creation scripts as well as more video demos in the Supplemental Material:
"codes/nay " is the script that creates the navigation scan, presenting the exact implementation.
"Creation Scanning Video.mp4" illustrates the creation process of the scanning video from both first-person and third-person perspectives.
"Scanning Video.mkv" presents the rendered scene resulting from the aforementioned scanning process.
25

---

12 12
C-10
Figure Screenshots from the sean creation process.
C.3 QA Definition
Table 7 provides representative templates and corresponding answer sources for each question type in ScanForgeQA. This information clarifies the coverage of different reasoning categories and indicates which scene annotations are used to derive ground-truth answers.
D More details for SpatialMind
D.1 Prompt for Object Description
As representative example, Figure 10 illustrates the prompt used to generate textual descriptions of object positions. This prompt guides the model to produce structured spatial layouts in natural language based on the visual input. Similar prompts are used for other reasoning components, following consistent design pattern. Other prompts are provided in the "codes/gen scene "henn_exp.py" file of Supplementary Material for reference and reproducibility.
D.2 Reasoning Steps for Different Questions
Figure provides example of the detailed reasoning steps used for the relative distance question type. Each question type paired with a concise, structured reasoning process that offers a generaliz- able solution strategy for VLMs. All reasoning steps for the various question types are included in "codes/reason_stepp"" stepssppptppp""" file of Supplementary Material for reference and reproducibility.
E
Limitations
This paper demonstrates that existing VLMs tend rely more heavily on understanding textual descriptions when performing spatial reasoning. However, textual descriptions alone may not capture spatial semanties intuitively as visual input, potentially limiting the upper bound of performance for such methods. Moreover, we primarily focus on basic single-room 3D scene. Although strong performance has been achieved on OpenEQA (52), which includes some multi-room scenes, more complex settings, such as large-scale indoor layouts and outdoor environments, remain underexplored and represent important directions for future work.
Broader Impacts
Our approach improves visual-spatial understanding in VLMs, enabling better interpretation of object layouts and spatial relationships. This enhancement offers potential benefits in areas such as embodied Al and AR/VR applications, where spatial reasoning is essential. However, similar to other prompting and fine-tuning methods, our framework may inherit biases from pre-trained models or training data, potentially leading to unfair inaceurate spatial interpretations. While our method reduces reliance expensive 3D sensors, evaluating spatial accuracy remains key challenge, especially for
26

---

downstream tasks that demand high robustness and trustworthiness. Despite these limitations, we believe that releasing our framework and dataset will foster transparency, support reproducibility, and accelerate progress toward trustworthy and robust spatial reasoning systems.
Table 7: Overview of question templates and their corresponding answer sources.
Type
Question Template
Answer Source
Attribute Estimation
How many <A> are there in the room?
Object Count
Name of object instance
What is the total number of <A>?
What is the length of the longest side of <A> in meters?
What the size of <A> in square meters?
Dimension of the object model
Object Size
What is the length of the shortest side of <A> in meters?
How tall is <A> in meters?
What the size the room square me-
Scene boundaries and scale
Room Size
ters?
Based on object layout, what is the most likely type of this room?
Room Type
Room labels
Is this space a living room, a kitchen, or something else?
Spatial Reasoning
Which of these objects (<A>, <B>, <C>) is the closest to <R>? Among the listed objects (<A>, <B>, <C>),
The minimum distance between the centroids of objects
Relative Distance
which is closest to <R>?
What is the distance between <A> and <B> in meters?
Euclidean distance between objects
Absolute Distance
Measure the distance from <A> to <B> in meters.
How far is <A> from <B> in meters? If I am standing by <A> and facing <B>, which side is object <R> on? From the viewpoint <A> facing <B>, where is <R>?
The direction of connections between objects
Relative Direction
there gap between <A> and <B>?
Comparison of distances and sizes between objects
Contact Relationship
Are <A> and <B> touching each other?
Hypothesis Analysis
Considering only object sizes, is there enough space to put <A> in <B>? Considering only object dimensions, feasible to place <A> on <B>?
Comparison of sizes between objects
Operation Feasibility
27

---

Prompt
[Task]
You are given a video (multiple frames) capturing an indoor scene. Your goal is recognize fcategories of interest) objects, analyze the spatial layout of the scene, and describe the relative position of each object.
[Instructions]
1. Per-frame analysis:
For each frame, choose one object as a **local reference point*s,
Predict the relative position of all other objects with respect this local reference point.
Express relative positions using simple terms like "left", "right", "front", "behind", "above", "below", and approximate distances (e.g., "2 meters to the right").
Global scene layout:
Take the **local reference point from the first frame** as the **global reference point** for the whole video.
Use overlapping objects between frames to align frames together.
Gradually build the spatial descriptions for all objects relative the global reference point.
[Rules]
If category has multiple instances (e.g., two chairs), describe each instance separately.
Preserve the real-world spatial relationships and distances as accurately as possible.
Use clear and consistent directional and distance terms.
[Output Format]
ONLY Return the result as JSON dictionary following STRICTLY this format:
"category name": "global reference point",
"another category name": "Position description relative the global reference point",
}
Example:
"chair": "The chair is located 1.5 meters to the left and 0.5 meters behind the global reference point",
"table": "The table located 2 meters the right of the global reference point",
"lamp": "The lamp is located meter above and meter behind the global reference point"
Figure 10: Example prompt used to generate textual descriptions of object positions.
28


Reviews:
[{"review_id": "9e7t7GRkNS", "reviewer": "Reviewer_YFSW", "summary": "- This paper presents a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. Specifically, the paper introduces SpatialMind, a structured Chain-of-Thought (CoT) prompting strategy, and ScanForgeQA, a large-scale synthetic QA dataset generated from diverse 3D simulation scenes.\n- Experimental results validate the effectiveness of both SpatialMind and ScanForgeQA.", "questions": "Regarding the ScanForgeQA dataset, are there examples of questions whose answers cannot be directly inferred from metadata alone?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper introduces SpatialMind, a novel structured CoT prompting strategy that enables step-by-step spatial reasoning in VLMs, improving interpretability and performance.\n- The paper proposes ScanForgeQA, a large-scale synthetic QA dataset using an automated scene construction pipeline.\n- The paper has evaluated the effectiveness of proposed methods on various VLMs.\n\n**Weaknesses**\n- The paper proposes SpatialMind, a structured Chain-of-Thought (CoT) prompting strategy, but it lacks comparison with existing baseline prompting strategies. \n- The paper would benefit from a comparative analysis between ScanForgeQA and other existing 3D QA datasets such as SPARTUN3D [1], MSQA [2], and 3D-LLM [3], etc. to highlight the novelty and advantages of ScanForgeQA.\n- It would be better if the paper could provide some sample instances from the ScanForgeQA dataset \n\n[1] SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models\n\n[2] Multi-modal Situated Reasoning in 3D Scenes\n\n[3] 3D-LLM: Injecting the 3D World into Large Language Models", "was_revised": true, "final_justification": "The paper proposes a scalable dataset generation pipeline, along with ScanForgeQA, a synthetic spatial question-answering dataset that could serve as a valuable resource for the research community. During the rebuttal, the authors also provide comparison with existing 3D QA datasets. Overall, I believe the paper is good to be accepted.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n> **W1**: The paper proposes SpatialMind, a structured Chain-of-Thought (CoT) prompting strategy, but it lacks comparison with existing baseline prompting strategies.\n\n**A1**: To establish comparative baselines, we utilized both GPT-4o (which we used to develop our SpatialMind) and GPT-4.1 (a more advanced model) to generate two distinct prompts focused on spatial understanding, as detailed below.\n\n```\n# Baseline prompt 1 from GPT-4o\n\nYou are a spatial reasoning expert. The input is a scanned indoor video. Your task is to analyze the video to extract accurate spatial information and use it to answer specific questions about the scene.\n\nStep 1: Scene Understanding\nFrom the video, identify:\n-The room structure: walls, doors, windows\n-Objects: such as a refrigerator, microwave, window, sofa, TV, etc.\n-Estimate the location of each object, using relative coordinates or spatial descriptions (e.g., \"near the left wall\", \"close to the door\", etc.)\n\nStep 2: Spatial Reasoning and Question Answering\n-Use the extracted spatial information to answer the given question.\n-You may refer to their positions using coordinates, directional terms (e.g., left, right, front), or visual layout based on the video.\n\nAssume that you have 3D spatial awareness and can mentally reconstruct the layout of the room from the video.\n```\n\n```\n# Baseline prompt 2 from GPT-4.1\n\nCarefully watch the indoor scan video and follow these steps:\n1.Identify and list all relevant objects and features visible.\n2.Locate each object in the room relative to fixed points (like doors or corners).\n3.Map the spatial layout of the room, marking object positions.\n4.Analyze the specific spatial question using visual cues (distance, direction, or size).\n5.Explain your reasoning step by step, citing evidence from the video.\n6.Provide a clear, justified answer.\n\nIf you are uncertain due to missing or unclear information, mention it and specify what would resolve it.\n```\n\nWe then evaluated the performance of different prompts on the VSI-Bench benchmark using both configurations of the Qwen2.5-VL model, as presented in the table below. \n\n| Source  | Prompt  | Avg (7B) | Avg (72B) |\n|------|------|------|------|\n| VSI-Bench (Default) | Answer with the option's letter from the given choices directly | 37.2 | 39.2 |\n| GPT-4o | Above prompt 1 | 37.6 | 41.9 |\n| GPT-4.1 | Above prompt 2 | 37.5 | 42.3 |\n| Ours | SpatialMind | **39.2** | **44.0** |\n\nThe results in the table indicate that different prompting strategies have a substantial impact on the spatial reasoning abilities of VLMs, and our SpatialMind prompting strategy proves more effective in enhancing spatial understanding. We will include the above comparison in the supplementary materials.\n\n> **W2**: The paper would benefit from a comparative analysis between ScanForgeQA and other existing 3D QA datasets such as SPARTUN3D, MSQA, and 3D-LLM, etc. to highlight the novelty and advantages of ScanForgeQA.\n\n**A2**: Following your suggestion, we compared SPARTUN3D, MSQA, 3D-LLM, and our proposed ScanForgeQA dataset from multiple aspects, as shown in the table below. \n\n| Dataset  | Data Source | Scene Number | Scene Fromat | Question Type | QA Number | Scalability |\n|------|------|------|------|------|------|------|\n| SPARTUN3D | 3RScan | 478|Point Cloud | 4 | 133K | Difficult |\n| MSQA | ScanNet、3RScan、ARKitScenes | 1.7K|Point Cloud | 9 | 251K | Difficult |\n| 3D-LLM | ScanNet、HM3D | 1.2K|Point Cloud | 9 | 300K | Difficult |\n| ScanForgeQA (Ours) | Simulation/Synthesis | 34K|Scan Video | 9 |925K | Easy |\n\nWe adopted a novel simulated data generation pipeline, which brings the following advantages:\n\n- **Diverse scenes**. Existing datasets all utilize fixed real-world scene data, resulting in limited and homogeneous scene types. In contrast, synthetic scenes feature more diverse furniture and spatial layouts, leading to a richer and more comprehensive variety of scene types.\n- **Large scale**. The number of QA pairs is substantially greater, ranging from three to seven times more than those found in the other datasets.\n- **Easily extensible**. The dataset is easily scalable, as scene compositions can be modified by adjusting furniture layouts or changing generation prompts. This level of flexibility is difficult to achieve with real-world scenes.\n- **Aligned with human perception**. The ScanForgeQA dataset utilizes simplified scene scanning videos to more closely simulate how humans perceive their surroundings. In contrast, other datasets typically employ globally captured point cloud data, which does not fully reflect the selective and incremental way humans observe and understand environments.\n\nWe will include this comparison in the revised manuscript and cite the relevant references to highlight the distinctions and advantages of our ScanForgeQA dataset.\n\n> **W3**: It would be better if the paper could provide some sample instances from the ScanForgeQA dataset.\n\n**A3**: Thank you for bringing this to our attention. In the supplementary materials, we included only the data construction templates and videos, but did not provide any sample instances. To address this, we have now included 20 randomly selected examples from the ScanForgeQA dataset below. Due to the rebuttal policy, we are unable to share the corresponding videos, and we apologize for any inconvenience this may cause. In the future, we will include two complete sample instances in the supplementary materials, and we will also release the constructed dataset as open source.\n\n```\nscene_id,video_id,question_type,question,answer,options\n39284751,39284751_1,Object Count,\"How many cushions are there in the room?\",3,\n40718296,40718296_1,Room Size,\"What is the size of the room in square meters?\",28.5,\n74382961,74382961_3,Relative Distance,\"Which of these objects (microwave, fridge, oven) is the closest to the sink?\",C,\"['A. microwave', 'B. fridge', 'C. oven']\"\n29538410,29538410_3,Absolute Distance,\"How far is the television from the coffee table in meters?\",1.3,\n61028437,61028437_3,Operation Feasibility,\"Considering only object dimensions, is it feasible to place the printer on the bookshelf?\",B,\"['A. Yes', 'B. No']\"\n48971263,48971263_1,Contact Relationship,\"Is there a gap between the refrigerator and the wall?\",B,\"['A. Yes', 'B. No']\"\n28714053,28714053_2,Object Size,\"What is the length of the longest side of the table in meters?\",1.8,\n54109823,54109823_3,Room Type,\"Based on object layout, what is the most likely type of this room?\",B,\"['A. Kitchen', 'B. Bedroom', 'C. Living Room', 'D. Bathroom']\"\n52813406,52813406_3,Object Size,\"What is the size of the sofa in square meters?\",2.4,\n94278315,94278315_2,Room Type,\"Is this space a living room, a kitchen, or something else?\",A,\"['A. Living Room', 'B. Kitchen', 'C. Bathroom', 'D. Office']\"\n10837429,10837429_1,Relative Distance,\"Among the listed objects (lamp, bookshelf, plant), which one is closest to the window?\",B,\"['A. lamp', 'B. bookshelf', 'C. plant']\"\n87419205,87419205_1,Relative Direction,\"From the viewpoint at the door facing the desk, which side is the chair on?\",B,\"['A. left', 'B. right', 'C. front', 'D. back']\"\n17823964,17823964_2,Operation Feasibility,\"Considering only object sizes, is there enough space to put a microwave on the countertop?\",A,\"['A. Yes', 'B. No']\"\n82357164,82357164_1,Object Size,\"What is the length of the shortest side of the bed in meters?\",1.2,\n28714053,28714053_2,Object Size,\"What is the length of the longest side of the table in meters?\",1.8,\n43582170,43582170_2,Relative Direction,\"If I am standing by the wardrobe and facing the mirror, which side is the lamp on?\",A,\"['A. left', 'B. right', 'C. front', 'D. back']\"\n57418230,57418230_2,Object Size,\"How tall is the bookshelf in meters?\",1.9,\n37651942,37651942_2,Object Count,\"What is the total number of towels in the room?\",7,\n92107345,92107345_1,Object Count,\"How many chairs are there in the room?\",4,\n61823907,61823907_3,Contact Relationship,\"Are the nightstand and the bed touching each other?\",A,\"['A. Yes', 'B. No']\"\n76429381,76429381_2,Absolute Distance,\"What is the distance between the dining table and the sofa in meters?\",2.9,\n```\n\n> **Q1**: Regarding the ScanForgeQA dataset, are there examples of questions whose answers cannot be directly inferred from metadata alone?\n\n**A4**: No, there are no such examples in the ScanForgeQA dataset. During data construction, we first collected all scene metadata and then designed various question templates based on this metadata to ensure that every defined question is supported by the corresponding information. Furthermore, the questions in our dataset focus on fundamental attributes of objects within scenes, such as dimension, quantity, and direction, and do not involve complex question types that require extensive reasoning."}, {"review_id": "RdmCFjt6s9", "reviewer": "Reviewer_9tnn", "summary": "This paper identifies key challenges faced by current video models in spatial reasoning tasks and proposes the SpatialMind prompting strategy, which enhances reasoning by first constructing a cognition map. In addition, the authors introduce ScanForgeQA, a large-scale 3D scene reasoning dataset, and provide detailed descriptions of its generation pipeline. Experimental results demonstrate the effectiveness of the proposed SpatialMind approach and ScanForgeQA data.", "questions": "- In the \"both\" setting, is the ScanForgeQA fine-tuning data also annotated using the SpatialMind prompting strategy?\n- How does SpatialMind prompting affect inference time?\n- Are there any observed cases where SpatialMind prompting misleads the model, leading it to predict incorrect answers?\n- What are the authors' thoughts on the effectiveness of general video pretraining for spatial reasoning tasks? For instance, in Table 1, if ScanForgeQA were replaced with a similarly sized dataset of general video QA, would performance still improve?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper clearly poses the connection between spatial reasoning and video training, a relationship that remains largely underexplored in current VLM research.\n- Developing a complicated dataset contributes valuable resources to the spatial reasoning research community.\n### Weaknesses\n- In Figure 4, the comparison involves training with ScanForgeQA and a small portion of ShareGPT4Video. For a more comprehensive ablation, it would be helpful to include results from fine-tuning on ShareGPT4Video data alone.\n- The performance results on ScanForgeQA should be reported explicitly.", "was_revised": true, "final_justification": "The two strengths: 1. discussing the connection between spatial reasoning and video training. 2. curating a complicated dataset. These two contributions both are valuable to the spatial reasoning research community. Therefore, I choose my final rating of borderline accept.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n> **W1**: In Figure 4, the comparison involves training with ScanForgeQA and a small portion of ShareGPT4Video. For a more comprehensive ablation, it would be helpful to include results from fine-tuning on ShareGPT4Video data alone.\n\n**A1**: Following your suggestion, we have fine-tuned Qwen2.5-VL-7B model using only the ShareGPT4Video data. The experimental results on the MVBench and Video-MME benchmarks can be found in the sixth row of the table below.\n\n| Fine-tuned data | MVBench  | Video-MME (w sub) |\n|------|------|------|\n| Base | 69.6 | 71.6 | \n| +ScanForgeQA | 69.8 | 70.7 |\n| +ScanForgeQA & 5%ShareGPT4Video| 69.9 | 71.5 |\n| +ScanForgeQA & 10%ShareGPT4Video| 69.9 | 71.8 |\n| +ShareGPT4Video | 69.7 | 72.0 |\n\nThe results indicate that for MVBench, which includes a subset of spatial reasoning tasks, incorporating ScanForgeQA yields better performance than ShareGPT4Video. However, neither dataset alone surpasses the performance achieved by combining both. In contrast, for Video-MME, which is designed to evaluate event understanding, the general-purpose ShareGPT4Video dataset is more closely aligned with the evaluation tasks and therefore achieves the best performance.\n\n> **W2**: The performance results on ScanForgeQA should be reported explicitly.\n\n**A2**: We apologize if we have misunderstood your intent, and we appreciate the opportunity to clarify. Our preliminary response is as follows: \n\nScanForgeQA is a fine-tuning dataset that we constructed, which does not include validation or test sets, and therefore, we do not provide evaluation results on this dataset. The fine-tuning results of various model architectures using this dataset can be found in Table 1 of the manuscript (rows marked with \"+ScanForgeQA\"). If you could kindly provide further clarification regarding this question, we would be more than happy to offer a more detailed and specific response.\n\n> **Q1**: In the \"both\" setting, is the ScanForgeQA fine-tuning data also annotated using the SpatialMind prompting strategy?\n\n**A3**: No, the ScanForgeQA fine-tuning data is not annotated using the prompting strategy. In the \"both\" setting, we first fine-tune the VLMs using the ScanForgeQA data, and then apply the SpatialMind prompting strategy for inference with the fine-tuned VLMs.\n\n> **Q2**: How does SpatialMind prompting affect inference time?\n\n**A4**: The impact of the SpatialMind prompting strategy on inference time is primarily due to the increased prompt length. To evaluate the effect of this strategy on inference speed, we conducted experiments using the Qwen2.5-VL-7B model on an A100 GPU. Specifically, we performed inference on 100 samples and report the average inference time per sample. The results are shown in the table below. \n\n| Method | Time | Time (with Flash Attention) |\n|------|------|------|\n| Base | 8.37s | 5.71s | \n| +SpatialMind | 10.84s | 6.03s|\n\nAs observed, applying the SpatialMind prompting strategy does lead to slower inference, which is attributable to the increased number of input tokens. However, this impact can be mitigated through inference acceleration techniques for VLMs such as Flash Attention. Moreover, when dealing with large-scale data, methods such as vLLM can be further employed to narrow the gap between the two.\n\n> **Q3**: Are there any observed cases where SpatialMind prompting misleads the model, leading it to predict incorrect answers?\n\n**A5**: Yes. Although the SpatialMind prompting strategy demonstrates outstanding performance in enhancing spatial reasoning abilities, there are indeed rare cases where the prompts may inadvertently mislead the model, resulting in incorrect predictions. This is mainly due to inaccurate object recognition or significant errors in distance estimation, which can cause intermediate mistakes to accumulate and affect the final outcome. Such issues are common among many prompting methods and can be alleviated by controlling the intermediate results. Despite these isolated cases, our experiments show that SpatialMind prompting is generally effective at decomposing tasks and guiding model reasoning, leading to a significant reduction in systematic errors.\n\n> **Q4**: What are the authors' thoughts on the effectiveness of general video pretraining for spatial reasoning tasks? For instance, in Table 1, if ScanForgeQA were replaced with a similarly sized dataset of general video QA, would performance still improve?\n\n**A6**: We argue that current mainstream general video pretraining datasets primarily focus on the semantic understanding of temporal events in videos, such as action recognition and event relationships, rather than on fine-grained 3D spatial perception and reasoning. Consequently, models pretrained on such datasets struggle to transfer effectively to 3D spatial reasoning tasks.\n\nTo further validate this, we conducted an experiment in which we replaced our spatial dataset, ScanForgeQA (containing 925K samples), with a similarly sized general video QA dataset, HowToVQA69M (925K samples randomly selected). The experiments were carried out using the Qwen2.5-VL-7B model, and the results on VSI-Bench are presented in the table below.\n\n| Method | Object Count | Relative Distance | Route Plan | Avg |\n|------|------|------|------|------|\n| Base | 40.3 | 38.0 | 31.4 | 37.2 |\n| +ScanForgeQA | **53.2** | **42.3** | **37.3** | **43.3** |\n| +HowToVQA69M (925K) | 40.7 | 35.9 | 29.8 | 35.6 |\n\nWe observe that this replacement does not improve performance on spatial reasoning tasks; instead, it leads to a performance drop due to differences in the data. This further highlights the limitations of general video QA datasets in effectively supporting 3D spatial reasoning. Therefore, we believe that dedicated spatial datasets are both essential and irreplaceable for enhancing a model’s spatial reasoning capabilities."}, {"review_id": "o1suJ4ENzN", "reviewer": "Reviewer_mMuG", "summary": "This paper aims to address the limitation of state-of-the-art vision-language models (VLMs) in performing spatial reasoning over 3D scenes. Specifically, it introduces the SpatialMind approach, which decomposes 3D spatial reasoning into structured multi-step reasoning for complex scenes and questions. Furthermore, to train the approach, the authors also propose the large-scale ScanForgeQA dataset, which includes questions on attribute estimation, spatial reasoning, and hypothesis analysis to encourage fine-grained spatial learning. The paper demonstrates the effectiveness of the proposed SpatialMind approach by showing consistent improvements in spatial reasoning performance across the VSI-Bench, OpenEQA, ScanQA, and SQA3D benchmarks.", "questions": "Please look at the abovementioned weaknesses. It will be much more insightful to include empirical comparisons to more VLMs that are targeted at spatial reasoning.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength 1 - In terms of clarity and quality of the paper, the paper is relatively well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the proposed SpatialMind  approach as well as the data generation pipeline.\n\nStrength 2 - The contributions of this work are significant since it addresses an important research problem. In particular, the data generation pipeline introduced to curate ScanForgeQA provides a cost-effective way to automatically generate large-scale, high-quality QA datasets using 3D simulation environments and LLM-guided scene synthesis. Both dataset and pipeline can be useful to further research in this topic. \n\nStrength 3 - Due to the way that the SpatialMind approach incorporates scene and question decomposition into step-wise reasoning, it can be beneficial for generating rationales that are more interpretable.\n\nWeakness 1 - The empirical results included in the paper do not compare to some VLMS that are specifically targeted at 3D spatial reasoning and understanding, such as 3D-LLM and 3DLLM-Mem among many others. It will be much more insightful if such empirical comparisons are included.", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n> **W1**: The empirical results included in the paper do not compare to some VLMS that are specifically targeted at 3D spatial reasoning and understanding, such as 3D-LLM and 3DLLM-Mem among many others. It will be much more insightful if such empirical comparisons are included.\n\n**A1**: To enable a more comprehensive comparison, we included two additional categories of baselines focused on spatial understanding. The first category uses 3D point clouds as input, such as 3D-LLM and Chat-Scene. The second category consists of concurrent works on arXiv that also take scanned videos as input, including Struct2D, SpaceR-7B, and Video-R1-7B. The experimental results are presented in the table below, with corresponding explanations provided as follows.\n\n| Method | Video-Input Only | ScanQA Val. (BLEU-1) | VSI-Bench (Avg. Accuracy)|\n|------|------|------|------|\n| 3D-LLM (BLIP2-FlanT5) | ✖ | 39.3 | -|\n| Chat-Scene | ✖ | 43.2 |-|\n| Struct2D | ✔ | - | 41.9 |\n| SpaceR-7B | ✔ | - | 41.6 |\n| Video-R1-7B | ✔ | - | 34.6 |\n| Qwen2.5-VL-7B | ✔ | 32.5 | 37.2 |\n| +Both (Ours) | ✔ | 37.9 | **43.9** |\n| Qwen2.5-VL-72B | ✔ | 35.4 | 39.2 |\n| +Both (Ours) | ✔ | **44.1** | **47.1** |\n\nExplanation:  \n1. Since VSI-Bench does not provide point cloud data, methods based on 3D point clouds (e.g., 3D-LLM and Chat-Scene) are unable to produce corresponding results.\n2. Baseline 3DLLM-Mem has not reported experimental results on either of the two datasets mentioned above.\n3. The concurrent works that take scanned videos as input (e.g., Struct2D, SpaceR-7B, and Video-R1-7B) have not reported results on the ScanQA dataset.\n\nBased on the experimental results shown in the table, we draw the following observations:\n1. Compared to spatial understanding baselines that use 3D point clouds as input, our method achieves competitive performance using only 2D scanned video as input.\n2. In comparison with concurrent works that also take scanned videos as input, our framework demonstrates a significant performance advantage.\n3. Our method is highly flexible and can substantially enhance spatial reasoning capabilities across different backbone models.\n\nFor completeness, we will include the above results and analysis, and cite all relevant references (including 3DLLM-Mem) in the revised manuscript."}, {"review_id": "L7cWPM7yeC", "reviewer": "Reviewer_qZj9", "summary": "This paper presents a general framework for enhancing the VLMs’ spatial understanding ability, consisting of two parts: \n-\tSpatialMind: A structured chain-of-thought prompting strategy that divides the spatial understanding problems into multiple steps for reasoning.\n-\tScanForgeQA: A pipeline for automatically constructing spatial understanding QA data in the simulation environments is proposed, and a large-scale QA dataset for spatial reasoning is constructed based on this pipeline.\nThe experimental results show that both parts can effectively improve the performance of various VLMs on multiple benchmarks. Moreover, combining the two parts can further enhance the performance.", "questions": "1.\tWhat are the specific experimental settings in the comparative experiments of different representation formats, and what are the detailed settings for fine-tuning on ScanForgeQA dataset?\n2.\tThe SpatialMind pipeline relies on GPT-4o for both scene decomposition and question decomposition. How it works if totally use the evaluated VLM in the pipeline?\n3.\tConcerns about VLM’s hallucination: During scene decomposition process, will there be a situation where the local coordinates estimated by VLM for each frame conflict with each other? If such a situation exists, then how can they be integrated into global coordinates?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n-\tThis paper is well-structed and the detailed information of the method is provided\n-\tThe proposed SpatialMind strategy and ScanForgeQA dataset is easy to apply and applicable to various VLMs, thus it’s valuable for future research\n-\tThe paper investigates which type of scene representation is more effective for VLMs.\n\n# Weaknesses\n-\tWill the accuracy of coordination estimation affect the model performance? The authors should explain more about it.\n\n-      What is the gap between synthesis data and real data?", "was_revised": true, "final_justification": "I recommend the paper to be accepted.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n> **W1**: Will the accuracy of coordination estimation affect the model performance? The authors should explain more about it.\n\n**A1**：The accuracy of coordinate estimation has a direct and significant impact on the overall performance of models in spatial reasoning tasks. Specifically, the estimated spatial coordinates of objects are provided to VLMs alongside the scanned video for answer prediction, meaning that model decisions are influenced by the estimated coordinates. If these estimates are inaccurate, for example if the predicted location of a target object deviates from its actual position, subsequent spatial reasoning (such as determining whether something is to the \"left\", \"right\", \"front\", or \"behind\") may be misled, resulting in incorrect answers.\n\nIn our experiments (as shown in Figure 3 of the manuscript), we observed that more detailed coordinate representations (e.g., 3D map) yield less improvement than coarser textual position descriptions. This is mainly because current VLMs are not yet capable of accurately interpreting fine-grained spatial coordinates, and more detailed representations may introduce additional noise (or hallucination). Therefore, improving the accuracy of coordinate estimation is essential for further enhancing a model's spatial reasoning capabilities.\n\n> **W2**: What is the gap between synthesis data and real data?\n\n**A2**: Regarding the gap between synthetic data and real data, we summarize the following four aspects:\n\n| Comparison Aspect | Synthetic Data  | Real Data |\n|-------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------|\n| Scene Diversity               | Covers various room types (living room, bedroom, kitchen, office, etc.), offering rich diversity | Usually limited to a few scene types (mostly kitchens and daily environments), with less diversity |\n| Layout Flexibility | Furniture types, quantities, and positions can be flexibly adjusted; scenes can be quickly generated or modified programmatically with low cost | Once scanned, layout and objects are fixed; modification and expansion are difficult and costly    |\n| Data Scale                    | Easy to generate large-scale, diverse datasets in bulk              | Data scale is limited by real-world collection and difficult to expand on a large scale            |\n| Realism                       | Visual and physical properties can be realistically simulated, but may still differ from real-world scenarios | Directly collected from the real world, providing high authenticity                                |\n\nAs shown in the table, synthetic data can be generated at scale, supports diverse scene types, and allows flexible modifications to layouts. However, it may lack the full authenticity of real-world environments. In contrast, real data offers high fidelity and realism but is constrained by limited scene diversity, difficulty in modification, and challenges in scalability.\n\n> **Q1**: What are the specific experimental settings in the comparative experiments of different representation formats, and what are the detailed settings for fine-tuning on ScanForgeQA dataset?\n\n**A3**: For the experiments involving different representation formats (as shown in Figure 3 of the manuscript), we performed inference using the default evaluation code provided by VSI-Bench, which is built on top of the lmms_eval framework. The experimental settings are as follows. For all other parameters not explicitly stated, we followed the default configurations.\n\n| Parameter|GPT-4o | Qwen2.5-VL | \n|------|------|------|\n| Version | 2024-08-06 | 7B/72B |\n| Frames | 16 | 16 |\n\nFor the fine-tuning experiments on the ScanForgeQA dataset, we used the official codebase provided by InternVL2. For the Qwen2.5-VL model, fine-tuning was conducted using the LLaMA-Factory framework. The detailed settings are as follows:\n\n| Parameter           | InternVL2 (8B) | InternVL2 (40B) | Qwen2.5-VL (7B) | Qwen2.5-VL (72B) |\n|---------------------|----------------|-----------------|-----------------|------------------|\n| Frames              | 8              | 8               | 16              | 16               |\n| Batch Size          | 32             | 16              | 32              | 8                |\n| Learning Rate       | 4e-5           | 2e-5            | 2e-5            | 1e-5             |\n| Epochs              | 1              | 1               | 1               | 1                |\n| Optimizer           | AdamW          | AdamW           | AdamW           | AdamW            |\n| Image Resolution    | 224×224        | 224×224         | 224×224         | 224×224          |\n| Max Sequence Length          | 4096           | 4096            | 2048            | 2048             |\n| Weight Decay        | 0.05           | 0.05            | 0.01               | 0.01                |\n| Warmup Ratio        | 0.03           | 0.03            | 0.03            | 0.03             |\n| LoRA Rank           | 8              | 8               | 8               | 8                |\n| LoRA Alpha          | 16             | 16              | 16              | 16               |\n\nAll experiments are conducted on 8 NVIDIA H20 GPUs. We will include the detailed experimental settings mentioned above in the supplementary materials and provide the full implementation details in the open-source repository.\n\n> **Q2**: The SpatialMind pipeline relies on GPT-4o for both scene decomposition and question decomposition. How it works if totally use the evaluated VLM in the pipeline?\n\n**A4**: In the scene decomposition of the SpatialMind pipeline, we employ GPT-4o to extract all objects mentioned across questions related to the same scene, and then prompt the VLMs (e.g., Qwen2.5-VL) to carry out the subsequent reasoning process. GPT-4o is selected for its high API concurrency efficiency, low preprocessing latency, and strong performance. Nevertheless, the entire pipeline can also be executed using the evaluation VLM itself, simply by replacing GPT-4o with the target model in this step.\n\nTo investigate the potential impact of this modification, we conducted an experiment on VSI-Bench using the Qwen2.5-VL-7B model. The results are shown in the table below.\n\n| Method | Average Accuracy | \n|------|------|\n| Qwen2.5-VL-7B | 37.2|\n| +SpatialMind (Original) | 39.2 |\n| +SpatialMind (w/o GPT-4o) | 38.6 |\n\nThe results show a decline in performance after the modification. Given that extracting objects from the question during scene decomposition is a relatively simple task with low potential for error, we speculate that the performance drop may stem from inaccuracies in the reasoning steps generated by the VLMs during question decomposition.\n\n> **Q3**: Concerns about VLM’s hallucination: During scene decomposition process, will there be a situation where the local coordinates estimated by VLM for each frame conflict with each other? If such a situation exists, then how can they be integrated into global coordinates?\n\n**A5**: Yes, hallucination is a common issue in VLMs, which can lead to conflicts or inconsistencies in local coordinate estimations across different frames. This phenomenon may explain why the generated 3D map and 2D grid perform worse than the text-based position descriptions in Figure 3 of the manuscript. As the granularity of the generated representation increases, the likelihood and severity of hallucinations also increase. In our experiments, we did not focus on addressing this issue; instead, we chose to use text-based position descriptions generated by VLMs as the scene representation format, as they yielded better overall performance."}]

Author rebuttal:

Meta-review: After the review, rebuttal and discussion all reviewers recommend acceptance. The AC agrees.

The reviewers appreciated the new data pipeline and resulting dataset.
Decision: Accept (spotlight)