Title: Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual-Spatial Understanding

Vision-Language Model

Prompting
Abstract: Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision-language models (VLMs). To address these challenges, we present a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes through an automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a general framework for enhancing the VLMs’ spatial understanding ability, consisting of two parts: \n-\tSpatialMind: A structured chain-of-thought prompting strategy that divides the spatial understanding problems into multiple steps for reasoning.\n-\tScanForgeQA: A pipeline for automatically constructing spatial understanding QA data in the simulation environments is proposed, and a large-scale QA dataset for spatial reasoning is constructed based on this pipeline.\nThe experimental results show that both parts can effectively improve the performance of various VLMs on multiple benchmarks. Moreover, combining the two parts can further enhance the performance.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n-\tThis paper is well-structed and the detailed information of the method is provided\n-\tThe proposed SpatialMind strategy and ScanForgeQA dataset is easy to apply and applicable to various VLMs, thus it’s valuable for future research\n-\tThe paper investigates which type of scene representation is more effective for VLMs.\n\n# Weaknesses\n-\tWill the accuracy of coordination estimation affect the model performance? The authors should explain more about it.\n\n-      What is the gap between synthesis data and real data?\n\nQUESTIONS:\n1.\tWhat are the specific experimental settings in the comparative experiments of different representation formats, and what are the detailed settings for fine-tuning on ScanForgeQA dataset?\n2.\tThe SpatialMind pipeline relies on GPT-4o for both scene decomposition and question decomposition. How it works if totally use the evaluated VLM in the pipeline?\n3.\tConcerns about VLM’s hallucination: During scene decomposition process, will there be a situation where the local coordinates estimated by VLM for each frame conflict with each other? If such a situation exists, then how can they be integrated into global coordinates?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the limitation of state-of-the-art vision-language models (VLMs) in performing spatial reasoning over 3D scenes. Specifically, it introduces the SpatialMind approach, which decomposes 3D spatial reasoning into structured multi-step reasoning for complex scenes and questions. Furthermore, to train the approach, the authors also propose the large-scale ScanForgeQA dataset, which includes questions on attribute estimation, spatial reasoning, and hypothesis analysis to encourage fine-grained spatial learning. The paper demonstrates the effectiveness of the proposed SpatialMind approach by showing consistent improvements in spatial reasoning performance across the VSI-Bench, OpenEQA, ScanQA, and SQA3D benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength 1 - In terms of clarity and quality of the paper, the paper is relatively well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the proposed SpatialMind  approach as well as the data generation pipeline.\n\nStrength 2 - The contributions of this work are significant since it addresses an important research problem. In particular, the data generation pipeline introduced to curate ScanForgeQA provides a cost-effective way to automatically generate large-scale, high-quality QA datasets using 3D simulation environments and LLM-guided scene synthesis. Both dataset and pipeline can be useful to further research in this topic. \n\nStrength 3 - Due to the way that the SpatialMind approach incorporates scene and question decomposition into step-wise reasoning, it can be beneficial for generating rationales that are more interpretable.\n\nWeakness 1 - The empirical results included in the paper do not compare to some VLMS that are specifically targeted at 3D spatial reasoning and understanding, such as 3D-LLM and 3DLLM-Mem among many others. It will be much more insightful if such empirical comparisons are included.\n\nQUESTIONS:\nPlease look at the abovementioned weaknesses. It will be much more insightful to include empirical comparisons to more VLMs that are targeted at spatial reasoning.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper identifies key challenges faced by current video models in spatial reasoning tasks and proposes the SpatialMind prompting strategy, which enhances reasoning by first constructing a cognition map. In addition, the authors introduce ScanForgeQA, a large-scale 3D scene reasoning dataset, and provide detailed descriptions of its generation pipeline. Experimental results demonstrate the effectiveness of the proposed SpatialMind approach and ScanForgeQA data.

STRENGTHS AND WEAKNESSES:
### Strengths
- The paper clearly poses the connection between spatial reasoning and video training, a relationship that remains largely underexplored in current VLM research.
- Developing a complicated dataset contributes valuable resources to the spatial reasoning research community.
### Weaknesses
- In Figure 4, the comparison involves training with ScanForgeQA and a small portion of ShareGPT4Video. For a more comprehensive ablation, it would be helpful to include results from fine-tuning on ShareGPT4Video data alone.
- The performance results on ScanForgeQA should be reported explicitly.

QUESTIONS:
- In the "both" setting, is the ScanForgeQA fine-tuning data also annotated using the SpatialMind prompting strategy?
- How does SpatialMind prompting affect inference time?
- Are there any observed cases where SpatialMind prompting misleads the model, leading it to predict incorrect answers?
- What are the authors' thoughts on the effectiveness of general video pretraining for spatial reasoning tasks? For instance, in Table 1, if ScanForgeQA were replaced with a similarly sized dataset of general video QA, would performance still improve?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
- This paper presents a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. Specifically, the paper introduces SpatialMind, a structured Chain-of-Thought (CoT) prompting strategy, and ScanForgeQA, a large-scale synthetic QA dataset generated from diverse 3D simulation scenes.
- Experimental results validate the effectiveness of both SpatialMind and ScanForgeQA.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The paper introduces SpatialMind, a novel structured CoT prompting strategy that enables step-by-step spatial reasoning in VLMs, improving interpretability and performance.
- The paper proposes ScanForgeQA, a large-scale synthetic QA dataset using an automated scene construction pipeline.
- The paper has evaluated the effectiveness of proposed methods on various VLMs.

**Weaknesses**
- The paper proposes SpatialMind, a structured Chain-of-Thought (CoT) prompting strategy, but it lacks comparison with existing baseline prompting strategies. 
- The paper would benefit from a comparative analysis between ScanForgeQA and other existing 3D QA datasets such as SPARTUN3D [1], MSQA [2], and 3D-LLM [3], etc. to highlight the novelty and advantages of ScanForgeQA.
- It would be better if the paper could provide some sample instances from the ScanForgeQA dataset 

[1] SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models

[2] Multi-modal Situated Reasoning in 3D Scenes

[3] 3D-LLM: Injecting the 3D World into Large Language Models

QUESTIONS:
Regarding the ScanForgeQA dataset, are there examples of questions whose answers cannot be directly inferred from metadata alone?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03