Title: Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal learning

spatial reasoning

multimodal large language models
Abstract: Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D representations derived from perception?
In this work, we introduce Struct2D, a perception-guided prompting framework that combines bird’s-eye-view (BEV) images with object marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MLLMs (e.g., GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with projected 2D inputs, effectively handling tasks such as relative direction estimation and route planning.
Motivated by these findings, we construct a large-scale instructional tuning dataset, \textbf{Struct2D-Set}, using an automated pipeline that generates fine-grained QA pairs grounded in 3D indoor scenes. We then fine-tune an open-source MLLM (Qwen2.5VL) using Struct2D-Set, relying on noisy 3D perception rather than ground-truth annotations. Despite this, the tuned model achieves strong performance across multiple spatial reasoning benchmarks, including 3D question answering, captioning, and object grounding, spanning eight diverse reasoning categories.
Our approach demonstrates that structured 2D inputs can effectively bridge perception and language reasoning in MLLMs—without requiring explicit 3D representations as input. We will release both our code and dataset to support future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Struct2D, a prompting solution for large multimodal models (LMMs) that prompts LMMs with bird’s-eye-view (BEV) images, object marks, object-centric metadata, and additional keyframe information. Zero-shot analysis on closed-source LMMs indicate that the proposed prompting strategy achieves promising performance. Therefore, additionally an instruction tuning dataset Struct2D-Set is proposed for finetuning the open-source LMMs on this specific prompting strategy. Experimental results show that finetuning with the newly curated Struct2D-Set achieves superior performance across different benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The workflow of this paper is smooth. First, the paper shows promising results with zero-shot Struct2D prompting strategy for closed-source LMMs. Then, the paper constructs an instruction tuning set for supervised finetuning on open-source LMMs, and further boosts the performance on open-source LMMs based 3D scene understanding.\n\n- The proposed method well leverages the powerful 2D foundation models like 2D LMMs and 2D SAM for 3D scene understanding, overcoming the limitations from the scarcity of 3D data.\n\n- The paper introduces a Struct2D-Set, which can be useful for future research in 3D scene understanding with instruction tuning.\n\n- The paper is clearly written and easy to understand.\n\nWeaknesses:\n\n- This work is so similar to GPT4Scene, although the paper made conceptual comparisons with GPT4Scene in Lines 172-180. However, first I think GPT4Scene also has object marks as part of its prompting strategy, called STO-Marker in the GPT4Scene paper. The major difference is that the proposed Struct2D only includes objects relevant to the question, while GPT4Scene gets all the instances in the 3D scenes. The metadata prompts for objects and the selected keyframes are essentially the major components added on top of GPT4Scene. Therefore, I think there is not enough innovation in the proposed prompting method Struct2D, making the contribution of this paper incremental.\n\n- For the experimental results, the comparison with GPT4Scene in Table 4 is unfair to me. As mentioned in Lines 261-262, all the visual inputs are resized to $640\\times 640$ in this paper. However, the numbers for GPT4Scene in Table 4 are using resolutions $128\\times 123$. Therefore, if using the HD or HDM version of GPT4Scene, the proposed Struct2D will have worse or comparable performance on many of the metrics. For example, for METEOR/ROUGE/CIDEr on ScanQA, and EM-1/EM-R1 on SQA3D, GPT4Scene-HDM will actually outperform the proposed method. Even in this case, GPT4Scene is using $512\\times 490$ resolution, which is still lower than the size of the visual inputs used for Struct2D.\n\n- It is unclear what is special about the curated Struct2D-Set dataset for instruction-tuning, compared with existing data for 3D instruction tuning. Is it specially curated for matching the QA types in VSI-Bench and other 3D understanding benchmarks like ScanQA, SQA3D, etc.?\n\n- The training cost of the proposed method is also relatively larger compared with GPT4Scene. The proposed method requires 8 hours on 8 H200 GPUs, while GPT4Scene needs 6 hours on 8 A100 GPUs. Intuitively, since Struct2D has only the object markers that are relevant to the question, the training process should be more efficient than the previous methods.\n\nQUESTIONS:\n- What are the additional insights of the prompting design in Struct2D? Especially compared with GPT4Scene.\n\n- What are the advantages and special designs of the curated Struct2D-Set for instruction tuning for 3D scene understanding?\n\n- If performing a fair comparison with GPT4Scene, how is the performance of the proposed solution compared with GPT4Scene?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
In this paper, the authors propose a perception-guided prompting framework to reason about 3D using only structured 2D representations. Besides, they also construct $Struct2D-Set$, a large-scale instructional tuning dataset. Compared with existing similar work, their dataset is filtered by object marks, prompted guided by their metadata as well as selected by key frame.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The topic of this paper is interesting.
2. The paper is well-structured and easy to follow.
3. The experiments and ablations are comprehensive.

Weaknesses:
1. This is maybe minor, but it will be great if the authors could unify the front of $Struct2D$.
2. There are some missing details, I will put them in Question Section.
3. The contributions seem inefficient since the improvement on some shown in experiment section seems not significant.

QUESTIONS:
As for Section 3:
1. Can author explain that how metadata is obtained? Do the authors also filter the metadata?
2. As for subset for evaluation, do 422 QA pairs randomly selected?
3. Can authors also explain what is the text query length of $T^{out}$ when incorporating with the $Struct2D$? If possible, can authors provide ablations on text query length?

As for Section 4:
1. Can the authors provide settings when using VLMs, such as temperature?
2. Can authors explain when using VLMs whether you run several times or only once?

As for Section 5:
1. It seems like the VLMs matter, but in Table 3 and Table 4 the Qwen model, authors comparing with, is Qwen2. Will the existing dataset incorporated with Qwen2.5 improve the performance?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Struct2D, a perception-guided prompting framework, While multimodal large language model has been used for 3D scene understanding with point cloud information as input, authors argue that structured 2D representation can also be used to reason about 3D space. Their structured 2D representations include BEV images with object marks, object-centric metadata, and optionally egocentric keyframes. They have also used guided prompt to improve zero-shot performance. They have constructed an instruction tuning dataset Struct2D-Set to tune open-source multimodal large language model (MLLM).\n\nFollowings are the contribution of the paper:\n1. They have proposes a perception-guided 2D prompting strategy to perform 3D spatial reasoning from structured 2D inputs alone,\n2. They have introduced instruction tuning dataset Struct2D-Set for this task,\n3. They have fine-tuned open-source MLLM and achieved state-of-the-art performance across several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is very well written, \n2. The experimental evaluation is thorough. They have also achieved state-of-the-art performance in the relevant benchmarks,\n3. They have shown how structured BEV image, combined with metadata, is often sufficient for accurate zero-shot 3D scene understanding. This key insight is valuable for computationally efficient 3D scene understanding.\n4. They have also contributed an instruction tuning dataset for this task.\n\nWeaknesses:\n1. Technical novelty of this paper is very limited. This paper is not the first work that tries to reason over 3D scene using 2D data with multimodal large language model (MLLM). GPT4Scene [56] also had the similar motivation and utilized 2D input to infer on 3D scene using MLLM. Main improvement over GPT4Scene is the use of object marks, metadata, guided prompt, and key frames. All of these are actually incremental novelty.\n\nQUESTIONS:\nPlease check weaknesses. \n\n1. Is guided prompt also used during instruction tuning?\n2. Suggestion: Should include more details on keyframe selection.\n\nLIMITATIONS:\nLimitations are included in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Struct2D, a perception-guided prompting framework designed to enable spatial reasoning in Large Multimodal Models (LMMs) using only structured 2D representations derived from 3D perception, rather than explicit 3D data. The approach combines bird’s-eye-view (BEV) images, object marks, object-centric metadata, and optionally egocentric keyframes to form the input for LMMs. The authors demonstrate, through zero-shot analysis with closed-source LMMs (like GPT-4o), that these models can reason effectively about 3D spatial tasks (e.g., direction estimation, route planning) from the structured 2D input. Building on this, the authors construct a large-scale dataset (Struct2D-Set) via automated QA pair generation from 3D indoor scenes, and instruction-tune an open-source LMM (Qwen2.5VL). Experiments across several benchmarks show that the Struct2D-tuned model achieves strong, sometimes state-of-the-art, performance on a suite of spatial reasoning tasks, without requiring explicit 3D inputs at inference time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework is practical, lowering the barrier for spatial reasoning with LMMs by removing the need for explicit 3D inputs at inference.\n2. The paper is technically sound and experimentally thorough. It provides a detailed ablation analysis (e.g., effects of metadata, prompt structure) and evaluates across multiple benchmarks (VSI-Bench, ScanQA, SQA3D, etc.).\n3. The approach bridges a major gap between 2D perception and 3D spatial reasoning in LMMs, showing that explicit 3D inputs are not necessary for strong performance.\n\nWeaknesses:\n1. The impact may be somewhat limited by the continued need for good 3D perception in the preprocessing stage, even if not at inference.\n2. The presentation of dataset generation is a bit unclear. It could benefit from a flowchart summarizing the pipeline.\n3. The manuscript lacks a deep exploration of limitations in highly cluttered or occluded environments, or how the framework generalizes to domains beyond indoor scenes.\n\nQUESTIONS:\n1. Can you provide further analysis or case studies on natural images or more complex, unstructured real-world scenes? Most benchmarks are on indoor scenes. Have you tried Struct2D on different domains (e.g., urban outdoors, street scenes, aerial images), and what would need to be changed for broader generalization?\n\n2. Please include and discuss representative failure cases -- when does Struct2D not help (or even hurt), and why?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}