Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
Authors: Fangrui Zhu
Hanhui Wang
Yiming Xie
Jing Gu
Tianye Ding
Jianwei Yang
Huaizu Jiang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal learning
spatial reasoning
multimodal large language models
Primary area: deep_learning

Abstract: Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D representations derived from perception?
In this work, we introduce Struct2D, a perception-guided prompting framework that combines bird’s-eye-view (BEV) images with object marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MLLMs (e.g., GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with projected 2D inputs, effectively handling tasks such as relative direction estimation and route planning.
Motivated by these findings, we construct a large-scale instructional tuning dataset, \textbf{Struct2D-Set}, using an automated pipeline that generates fine-grained QA pairs grounded in 3D indoor scenes. We then fine-tune an open-source MLLM (Qwen2.5VL) using Struct2D-Set, relying on noisy 3D perception rather than ground-truth annotations. Despite this, the tuned model achieves strong performance across multiple spatial reasoning benchmarks, including 3D question answering, captioning, and object grounding, spanning eight diverse reasoning categories.
Our approach demonstrates that structured 2D inputs can effectively bridge perception and language reasoning in MLLMs—without requiring explicit 3D representations as input. We will release both our code and dataset to support future research.

Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
Fangrui Zhu'; Hanhui Wa Yiming Xie', Jing Gu', Tianye Ding'
Jianwei Yang", Huaizu Jiang'
Northeastern University 2 Microsoft Research
University of Southern California University of California, Santa Cruz
(zhu. fang, wang hanh, xie yim, ding. tian, h. joottaasttorttheeteeen edu, edu, "igu110@ucsc. edu
2 yangoggal com, com/neu-vi/structaa https://gitee com
Obtain structured inputs via perception module
Enables spatial reasoning across multiple tasks
Struct2D
Spattal Reasoning, Prediction
Metadata (3D prior)
MLLM
1: sofa: (2,4, 0.5) 2: tv: [-3.5, 0.2] 3: stove: (-0.7, -0.9, -0.9.
<image>>
Object marks are shown in
Belettie Direction
Route Planning
She Estimation
ONjec Counting
RGB-3 Video Input
bird's eye view image.
Metadata: sofa: [-2.4,
3D Dense Captioning 30 Object
<image> < image> <image>
Selected keyframes are
given.
Question:
37.0%
Ifl am standing by the stove and facing
BEV Image w/ Marks
the sofa
my front-left, front-right, back-left, or backrrigh? <thinks by canswer> front right
12.5%
Keyframes (optional)
Red nirection
Aaste Piez
olj Coust
Abc. Ditame
canswer>
Accuracy VSi-Bench Tasks
Figure 1: Overview of our Struct2D framework for enabling spatial reasoning in Multi- modal Large Language Models (MLLMs). From an RGB-D video, we generate structured 2D inputs- BEV images with filtered object marks, object-centric metadata, and optional keyframes-via a 3D perception module. These inputs prompt an MLLM with spatial priors and visual context, enabling diverse spatial reasoning tasks without explicit 3D input inference.
Abstract
Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D represen- tations derived from perception? We introduce Struct2D, a dnnception-guider (BEV) images with object
prompting framework that combines bird's-eye-ue marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MIL.Ms (e.g., GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with structured 2D inputs, effectively handling tasks such relative direction estimation and route planning. Building
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

on these insights, we construct Struct2D-Set, a large-scale instruction tuning dataset with 200K tineggrained QA pairs across eight spatial reasoning categories, generated automatically from 3D indoor scenes. We fine-tune an open-source MI.LM (Qwen2.5VIL) on Struct2D-Sett achieving competitive performance on multiple benchmarks, including 3D question answering. dense captioning, and object grounding. Our approach demonstrates that structured 2D inputs can ef- fectively bridge perception and language reasoning in MLLMs- explicit 3D representations as input. We will release both our code and dataset to support future research.
requiring
Introduction
Understanding objects and their spatial relationships in 3D space is a comnertone of intelligent inter- action in complex physical environments. Tasks such as robotic manipulation [38, 67]. autonomous navigation [25 [56, and visual reasoning 10 14 53 82 [0I] all depend on accurate spatial under- standing of scenes. At the core of these tasks lies the ability to localize objects precisely and reason about their configurations in 3D space. Morcover, grounding such spatial understanding in natural language enhances AI system's s ability to interpret, explain, and act upon spatial information in human-centric contexts.
Traditional task mask-specific models rely on explicit 3D representations as input, such as point clouds or reconstructed environments [ [34 [53] [02], providing detailed geometric information. However, these models, often trained on limited data sources, making them less adaptable and struggle to generalize to diverse and complex textual queries. As a result, they fail to effectively bridge spatial reasoning with language comprehension, limiting their applicability for embodied AI.
96 developed with Large Language Models (LLMs) have achieved significant advances in perception and reasoning tasks for images and videos. To extend MLLMs' capabilities 3D understanding, point cloud-based
In recent years, Multimodal Large Language Models (MLLMs) 2
28 31 [54 61 62 73 79 have emerged, incorporating 3D spatial features by aligning point cloud data with LLMs. This integration enhances spatial reasoning and provides a richer understanding of the 3D physical world. However, they often rely on well-annotated datasets for instruction tuning and require point-cloud features as input, which limits their flexibility.
LLMs 13
23
27
Unlike models that take explicit 3D representations as input, humans perceive the world as a continuous stream of 2D visual inputs akin video, and naturally infer spatial relationships and object configurations by building mental representations subconsciously (58) (7). Naturally, we ask "Can MLLMs perform spatial reasoning without using explicit 3D features as direct inputs?" ?" Recent work has begun to explore this direction by leveraging cognitive maps [82) and Bird's Eye generated from video as 2D spatial cues, enabling MLLMs perform
View (BEV) images (63
spatial reasoning [8 85 While promising, these approaches often omit object appearance and detailed priors (e.g.. coordinates, categories), which are critical for comprehensive 3D understanding.
We conduct an in-depth analysis of MILMs' spatial reasoning abilities using a perception-guided 2D framework called Struct2D Prompting. This strategy transforms 3D perception outputs-obtained from off-the-shelf detectors-intt structured 2D inputs, consisting of (1) a rendered bird's seeev-ww (BEV) image with projected object barks]Jand (2) object-centric metadata such category labels and 3D coordinates. When appearance cues are needed, we optionally incorporate egocentric key framee selected based on object visibility. This design enables MLLMs reason about complex 3D scenes using only structured 2D visual and textual cues, eliminating the need for explicit 3D inputs. We begin by evaluating this approach on GPT-o3 60). a representative closed-soure MLLM, assess its zero-shot spatial reasoning capabilities.
To better understand the spatial reasoning capabilities of existing MLLMs, we begin with zero- shot analysis using our proposed Struct2D Prompting strategy. The goal is to evaluate whether a pretrained, closed-source model such GPT-o3 can accurately infer 3D spatial relationships when given only structured 2D visual and textual inputs. We use rendered bird's s-eee-veww (BEV) images with projected object marks and object-centric metadata, allowing the model reason about 3D scenes without access to explicit 3D features. This analysis yields several key insights: (1) A single We follow the term "mark" as used in (1].

---

informative BEV image, combined with metadata, is often sufficient for accurate zero-shot 3D scene understanding; (2) Prompt composition critical- differrnt spatial reasoning tasks benefit from tailored input formats; (3) For challenging tasks in VSI-Bench [82, such as egocentric-to-alle to-allocentric transformations, MLLLMs can perform robustly when provided with well-structured 2D projections of the 3D scene.
Guided by the findings from our zero-shot analysis, we construct large-scale instructional tuning dataset, named Struct2D-See using an automated pipeline. The dataset consists of 200K QA pairs generated from 6K 3D indoor scenes, leveraging ground-truth object annotations provided by the original 3D datasets. It spans eight categories of spatial reasoning tasks relevant to embodied AL To ensure data quality, we use ChatGPT to both enrich the QA pairs with step-by-ster reasoning traces and identify potentially low-quality samples. Additionally, we incorporate review process to further refine and validate the dataset. We then fine-tune an open-source MI.LM (Qwen2.5V1 (72)) using Struct2D-Sed Although the fine-tuned model is evaluated under noisy 3D perception conditions, achieves strong performance across multiple spatial reasoning benchmarks, including 3D question answering
3
53 [2) spatial captioning [I4], and object grounding (10] 955.
demonstrating the practicality and robustness of our approach.
Our main contributions are as follows:
We propose perception -urided 2D prompting strategy, Struct2D Prompting, and conduct detailed zero-shot analysis that reveals MLLMs' ability to perform 3D spatial reasoning from structured 2D inputs alone.
We introduce Struct2D-Sert a large-scale instructional tuning dataset with automatically gener- ated, fine-grained QA pairs covering eight spatial reasoning categories grounded in 3D scenes.
We fine-tune an open-source MLLM to achieve competitive performance across several spatial reasoning benchmarks, validating the real-world applicability of our framework.
2 Related Work
3D Spatial Reasoning with MLLMs. Developing rorldworld embodied agents requires equipping Multimodal Large Language Models (MLLMs) with robust 3D spatial reasoning abilitics 88 9 16
42,44 48)
83 99]. Recent efforts have explored spatial understanding through language [57] 65 77].
Our work builds upon the video-input setting, but diverges by enabling spatial reasoning in MLLMS using only structured 2D inputs-BEB BEV images, object marks, and metadata wrtthut wwthout relying on explicit 3D encoder / representations at inference.
static 2D images [52 55 64 69 81], or videos [26 45 63 [2]
has explored instruction tuning to enhance MLLMs" capabilities for 3D spatial reasoning, targeting tasks such as 3D visual grounding [2] 10 95]. 3D dense captioning [14. and 3D question answering 3 (53. M3DBench [43 provides region- and scene-level instruction-respnne pairs for general 3D 44 focuses scene-centric instructions. LL3DA [33 supports interactive planning and reasoning across omni-3D inputs. Robin3D (35 introduces a 3D LLM trained on diverse tuning dataset and a GRPO-based training method to enhance spatial reasoning Qwen2-VL VL, yet its QA pairs involve limited reasoning complexity, cover fewer task types, and yield marginal performance gains. In contrast, we propose Struct2D-Set, a large-scale dataset that enables open- source MLLMS to acquire rich 3D spatial reasoning skills through instruction tuning- only structured 2D representations, without requiring direct access to 3D point clouds.
Instruction Tuning for 3D Spatial Reasoning. Recent work 13 35
45
understanding, while 3DMIT
examples. R1-Zero-VS. [45] proposes a Hideo-based instruction
3D Point Cloud LLMs. Recent advances in 3D point cloud LLMs enable natural language generation and interaction grounded in 3D geometry by directly processing point clouds as input. These models benefit from the geometric precision and texture richness of point clouds, offering stronger spatial understanding than raw image or video inputs. Prior work has focused on object-level [27 61 62 79
and scene-level [13 23 28 31 54 73 spatial reasoning. However, directly using point cloud features requires additional training and infrastructure, limiting flexibility and scalability in real- world applications.
Prompting L.Mss. Despite the rapid scaling of large language models (LLMM) 17 93 their reasoning capabilities remain heavily dependent effective prompt design. In-context
3

---

(1) Input:
(2) Perception module
(4) Prompt Construction
agocentric video
Point reconstruction 3D object detection
Rotated BFV Image
Question: If am stonding by the safe and facing the the my left. right, bock?
Guide Prompt
Follow the steps below to complete the task accurately: point where standing
Line
the object which perpendicular Line
(3) BEV Generation
Line 5
intersects my standing pocition.
determine object's relative posttion
Line A
question: which
abject
the osject
same side as objecti facing. frant: otherwise, it
Object centric Metadata
Next, determine the stictts scattion rolacuue to Line & it to the left of Line it is on the left. If it to the right of Line A, is on the right. Left and right should always be diteeriieed based on the direction in which am facing.
back
sofa:: (-2.0.30.0005)
stove:: [-0.7. 0.9, -0.1]
18: tv: [-3.8, 0.8, 0.5]
Figure 2: Illustration of Struct2D prompting. Given an egocentric video, we first reconstruct point cloud and detect 3D objects. A bird's seyee-ve (BEV) image is rendered and drawn with object marks related with the question. To facilitate reasoning about relative directions, the BEV is rotated align with the agent's facing direction. We further construct object-centric metadata and structured guide prompt to support the model in understanding spatial relationships between objects.
learning [5] 21]. which conditions models on a few representative examples, has become a widely adopted technique for improving Dnstruction-follovii behavior: To further enhance reasoning. strategies such as chain-of-thought (74) and of-h-of-th (86) prompting have been proposed.
More recently, Multimodal Large Language Models (M 20 24 40 41
46
(5) 60) [80
91 92, 96 [00] have gained prominence for their ability to reason over multiple input modalities. This has led to surge of research into prompting techniques tailored for MLLMs 16 29 30
39
47
59
68
75 76
78 8
89 94 988. Building on this direction, we propose
Struct2D, structured 2D prompting strategy that enables MLLMs to perform 3D spatial reasoning requiring explicit 3D input representations.
3 Analysis on Struct2D prompting with GPT-o3
3.1 Struct2D Prompting
Given video V as input, an MLLM F processes set of N sampled video frames, denoted as I= I= (4,,22.... IN} J. where each frame I, has dimensions R "Ix W for n E { {l,.... N]. Alongside visual input, the MILMM receives text query of length 4, represented as Tin [ f The
model then generates textual response of length lo denoted as Tout_ [tq ......... 1f,1, formulated as:
T = T"".
However, directly using video frames for spatial reasoning introduces two major limitations: (1) Incomplete perception Video frames are typically sampled sparsely and from limited viewpoints, which can result in missing critical visual evidence required for spatial understanding. For instance, consider scene where chair tucked partially under table. If most sampled frames taken from frontal views or from a standing height, the chair's presence might be obscured or entirely invisible, leading the model to incorrectly assume there is empty space beneath the table. This limitation becomes more severe in cluttered or occluded environments, where small objects or those blocked by other furniture. (2) Lack of global context Video frames offer fragmented, egocentric views that often fail to capture the overall structure of the scene. For example, determining whether a lamp is closer to the couch or the bookshelf may be impossible if the two objects never co-occur in the same frame. Without consistent top-down or holistic representation, the model must rely on spatial memory or reasoning across disjoint perspectives- an ability that remains weak in most MI..Ms. This fragmentation also impedes the understanding of traversability (e.g., identifying clear path from the door to the kitchen) or relational queries (e.g., which chair directly behind the dining table).

---

Table
Zero-shot evaluation of GPT-o3 on the VSI-Bench subset. The first row simply uses 16 frames from the input video, proposed in VSI-Bench [82]. For our prompting, we only input BEV image with object marks on it along with object-centric meta information.
Numerical
Multiple-Chaiee
Settings
# images
Cost (S)
Avg.
Obj.
Count
Abs. Dist.
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
VSI Bench
82
16
105.07
48.6
44.3
34.1
50.9
51.0
49.4
61.9
GPT4Scene
51.5
50.5
58.8
Ours (Noisy Objects) Ours (GT Objects)
27.25
56.1
52.8
38.4
48.9
60.0
60.1
76.2
27.25
83.8
93.8
90.6
47.4
96.5
94.4
80.1
To address these issues, Struct2D incorporates a perception module Opercope that extracts point clouds P and object detections O from the input video V. We then generate a top-down bird' s-eye-veeww image with filtered object marks-onle including objects relevant the question, illustrated in Figure 2 Additionally, we construct object-centric metadata Tumea (e.g., categories, coordinates) as textual input guide spatial reasoning. Formally, we redefine Eq. Tas:
T [[Struct2D(0pxooopp (V), T T
For questions requiring appearance depth cues (e.g., object color size), supplement the BEV view with selected egocentric keyframes Kkyframe that capture clear views of the relevant objects. Instead of uniformly sampling keyframes, we use 3D projections select frames that better capture the spatial coverage of the scene. The full formulation becomes:
T F(Sttruct2D(dpemer VV, Tmeta Tkeyframe) ); T
This formulation illustrates how Struct2D leverages 3D perception as an intermediate step to generate informative 2D inputs that preserve spatial structure. Although 3D point clouds used during preprocessing, they are not directly provided to the MLLM. Instead, they are transformed into BEV images and metadata used for prompting. As a result, the model performs spatial reasoning effectively without requiring explicit 3D representations as input.
Evaluation Setup. We sample questions from VSI- Bench 82 which is designed to evaluate complex spatial reasoning skills. Compared traditional 3D QA datasets fine-grained object perception requirements, intri- cate global spatial relationships, and egocentric-to- allocentric transformations. It also features diverse indoor scene sources and robust evaluation metrics that go beyond rule-based NLP scoring.
Counting
Abs.Distance 26%
18%
3 53
VSI-Bench covers more
Route Planning
5%
Rel. Direction 30%
Room Size 8%
Comparison to GPT4Scene Prompting 663] While GPT4Scene pioneered 2D spatial prompting using BEV images, our Struct2D strategy introduces sev- eral key improvements:
Distance 13%
Figure 3: Distribution of question types in the selected VSI-Bench subset. This follows the distribution of the full set.
(1) Filtered object marks tai- lored to the query improve visual relevance and re- duce distraction; (2) Guided metadata prompts pro- vide additional spatial priors; (3) KeyTrame selection optimized using depth-aware 3D projection instead of uniform sampling, making them both fewer and more informative (training drops from 6 to 4 hours).
3.2 Zero-shot Analysis of Struct2D Prompting
We construct a subset of 422 QA pairs for evaluation, selected due to API call budgets. As shown in Figure 3 the distribution of question types is consistent with the full benchmark. For our analysis, generate object marks using both ground-truth 3D annotations and noisy detections (following (3])) ensuring comprehensive object coverage while eliminating perception errors. This also enables a fair comparison with prior work, particularly (63).
Zero-shot Prompting Results. Table shows that GPT-o3 exhibits strong spatial reasoning capabili- ties when prompted with structured 2D inputs. Specifically, providing both object-centric metadata
5

---

and filtered object marks significantly boosts performance, achieving 96.5 on relative distance, 94.4 on relative direction, and 80.1 on route planning. This highlights that explicit 3D representations MMLMM can reason effectively with carefully structured 2DD projections. The ablation further reveals that rotation alignment and structured guide prompt each contribute to improved accuracy on relative direction tasks, with the combination of both yielding the best performance (94.4). These results underscore the importance aligning spatial context and guiding the model through geometric reasoning steps. Notably, method requires only single BEV image and lightweight metadata, making it low-cost and robust alternative to multi-frame prompting strategies 63
are not strictly
Table Ablation on different prompting strategies.
Filtered Marks
Guide Prompt
Metadata
Rel. Dist.
Rel. Dir.
Route Plan
Rotation
Rel. Dir.
67.5
82.1
74.3
75.3
/
72.1
88.3
78.3
v
89.2
75.3
89.5
50.6
80.2
96.5
94.4
80.1
94.4
Effects metadata and filtered marks.
Effects of rotation and guide prompt.
What makes good prompt for spatial reasoning? Table highlights the impact of key components in our prompting strategy. Incorporating object-centric metadata consistently improves performance across tasks maasss-risiin raising relative distance accuracy from 67.5 96.5 and route planning from 74.3 to 80.1 30.h-higgl its importance for grounding spatial context. Filtering object marks based on question relevance further reduces ambiguity, yielding substantial gains route planning (from 50.6 to 80. For relative direction, both the use of a structured guide prompt and rotation alignment prove essential. While each individually improves accuracy (89.2 and 80.2 respectively), their combination leads to the best performance (94.4). We focus on these question types in ablation because they represent core challenges in spatial understanding.
Large-Scale Instruction Tuning with Struct2D-Set
Building on the insights from our zero-shot analysis (Sec. 3 construct large-scale instruction tuning dataset, Struct2D-Set, tailored to support diverse spatial reasoning tasks grounded in realistic 3D indoor scenes. Notably, the dataset designed require only 2D projected inputs during training, enabling efficient supervision without reliance on full 3D data.
In this section, first describe the design and construction of Struct2D-Set highlighting its coverage, annotation pipeline, and task diversity. We then present the supervised fine-tuning (SFT) setup using open-source MI.LMs, detailing the model configurations and training procedures. Fi- nally, we evaluate the effectiveness our instruction-tunde model across multiple spatial reasoning benchmarks, assessing its generalization and reasoning capabilities.
4.1 Struct2D-Set
Overview. Struct2D-Set consists of 200K QA pairs generated from over 6K richly annotated indoor scenes, sourced from large-scale 3D recon- struction ARtitteeeAR (4) ScanNet [19] and ScanNet++[S33 These datasets capture diverse real- world environments, including homes, offices, and industrial settings. The QA pairs cover eight categories of spatial reasoning tasks. Figure E shows the distribution of question types across the dataset.
Spelial Localization (25.3%)
Spatial Relationship (32.3%)
(16.0%)
Struct2D-Set
Object Counting (7.2%)
Binary Attribute Verification (4.9%)
Egocentric
Orientation
Quantitative Spatial Measurement (5.3%)
Comparative Attribute Reasoning (4.0%)
Figure 4: Distribution of QA types in Struct2D-Sed The dataset covers diverse range of spatial reasoning skills, with focus on spatial relationships and localization tasks that require strong geometric understanding.

---

Object Attribute identification
Object Counting
Egocentric Navigation /
Orientation
color front couch?
many green on bod?
The table front the blue.
There are pillovs placed on head the bed.
paper and facing beginning holder, You want to nevigate to the brown door. You will perform the falkowing actions (Note: for each [please fill in], choose either tuum back," 'tum left," "tum right, 'pass by 1. (please
You
Spatial Relation (Relative)
Comparative Reasoning
Go forward until the forward until forward until white shower curtain [please fill in] 8. Go forward until the brown door. You have reached the final destination.
jplease
(please
Short answer: "tur right", "turn right", "pass by", "tur right".
Q: It am standing next to the oven and facing the sotn, is the chair front-ieff, front-rigtt, back-left, back-right?
Q: Which of the following abjects (toilet, bethtub, sini) is closest to the washer when measuring from their nearest points?
Augmented answer: The robot is now at the white tailet, facing the black toilet paper holder. The object, the wall, right, should right. The
Short anewer: back-right.
object, another wall, directly ahead, so it should turn right agai. The robot is now facing the white shower curtain. The next object, the shower curtain directly ahead, so should pass faces object. the should right. The robot is now at the brown door. having reached its final destination.
facing
Short answer: Tollet.
Augmented answer: chair located back-right position ralative to the oven (ID 6) and the sofa (ID 11), as situated towards the lower right quadrant when facing north along the poeitive y-axis.
Augmented answer: closeet object to the washer the tollat, with coordinates (0.25, 1.27, -0.09), which is approximately 0.54 units away from the washer at (0.79, 1.35, 0.07), while the other abjects farther away.
Continuing forward, the rabot door. directly ahead,
QA examples of Struct2D-Saa Examples cover diverse spatial reasoning tasks, including object attributes, counting, relative positioning, navigation, and comparative reasoning. Each QA pair includes short answer from 3D geometry and augmented answer with detailed reasoning generated by ChatGPT.
Figure 5:
Construction pipeline. We generate
two types of QA pairs support both spatial reasoning and scene understanding tasks. Each type encompasses multiple subtypes targeting distinct reasoning skills. Representative examples from both types are shown in Figure
The first type, inspired by VSI-Bench (22. involves tasks that require understanding global spatial relationships in 3D, such spatial relation identification, egocentric navigation, and comparative reasoning. These questions cannot be answered from single key frame alone. We begin by extracting ground-truth object annotations from the training sets of the 3D datasets, including object boxes, depth maps, and camera poses. Using structured templates, we generate initial QA pairs based on this meta information, and then enrich them using ChatGPT to produce step-by-ster reasoning traces and more natural language formulations. Each QA pair includes a short answer derived from geometry templates and long-form answer elaborating on the reasoning process.
The second type of QA pairs adapted from existing 3D scene understanding benchmarks, including ScanQA (3) SQA3D (53]. Scan2Cap [1], ScanRefer [10]. and Multi3DRefer (95]. These examples cover tasks such as object attribute identification, counting, and binary verification. We augment the original training set questions and descriptions using ChatGPT to improve clarity and reasoning depth. These tasks typically benefit from selecting keyframes where relevant objects are clearly visible, allowing the model to ground spatial reasoning in egocentric frames.
4.2 Experiment Setup
We fine-tune the open-source MILLM Qwen2.5VI. 72 using our proposed dataset, Struct22-Set. For evaluation, we primarily focus VSI-Bench 82) which includes complex spatial reasoning tasks
7

---

Table 3: Performance comparison of various models VSI-Bench [82] The model fine-tuned
with Struct2D-Set. surpasses both the Struct2D prompting and the video-based tuning baseline.
Numerieal Answer
Multiple-Choke Answer
Methods
Avg.
Obj. Count
Abs. Dist.
Room Size
Obj. Size
Rel. Dist.
Rel. Dir.
Route Plan
Open-souzze Models
21.8
35.0
22.0
33.8
44.2
30.5
IntemVL2-88
15
33.9
23.1
28.7
39.8
48.2
36.7
30.7
29.9
LongVILA
VII.A-1.5.8B 46
29.5
17.4
21.8
18.8
50.3
32.1
34,8
31.0
LongVA-7B 91]
22.2
33.1
25.4
LLaVA-NeXT-Vidan
96
36.3
48.5
14.0
24.2
47.8
43.5
42.4
34.0
ILaVA-OmeVision-00.11
28.3
36.9
34,5
LLaVa-OneWision-1
40
33.5
47.7
20.2
12.3
47.4
42.5
35.2
29.4
R1 VSI
35
(Qwen2-VL-
32.1
39.4
25.0
43.2
25.8
32.6
30.9
27.8
R1-Zem- VSI
45
(Qwen2 V..-7-B- SFT
38.8
44.7
27.6
50.4
46.1
34.0
35.7
33,0
Ours
Qwen2.5-V1.35
28.9
25.6
Owen2.5 5-V-2.B (Struct20 Prompting)
29.4
46.6
24.6
22.3
33.6
21.2
30.5
27.2
- (Baseline)
33.9
24.6
46.4
53.5
21.2
30.5
27.2
5-v-2.5-VL-3B (SFT)
41.9
46.0
34.7
42.6
56.4
35.1
44.9
33,5
Qwen2.5-VL-72 (SFT)
43.6
47.1
35.1
48.9
57.1
35.1
45.9
35,8
such as relative direction and route planning. Additionally, we assess model performance on three standard 3D understanding tasks built ScanNet [19] 3D question answering (ScanQA (3), SQA3D
(33 ), 3D dense captioning (Scan2Cap (14). and 3D) visual grounding (ScanRefer [1O],
Multi3DRef (95)). For VSI-Bench,
input only BEV images with filtered object marks and
metadata, as the tasks focus purely on spatial relationships. For the other benchmarks, which often
involve object attributes visual details, additionally provide selected egocentric keyframes to
support reasoning.
4.3 Implementation Details
We adopt Qwen2.5VL [2] as our base MILM for instruction tuning. During training, the model receives BEV images with filtered object marks and object-centric metadata. For tasks that require appearance or attribute information (e.g., object color or count), we additionally provide egocentric keyframes, All visual inputs resized 480 480, and object marks adaptively scaled based on their original resolution.
For questions involving complex spatial reasoning, such relative direction or route planning, we insert special tokens <think> and </think> to guide the model to generate step-by-step reasoning process, followed by the final answer enclosed within <answer> and </ </answer>, For simpler questions involving object appearance or quantitative estimation, the model trained to directly produce short answers without reasoning traces. We train the model for epoch using a base learning rate of 2e-6 with cosine annealing. Training with the whole Struct2D-Se takes approximately hours 8xH200 GPUs. For evaluation, follow BI 63) by reconstructing point clouds offline using BundleFusion [18), detecting 3D object boxes with Mask3D and UniDet, and projecting them into BEV images and 2D object marks.
4.4
Main results
We present quantitative results on VSI-Bench 82 in Table and on ScanQA and SQA3 53 in Table 4 Additional benchmark results are provided in the Appendix due to space limitations.
As shown in Table 3 our model fine-tuned with the Struct2D-Ses -St dataset achieves the highest average score (43.6) among all open-source models evaluated on VSI-Bench. Notably, surpasses both the Struct2D prompting variant (29. 4)) and the standard baseline trained with uniformly sampled 16 video frames (33.9). confirming the effectiveness of our full instruction tuning approach. The performance gains are especially prominent on spatial reasoning tasks like relative direction (45.9) and route planning (35.8), where the model must integrate both geometric understanding and egocentric context. Compared with R1-Zero- VSI 45
(38.3 8), recent method that trains Dwen2-VLL-7B using video-based supervision, our tuned model not only achieves stronger average performance but also uses fewer visual frames and does not rely on dense temporal input. These results highlight the

---

Table 4: 3D Question Answering Evaluation on Sc [3] and SQA3D (33 datasets.
ScanQA(val)
SQA3D(val)
Methods
BLEU-1
BLEU-4
METEOR
ROUGE
CIDEr
EM-1
EM-RI
Task-Specific Model
SeanQA 3
30.2
10.1
13.1
33.3
64.9
SQ3D 53
46.6
3D-VLP
30.5
11.2
13.5
34.5
3D-Vista
102
13.9
35.7
48.5
3D LLM Based Model
Chat-3D 77
29.1
6.4
11.9
28.5
53.2
Chat-3D v2
38.4
16.1
40.1
3D-LLM
39.3
12.0
14.5
37.3
69.4
LL3DA 13
13.5
15.9
37.3
76.8
PQ3D [03
47.1
LEO 32
11.5
16.2
39.3
80.0
50.0
50.0
Chat-Scene
31
43.2
14.3
18.0
41.6
87.7
54.6
57.5
Vision LLM Based Model
Intem VL-2-8B [[5
23.9
3.3
14.5
34.3
62.5
33.0
45.3
MM-22PM- 87
25.1
8.4
11.8
31.5
60.1
42.6
46.6
Qwen2-VL. VL-7B (GPT4Scene)
43.4
14.6
17.7
43.6
57.4
Qwen2.5-VI--BB (Ours)
45.2
15.8
17.4
44.1
92.1
58.5
Table Ablation on different variants. To save computational resource, models are trained with
Qwen2.5VL-3B model by default.
Numerical Answer
Multiple-Choice Answer
Settings
Avg.
Obj. Count
Abs. Dist.
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Taning Data Format
wol augmented QA
31.5
43.7
33.1
34.1
32.1
14.7
31.5
w/ augmented QA
38.0
44.4
33.6
41.5
33.3
42.2
33.0
Evaluation Strategy wo/ </think> w/ </think> 36.1
36.2
44.1
33.6
41.5
33.3
38.6
26.3
44.4
30.0
35.6
31.5
42.2
33.0
scalability and efficiency of Struct2D-Set for training capable spatial reasoners without explicit 3D features at inference.
Table shows shows results two traditional 3D question answering benchmarks. ScanQA and SQA3D. Our model outperforms most existing methods, including several that rely on explicit 3D point cloud inputs. Compared with GPT4Scene [63]. model performs on par across most metrics. However, these benchmarks primarily require identifying relevant keyframes and generating free-form textual answers. As result, models often rely memorizing object-level attributes, and the rule-based evaluation metrics (e.g., BLEU, CIDEr) may not fully reflect the correctness or reasoning quality of the generated answers. Please refer to Appendix for qualitative results. Ablation Study To better understand the impact of individual components in our framework, we conduct a series of ablation studies using the Qwen2.5VL-3B model for efficiency, shown in Table First, we evaluate the effect of QA augmentation. Incorporating enriched QA pairs generated with ChatGPT leads to substantial improvement in overall performance (Avg: 38.0 31.5), especially on reasoning-heavy tasks such as relative direction (42.2 vs. 14.7). This supports our earlier claim that step-by-step reasoning traces help guide the model's attention and inference. We further assess the role of explicit reasoning supervision using the <think> and <answer> tokens. While the average scores comparable, including <think> tokens improves performance on sessoning-intensive tasks like relative direction and route planning (42.2 vs. 38.6 and 33.0 vs. 26.3, respectively), indicating that instructing the model to explicitly reason can enhance output quality on complex spatial questions. Figure illustratsss step-by-step reasoning generated using our <think> and <answer> prompting format in egocentric navigation tasks. In each case, the model is prompted to analyze object positions and spatial transitions from top-down BEV image with object markers. These examples highlight the model's 's ability decompose spatial reasoning tasks into interpretable steps and generate structured answers grounded in visual context.

---

Here object labels and ids: table:2, sofa:6, tv:9, table:15 in the bird's eye view image. If am standing by the tv and facing the sofa, is the table to my front-left, front-right, back-left, back-right? Here the options: ['A, front- right', 'B. front-left', 'c. back-left', 'D. back-right'.. Please think step by step and enclose your reasoning in <think> </think> tags and then provide the short answer with only the option letter in <answer> </answer>.
Model's reasoning steps:
<think>Thhe table, identified as marker ID 2, is located in the front-right quadrant relative to my position by the TV (marker 9) while facing the sofa (marker ID 6). </think>
Final prediction: <answer> A </answer>
Here are object labels and ids: window:28, monitor:111, door:89, monitor:126, monitor:16 the bird's eye view image. You robot beginning the door and facing the window. You want to navigate to the monitor. You will perform the following actions (Note: for each [please fill in), choose either 'turn back,' 'turn left,' or 'turn right. '): Go forward until the heater 2. [please fill in] 3. Go forward until the monitor. You have reached the final destination. Here the options: ('A. Turn Left', 'B. Turn Back', 'C. Turn Right'). Please think step by step and enclose your reasoning <think </think tags and then provide the short answer with only the option letter <answer> </answer>.
B9
Model's reasoning steps:
<think> The robot is now at the d8or facing the window [28). The next object, the heater, is straight ahead, so should go forward. Upon reaching the heater, the robot still facing window. The next object, the monitor, is to its right, so it should turn right. Now, the robot is facing the monitor. It goes forward until reaches the monitor, which the final destination. </think
Final prediction: <answer> A </answer>
Figure 6: Example reasoning traces using <think> and <answer> prompting. The top example shows relative direction reasoning. where the model infers the spatial relation between objects from fixed viewpoint the BEV image. The bottom example shows step-by-step planning, requiring the model to simulate orientation changes across sequence of waypoints.
5 Limitations and future work
While Struct2D demonstrates strong spatial reasoning capabilities with structured 2D inputs, there remain areas where future work could extend its applicability:
3D preprocessing requirements. Although Struct2D does not use 3D features during infer- ence, it currently relies on 3D perception modules to generate BEV images and object- object-centric metadata. This may pose a challenge in latency-sensitive or resource-constraine environments. However, since Struct2D is agnostic to the specific perception backbone, it can readily integrate with ongoing advances in real-time and lightweight 3D reconstruction systems.
Indoor scene focus. The current version of Struct2D- annotated indoor scenes, including homes, offices, and classrooms. While this enables detailed reasoning in structured environments, generalization to outdoor or open-workd scenes remains less explored. Incorporating diverse spatial layouts and object categories from outdoor domains is promising direction for future dataset expansion.
is constructed from over 6K richly
6 Conclusion
We present Struct2D, perception -uuided framework that enables MLLMs to perform 3D spatial reasoning using structured 2D inputs. Through zero-shot analysis and instruction tuning, we show that BEV images, object-centric metadata, and keyframes are sufficient to unlock strong spatial reasoning capabilities withhut requiring explicit 3D inputs. Our curated dataset, Struct2D-Sett supports scalable instruction tuning with tine-grainet QA pairs grounded in real 3D scenes. Fine-tuning with
10

---

Struct2D SSt yields significant gains across spatial reasoning benchmarks, outperforming prior open-source methods even under noisy perception. These findings demonstrate that structured 2D projections are practical and effective alternative to direct 3D representations, offering scalable path toward robust multimodal spatial understanding in MLL.M.
Acknowledgment
We thank Zhangyang Qi for the thoughtful discussions.
References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
[2] Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., and Guibas, L. (2020). Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 422 440. Springer.
[3] Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. (2022). Scanqa: 3d question answering for spatial scene understanding. CVPR.
[4] Baruch, G., Chen, Z., Dehghan, A., Dimry, T.,, Feigin, Y., Fu, P., Gebauer, T., Joffe, B., Kurz, D., Schwartz, A., et al. (2021). Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.088.
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances neural information processing systems, 33, 1877-1901.
[6] Cai, D., Zhao, L.,, Zhang, J., Sheng, L., and Xu, D. (2022). 3djeg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In CVPR.
[7] Cai, M., Liu. H., Mustikovela, S. K., Meyer, G. P., Chai, Y., Park, D., and Lee, Y. J. (2024a). Vip-Ilava: Making large multimodal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12914-12923.
[8] Cai, W., Ponomarenko, I., Yuan, J., Li, X., Yang, W., Dong, H., and Zhao, B. (2024b). Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406. 13642.
[9] Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. (2024a). Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455 -4455-14465.
[10] Chen, D. Z., Chang, A. X., and Nießner. M. (2020). Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202-221. Springer.
[11] Chen, D. Z., Wu, Q. Nießner, M., and Chang, A. X. (2021a). D3net: A speaker-listener architecture for semi-supervised dense captioning and visual grounding in rgb-d scans.
[12] Chen, S., Zhu, H., Chen, X., Lei, Y., Yu, G., and Chen, T. (2023). End-to-end 3d dense captioning with vote2cap-dett In CVPR.
[13] Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., and Chen, T. (2024b). Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428-22 26438.
[14] Chen, Z., Gholami, A., Nießner, M., and Chang, A. X. (20216). Scan2cap: Context-aware dense captioning in rgb-d seans. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3193-3203.

---

[15] Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L., et al. (2024c). Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR.
[16] Cheng, A.-C., Yin, H., Fu, Y., Guo, Q.. Yang, R., Kautz, J., Wang, X., and Liu, S. (2024). Spa- tialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.01588
[17] Chowdhery, A., Narang, S., Devlin, J.. Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240), 1-113.
[18) Dai, A., NieBner, M., Zollhöfer, M., Izadi, S., and Theobalt, C. (2017a). Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Gruphics (ToG), 36(4), 1.
[19] Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. (2017b). Scannet: Richly-anin aanooaee 3d reconstructions of indoor scenes. In CVPR.
[20] Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. (2023). InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Systems.
[21] Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2022).
A survey on in-context learning. arXiv preprint arXiv:2301.00344
[22] Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten,
A., Yang. A., Fan, A., al. (2024). The llama 3 herd of models. arXiv preprint 2477:2240...178..
[23] Fu, R., Liu, J., Chen, X., Nie, Y., and Xiong, W. (2024). Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401.
[24] Gemini (2024). Gemini: A family of highly capable multimodal models. https:///gemin google. www. app
[25] Gu, Stefani, F.,, Wu, Q., Thomason, J., and Wang. X. (2022). Vision-and-languaae navigation: A survey of tasks, methods, and future directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7606-7623.
[26] Gu, J., Liu, X., Zeng, Y., Nagarajan, A., Zhu, F., Hong. D., Fan, Y., Yan, Q. Zhou, K., Liu, M.-Y. et al. (2025). phyworldbbnnhh"; comprehensive evaluation of physical realism in text-to-video models. arXiv preprint arXiv:2507.13428
[27] Guo, Z., Zhang, R., Zhu, X., Tang, Y., Ma, X., Han, J., Chen, K., Gao, P., Li, X., Li, H., al. (2023). Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint Xiiv:23090006/55
[28] Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. (2023). 3d-llm: Injecting the 3d world into large language models. NeurIPS.
[29] Hu, J., Lin, J., Yan, J., and Gong, S. (2025a). Leveraging hallucinations reduce manual prompt dependency in promptable segmentation. Advances in Neural Information Processing Systems, 37, 107171-107197.
[30] Hu, Y., Shi, W., Fu, x., Roth, D., Ostendorf, M., Zettlemoyer, L.,, Smith, N. A., and Krishna, R. (2025b). Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. Advances in Neural Information Processing Systems, 37, 139348-139379.
[31] Huang, H.. Chen, Y., Wang, Z., Huang, R.. Xu, R., Wang. T., Liu, L.. Cheng, X., Zhao, Y., Pang, J., et al. (2023a). Chat-scene: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:23/2.088
[32] Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang, Y., Li, Q., 7hu, S.-C,, Jia, B., and Huang, S. (2023b). An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871.
12

---

[33] Hurst, A.. Lerer, A., Goucher. A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276.
(34] Jin, Z., Hayat, M., Yang, Y., Guo, Y., and Lei, Y. (2023). Context-aware alignment and mutual masking for 3d-language pre-training. In CVPR.
[35] Kang, W., Huang, H., Shang, Y., Shah, M., and Yan, Y. (2024). Robin3d: Improving 3d large language model via robust instruction tuning. arXiv:2410.00255.
[36] Kirillov, A., Mintun, E.,, Ravi, N.. Mao, H.,, Rolland, C.,, Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., el al. (2023). Segment anything. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4015-4026.
[37] Kolodiazhnyi, M., Vorontsova, A., Skripkin, M., Rukhovich, D., and Konushin, A. (2024). Unidet3d: Multi-dataset indoor 3d object detection. arXiv preprint 4rX1v:2409.04234.
[38] Landsiedel, C., Rieser, V., Walter. M.. and Wollherr, D. (2017). A review of spatial reasoning and interaction for real-world robotics. Advanced Robotics, 31(5), 222-242.
[39] Lee, O. Y., Xie, A., Fang, K., Pertsch, K., and Finn, C. (2024). Affordance-guidee reinforcement learning via visual prompting. arXiv preprint arXiv:2407. 103444
[40] Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z.,, et al. (2024a). Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333
[41] Li, C., Gan, Z., Yang, Z., Yang, J., Li. L., Wang, L., Gao, J., et al. (2024b). Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2), 1-214.
[42] Li, C., Zhang, C., Zhou, H., Collier, N., Korhonen, A., and Vulić, I. (2024c). Topviewrs: Vision-language models top-view spatial reasoners. arXiv preprint arXiv:2406.02537.
[43] Li, M., Chen, X., Zhang, C., Chen, S., Zhu, H., Yin, F., Yu, G., and Chen, T. (2023). M3dbench: Let's instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763.
[44] Li, Z., Zhang, C., Wang, X., Ren, R., Xu, Y., Ma, R., Liu, X., and Wei, R. (2024d), 3dmit: 3d instruction tuning for scene understanding. In 2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pages 1 1-5. IEEE.
[45] Liao, Z., Xie, Q., Zhang. Y., Kong, Z., Lu, H., Yang, Z., and Deng, (2025). Improved visual-spatial reasoning via rl-zero-like training. arXiv preprint arXiv:2504.0088:
[46] Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. (2023). Vila: On pre-training for visual language models.
[47] Lin, W., Wei, X., An, R., Gao, P., Zou, B., Luo, Y., Huang, S., Zhang, S., and Li, H. (2024). Braw-and-g understand: Leveraging visual prompts to enable milms to comprehend what you want. arXiv preprint arXiv:2403.20277
[48] Liu, B., Dong, Y., Wang, Y., Rao, Y., Tang, Y.. Ma, W.-C., and Krishna, R. (2024a). Coarse correspondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint arXiv:2408. 007554
[49] Liu, D., Dong, X., Zhang, R., Luo, X., Gao, P., Huang. X., Gong. Y., and Wang. Z. (2023a). 3dax- iesprompts: Unleashing the 3d spatial task capabilities of gpt-4v. arXiv preprint arXiv:3312.09733
[50] Liu, H., Li, C., Wu, Q., and Lee, Y. (2023b). Visual instruction tuning. Advances in neural information processing systems, 36, 34892-34916.
[51] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q. Li, C., Yang, J., Su, H., el al. (2024b). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, pages 38-55. Springer.
13

---

[52] Ma, W., Chou, Y.C.., Liu, Q., Wang, X., de Melo, C., Chen, J., Xie, J., and Yuille, A. (2025). Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning. arXiv preprint arXiv:2504.2000
[53] Ma, X., Yong, S., Zheng, Z., Li, Q. Liang, Y., Zhu, S.-C., and Huang, S. (2022). Sqa3d: Situated question answering in 3d scenes. In ICLR.
[54] Man, Y., Zheng, S., Bao, Z., Hebert, M., Gui, L.-Y., and Wang, Y.X. (2024). Lexi- con3d: Probing visual foundation models for complex 3d scene understanding. arXiv preprint arXiv:2409.0372
[55] Marsili, D., Agrawal, R., Yue, Y., and Gkioxari, G. (2025). Visual agentic ai for spatial reasoning with dynamic api. arXiv preprint wxXv:25022.6687.
[56] Marza, P., Matignon, L.,, Simonin, O., and Wolf, C. (2022). Teaching agents how to map: Spatial reasoning for multi-object navigation. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1725-1732. IEEE.
[57] Momennejad, I., Hasambeig, H., Vieira Frujeri, F., Sharma, H., Jojic, N., Palangi, H., Ness, R., and Larson, J. (2023). Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 69736-69751.
[58] Nadel, L. (2008). The Hippocampus and Context Revisited. Oxford University Press.
[59] Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I,, Xie, A., Driess, D., Wahid, A., Xu, Z., et al. (2024). Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877.
[60] OpenAl (2024), Gpt-o3: A large language model by openai. https://oppeaa..com/nnnee hello-gpt-4o/
[61] Qi, 7., Fang, Y., Sun, Z., Wu, X., WU, T., Wang, J., Lin, D., and Zhao, H. (2024a). Gpt4point: A unified framework for point-language understanding and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26417-26427
[62] Qi, Z., Dong, R., Zhang, S., Geng, H., Han, C., Ge, Z., Yi, L., and Ma. K. (2024b). Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pages 214 214.238. Springer.
[63] Qi Z., Zhang, Z., Fang, Y., Wang, J., and Zhao, H. (2025). Gpt4scene: Understand 3d scenes from videos with vision- languaee models. arXiv preprint arXiv:2501.0144 001428..
[64] Ramakrishnan, S. K., Wijmans, E., Kraehenbuchl, P., and Koltun, V. (2024). Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.06465
[65] Rozanova, J., Ferreira, D., Dubba, K., Cheng, W., Zhang, D., and Freitas, A. (2021). Grounding natural language instructions: Can large language models capture spatial information? arXiv preprint wwXiv:2109.08634
[66] Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., and Leibe, B. (2023). Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE.
[67] Sisbot, F. A., Marin, L.. F., and Alami, R. (2007). Spatial reasoning for human robot interaction. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2281-2287 IEEE.
[68] Sun, F.Y.,, Liu, W., Gu, S., Lim, D., Bhat, G., Tombari, F., Li, M., Haber, N., and Wu, J. (2024). Layoutvim: Differentiable optimization of 3d layout via vasion-language models. arXiv preprint arXiv:2412.02193.
[69] Tang, Y., Qu, A., Wang, Z., Zhuang, D., Wu, Z., Ma, W., Wang. S., Zheng, Y., Zhao, Z., and Zhao, (2024). Sparkle: Mastering basie spatial capabilities vision language models elicits generalization composite spatial reasoning, arXiv preprint 2rXiv:2410.16162
14

---

[70] Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4), 189-208.
[71] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B.. Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302. 13971.
[72] Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., el al. (2024). Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 4rXiv:2409.12191.
[73] Wang, Z., Huang, H.,, Zhao, Y., Zhang. Z., and Zhao, Z. (2023). Chat-3d: Data-cfficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv: 087669
[74] Wei, J.. Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thoupee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837
[75] Wu, J., Zhang, Z.,, Xia, Y., Li, X., Xia, Z.,, Chang, A., Yu, T., Kim, S., Rossi, R. A., Zhang, R., et al. (2024a). Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.1533 15310.
[76] Wu, M. Cai, X., J., Li, J., Huang, O., Luo, G., Fei, H., Jiang, G., Sun, X., and Ji, R. (2025). Controlmllm: Training-free visual prompt learning for multimodal large language models. Advances in Neural Information Processing Systems, 37, 45206-45234.
[77] Wu, W., Mao, S., Zhang, Y., Xia, Y., Dong, L., Cui, L., and Wei, F. (2024b). Visualization-of- thought elicits spatial reasoning in large language models. arXiv e-prints, pages arXiv-2400
[78] Wu, Y., Wang, Y., Tang, S., Wu, W., He, T., Ouyang, W., Torr, P., and Wu, (2024c). Dettoolchain: A new prompting paradigm to unleash detection ability of milm. In European Conference on Computer Vision, pages 164 182. Springer.
[79] Xu, R., Wang. X., Wang, T., Chen, Y., Pang. J., and Lin, D. (2024). Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pages 131-147. Springer.
[80] Xue, F, Chen, Y., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al. (2024). Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188.
[81] Yang. J., Zhang, H., Li. F., Zou, X., Li, C., and Gao, J. (2023a). Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.1144!
[82] Yang, J., Yang. S., Gupta, A. W., Han, R., Fei-Fei, L., and Xie, S. (2024a). Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprini arXiv:2412.14 14171.
[83] Yang, J., Ding, R., Brown, E.,, Qi, X., and Xie, S. (2024b). V-irl: Grounding virtual intelligence in real life. In European Conference Computer Vision, pages 36-55. Springer.
[84] Yang. J., Tan, R., Wu, Q., Zheng, R., Peng, B., Liang, Y., Gu, Y., Cai, M., Ye, S., Jang, J., et al. (2025). Magma: A foundation model for multimodal agents. arXiv preprint arXiv:2502.13130.
[85] Yang, Z., Li. L., Lin, K., Wang, J.. Lin, C.-C., Liu, Z., and Wang. L. (2023b). The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 9(1),
[86] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2023), Tree of thoughts: Deliberate problem solving with large language models. Advances neural information processing systems, 36, 1 11802-11822.
[87] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, 7.,, al. (2024). Minicpm-v: A gpt-4v level milm your phone. arXiv preprint 2tt.:2400008000
15

---

[88] Yeshwanth, C., Liu, Y.-C., Nießner, M. and Dai, A. (2023). Scannet++: A high- tigh-fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 12-22.
[89] Yu, R., Yu, W., and Wang. X. (2024). Attention prompting on image for large vision-language models. In European Conference on Computer Vision, pages 251-268. Springer.
[90] Yuan, Z., Yan, X., Liao, Y., Guo, Y., Li, G., Cui, S., and Li, Z. (2022). X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. In CVPR.
[91] Zhang. Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C.,, and Liu, (2024a). Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852.
[92] Zhang, R., Han, J.. Liu, C., Gao, P., Zhou, A.. Hu, X., Yan, S., Lu, P., Li, H., and Qiao, Y. (2023a). Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199.
[93] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li,X., X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2055.0166.
[94] Zhang, W., Cai, M., Zhang, T., Zhuang, Y., Li, J., and Mao, X. (2024b). Earthmarker: A visual prompting multi-modal large language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing.
[95] Zhang, Y., Gong, Z., and Chang, (2023b). Multi3drefer: Grounding text description to multiple 3d objects. In ICCV.
[96] Zhang, Y., Li, B., Liu, h., Lee, Y. Gui, L., Fu, D., Feng, J., Liu, Z., and Li, C. (2024c). Llava-next: A strong zero-shot video understanding model.
[97] Zhao, L., Cai, D., Sheng, L., and Xu, D. (2021). advg-transformer: Relation modeling for visual grounding point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2928-2937
[98] Zheng, R., Liang, Y., Huang, S., Gao, J., Daumé III, H., Kolobov, A., Huang, F,, and Yang, J. (2024). Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint arXiv:2412.10344
[99] Zhu, C., Wang, T.. Zhang, W., Pang, J., and Liu, X. (2024a). Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. arXiv preprint 18125
[100] Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023a). Minigpt-4: Enhancing vision-
language understanding with advanced large language models. arXiv preprint 10X55
[101] Zhu, F., Yang, J., and Jiang, H. (2024b). Towards flexible visual relationship segmentation. Advances in Neural Information Processing Systems, 37, 107633-107661.
[102] Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., and Li, Q. (2023b). 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV.
[103] Zhu, Z., Zhang, Z., Ma, X., NIu, X.. Chen, Jia, B., Deng, Z., Huang, S., and Li, Q. (2024c). Unifying 3d vision- laiounlanguage understanding via promptable queries. In ECCV.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract explicitly lists the key claims claims-introducing introducing the Struct2D perception-guidde prompting framework, demonstrating strong zero-shot spatial reasoning in closed-source LMMs, constructing the large-scale Struct2ID-Se dataset, and achieving competitive results after fine-tuning open-source model across multiple benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provided that in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
This paper does not contain theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper names the base model, datasets, and high-level inputs, and it contains the parameters for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We use open-source datasets in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/ppll wwtpcccpppppppppeeeeeeeeeeeeeeee httpe//ggideee/eeeeebbissssssss /Cc////oddiiisssseeeeeee ssiiiiiiscyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The main paper and appendix included the training details and hyperparame- ters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report different metrics and their meanings to the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational resources and time needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We followed the code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide that in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work is for object spatial reasoning ability for 3D scene. We carefully examined the model to avoid such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the datasets used in our paper, and all resources used in this paper open for research usage.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We had tuning set introduced and had detailed introduction about
it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
22

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We used LLM the backbone of our method conduct spatial reasoning task for 3D environment.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy for what should or should not be described.
23

---

A Details of Struct2D Prompting Strategy.
Figure illustrates the overall Struct2D prompting framework, which transforms egocentric 31> scene input into structured 2D representations for spatial reasoning. Given an input video and a spatial question, first reconstruct 3D point cloud from RGB-D frames and remove the ceiling obtain a clear top-down view of the scene. Object detection is then performed in 3D space, and detected objects are projected onto bird's e-ye--viw (BEV) image produce layout of the environment. These object marks are filtered to retain only those relevant to the input question.
We optionally extract egocentric keyframes to capture detailed object appearances. Keyframes selected by projecting 3D object bounding boxes onto sampled video frames and depth maps, and iden- tifying views where each object is both visible and unobstructed. Object-centric metadata- -iniuuuiiiinnnnnnggggg object categories and 3D coordinates encoded as text and used as part of the prompt input.
A oottlines the core procedure for constructing the Struct2D prompt. Given an input video V. depth frames D, a reconstructed 3D scene P, and a set of target objects O we begin by rendering BEV image and projecting each object E O into the view using the RGB camera parameters Crp.- The 2D projections are then drawn as object marks on the image.
To select keyframes, we sample N RGB-D frames and iteratively check for visibility of objects not yet covered the BEV. For each candidate frame 4, we project the remaining unseen objects onto both the frame and its depth map. If valid projection exists (i.c., the projected location lies within the image and has valid depth), the object mark is rendered and the frame added the keyframe set T This process continues until all relevant objects are covered. The final prompt consists of BEV image with filtered marks, @ optional keyframes containing visible objects, and object metadata text, all of which are passed to multimodal large language model for reasoning.
This framework allows the MLLM to perform 3D spatial reasoning from 2D visual and textual inputs, without requiring direct access to raw 3D data at inference time. It enables scalable, flexible spatial understanding grounded in realistic perception outputs.
Algorithm Struct2D Visual Prompting
Input: Input video V, Depth frames D, Reconstructed 3D scene P, Objects interest O, RGB camera parameters Cigb. Depth camera parameters Ca
1: Render a Bird's Eye View image: v +- BEV(P)
2: for o; O do
3: Project o onto v: p; <- Project(o,, v. Cigh) Add-Mark(Y (vark(v,2)) P:)
4: Update view:
5: end for
6: Sample N frames: I, Dj Sample D)
7: Initialize key frame set: Reys < 0
8: Initialize found objects set: Op 0
9: for I E I and D, Dí do
b. False
11:
for 0j O and & Op do
12:
Project 0y onto I, and Dj: 1/ +- Project(o,, I, Cigb): D9 < Project(oy, D,, Ct)
13:
if p and p E D; and p 2 0 then
14:
by + True
15:
Update frame: I < Add- MarlMMrr p/) Add object set: OF Op U (0i)
16:
17:
end end for
18:
19:
if b, then
20: Add to key frame set: Ieys Ikyss {Li}
21: end if
22:
end for
Return: Informative BEV view v and key frame set Ieess
Qualitative comparison of our Struct2D prompting. better understand the impact of prompt design spatial reasoning, conduct qualitative analyses highlighting two key aspects of our

---

Object Locations:
Egocentric Keyframes (optional)
Questhon
sofa
and facing the sofa my from-llefh, frant-nghht back-laft bock-rightt
Object Detection
Multimodal Large Language Model
BEV Image with filtered object marks
Object-centric Me tadata
Here are objects with their mark ids and coordinates 30 space.
Celling removal
Point
sofa: -0.5) [-3.5, 0.2]
reconstruction
[-0.7, -0.1]
Input
Perception Module
Struct2D Prompting Module
Reasoning
Figure 7: Overview of the Struct2D Prompting Framework. Given an egocentric video and spatial question, we first reconstruct a 3D point cloud and remove the ceiling for clear top-down view. Objects are detected in 3D space, and marks projected onto the floor plane. These object marks are filtered based on the content of the question. We also extract egocentric keyframes by projecting 3D bounding box centers onto the video, when appearance cues are needed. Object-centric metadata-incclude coordinates is encoded as text. The structured 2D visual and textual inputs are then fed into multimodal large language model for spatial reasoning.
seel'---eeeeeee (BEV) image is rendered with object
object IDs and 3D)
framework: reasoning guidance, object orientation, and structured metadata. As shown in Figure g when the model prompted only with a BEV image and object marks, it struggles to accurately resolve relative spatial relationships. Adding structured guide prompt enables the model to decom- pose the task into interpretable geometric steps, though it may still fail without an aligned reference frame. Once the BEV is rotation-aligned with the agent's viewpoint, the reasoning becomes more intuitive, leading to the correct answer. Similarly, in Figure 9 we illustrate the benefit object- centric metadata. Without access to precise coordinates, the model must estimate distances visually, which can lead to errors. When provided with 3D object positions, the model can directly compute spatial relations such Euclidean distances, significantly improving its accuracy localization tasks. These examples highlight how prompt structure throuu guided reasoning and geometric plays crucial role in unlocking spatial understanding in MLLMs.
B Details of Struct2D-Set
Overview. Struct2D-Sed is large-scale instruction tuning dataset aimed at enabling spatial reasoning and scene understanding in indoor 3D environments using only 2D projected inputs. contains 200K question-answer (QA) pairs derived from 6K richly annotated indoor scenes drawn from ScanNet [19]. SeanNet++ 188) and ARKitScenes 4. Fach QA instance is paired with structured scene- and object-level metadata, allowing models learn spatial concepts without relying on explicit 3D feature representations during training.
The dataset spans eight categories of spatial reasoning tasks, such as object attribute identification, relative localization, and egocentric navigation. Each QA pair follows an instruction-style format and includes:
A natural language question,
A concise short-form answer,
A long-form answer, when applicable, containing step-by-step reasoning or contextual elabora- tion,
25

---

Original
Quesiion: You are 30 indoor scene assistant. We provide Bird's Eye (iE image with object marks and object categories for analysis. Object marks objects that mentloned the question. The category names of object marks are: Lity, 2:5ofa, 3-table. If am standing the tv and facing the table, the sofa to my front-left, front-right, back-eftt backright?
Options: A. front-right back-left C, back-rgght front-left
Please refer to the object locations through their marks.
Answer:
unable confidently determine the sofa's relative position based on the current image.
w/ Guide Prompt
Guide Promot: Follow the steps below to complete the task accurately.
1. Draw Line A from the point where am standing to the object am facing.
Draw
which perpendicular
intersects at my standing position.
3. To determine the object's relative position in the question: Check which side of Line B the object on. the
object
side the object facing, the front; atherwise, It the back. Nextt, determine object's position relative to right of Line A, is an the right. Left and right should always be determined based on the direction in which am facing.
the left
the left. it to the
Rotation
Answer: Lat's follow the steps:
1.Line goes from object [tv] to object (table), roughly from to right.
2.ne iss perpendizular to that, so it goes top to bottom.
Objectt [sofa] appears to bohind a (l.., the same side table), so it's the back. 4 to the left Line
So, conclude: back-loft
Rotation
Answer: Let's follow the steps:
goes from object
object [table), which exactly vertical and straight up.
2.Line B perpandicular Line A, so it goes loft to right through the Iv. clearly above Line (same side table) + it's the front.
3.The sofa (object
4.The is the left
So, I conclude: The sofa is the front-left relative to my orientation. The correct answer should be D. front-left,
Figure 8: Effects of guide prompting and rotation correction in relative spatial reasoning of the zero-shot prompting. We show different variations of our Struct2D. The original model struggles to interpret spatial relationships using the raw BEV image and object marks alone, failing to answer correctly. With the addition of a structured guide prompt, the model is able to decompose the reasoning process into geometric steps, though it still arrives at an incorrect answer. When object orientation is further corrected via rotation alignment, the spatial reference frame becomes more intuitive, and the model's reasoning becomes clearer and more accurate. This demonstrates the importance of both prompting design and canonical alignment in unlocking MLLMs' spatial understanding abilities. (Red texts are wrong answers; Green texts are correct ones.)
Accompanying metadata including relevant object marks, spatial coordinates, and references to visual input modalities (e.g., BEV image, selected keyframes).
Long-form answers are provided selectively for tasks that benefit from explicit reasoning or contextual understanding. For categories requiring direct factual responses-suce such as object counting or binary verification- short-form answers are used. This balanced design ensures effective supervision across tasks of varying complexity, while maintaining interpretability and richness in reasoning. We next describe the construction process for each task category detail.
Object counting.To To construct object counting questions, we begin by sampling a scene from the training split of the source datasets and extracting its ground- object annotations. A target object category (e.g., chair) is randomly selected from the annotated instances within the scene. A QA pair is then generated using templated prompt such as "How many class label(s) are there in this room?", paired with the correct numerical count as the answer. To improve linguistic diversity and fluency, we further augment these questions by prompting ChatGPT to generate alternative phrasings with equivalent semantic meaning.
Spatial Relationship. This category evaluates a model's ability to reason about the directional relationships between objects in a 3D scene from an egocentric perspective. Following the formulation in VSI-Bench [82). we focus on the subtask of relative direction, where the goal is to identify the directional location of a target object based on a specified standing point and facing direction.
To construct each QA pair, we begin by computing the 3D centers of all objects in the scene and projecting them onto the 2D BEV image. We then sample a triplet of objects representing the roles of standing, facing, and target, while filtering out ambiguous categories (e.g., object clusters or large connected instances) and enforcing minimum pixel-distance threshold to ensure spatial distinguishability. The <standing, facing vector defines the forward direction of the agent, and the
26

---

Original
Object centric Metadata
Questen: You are 30 indecr scene auiatant, Wa procide Brrc's Eye thewettee) image with cbject marks and elject categories for onalysts Ofject marks are objects that are mentinned the qu eation.
The center ecoed the objecn the point doud are oupt: [1,45, 0.55, 2.04),
door 10 89) 1.88, 108) 13: tolle [1 68, 1.80, 0.45) 94 colling Light (1 80, 1.11 A4]
cotegrey
hester
Miusuring
owlling right) the sloned the Sption liotte dear
deses
desestio to the tokket lelject 15% We aperix mate thi computtie the Eilen obstance cetwean the bolket and enct other onjects:
Anower: We are gaan the center coend the odjecti and anker bo five which one
Ploas relar oljoot locations through their marks.
Coordinates:
Ansuan Let's andlyse the relarive distances BEV
Talke
center.
Colling light (94) direclyy above jus sightlyyyyy front the tallet.
shoer (51 13.08 1.24, 1.05]
cup (P) is shoed under the onn, comeator tytther tosord the bottom of the image.
*Coling lilht (34): [1.89, 2.12, 2.44]
Hoater (98) on the fariif wall.
kjjester (30) 1.15, 1.64 (.S))
Deor (s] is on the tep-lett implunng the fooo plane.
Ossances the talex fnsing Budiklean distncely
compane closest paint object tallet
+tup vitl 2.-26 (2 (2.09-2230" 20-0-5 201-2.88f1 1.344
by
dizance
*The (2) from and below deartly fatther.
*ilesten s11 (8-0 15/7 65-1 144 (0.55-0 (035 1.816
<The heater (34) semos the mom bytther
Condusion:
"The door (6) on the not upt.iputilly ucjacon.
door the closss to the tollet.
Correct Answer D. colling light
Answer: C. door
Figure 9: Effect of object-centric metadata for precise spatial reasoning. Originally, the model attempts to estimate distances based solely on the spatial layout in the BEV image but fails to identify the correct object closest the toilet. In contrast, with access object-centric metadata- siciittaassseiilllllll. 3D) coordinates of each object-the model can compute accurate Fuclidean distances and correctly identify the nearest object. This example highlights how structured metadata enhances geometric reasoning and helps avoid ambiguity in visual interpretation. (Red text indicates incorrect reasoning; Green text indicates the correct answer.)
<standing, target> vector is used to determine the relative orientation of the target object. The angular offset between these vectors is then discretized into directional bins such as front-left, right, or buck, producing the correct label.
We format each QA pair using a natural language template (e.g.. I am standing by the TV and facing the refrigerator, is the sink to my left. right, or back?") and provide the short-form directional answer. To enhance both linguistic variation and model supervision, we further augment each instance using ChatGPT, which paraphrases the question and generates a long-form answer that walks through the step-by-step reasoning process under the egocentric frame of reference.
Comparative Reasoning. This category involves tasks where the model must compare spatial attributes among multiple objects. We focus on relative distance comparison, where the objective is to identify which candidate object is closest or farthest from given reference object.
To construct such questions, we first select a reference object whose identity is unambiguous based on its class label. Next, we sample set of candidate objects, including multiple peranccs-potentially of the same class-to encourage instance-level discrimination. In contrast to reference selection, we do not filter ambiguous or repeated categories among the candidates, the goal is to challenge the model to reason over instance-spenn -ppecfffc spatial relations.
We compute the 3D centroid of each object using the center of its oriented bounding box and measure pairwise Euclidean distances between the reference and each candidate. Based on the ranking of these distances, we generate templated question, such as "Measuring from the closest point of each object, which of these objects (candidate labels) closest to the reference object?". along with the correct answer derived from the computed rankings.
27

---

To enhance linguistic variation and encourage deeper reasoning, we further augment each instance using ChatGPT, which paraphrases the question and generates long-form answer. These enriched responses guide the model through comparative spatial reasoning before producing the final answer.
Quantitative Spatial Measuring. This category targets tasks requiring the model to reason about metric properties in 3D space, such as object size, spatial extent, and inter-object distance. We focus on the object absolute distance subtask, where the model needs to estimate the physical distance between two specified objects within scene.
To construct these questions, begin by selecting two distinct objects with clearly identifiable class labels to avoid semantic ambiguity. Using the oriented bounding box annotations, we extract all eight corner points for each object and compute the minimum Euclidean distance across all point pairs-this serves as the ground-truth physical distance between the two objects. Based on this calculation, generate templated questions such as: "Measuring from the closest point of each object, what is the distance between the object/ and the object2 (in meters)?"
To enhance supervision and promote reasoning transparency, we further use ChatGPT to produce long-form answers. These responses walk through the spatial computation process, prompting the model to conceptually simulate pairwise distance comparisons before arriving at the correct numerical answer.
Egocentric Navigation. This category focuses on tasks that require the model to plan navigation routes from an egocentric perspective, reasoning about object references, turning actions, and scene layout. The goal is to simulate how an embodied agent would traverse 3D space by following instructions grounded object-level references.
To construct these tasks, we first sample up to 15 candidate objects per scene and project their 3D centers onto the BEV image. Each object is visually marked in the BEV, and a mark-to-label dictionary string is generated to facilitate object identification. These scene representations are then passed to ChatGPT to generate plausible navigation routes in natural language.
Route generation is guided by several constraints: Each route must consist of a sequence of consecutive object marks (IDs) that agent follow. @ At cach step, the agent must perform a local navigation action (e.g., turn left, turn right, pass by). @ Routes must avoid collisions with irrelevant obstructing objects. t Each path should span 3 to 5 objects to ensure sufficient reasoning complexity.
All generated routes undergo human review ensure spatial plausibility. Invalid routes are discarded, and valid ones are further augmented via route reversal and sub-segmentation to increase diversity.
To determine the action sequence along the path, we randomly choose a facing object at the starting point to establish the initial egocentric orientation. For each transition between objects, we compute the vector from the current object to the next and compare it with the current facing direction to infer the correct action (e.g., turn left, go forward). These navigation actions form the short-form answer.
For each object along the route, we apply our keyframe selection algorithm to extract egocentric views from the original video. These keyframes, combined with the object labels, used to prompt ChatGPT to generate rich textual descriptions of each waypoint. Finally, we instruct ChatGPT to produce long-form answers that walk through the full navigation route, step by step, reasoning over orientation shifts and identifying the appropriate navigation action at each stage.
Other Categories. The remaining task typcs-such object attribute identification and binary attribute verification- are constructed by augmenting QA pairs from existing 3D rision-language datasets, including ScanQA B. SQA3D (53), Scan2Cap [14], ScanRefer [10], and Multi3DRefer (95. These datasets provide scene-specific questions grounded in the ScanNet environment and collectively cover all eight spatial reasoning categories defined in Struct2D-Set.
To adapt these examples for instruction tuning, we first use ChatGPT to rephrase each question into a more natural and instructional style. For tasks requiring reasoning, we also prompt ChatGPT to generate long-form answers that walk through the inference process. For visual grounding, we localize referenced objects using two approaches: when object IDs are available, we apply our keyframe detection method to extract representative egocentric views. In datasets with descriptive references (c.g., SQA3D), we extract referring expressions with ChatGPT and apply Grounding DINO [5 and SAM (6) to identify and segment the mentioned objects.
28

---

Spatial Location
Object Counting
Q: What is placed on the bed the room
Q: How many black chairs are located beside the table A: There are black chairs on the right side of the table.
A: On the bed, there is a blue pillow with blue arms, which is visible in the image where the bed prominently located towards the right side, suggesting cary setup.
Spatial Relation (Relative)
Object Attribute Identification
Q: What is to the right of the picture? ?
What color the curtain
the right doorframe. This indicated in the bird's-eye view and the keyframe image, where the doorframe is positioned adjacent the seating area, suggesting clear exit point in the layout.
of the image, there is
white seen image hangs beside the bed, contrasting with the surrounding décor.
Figure Examples of QA pairs from Struct2D-Set used for instruction tuning. Each example illustrates distinct category of spatial reasoning: spatial localization, object counting, spatial relationship, and object attribute identification. For each question, the model is provided with BEV image annotated with object marks, and optionally an egocentric keyframe to enhance visual grounding. The answers include descriptive reasoning grounded in object positions and appearances, enabling the model to learn to associate structured 2D inputs with fine-graie fineggrinedd spatial understanding.
The resulting keyframes are paired with each QA instance to serve as visual inputs during fine-tuning. This pipeline enables instruction tuning complex, object-centric spatial tasks while relying only on 2D visual projections and avoiding the need for explicit 3D geometry at training time.
C Implementation Details
We use Qwen2.5VL 72 as the base multimodal large language model (MLLM) for instruction tuning. During training, the model receives BEV images with filtered object marks and object- centric metadata as core inputs. For tasks requiring visual cues such as object color or quantity, we additionally provide egocentric keyframes. The BEV images resized to 640 640, with object marks adaptively scaled based on their original image resolution. Keyframes are resized to 256 x 246 and stitched into compact 1x2 2x4 grids, enabling efficient batch loading and reducing GPU memory consumption.
To support reasoning supervision, adopt task-specific output format. For complex spatial reasoning tasks-such as relative direction estimation or route planning-we wrap the reasoning process between special tokens <think> and </think>, followed by the final answer enclosed within <answer> and </ </answer>. For tasks focused on appearance or simple measurements, the model is trained to generate direct short-form answers without explicit reasoning traces. The model is trained for one epoch using a base learning rate of 2e-6 with cosine annealing, taking approximately 8 hours 8xH200 GPUs.
At evaluation time, we follow standard practices from prior work 31 63 reconstructing point clouds offline using BundleFusion (18), detecting 3D objects using Mask3D and UniDet (37).

---

Binary Attribute Verification
Egacentric Navigation / Orientation
Q: the bedside lamp on off A: The bedside lamp between the two beds is on.
facing the window
Q:
you were
the left side of the there large whiteboard, which visible the image and occupies significant part of that wall area..
facing the window,
Quantitative Spatial Measurement
Comparative Reasoning
Q: What is the size of this room (in meters)? if multiple rooms shown,
the combined space.
Short answer: 21.2.
Augmented answer: The calculated 2D distance between the two bods approdmately 2.0 meters. This scale reference to estimate the width and length of the room. Given the visible estimate the room dimensions to be approdimatll 4.0 meters in width and 5.3 meters in length. Thus, the dimensions are approximately
What is smaller of the objects hanging on the wall? soap dispenser, which is typically mare compact compared to other wall mounted items, such towel holder larger storage units that may also be present in kitchen area. This can be inferred from the layout and features of the kitchen visible the images
object hanging on
such the walls and floor boundaries,
"4,0m "5.3m, yiclding total
square meters, which matches the original room size value 21.2.
Figure 11: Additional QA examples from Struct2D-Set covering diverse spatial reasoning categories. This figure showcases examples from the remaining categories our dataset: binary attribute verification, egocentric navigation and orientation, quantitative spatial measurement, and comparative reasoning. Each QA pair is grounded structured 2D visual inputs (BEV views and keyframes) and enriched with object marks and contextual metadata. These examples demonstrate the model's ability to reason about object states, egocentric spatial references, metric estimations, and relative comparisons key competencies for embodied spatial understanding.
and projecting the results to produce BEV images and 2D object marks. For object-level grounding, apply rule-based method to identify the relevant objects mentioned in each question.
D Results on 3D Grounding and 3D Dense Captioning
Quantitative results. Tables and present our model's performance on 3D grounding (ScanRefer, Multi3DRefer) and dense capuoning (Scan2Cap) benchmarks. While our method does not achieve the highest scores under rule-based metries such as delivers competitive results compared existing vision- vision-language baselines. Importantly, our approach does not rely on point cloud features during training or evaluation, contrast task-specific and LIM models that depend heavily explicit 3D representations. In addition, our approach requires substantially fewer egocentric keyframes on average (2 compared to in GPT4Scene (63)), resulting in more efffcient and scalable training process. Compared to models designed for narrow tasks, framework is more general and supports a wider range of spatial reasoning types, including relative direction and route planning, which are not covered by these benchmarks, It is also worth noting that the current evaluation metrics are rule-based and limited in expressiveness, which may not fully reflect a model's capability in spatial understanding.
@0.25/00. and BLEU/ROUGE, consistently
Qualitative results. Figure 22 illustrates qualitative examples our fine-tuned Qwen2.5-VL--7B model across three major spatial reasoning tasks: 3D dense captioning, object grounding, and 3D question answering. In each case, the model receives a BEV image with object marks, optionally supplemented with egocentric keyframes and metadata, and produces either descriptive caption, an
30

---

Table 6: 3D Grounding Evaluation on ScanRefer 10] and Multi3DRefer 95 datasets.
ScanRefer (val)
Multi3DRefer (val)
Methods
Acc@0.25
Acc@0.50
all Fl @0.25
all Fl @0.50
Task-Specific Model
3DVG-Transformer
97
47.6
34.7
25.5
3DICG
6
49.6
37.3
26.6
D3Net
III
37.9
32.2
M3DRef-CLIP
95
51.9
44.7
42.8
38.4
3D LLM Based Model
Chat- Scene
31
55.5
50.2
57.1
52.4
Vision LLM Based Model
Owen2-VL-77 (72
5,4
5.1
21.1
19.9
Qwen2-VL-7B (GPT4Scene 633
40.5
36.7
45,4
42.1
Qwen2.5-VL-7B (Ours)
51.7
48.5
42.1
40.6
Table 7: 3D Dense Captioning Evaluation on Scan2Cap [14] dataset.
IoU@0.5
Methods
BLEU-4
ROUGE
BLEU-4
ROUGE
Task- Spec-fpe Model
Scan2Cap 14
34.2
55.3
23.3
44.5
3DICG 6
40.2
59.2
31.0
50.8
X X-Trans2Cap 990
35.7
54.7
25.1
45.3
3D-VisTA 1102
36.5
57.6
34.0
54.3
Vote2Cap-DETR
12
39.3
59.3
34.5
54.4
3D I.M Based Model
LL3DA 16
41.4
59.5
36.8
55.1
LEO [2
36.9
57.8
Chat-Scene 31
38.2
60.6
36.3
58.1
Robin3D 35
38.4
Vision LLM Based Model
Owen2-VL-H VL-7B 722
3.8
24.7
3.8
24.6
Owen2-VL--77 (GPT4Scene
63
36.3
57.6
34.2
55.2
Qwen2.5-VL-7B (Ours)
34.8
57.0
32.7
54.5
object ID, or a short-form answer. The examples demonstrate the model's ability to reason about visual attributes (e.g., "a brown rectangle"), relative spatial positions (e.g., "the table is to the right of the couch"), and numerical or commonsense questions. We observe that the model often produces answers consistent with the ground truth, and in some cases offers additional descriptive clarity grounded the visual context. These results highlight the effectiveness of our Struct2D prompting strategy in enabling rich spatial understanding from structured 2D inputs.
E Failure cases
To better understand the limitations of our approach, we conducted qualitative error analysis on 30 representative questions spanning multiple QA types in VSI-Bench. Among the 16 failure cases, we identified two dominant causes. First, in 11 cases, the underlying 3D reconstruction was noisy or incomplete, producing degraded BEV projections that obscured critical spatial layouts. Second, in 5 cases, missing setections-ofie involving small heavily occluded objects- ledd to incomplete structured inputs. Both factors reduce Struet2D'ss ability to encode accurate spatial cues, thereby hindering its reasoning capability. Representative visualizations of these failure modes are shown in
31

---

3D dense captioning
3D dense captioning
3D grounding
Pann a picture of the visual stt fttributes <0B111>, than osplore how rolates sputially to other elements in the scette
Questtor:
Question: picture the visul atributes of
alggrs Icenctt iption tabli, slaced the midele roam, on brown chairs on the right are brown chairs, the front a brown door with light shining ont" Ground Truth: 23.
<OB1002>, than exalore hew it relates sponilly to other elements in the scome.
Ground Truth: black
furthest
the alad.
Dure The table to the right of the couch. The table a brown rectangle.
3D Question Answering
3D grounding
3D Question Answering
Question: in and my baekpwckk right Ban amount book in front of me diviiee by three? Ground Ours: No.
Qussioo:: idantify the iD of the object that closely with description "this pulles doarony
Questton: am walking up the starss near the middle the stain going to the sucund floor. What is onvering the window my righty
Curtales. Ours: Curaanss
Ground Truth: 12.
Dune 12.
Figure 12: Output examples from fine-tuned Qwen2.5-VL-7B model across multiple 3D spatial reasoning tasks.The figure showcases model responses on 3D dense captioning, object grounding, and 3D question answering tasks. Each example includes the question, BEV and keyframe inputs with object marks, the ground-truth answer, and our model's prediction. These examples illustrate the model's ability localize, describe, and reason about spatial relations using structured 2D prompts derived from 3D scenes. Across tasks, the model demonstrates strong alignment with ground-truth answers, even when questions require appearance attributes, relative spatial context, or numerical reasoning.
Question: How many sofa(s) are in this room?
Question: What the length the langest dimension (length, width, height) table, measured centimeters?
Figure 13: Examples of failure cases caused by 3D reconstruction and detection.
32

---

These examples highlight the importance of robust 3D reconstruction and reliable object grounding for spatial reasoning complex indoor scenes.
Figure 13
Broader impacts
Our work introduces Struct2D, perception-guiee prompting framework that enables Multimodal Large Language Models (MLLMs) to perform robust spatial reasoning in 3D environments using only structured 2D and text inputs. This direction offers several broader implications for research, society, and the reasonable development AI systems.
Social Benefits. Struct2D lowers the barrier to 3D spatial reasoning by leveraging RGB-D percep- tion instead of requiring dense 3D annotations or point cloud inputs during inference. This makes spatial understanding more accessible to wide range of applications, especially in settings where real-time 3D sensing is noisy, sparse, or unavailable. Potential downstream applications include:
Assistive robotics, where spatial-language understanding is critical for navigation and object manipulation in dynamic home environments;
Augmented reality interfaces, where natural-language spatial queries must be resolved in partially reconstructed environments;
Accessibility technologies, especially for users with visual impairments, by enabling robust, language-driven scene understanding with minimal hardware.
Potential Negative Impacts. The preprocessing pipeline relies on egocentric video and 3D recon- struction, which may involve scenes from private homes workplaces. If deployed in real-world applications, such systems may inadvertently capture sensitive spatial or personal data. Ensuring strict anonymization, access control, and user consent mechanisms is essential.
Research Contributions. By decoupling MLLM training from explicit 3D input requirements, Struct2D promotes research into modular, scalable instruction-tuninn pipelines that can generalize across environments with different sensor setups. Furthermore, our public release of Struct2D- -Sett large-scale spatial instruction dataset built with principled blend of structured prompts, egocentric frames, and metadata- Baaedatt-cootribteee contributes valuable benchmarks to the broader vision-langen lasion-langugge community.
33


Reviews:
[{"review_id": "P3aoQX46DV", "reviewer": "Reviewer_a2sz", "summary": "This paper presents Struct2D, a perception-guided prompting framework designed to enable spatial reasoning in Large Multimodal Models (LMMs) using only structured 2D representations derived from 3D perception, rather than explicit 3D data. The approach combines bird’s-eye-view (BEV) images, object marks, object-centric metadata, and optionally egocentric keyframes to form the input for LMMs. The authors demonstrate, through zero-shot analysis with closed-source LMMs (like GPT-4o), that these models can reason effectively about 3D spatial tasks (e.g., direction estimation, route planning) from the structured 2D input. Building on this, the authors construct a large-scale dataset (Struct2D-Set) via automated QA pair generation from 3D indoor scenes, and instruction-tune an open-source LMM (Qwen2.5VL). Experiments across several benchmarks show that the Struct2D-tuned model achieves strong, sometimes state-of-the-art, performance on a suite of spatial reasoning tasks, without requiring explicit 3D inputs at inference time.", "questions": "1. Can you provide further analysis or case studies on natural images or more complex, unstructured real-world scenes? Most benchmarks are on indoor scenes. Have you tried Struct2D on different domains (e.g., urban outdoors, street scenes, aerial images), and what would need to be changed for broader generalization?\n\n2. Please include and discuss representative failure cases -- when does Struct2D not help (or even hurt), and why?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The framework is practical, lowering the barrier for spatial reasoning with LMMs by removing the need for explicit 3D inputs at inference.\n2. The paper is technically sound and experimentally thorough. It provides a detailed ablation analysis (e.g., effects of metadata, prompt structure) and evaluates across multiple benchmarks (VSI-Bench, ScanQA, SQA3D, etc.).\n3. The approach bridges a major gap between 2D perception and 3D spatial reasoning in LMMs, showing that explicit 3D inputs are not necessary for strong performance.\n\nWeaknesses:\n1. The impact may be somewhat limited by the continued need for good 3D perception in the preprocessing stage, even if not at inference.\n2. The presentation of dataset generation is a bit unclear. It could benefit from a flowchart summarizing the pipeline.\n3. The manuscript lacks a deep exploration of limitations in highly cluttered or occluded environments, or how the framework generalizes to domains beyond indoor scenes.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer a2sz for the thoughtful and constructive feedback. We appreciate the recognition of Struct2D as practical, technically sound, and experimentally thorough. Below, we address the reviewer’s questions and concerns point by point.\n\n**1. 3D Preprocessing Requirement**\n\nThank you for raising this point. As we noted in our *supplementary*, Struct2D does rely on 3D perception in preprocessing, but this step is ***lightweight*** (under 5 seconds per scene on a single GPU) and entirely offline. Importantly, this design choice decouples perception from inference, enabling flexible and efficient deployment. Accurate 3D perception is a standard practice in recent 3D-LLM research and plays a critical role in enabling robust spatial reasoning. Continued progress in this area will further amplify the effectiveness of models like Struct2D:)\n\n**2. Failure Cases of Struct2D**\n\nThank you for the valuable suggestion. We conducted a qualitative analysis on 30 representative questions across different QA types in VSI-Bench. Among the 16 failure cases, we identified two primary causes: (1) in 11 cases, ***noisy or incomplete 3D reconstructions*** led to degraded BEV inputs, and (2) in 5 cases, ***missing detections***, particularly for small or occluded objects, resulted in incomplete or incorrect structured inputs. These factors hinder Struct2D’s ability to provide accurate spatial cues, which are essential for effective reasoning.\n\nWhile the NeurIPS rebuttal policy prohibits image inclusion, we will present representative visual examples and a more detailed breakdown of these failure modes in the revised manuscript.\n\n**3. Case Study on Outdoor / Open-world Domains**\n\nThank you for the suggestion. We conducted a case study using scenes from DL3DV-10K, which includes various ***outdoor environments such as tourist sites, educational institutions, and natural settings***. We reconstructed point clouds from RGB frames using VGGT, manually labeled object marks, and constructed 15 spatial reasoning QA pairs (*e.g.*, *“What is in front of the door of the public restroom?”* → *“Three trash bins”*). Using Struct2D prompting with GPT-O3 in a zero-shot setting, we achieved *72%* accuracy. Most failures were due to noisy or incomplete reconstructions, which are more common in outdoor scenes. \n\nWhile our current focus aligns with existing indoor benchmarks, we agree that broader generalization is important (as in *supplementary*) and plan to incorporate outdoor scenes into future training and evaluation.\n\n**4. Dataset Generation a bit Unclear**\n\nThank you for the helpful suggestion. Our dataset generation process involves: (1) sampling scenes from existing 3D indoor datasets; (2) leveraging object annotations to generate diverse QA types using templated codes; (3) manually balancing the distribution of QA types; (4) enriching questions and answers with explicit reasoning steps; and (5) conducting quality control to ensure clarity and correctness. \n\nWe agree that a flowchart would significantly clarify this pipeline. Although we cannot include figures in the rebuttal due to PDF restrictions, we will incorporate a clear flowchart in the revised paper to enhance presentation:)"}, {"review_id": "wcAmpldGIU", "reviewer": "Reviewer_54eJ", "summary": "This paper introduces Struct2D, a perception-guided prompting framework, While multimodal large language model has been used for 3D scene understanding with point cloud information as input, authors argue that structured 2D representation can also be used to reason about 3D space. Their structured 2D representations include BEV images with object marks, object-centric metadata, and optionally egocentric keyframes. They have also used guided prompt to improve zero-shot performance. They have constructed an instruction tuning dataset Struct2D-Set to tune open-source multimodal large language model (MLLM).\n\nFollowings are the contribution of the paper:\n1. They have proposes a perception-guided 2D prompting strategy to perform 3D spatial reasoning from structured 2D inputs alone,\n2. They have introduced instruction tuning dataset Struct2D-Set for this task,\n3. They have fine-tuned open-source MLLM and achieved state-of-the-art performance across several benchmarks.", "questions": "Please check weaknesses. \n\n1. Is guided prompt also used during instruction tuning?\n2. Suggestion: Should include more details on keyframe selection.", "limitations": "Limitations are included in the supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is very well written, \n2. The experimental evaluation is thorough. They have also achieved state-of-the-art performance in the relevant benchmarks,\n3. They have shown how structured BEV image, combined with metadata, is often sufficient for accurate zero-shot 3D scene understanding. This key insight is valuable for computationally efficient 3D scene understanding.\n4. They have also contributed an instruction tuning dataset for this task.\n\nWeaknesses:\n1. Technical novelty of this paper is very limited. This paper is not the first work that tries to reason over 3D scene using 2D data with multimodal large language model (MLLM). GPT4Scene [56] also had the similar motivation and utilized 2D input to infer on 3D scene using MLLM. Main improvement over GPT4Scene is the use of object marks, metadata, guided prompt, and key frames. All of these are actually incremental novelty.", "was_revised": true, "final_justification": "Thanks to the authors for their effort. I think this paper has some limited technical novelty but provides useful insights that will help the community. Therefore, I will keep my rating.", "rebuttal": "We sincerely appreciate reviewer 54eJ’s thoughtful and constructive feedback. We are grateful for the positive remarks on the clarity of our writing, the depth of our evaluation, and the value of our insights. Below, we address the reviewer’s comments and questions in detail.\n\n**1. Guided Prompt used in Instruction Tuning**\n\nYes, we incorporate guided prompts during instruction tuning, particularly for \"relative direction\" questions. This design is inspired by our findings from zero-shot prompting, where structured guidance proved especially beneficial for spatial reasoning tasks:)\n\n**2. Details on Keyframe Selection**\n\nThank you for the suggestion. We will include more details and a pseudocode block in our revision. Our keyframe selection aims to enhance the BEV representation by providing better object detail, particularly for questions involving object attributes and viewpoint-based spatial relations. \n\nTo build the keyframe set $I_{\\text{keys}}$, we sample $N=100$ RGB-D frames and iteratively select those that improve coverage of objects not yet captured in the BEV. For each candidate frame $I_i$, we project the remaining uncovered objects onto the image and its depth map. If the projection lies within the image bounds and has valid depth, the object is marked as visible and added to the BEV. The corresponding frame is then included in $I_{\\text{keys}}$. This process repeats until all target objects are covered.\n\n**3. Limited Novelty**\n\nThank you for the feedback. While GPT4Scene also leverages 2D inputs for 3D reasoning with MLLMs, our work tackles a different and more challenging ***problem scope*** -- complex spatial reasoning tasks such as *relative direction* and *route planning*, which require multi-step geometric inference rather than simple object-level understanding.\n\nMoreover, our contributions go beyond incremental improvements. We introduce a data generation pipeline and a tunable dataset specifically designed for spatial reasoning instruction tuning. Our structured reasoning framework integrates object-centric BEV representations, guided prompts, and keyframe selection to support fine-grained geometric reasoning. \n\nLastly, in contrast to GPT4Scene’s primarily rule-based evaluation, our work adopts accuracy-based metrics that more directly reflect reasoning performance."}, {"review_id": "LXOI9V3Jfi", "reviewer": "Reviewer_QoNX", "summary": "In this paper, the authors propose a perception-guided prompting framework to reason about 3D using only structured 2D representations. Besides, they also construct $Struct2D-Set$, a large-scale instructional tuning dataset. Compared with existing similar work, their dataset is filtered by object marks, prompted guided by their metadata as well as selected by key frame.", "questions": "As for Section 3:\n1. Can author explain that how metadata is obtained? Do the authors also filter the metadata?\n2. As for subset for evaluation, do 422 QA pairs randomly selected?\n3. Can authors also explain what is the text query length of $T^{out}$ when incorporating with the $Struct2D$? If possible, can authors provide ablations on text query length?\n\nAs for Section 4:\n1. Can the authors provide settings when using VLMs, such as temperature?\n2. Can authors explain when using VLMs whether you run several times or only once?\n\nAs for Section 5:\n1. It seems like the VLMs matter, but in Table 3 and Table 4 the Qwen model, authors comparing with, is Qwen2. Will the existing dataset incorporated with Qwen2.5 improve the performance?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The topic of this paper is interesting.\n2. The paper is well-structured and easy to follow.\n3. The experiments and ablations are comprehensive.\n\nWeaknesses:\n1. This is maybe minor, but it will be great if the authors could unify the front of $Struct2D$.\n2. There are some missing details, I will put them in Question Section.\n3. The contributions seem inefficient since the improvement on some shown in experiment section seems not significant.", "was_revised": true, "final_justification": "The authors' explanations resolved most of my main concerns. Therefore, I will raise my rating to borderline accept.", "rebuttal": "We are grateful to Reviewer QoNX for the helpful and encouraging feedback. We appreciate the recognition of our paper’s organization and the breadth of our experimental analysis. Please find our detailed responses to the reviewer’s questions and suggestions below.\n\n**1. Metadata Extraction and Filtering**\n\nWe extract metadata by detecting 3D object bounding boxes and categories from the reconstructed point cloud using off-the-shelf 3D detectors such as Mask3D and UniDet3D. Each object’s center coordinates are taken as its location. This structured metadata is used to provide grounding context in prompts. To ensure relevance and reduce noise, we filter metadata to only include objects referenced in the question.\n\n**2. Evaluation Subset Construction**\n\nWe do not randomly sample the subset. Instead, we first verify that all objects referenced in each question are clearly annotated in the 3D point cloud. Then, we select a subset that preserves the distribution of question types from the full dataset. This ensures a *balanced and representative* evaluation set.\n\n**3. Text Query Length and Ablation**\n\nThe text query length includes ***image tokens***, ***structured metadata (converted to text)***, and ***the input question***. We set the maximum input length to 4096 tokens, which empirically accommodates all required information without truncation. Reducing the length to 2048 degrades performance (accuracy drops from 41.3 to 33.5) due to incomplete input encoding. Increasing it to 6000 brings no additional gains, indicating 4096 is sufficient.\n\nThe $T^{out}$ refers to model output. We use `max_new_tokens=2048` to allow complete answer generation.\n\n**4. VLM Usage Settings**\n\nFor zero-shot prompting, we use a temperature of 1.0 and generate five responses per input. The final answer is chosen by majority vote. For QA pair enrichment, we also set the temperature to 1.0 and generate each sample with a single forward pass.\n\n**5. Comparison with Qwen2-VL**\n\nThank you for the suggestion. We added a comparison using Qwen2-VL-7B to evaluate the impact of newer models. As shown below, Qwen2.5-VL provides slight performance gains over Qwen2-VL. Importantly, our pipeline improves performance in both cases and consistently outperforms GPT4Scene.\n\n| Benchmark           | ScanQA                      |                           |        | SQA3D                |         |\n|---------------------|-----------------------------|---------------------------|--------|----------------------|---------|\n| Metric              | BLEU-1                      | METEOR                   | ROUGE | EM-1                | EM-R1   |\n| Qwen2-VL-7B (GPT4Scene) | 43.4                        | 17.7                      | 43.6   | 57.4                | 60.7    |\n| Qwen2-VL-7B (Ours)     | 44.9                        | 17.5                      | 44.0   | 58.3                | 60.9    |\n| Qwen2.5-VL-7B (Ours)   | 45.2                        | 17.4                      | 44.1   | 58.5                | 61.3    |\n\n**6. Limited Improvements on Some Benchmarks**\n\nThe improvements on benchmarks such as ScanQA and Scan2Cap are relatively modest (44.9 Vs 43.4, 34.8 Vs 36.3), as these tasks largely depend on localized or *viewpoint-level* understanding and can often be addressed effectively with just a single or dual-frame input.\n\nOur method is designed for complex, global spatial reasoning, which is better reflected in VSI-Bench tasks (*e.g.*, multi-object relationships, navigation). Limitations in keyframe selection, missing object detections, and occlusion may hinder performance on traditional benchmarks that lack comprehensive spatial annotations.\n\nAdditionally, standard benchmarks often use rule-based metrics (*e.g.*, BLEU, METEOR, ROUGE scores) that may bias models toward templated or memorized responses. We instead emphasize accuracy-based evaluation to more reliably measure spatial reasoning capabilities, as in VSI-Bench."}, {"review_id": "Wwuc8mfuOy", "reviewer": "Reviewer_D1rW", "summary": "This paper presents Struct2D, a prompting solution for large multimodal models (LMMs) that prompts LMMs with bird’s-eye-view (BEV) images, object marks, object-centric metadata, and additional keyframe information. Zero-shot analysis on closed-source LMMs indicate that the proposed prompting strategy achieves promising performance. Therefore, additionally an instruction tuning dataset Struct2D-Set is proposed for finetuning the open-source LMMs on this specific prompting strategy. Experimental results show that finetuning with the newly curated Struct2D-Set achieves superior performance across different benchmarks.", "questions": "- What are the additional insights of the prompting design in Struct2D? Especially compared with GPT4Scene.\n\n- What are the advantages and special designs of the curated Struct2D-Set for instruction tuning for 3D scene understanding?\n\n- If performing a fair comparison with GPT4Scene, how is the performance of the proposed solution compared with GPT4Scene?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The workflow of this paper is smooth. First, the paper shows promising results with zero-shot Struct2D prompting strategy for closed-source LMMs. Then, the paper constructs an instruction tuning set for supervised finetuning on open-source LMMs, and further boosts the performance on open-source LMMs based 3D scene understanding.\n\n- The proposed method well leverages the powerful 2D foundation models like 2D LMMs and 2D SAM for 3D scene understanding, overcoming the limitations from the scarcity of 3D data.\n\n- The paper introduces a Struct2D-Set, which can be useful for future research in 3D scene understanding with instruction tuning.\n\n- The paper is clearly written and easy to understand.\n\nWeaknesses:\n\n- This work is so similar to GPT4Scene, although the paper made conceptual comparisons with GPT4Scene in Lines 172-180. However, first I think GPT4Scene also has object marks as part of its prompting strategy, called STO-Marker in the GPT4Scene paper. The major difference is that the proposed Struct2D only includes objects relevant to the question, while GPT4Scene gets all the instances in the 3D scenes. The metadata prompts for objects and the selected keyframes are essentially the major components added on top of GPT4Scene. Therefore, I think there is not enough innovation in the proposed prompting method Struct2D, making the contribution of this paper incremental.\n\n- For the experimental results, the comparison with GPT4Scene in Table 4 is unfair to me. As mentioned in Lines 261-262, all the visual inputs are resized to $640\\times 640$ in this paper. However, the numbers for GPT4Scene in Table 4 are using resolutions $128\\times 123$. Therefore, if using the HD or HDM version of GPT4Scene, the proposed Struct2D will have worse or comparable performance on many of the metrics. For example, for METEOR/ROUGE/CIDEr on ScanQA, and EM-1/EM-R1 on SQA3D, GPT4Scene-HDM will actually outperform the proposed method. Even in this case, GPT4Scene is using $512\\times 490$ resolution, which is still lower than the size of the visual inputs used for Struct2D.\n\n- It is unclear what is special about the curated Struct2D-Set dataset for instruction-tuning, compared with existing data for 3D instruction tuning. Is it specially curated for matching the QA types in VSI-Bench and other 3D understanding benchmarks like ScanQA, SQA3D, etc.?\n\n- The training cost of the proposed method is also relatively larger compared with GPT4Scene. The proposed method requires 8 hours on 8 H200 GPUs, while GPT4Scene needs 6 hours on 8 A100 GPUs. Intuitively, since Struct2D has only the object markers that are relevant to the question, the training process should be more efficient than the previous methods.", "was_revised": true, "final_justification": "In all, this paper is very similar to GPT4Scene (also mentioned by Reviewer 54eJ, and the object marks mentioned by Reviewer 54eJ already exist in GPT4Scene at least in some format), with notable contributions being designing more efficient prompts for 3D reasoning, as seen in the authors reply to my question ***Comparison with GPT4Scene: Novelty and Prompting Design***, basically the first four points are saying how to design a more efficient prompt for selecting objects related to the question, enabling explicit reasoning, etc. But the authors show in the rebuttal that in a fair comparison setting, the proposed solution indeed has advantages over the baseline GPT4Scene, making the method valuable in practical usage. Therefore, I would like to raise my score to borderline accept.", "rebuttal": "We greatly appreciate your careful review and insightful comments. Thank you for recognizing the contributions of Struct2D and Struct2D-Set. Below, we provide detailed responses to your concerns.\n\n**1. Comparison with GPT4Scene: Novelty and Prompting Design**\n\nWe respectfully clarify that although GPT4Scene and Struct2D both use BEV representations and object markers, our method differs significantly in motivation, design, and technical execution:\n\n- ***Targeted Spatial Reasoning Focus:*** Our primary goal is to enable explicit spatial reasoning, particularly global spatial tasks such as relative directions and route planning, as proposed in VSI-Bench. In contrast, GPT4Scene focuses more on scene understanding and QA in relatively constrained settings. The task scope and required reasoning complexity are therefore different.\n\n- ***Selective Object Prompting:*** While GPT4Scene includes all objects in a scene (via STO-Markers), Struct2D only includes objects relevant to the question, significantly reducing visual clutter in the BEV image. This makes the image more interpretable and improves reasoning performance.\n\n- ***Guide Prompts & Reasoning Chains:*** Our prompting strategy integrates guide prompts and intermediate reasoning steps during both training and inference (akin to chain-of-thought prompting). This improves model interpretability and performance and offers a more structured approach to 3D scene reasoning.\n\n- ***Efficiency and Generalizability:*** Our prompting method is more concise and efficient, as it avoids exhaustive object rendering. In addition, we introduce a generalizable data generation pipeline, enabling scalable tuning of MLLMs for spatial reasoning, which goes beyond the static pipeline of GPT4Scene.\n\n- ***Key Insight:*** Our zero-shot results suggest that the performance bottleneck in modern MLLMs lies in 3D perception, not reasoning. We believe this is an important insight that can shape future work on 3D vision-language models.\n\n**2. Fair Comparison with GPT4Scene: Resolution and Training Cost**\n\nWe appreciate your concern regarding fair comparisons! We have carefully matched the experimental settings:\n\n- ***Resolution Matching:*** GPT4Scene stitches 16 frames into a 2×8 grid. Each frame has a resolution of 128×123, resulting in a composite keyframe image of 512×246. Their BEV images are also resized to 512×512. We adopted the same resolution setup to ensure a fair comparison.\n\n- ***Data Volume Differences:*** GPT4Scene uses all 16 video frames, whereas our Struct2D approach averages only ~2.5 keyframes per sample, with selection driven by object relevance. Despite this much lower visual input volume, our method matches or outperforms GPT4Scene, and can do well at reasoning-heavy benchmarks like VSI-Bench.\n\n- ***Updated Quantitative Comparison (Qwen2-VL-7B):***\n\n| Benchmarks               | ScanQA                              |       |        |        |        | SQA3D         |        |\n|--------------------------|--------------------------------------|-------|--------|--------|--------|----------------|--------|\n| Metric                         | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr | EM-1          | EM-R1 |\n| Qwen2-VL-7B (GPT4Scene)  | 43.4   | 14.6   | 17.7   | 43.6  | 90.9  | 57.4          | 60.7  |\n| Qwen2-VL-7B (Ours)       | 44.5   | 14.9   | 17.4   | 44.0  | 91.6  | 58.1          | 61.0  |\n\n- ***Training Cost Clarification:*** Using the same hyperparameters as GPT4Scene (batch size, learning rate, dataset), our training takes 6.5 hours on 8×H200 GPUs. The 8-hour figure mentioned in the paper includes additional training with extra curated QA types. We will clarify this in the revised manuscript.\n\n**3. Struct2D-Set: Dataset Design and Advantages**\n\nWe designed Struct2D-Set to specifically address spatial reasoning gaps in existing 3D instruction-tuning datasets:\n\n- ***Reasoning-Centric QA Pairs:*** Most prior datasets (ScanQA, SQA3D, Scan2Cap) emphasize object attributes, descriptions, or basic viewpoint understanding. In contrast, Struct2D-Set focuses on global spatial reasoning, such as route planning, orientation, and relative positioning.\n\n- ***Instruction-Tuning Compatibility:*** We enrich both QA types and formats to better support instruction tuning of MLLMs, incorporating tasks like step-by-step spatial deduction, instruction following, and spatial transformations.\n\n- ***2D BEV Representation:*** All samples include BEV images with minimal but relevant object annotations, allowing efficient and scalable instruction tuning without the need for complete 3D scene representations as used in existing 3D tuning datasets.\n\n- ***Diversity and Coverage:*** Struct2D-Set includes both complex reasoning questions and standard QA formats such as from ScanQA and SQA3D, ensuring broader coverage for generalization while staying focused on our spatial reasoning objective.\n\nWe hope this clarifies our contributions and positioning relative to GPT4Scene. Thank you again for the valuable feedback! We will revise the manuscript accordingly to better communicate these points."}]

Author rebuttal:

Meta-review: The paper introduces Struct2D, a prompting framework designed to enable spatial reasoning in Large Multimodal Models (LMMs) using structured 2D representations derived from 3D perception, rather than explicit 3D data. Zero-shot analysis on closed-source LMMs shows that the proposed prompting strategy achieves promising performance. Moreover, fine-tuning open-source LMMs with this strategy on the newly curated Struct2D-Set yields superior performance across different benchmarks.

Overall, all four reviewers remain positive about the paper, noting that the experiments and ablations are comprehensive and effectively demonstrate the framework’s effectiveness. They also recognize that the proposed Struct2D-Set will be valuable for future research. One main initial concern raised by several reviewers was the limited novelty compared with prior work, particularly GPT4Scene. The authors and reviewers (54eJ and D1rW) further discussed this point, and both reviewers agreed that although the technical novelty is limited, the paper still provides useful insights and demonstrates performance that can outperform existing methods.
Decision: Accept (poster)