3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
Authors: Xiaohu Huang
Jingjing Wu
Qunyi Xie
Kai Han
Venue: NeurIPS 2025 poster ()
Keywords: Scene Understanding
Multimodal Large Language Models
Primary area: deep_learning
TL;DR: Improving 3D representation learning for multimodal large language models.

Abstract: Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs—including visual grounding, captioning, and question answering—demonstrate consistent performance gains. Project: https://visual-ai.github.io/3drs/

3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
Xiaohu Huang'
Jingjing Wu²
Qunyi Xie
Kai Han
Visual AI Lab, The University of Hong Kong Department of Computer Vision Technology (VIS), Baidu Inc.
hunggziaohu&connet.. aku kaihanxChku.
Abstract
Recent advances in understanding have leveraged multimodal large language models (MILMs) for 3D) reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 31) data during MILLM pretraining limits 3D repre- sentation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, framework that enhances MILM 3D Representation learning by introducing Supervision from pretrained 3D foundation models. Our approach aligns MILM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive ex- periments across multiple benchmarks and MLL.Msiinnldd visual grounding, captioning, and question answering demonstrate consistent performance gains. Project page: httts:://viiieiii
github....... o/drrs
(a) Overview 3DRS.
Reprosnntiiie Seperriioo
Seperrision
Maltzesdal Large Langangs Model
Foundation Mudel
Denonii the deaetl the secnos
(b) Performance Improvement.
(PAA At
VC.8
Figure 1: Enhancing 3D awareness of MLLMs to improve downstream performance. (a) Besides the common text supervision for MLLMs, 3DRS adopts 3D foundation models to supervise 3D- aware visual representation learning in MLLMs. (b) Combined with 3DRS, we achieve consistent performance improvement across multiple MLLMs and benchmarks.
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Scene understanding serves as a cornerstone for interpreting 3D environments, enabling a wide range of critical applications ranging from robotic navigation to augmented reality. The recent emergence of large language models (LLMs)) [43, 17, 18] has sparked innovative research aimed at endowing these models with scene comprehension capabilities. One major line of research [51, 24, 36, 19, 24, 40, 41, 9, 14, 8, 55] utilizes point cloud encoders- either independently or in combination with multi-view images-to extract 3D representations, which are subsequently projected into a language-aligned space for LLMs. However, these approaches are constrained by the scarcity of paired 3D-text datasets, which impedes effective mross-modal feature alignment.
In response to this challenge, recent ofte-of-the-eee art methods [65, 63, 16, 22, 37] have shifted towards leveraging multi-view images exclusively, drawing inspiration from the success of large-scale visual- language pretraining in multimodal LLMs (MLLMs) [29, 27, 60, I, 47, 42]. These approaches aim to transfer 2D visual understanding 3DD scene comprehension by injecting 3D priors, such 31) positional embeddings, into the models, thereby allowing MILMss to capitalize on their extensive pretrained 2D knowledge for 3D interpretation.
Despite these advancements, genuine 3D scene understanding fundamentally requires models to capture intrinsic 3D attributes and spatial structures to comprehend scenes. The absence of explicit 3D data during MLLM pretraining reveals a significant gap, which motivates our core investigation centered around the following questions: (1) How can evaluate the ability of MLLMs to learn 3D-aware representations? (2) How does the quality of 3D feature learning inffuence downstream scene understanding performance? (3) What methods can enhance 3D-aware representation learning within MILMM frameworks? While several prior works [54, 28, 15, 33] have attempted to probe the 3D awareness of 2D vision foundation models, systematic investigation into 3D-aware representation learning in MLLMS remains largely unexplored. This gap is particularly crucial given the growing adoption of MLLMs in multimodal 3D understanding tasks. Our study aims to address this overlooked area and provide new insights into 3D representation learning within the MLLM paradigm.
For the first question, we conduct comprehensive experiments to evaluate the 3D awareness on three representative MLLMs, including LLaVA-NexL -ext--idee [60], LLaVAA-ns -nne--innvioonnn [27]. and Qwen2- VL [47]. following the finetuing settings of Video-3D LLM [63]. Specifically, we assess 3D awareness via view equivariance, quantifying it by computing the feature similarity between corresponding pairs from the same 3D voxel across different views. This evaluation requires MILMS associate the same object across multiple views, thereby reflecting their capacity for 3D representation. Our analysis encompasses six datasets spanning tasks such 3D grounding [5], captioning [12], and question answering [2].
To address the second question, we systematically analyze model performance across these datasets and observe that samples with higher correspondence scores-i.e., those exhibiting stronger 3D awareness consissettt lead to improved performance. This finding demonstrates a strong positive correlation between the quality of 3D-awre representations and downstream scene understanding performance, highlighting the necessity of enhancing 3D feature learning in MLLMs.
In response to the third question and building upon our earlier findings, we first introduce a view equivalence supervision strategy for MLLMs, encouraging alignment between feature pairs corre- sponding to the same 3D location across different views (positive pairs) while discouraging similarity among unrelated pairs (negative pairs). While this approach results in some performance gains, the supervision provided by such handcrafted, single-task objectives is inherently limited for 3D learning.
In contrast, recent 3D foundation models such as VGGT [46] and FLARE [58] are pretrained end- to-end on multi-view image sequences spanning diverse set of 3D geometric aasksiincluding not only correspondence learning, but also depth estimation and camera parameter prediction. This comprehensive pretraining enables them to encode rich 3D properties within their features. Building on this, we propose a framework, 3DRS, that leverages these pretrained models by using their features as alignment targets for the visual outputs of MILMs, thereby facilitating more effective 3D-aware representation learning. Unlike previous 3D MLLM approaches, in addition to traditional text token supervision, our framework employs explicit 3D-specific supervision directly on scene visual tokens. As demonstrated in our experiments (see Fig. 1), incorporating this form of 3D supervision consistently improves performance across range of MILLMs and benchmarks. Notably, our approach incurs no additional training overhead, since the supervisory features can be pre-extracted offline.
2

---

We believe this design offers valuable new insights for applying 3D foundation models in scene understanding, The key contribution of this paper can be summarized as follows:
We conduct systematic evaluation of the 3D-awareness of MILMS using multi-view correspondence metrics, and observe strong positive correlation between 3D-aware repre- sentation quality and downstream scene understanding performance across diverse tasks, datasets, and models.
We propose 3D-aware representation supervision framework that aligns MLLM visual features with those of 3D geometry-pretent pretrained model, enabling effective 3D feature learning.
Extensive experiments demonstrate consistent performance improvements across multiple MLLMs and 3D scene understanding benchmarks, validating the effectiveness and generality of our approach.
Method
2.1
Investigating 3D-Aware Representation Learning in MILMs
2.1.1 Preliminaries
A MLLM typically consists of two main components: an image encoder Eing and text decoder T. In this work, the input to our MLLM comprises set of N multi-view images I {I1 Iy...., IN each associated with per-pixel 3D coordinates C {C1. C..... Cx), J. where C, E R fx x3 for image of size x W. The 3D coordinates for each pixel are computed from the depth map and the corresponding camera intrinsic and extrinsic parameters; detailed formulas and procedures can be found in the App. A.I.
The MLLM receives both multi-view images and language instructions as input. Internally, for each image 4. the image encoder &ing extracts visual features F; #x X'ddd where is the feature dimension. Following Video3DLLM [63], we encode the per-pixel 3D coordinates via a positional encoding function o(-) and inject this information into the image features by addition:
F F.+ ((C.).
This design allows the MLLM to inherit 2D perceptual knowledge from pretraining while equipping with explicit 3D priors.
During finetuning, the MLLM- we denote as 1i-passes visual features (E?D)N with the instruction tokens the text decoder for autoregressive text generation. After the processing of the text decoder, we refer to the final per-pixel visual embedding of pixel p in image I, from LLM as (p). The model is optimized by minimizing the standard cross-entropy loss:
T
LCE
log po (ye yet, (Ti, C, ) instruction),
where yr the 1-th output token, and pa is the probability predicted by the model given all previous tokens and the multimodal context (i.e,, images and instructions).
2.1.2 Assessing 3D Feature Learning via Multi- View Correspondence
Inspired by the crucial role of cross-view correspondences in 3D modeling [20], propose correspondence- evaluation framework. Multi-view correspondences are fundamental in 3D vision, serving as essential cues for core tasks such as ray retriangulation [20], bundle adjustment [44], and pose estimation [39]. They are also critical for downstream applications like instance recognition and retrieval [39, 52, 50]. Therefore,
adopt -iet correspondence analysis evaluate the 3D representations of MLLMs. This approach requires the model to accurately associate and align objects regions that occupy the position in 3D space across different viewpoints.
proxy to
Voxelization and correspondence pair construction. We first voxelize the 3D scene into a regular grid of voxels V {U....., UM J. For each view 1, given its per-pixel 3D coordinates C,, we assign
3

---

every pixel's 's feature f (p) to voxel according to its 3D position. Features from different views that fall into the same voxel U2 are regarded as correspondence pairs.
Feature similarity and correspondence scores. Let Pk denote all correspondence feature pairs in voxel Uh, Le., all pairs (f)) (q)) where both pixels p and g from images and 1 are assigned to en with of j. For any pair of visual features (fa, f6) from the last layer of MLLM, feature similarity measured by the cosine similarity:
f f ||fa
S(f,,
[|
For each sequence, we compute:
$ = PIIE Σ fb),
(fa.E))
where 5 and P denote the correspondence score for each sequence and all the correspondence pairs in this sequence. A higher correspondence score indicates that the model produces more consistent features across views for the same 3D spatial location, reflecting stronger 3D-aware representation learning.
2.1.3 Quality of 3D Feature vs. Downstream Task Performance.
We evaluate three representative MLLMs, LLaVA-Next- LLaVA-One Vision, and Qwen2- VL, on five diverse 3D scene understanding benchmarks, including visual grounding (Multi3DReferr, ScanRefer), captioning (Scan2Cap), and question answering (ScanQA, SQA3D). All benchmarks are based on multi-view RGBD sequences. The three MILMs respectively emphasize video understand- ing, joint image-video reasoning, and advanced arbitrary-resolution visual encoding.
To analyze the relationship between 3D feature learning and downstream task performance, we sort samples within each dataset by their correspondence scores and divide them into four quartiles (Q1-Q4, lowest highest). From Fig. 2, we observe a clear trend: as the correspondence score increases, the model's performance on the downstream task consistently improves. This strong positive correlation demonstrates the critical importance of 3D-aware representation quality for effective scene understanding in MLLMs.
Mint-MMidee Scnemler
Manndeeer
Matti/reter
Scoereter
Sear2cap
Seate
04
67
91
01
42
01
94
Scanseee
Scarmhrr
Seas2cap
Scurcn
Spa36
Qween2-VL IOU 30
100, 35
109 $3
101, 35, AB
GDtr
GDT
AI
SW
Figure 2: Performance across correspondence score quartiles. Model performance acToss corre- spondence score quartiles (Q1-Q4, lowest to highest) for each dataset. Samples were divided into quartiles by their correspondence scores. A clear trend is observed: model accuracy improves the correspondence score increases.
These findings highlight the need for strategies to further enhance 3D-aware representation learning in MLLMs, which we address in the next section.

---

2.2 Enhancing 3D-Aware Representation Learning in MLLMs
2.2.1
Correspondence-basee 3D Supervision Loss
Inspired by our correspondence based evaluation, a straightforward approach to directly supervise the MLLM's visual features to be consistent for matched 3D locations across views and dissimilar for mismatched locations. We let P denote all positive feature pairs in voxel Uk. i.e., all pairs (5()), where pixels p and y from images I; and are assigned to ep with i + j. Similarly, Pe denotes negative pairs between Uk and any other voxel (1 + k). We supervise these objectives directly using simple loss function by maximizing the feature similarity in p+ and minimizing that in P
Stom
Σ
[1-
S(fa, fo]],
P-
(Putte)EPt
S(far fo).
P
(fo.P))EP-
The overall correspondence loss is weighted sum:
Leoor
L
feoor.
By directly supervising positive pairs to be similar and negative pairs to be dissimilar, this corre- spondence loss encourages the model learn multi-view 3D correspondences, thus enhancing the 3D-awareness of the learned representations. As will be shown in the experiments Sec. 3, supple- menting the standard cross-entropy objective with Loorr leads to improvements downstream task performance. However, as this loss primarily targets view equivariance, the range of 3D properties captured remains limited, motivating the need for richer supervision.
2.2.2 3D Foundation Model-Guided Feature Distillation
To overcome the inherent limitations of single-task supervision, further introduce knowledge distillation framework, 3DRS, that leverages the rich 3D priors embedded in 3D foundation models, wide array of 3D geometric tasks
e.g, FLARE and VGGT. These models are pretrained including correspondence learning, camera parameter estimation, multi-view depth prediction, and dense point cloud reconstruction-wiie enables them extract robust and highly 3D-aware visual features from multi-view image sequences.
Distillation target preparation. As shown in Fig. 3a, given a of multi-view images I for scene, we first input them into a pretrained 3D foundation model g. which outputs collection of per-pixel visual features (f.D(p) (p)} for each image and pixel p. Since the spatial resolution of these features may differ from those of the MLLM outputs {f, (p)), we apply 2D average pooling to the 3D foundation model's output match the MILMM feature map size.
Feature alignment and loss. To align the MLLM's per-pixel visual features with the 3D foun- dation model, we first process each (((p) with a two-layer MLP (denoted as MLP MLPatigu) to ensure compatibility in feature dimension:
{p)) MLP MLPalign (p)).
We then employ distillation loss based euclidean similarity to maximize the alignment between the MILMM features (p) and the corresponding 3D foundation model features f,3D (p):
N
Calign
Σ Σ s
NHW
1
PEL
where .(,)) denotes euclidean similarity, and the sum is calculated over pixels and views in the batch.
5

---

Text Tokens
Instruction
Cadt-andtian LLat aln wOIORS
Been2.t Owers waoss
MILMM
Visuaal Tokenns
Correpondion
Similarity
Visual Takens fo
Multi-view Images
Scartlater
Scanzcap
ScanqA
SQ43D
Details 3DRS.
Comparison of correspondence learning.
Figure 3: (a) 3DRS uses a 3D foundation model to supervise the visual representation of the MILMM.
(b) 3DRS effectively improves the correspondence learning for MLLMs.
Overall training objective. The final training objective for the MLLM combines the standard cross-entropy loss for text generation and the 3D foundation model distillation loss:
Liotal = LCe + Calign.
This approach enables the MILMM to inherit comprehensive 3D knowledge from powerful geometry- pretrained models, facilitating the learning of richer and more robust 3D-awa representations. Importantly, the distillation targets from the 3D foundation model can be precomputed offline, introducing no additional overhead during MLLM fine-tuning.
As illustrated in Fig. 3b, we compare the correspondence scores before and after applying our 3DRS. where VGGT serves as the foundation model. The results consistently demonstrate that introducing 3DRS leads to substantial improvements in correspondence learning ability across all evaluated MLLMs and benchmarks. This proves the effectiveness of leveraging pretrained 3D foundation model as a teacher model for enhancing 3D-aware representation learning in MI.LMs. More comprehensive experimental results and analyses are detailed in Sec. 3.
Experiments
3.1 Datasets and Evaluation Metrics
Datasets. We evaluate our approach on six benchmarks that collectively span key challenges in 3D scene understanding. ScanRefer [5] focuses localizing objects using free-form language, while Multi3DRefer [59] generalizes this to queries referencing zero, one, or multiple objects, better reflecting real-world ambiguity. Scan2Cap [12] addresses dense captioning by pairing detected objects in 3D scans with natural language descriptions. For question answering, ScanQA [2] tasks models with answering open-ended questions grounded in 3D geometry and semanties, and SQA3D [32] goes further by requiring situated reasoning: agents must interpret their position and context to answer complex queries. All these datasets are sourced from the richly annotated ScanNet [13] corpus, and we follow standard validation and test splits as established in prior work [24, 65, 9, 63]. Besides, VSI-Bench [53] used to evaluate the performance on visual-based spatial understanding tasks, which are composed of numerical and multiple-chote choice questions. The statistics of training sets are detailed in the App. A.2.
Evaluation metrics. For ScanRefer, we report accuracy at loU thresholds of 0.25 and 0.5 (Acc@0.25, Acc@0.5). Multi3DRefer uses Fl scores at matching IoU thresholds. Scan2Cap is evaluated by CIDEr and BLEU-4 scores at 0.5 IoU (C@0.5, B-4@0.5). ScanQA is assessed by CIDEr and exact match accuracy (C, EM), while SQA3D uses match accuracy as the metric.
3.2 Implementation Details
Our experiments leverage several MILMs, including VA-Neeee Video 7B [60], V.LVVA-OneVis Vision 7B [27], and Qwen2-VL LL 7B [47]. In addition these baselines, systematically compare the

---

Table 1: Performance comparison 3D scene understanding benchmarks. Specialists are single-task methods, while generalists target multiple tasks. Bold denotes best performance.
ScanRefer
Mulr3DRefer
Scan2Cap
ScanQA
SQA3D
Method
Acc@0.25
Acc@0.5
F1
F1@0.5
C@0.5
B-400.5
Specialists
ScanRefer [5]
24.3
MVT [26]
33.3
3DVG-Trans [61]
45.9
34.5
ViL3DRel
47.9
37.7
M3DRef-CLIP [59]
51.9
44.7
42.8
38.4
Scan2Cap [12]
35.2
22.4
ScanQA [2]
64.9
21.1
47.2
3D-VisTA [66]
45.8
22.4
Generalists
3D-LLM(Plamingo) [22]
21.2
59.2
20.4
3D-LLMMBLIP2-AAang [22]
30.3
69.4
20.5
Chat-3D [48]
53.2
Chat-3D v2 [24]
42.5
38.4
45.1
41.6
63,9
31.8
87.6
54.7
LL3DA [8]
62.9
36.0
76.8
SceneLLM [16]
80.0
27.2
53.6
LEO [25]
72.4
38.2
101.4
21.5
Grounded 3D-LLM
47.9
44.1
45.2
40.6
70.6
35.5
72.7
PQ3D [67]
57.0
51.2
50.1
80.3
36.0
47.1
ChatScene [24]
55,5
50.2
57.1
52.4
77.1
36.3
87.7
21.6
54.6
LLaVA-3D [65]
54.1
42.4
79.2
41.1
91.7
27.0
55.6
Inst3D-LLMM [55]
57.8
51.6
58.3
53.5
79.7
38.3
88.6
24.6
3D-LLaVA [14]
40.6
36.9
92.6
Video-3D LLM [63]
58.1
51.7
58.0
52.7
83.8
41.3
102.1
30.1
58.6
3DRS
62.9
56.1
60.4
54.9
86.1
41.6
104.8
30.3
60.6
effect of using 2D versus 3D foundation models as teachers for MLLM finetuning, The 2D teacher models include DINOv2 [34], MAE [21], and SigLIP [45], while the 3D teacher models comprise FLARE [58] and VGGT [46]. Unless stated otherwise, we use LLaVA-Next Video as the MLLM and VGGT as the representation teacher for our experiments.
For both training and inference, we uniformly sample 32 frames per scan to construct multi-view image sets. For evaluating the correspondence score, All models are optimized using Adam, with a batch size of 16 and a warm-up ratio of 0.03. The learning rates set maximum of 1 10-5 for the language model and x 10-6 for the visual backbone during the warm-up period. During training for visual grounding and dense captioning, ground truth object regions are used candidates. whereas during inference, follow the procedure of [24, 25, 63] and employ Mask3D [38] to generate object proposals. For LLaVA-Next Video and LLLVVA-One Vision, finetune all model parameters. For Qwen2-VL, due to GPU memory constraints, we finetune only the projector and the LLM components. We use H100 NVIDIA GPUs for all experiments.
the voxel size of 0.1 for voxelization.
3.3 Comparison with State-of-the-Art Models
Table presents a comprehensive comparison between our approach, task-specific specialist models- which require fine-tuning on individual datasets-and 3D generalist models that capable of handling multiple tasks. Compared to specialist models, our approach achieves substantial per- formance improvements. This demonstrates the significant benefits brought by joint training and the LLM-based architecture, which contribute to superior generalization and feature integration compared methods tailored for specific tasks. Furthermore, our method consistently outperforms 3D generalist approaches that utilize point clouds as input, such as LL3DA, Chat-3D, Grounded 3D-LLM, and 3D-LLaVA. Compared to Inst3D-LLM which fuses multi-view images and point clouds our approach also shows clear advantages, highlighting the strength of leveraging MLLMs as the backbone. Additionally, our method achieves considerable improvements over other MILM- based methods, including LLaVA-31 and Video-3D LLM. These results collectively indicate that enhancing the 3D-awareness of MI.LMs highly effective for 3D scene understanding tasks, further validating the effectiveness of proposed strategy.

---

Table 2: Performance comparison VSI-Bench.
Method
Avg.
Obj. Count
Obj.
Rel.
Appr. Order
4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5 Pro [42]
45.4
56.2
30.9
64.1
43.6
$1.3
46.3
36.0
34.6
LongVA-7B (57]
29.2
22.2
IntemVLL2-40B [11]
37.0
41.3
26.2
48.2
27.5
47.6
32.7
27.8
44.7
LLaVA-Video-7B [60)
35.6
48.5
14.0
47.8
24.2
43.5
42.4
34.0
30.6
LLaVA-Video-728 [60]
40.9
48.9
22.8
57.4
35.3
42.4
36.7
35.0
48.6
LLaVA-OneVision-72 [27]
32.4
47.7
20.7
47.4
42.5
35.2
29.4
24.4
LLaVA MOWVVA--VVVVV OnsViniin [27]
40.2
43.5
23.9
57.6
37.5
42.5
39.9
32.5
44.6
ADRS
45.9
68.7
34,8
53.6
56.6
40.9
43.2
30.4
39.2
Table 2 compares our method with leading proprietary APIs and open-source models on VSI- Bench [53], covering tasks such as object counting, spatial distance estimation, size reasoning, and sequence understanding. 3DRS achieves the best open-source results most peetics--iclusee object count, absolute distance, room size, and appearance oder oder-and and remains competitive with proprietary models. These results demonstrate the strong spatial reasoning, generalization, and comprehensive scene understanding capabilities our approach across diverse 3D vision tasks.
Table 3: Performance comparison of 3DRS when using with different MILMs.
ScanRefer
Multi3DRef
Scan2Cap
ScanQA
SQA3D
Method
Acc@0.25
Aco@0.5
@0.25
F1@0.
B-400.5
C@0.5
EM
EM
LLaVA Next- Video [60]
58.1
$1.7
58.0
52.7
41.3
83.8
102.1
30.1
58.6
L. V.aVV-NNxt- Video w/ 3DRS
62.9
56.1
60.4
54.9
41.6
86.1
104.8
30.3
60.6
LLaVA-One Vision [27]
57.3
51.0
57.1
51.9
40.4
81.7
101.7
29.0
57.4
LLa VLaVA-OneVision w/ 3DRS
6188
55.0
60.5
55.0
41.2
83.1
104.0
29.5
59.4
Qwen2-VL [47]
57.0
50.8
56.2
51.4
39.5
79.4
97.5
28.7
58.6
Qwen2-VL. w/ 3DRS
60.1
53.5
58.5
54.5
40.9
81.6
99.2
28.9
60.0
3.4 Diagnostic Study
Effectiveness with different MILMs. Table demonstrates that integrating 3DRS with different MLLMs LLalssLLaVie LLLV--LLL----tt------ Video, LLaVA-OneViis Vision, and Qwen2-VL VL- "IL-consittenlly boosts perfor- mance across all evaluated benchmarks. For example, LLaVA-Next CIVA-Next-Viiie Video w/ 3DRS improves ScanRefer Acc@0.25 from 58.1 to 62.9, and Multi3DRef F1 @0.25 from 58.0 to 60.4. Similar gains observed for LLaVA-One Vision and Qwen2-VL, where 3DRS brings improvements on every dataset and metric. These results highlight the general applicability of our approach and its effectiveness in enhancing 3D
understanding for various MILMSs.
Comparison between 2D and 3D foundation models. Table compares the performance of using 2D and 3D foundation models representation supervisors. It is clear that 3D foundation models (FLARE and VGGT) outperform all 2D foundation models (MAE, Siglip2, Dinov2) across almost every metric. This performance gap be attributed to the inherent difference in the prior knowledge captured by 2D and 3D foundation models. 3D models are pre-trained on large-scale 3D data and thus better capture geometric structure, spatial relationships. and depth information, which are critical for 3D scene understanding tasks. In contrast, 2D foundation models, trained on images, lack explicit 3D spatial priors and struggle provide effective supervision for learning 3D-aware representations. This highlights the importance of 3D-specific foundation models for achieving strong results in downstream 3D tasks.
Comparison of supervision signal. Table shows that using correspondence loss for supervision brings improvements over the baseline, demonstrating the effectiveness of encouraging the model to learn multi-view correspondences. However, when 3D foundation model supervision applied, the performance increases further across all metrics. This indicates that 3D foundation models, with their rich 3D prior knowledge learned during pre-training, can more effectively enhance the 3D representation ability of MILMSs and yield greater gains for 3D understanding tasks.
Comparison supervision different layers. Table examines the effect of applying 3D foundation model supervision at different layers of the network. The results reveal that supervision

---

at deeper layers, especially the last layer. leads to the highest performance. This is likely because deeper layers are closer to the output and thus have a more direct impact on the final predictions. Additionally, these layers possess more parameters and a greater capacity to fit 3D features, which results in larger improvements on downstream tasks.
Table 4: Ablation study using different 2D/3D foundation models the representation supervisor. Bold denotes the best in each group.
ScanRefer
Multi3DRel
Scan2Cap
ScanQA
SQA3D
Representation Supervisor
Acc
Acc@0.5
F1
F1
C@0.5
B-4@0.5
EM
Buseline
58.1
51.7
58.0
52.7
83.8
41.3
102.1
30.1
58.6
2D Foundation Models
Siglip2 [45]
58.2
52.9
59.7
53.1
81.7
40.2
100.2
29.1
59.4
MAE [21]
59.1
53.7
60.0
53.7
82.8
40.4
102.5
29.5
59.2
Dinov2 [34]
59.8
53.3
58.5
53.5
80.3
39.3
103.5
60.1
3D Foundation Models
FLARE [58]
62.1
55.7
59.8
54.8
86.6
42.5
104.4
30.1
60.1
VGGT [46]
62.9
56.1
60.4
54.9
86.1
41.6
104.8
30.3
60.6
4 Related Work
4.1 Scene Understanding with Large Language Models
LLMs, owing their strong reasoning capabilities and remarkable success in 2D image understanding, have been widely applied to scene understanding tasks. Early works such as PointL [51], Point- Bind [19], GPT4Point [36], MiniGPT-3D [40], and Chat-3D [48] leverage the alignment between point cloud and text features to facilitate 3D scene comprehension. Building on this foundation, methods like Grounded 3D-LLM [9], LL3DA [8], 3D-LLa [14], and Inst3D-LLM [55] design more advanced cross-modal modules to better fuse multi-modal features, thereby enhancing scene representations. Furthermore, Chat-Scene [24] and Inst3D-LLM [55] exploit the complementary nature of 2D and 3D features to further boost scene understanding.
Some recent approaches, such as 3D-LLM [22] and Scene LLM [16], employ multi-view inputs and introduce 3D priors to transform 2D representations into 3D-aware format. Thanks to pre-training on large-scale image-text datasets, methods based MLLMs are gaining increasing popularity in the field of scene understanding. For instance, LLaVA-3D [65] takes multi-view images as input and utilizes voxelization to reduce the dimensionality of representations, thus lowering computational costs while leveraging the strengths of MLLMs. However, many MLLMS require specially structured inputs, making them incompatible with certain approaches. Video 3D-LLM [63] and GPT4Scene [37] more naturally inherit the MILM pipeline by introducing 3D priors-succ as positional embeddings or spatial markers- enablling the model better comprehend 3D scene content.
Our work follows this line of MLLM-bbssed scene understanding, aiming to probe the 3D-awareness of MLLMs and analyze their relationships with downstream tasks. In particular, demonstrate that introducing guidance from 3D foundation models can effectively enhance the representational capability of MLLMs for 3D scene understanding.
4.2 3D-A Awareness in Vision Models
Several studies have investigated 3D-awareness; however, most prior work has focused on pure vision models rather than MLLMs, and primarily leveraged 2D foundation models instead of 3D ones. For example, FiT3D [56], Probe3D [15], and Lexicon3D [33] empirically analyze the awaaenesssss of visual foundation models. CUA-03D [28] proposes integrating multiple 2D foundation models for 3D seene understanding, while Yang et [l.[54] evaluates and enhances the 3D-awareness of ViT-based models various downstream tasks. Some previous 3D detection works [10, 49, 3] have focused on improving 3D representations for pure vision models or CLIP-style vision-language models (VL.ss,, primarily aiming to enhance geometric understanding and spatial localization within unimodal or early multimodal frameworks. In addition, several studies on scene understanding [30, 23, 35] have

---

Table 5: Comparison of different supervision strate- gies.
Table 6: 3D foundation model supervision different layers.
Method
Multi3Reel
ScunRefer
Layer
MultiDRRef
ScanRefer
Acc@0.25
Acc@0.5
F160.25
F1@0.5
Auc85.25
Acc
F160025
(10.5
Baseline
58.0
52.7
58.1
51.7
Last Layer
3rd Last Layer
61.7
62.9
56.1
54.9
60.4
59.7
54.3
54,9
Correpanneecee Lows
60.1
53.3
59.1
53.7
5th Last Layer
61.4
54.8
59.3
54.0
Supervision
62.9
56.1
60.4
54,9
10th Layer
investigated various strategies for distilling 3D representations, such as transferring knowledge from 3D models to 2D networks or promoting cross-moda alignment. However, these efforts have not addressed the unique challenges presented by MILMs, which require a more holistic integration of visual, linguistic, and spatial information. As result, the potential of distilling 3D awareness into MLLMMs for richer and more generalizable scene understanding remains largely unexplored.
In contrast, our work specifically targets the 3D-awareness of MLLMs. Rather than enhancing 3D feature learning via 2D foundation models, we introduce 3D foundation models as supervisors to directly guide and improve the 3D representation capabilities of MLLMs.
Conclusion
In this paper, we present a comprehensive study the 3D representation capabilities multi-modal large language models (MLLMs) in the context of scene understanding. While most existing research has centered on leveraging 2D foundation models to improve visual reasoning in MLLMs, the role and utility of 3D foundation models in this setting remain largely unexplored. To bridge this gap. we propose 3DRS, a novel framework that introduces direct 3D-aware supervision to MLLMs by leveraging pretrained 3D foundation models as teachers. Our approach enables MLLMs to acquire richer geometric and spatial representations, facilitating more accurate and robust understanding of complex 3D scenes. Through extensive experiments diverse 3D scene understanding benchmarks, we demonstrate that 3DRS consistently improves performance across a variety of tasks, such as object localization, spatial reasoning, and 3D question answering. These results highlight the unique advantages and significant potential of integrating 3D foundation models for advancing multimodal scene understanding.
Limitation
While our paper aims to enhance the 3D-aw awaaaeesess of MLLMs, the relatively limited size of the dataset used for finetuning especially when compared to that used during the MLLM pretraining stage-may restrict the full realization of our approach's 's potential. Consequently, the improvements demonstrated this work may only represent an initial step toward more robust 3D understanding. promising direction for future research is to incorporate 3D-awareness learning into the pretraining stage of MLLMs, which could lead to fundamentally stronger models with deeper 3D comprehension. Besides, due to the distillation-bar based nature of our approach, the performance of our method is upper- bounded by the quality of the teacher 3D foundation model. Any limitations or failure modes of the teacher-suae such as inaccurate correspondence, erroneous depth estimation, or incomplete geometric representations can be propagated to the student MLLM and may potentially mislead it during training. While our experiments demonstrate consistent improvements possible that errors or biases the teacher's ' predictions can negatively impact the downstream 3D reasoning abilities of the student model. We believe that, as 3D foundation models continue rapidly advance, this limitation becomes less pronounced over time.
strong baselines, it is
10

---

Acknowledgments
This work is supported by Hong Kong Research Grant Council General Research Fund (Grant No. 17213825). We would like to thank Weining Ren for the valuable and insightful discussions.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilee Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08777 2023.
[2] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, 2022. License: Creative Commons Attrbution-NonCommmerrr NonCommercial ShareAlike 3.0 Unported License.
[3] Geonho Bang, Kwangjin Choi, Jisong Kim, Dongsuk Kum, and Jun Won Choi. Radardistill: Boosting radar-based object detection performance knowledge distillation from lidar features. In CVPR, 2024.
[4] Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djeg: unified framework for joint dense captioning and visual grounding on 3d point clouds. In CVPR, 2022.
[5] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020. License: Creative Commons Attribution-Nond Conaurooonn ShareAlike 3.0 Unported License.
[6] Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X. Chang. D net: A unified speaker-listener architecture for 3d dense captioning and visual grounding, In ECCV, 2022.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. NeurIPS, 2022.
[8] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu. Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LL3DA: visual interactive instruction tuning for omni-3d understanding, reasoning, and planning. In CVPR, 2024.
[9] Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405. (tf.:2405..03700 2024.
[10] Zehui Chen, Zhenyu Li. Shiquan Zhang, Liangji Fang. Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint 1xXiv:2211.003880 2022.
[11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguiiin tasks. In CVPR, 2024.
[12] Zhenyu Chen, Ali Gholami, Matthias Nicßner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In CVPR, 2021. License: Creative Commons ShareAlike 3.0 Unported License.
[13] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions indoor scenes. In CVPR, 2017. License: ScanNet Terms of Use.
[14] Jiajun Deng. Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, and Ian Reid. 3d-llava: Towards generalist 3d Imms with omni superpoint transformer. In CVPR, 2025.
[15] Mohamed El Banani, Amit Raj. Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas. Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. CVPR, 2024.
[16] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arxiv preprint arXiv:2403. 11401, 2024.
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arxiv:2407.21788, 2024. 2024.
11

---

[18] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[19] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang. Xianzheng Ma. Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi- modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2.30.0000
2023.
[20] Richard Hartley. Multiple view geometry in computer vision, volume 665. Cambridge university press, 2003.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[22] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In NewrIPS, 2023.
[23] Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias NieBner. Pri3d: Can 3d priors help 2d representation learning? In ICCV, 2021.
[24] Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. CoRR, 2023.
[25] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICMI, 2024.
[26] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In CVPR, 2022.
[27] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326. 2024.
[28] Jinlong Li, Cristiano Saltori, Fabio Poiesi, and Nicu Sebe. Cross-modal and uncertainty agglomera- tion for open- vocc-oocabulary 3d scene understanding. In CVPR, 2025.
[29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[30] Zhengzhe Liu, Xiaojuan Qi, and Chi- Wing Fu, 3d-to-2d distillation for indoor scene parsing. In CVPR, 2021.
[31] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx miim: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409. /2961, 2024.
[32] Xiaojian Ma, Silong Yong. Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. SQA3D: situated question answering in 3d scenes. In ICLR, 2023. License: CC-BY-4.0.
[33] Yunze Man, Shuhong Zheng, Zhipeng Bao. Martial Hebert, Liangyan Gui, and Yu-Xiong Wang. Lexicon3d: Probing visual foundation models for complex 3d scene understanding. In NeurIPS, 2024.
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learingg robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023.
[35] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocahularies. In CVPR, 2023.
[36] Zhangyang Qi. Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. In CVPR, 2024.
[37] Zhangyang Qi, Zhixiong Zhang, Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.0144 01248,,, 2025.
[38] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In ICRA, 2023.
[39] Stefan Stojanov, Anh Thai, Zixuan Huang, and James M Rehg. Learning dense object descriptors from multiple views for shot category generalization. In NewrIPS, 2022.
12

---

[40] Yuan Tang, Xu Han. Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiendy aligning 3d point clouds with large language models using 2d priors. In ACM Mulii, 2024.
[41] Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Jinfeng Xu, Yixue Hao, Long Hu, and Min Chen. More text, less point: Towards 3d data-efficient point-language understanding. In AAAI, 2025.
[42] Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint X1.::332../1805 2023.
[43] Qwen Team et al. Qwen2 technical report. arXiv preprint 4XXi:2247711660 2(8), 2024.
[44] Bill Triggs, Philip Mclaaun Richard Hartley, and Andrew Fitzgihbon. Bundle adjustment- modern synthesis. In International workshop on vision algorithms. Springer, 1999.
[45] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual languaea encoders with improved semantic understanding, localization, and dense features. arXiv preprint 250. 2025.
[46] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025.
[47] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing dision-languen model's perception of the world at any resolution. arXiv preprint awii::2409.00.....a /211911999111,,,, 2024.
[48] Zehan Wang, Haifeng Huang. Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes, arXiv preprint arXiv: 2023.
[49] Zeyu Wang, Dingwen Li, Chenxu Luo, Cihang Xie, and Xiaodong Yang. Distillbev: Boosting multi-camera
3d object detection with cross-modal knowledge distillation. In ICCV, 2023.
[50] Hao Wu, Ruochong Li, Hao Wang, and Hui Xiong. Com3d: Leveraging cross-view correspondence and cross-modal mining for 3d retrieval. In JCME, 2024.
[51] Runsen Xu, Xiaolong Wang, Tai Wang. Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In ECCV, 2024.
[52] Yong Xu, Chaoda Zheng, Ruotao Xu, Yuhui Quan, and Haibin Ling. Multi-view 3d shape recognition via correspondence-an aware deep learning. IEEE Transactions on Image Processing, 2021.
[53] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xic. Thinking in space: How multimodal large language models see, remember, and recall spaces. In CVPR, 2025.
[54] Yang You, Yixin Li. Congyue Deng. Yue Wang, and Leonidas Guibas. Multiview equivariance improves 3d correspondence understanding with minimal feature finetuning, arXiv preprint arXiv:2411.19458. 2024.
[55] Hanxun Yu. Wentong Li. Song Wang. Junbo Chen, and Jianke Zhu. Inst3d- Imm: Instance-aware 3d scene understanding with multi-modal instruction tuning. In CVPR, 2025.
[56] Yuanwen Yue, Anurag Das, Francis Engelmann. Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine-tuning. In ECCV, 2024.
[57] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang. Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[58] Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein. Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views. CVPR, 2025.
[59] Yiming Zhang, ZeMing Gong, and Angel X. Chang. Multi3drefer: Grounding text description to multiple 3d objects. In ICCV, 2023. License; MIT.
[60] Yuanhan Zhang, Jinming Wu. Wei Li. Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02113, 2024.
[61] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg- tranforrnsorrmr:: Relation modeling for visual grounding on point clouds. In ICCV, 2021.
13

---

[62] Duo Zheng. Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing milmss with 3d vision geometry priors. In NewrlPS, 2025.
[63] Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Um: Learning position-awand aware video representation for 3d scene understanding. In CVPR. 2025.
[64] Duo Zheng, Shijia Huang, Lin Zhao. Yiwu Zhong, and Liwei Wang. Towards learning generalist model for embodied navigation. In CVPR, 2024.
[65] Chenming 7hu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway empowering Imms with 3d-awareness. arXiv preprint arXiv:2409. 18122 2024.
[66] Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng. Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. In ICCV, 2023.
[67] Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng. Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. In ECCV, 2024.
14

---

A Technical Appendices and Supplementary Material
A.1 World Coordinate Computation
Given a set of N images I {41, .2.... |, each image 4 is paired with its depth map D; e ## camera intrinsic matrix K2 _2xx and camera-to-world extrinsic matrix E _2x4 For pixel at (u, v) in image 1,, the corresponding 3D coordinate in the global coordinate system, denoted as C (u,, 2) E RR, computed as:
v)) = T:
Di(u,v) K,
(I)
Repeating this process for all pixels yields the per-pixel 3D coordinate map C, = for each image 4. The complete set coordinate maps is denoted as C {C1, Co...., Cv}.
A.2 Datsests Training
For model fine-tuning, we utilize collection of well-established 3D vision-language datasets. Specifically, we follow the model finetuning settings of Video-3D LLM [63] by using the validation splits of ScanRefer, Multi3DRefer, Scan2Cap, and ScanQA, well as the test split of SQA3D. Across these datasets, the number of data samples varies significantly: ScanRefer and Scan2Cap each provide 36,665 samples, while Multi3DRefer offers 43,838 entries. ScanQA contains 26.515 instances, and SQA3D is the largest with 79,445 samples. Most datasets are derived from 562 unique scans, except SQA3D, which includes 518 scans. We further report the average lengths of questions and answers for each dataset. For example, question lengths range from approximately 13 to 38 words, with Scan2Cap and ScanQA also providing answer texts, averaging 17.9 and 2.4 words in length, respectively. In SQA3D, the average question and answer lengths are 37.8 and 1.1 words. For the evaluation VSI-Bench, use the pre- pre-training data from VG-LLM [62].
A.3 Detailed Comparison
In this section, we provide a detailed comparison with other methods using all metrics across benchmarks.
Scanrefer. Tab. shows that our method 3DRS achieves the best overall performance on the ScanRefer validation set, especially in the challenging "Multiple" scenario where precise target
Table 7: Performance comparison on the validation set of ScanRefer [5]. "Unique" and "Multiple" depends whether there are other objects of the same class the target object.
Unique
Multiple
Overall
Method
Acc Acc@0.25
Acc
Acc Acc@0.25
Aco
Acc@0..5
Acc@0.5
SeanRefer [5]
53.5
27.4
MVT [26]
77.7
66.4
31.9
25.3
40.8
33.3
[61]
81.9
60.6
39.3
28.4
47.6
34.7
ViL.DRel [7]
81.6
68.6
40.3
30.7
47.9
37.7
3DJCG [4]
83.5
64.3
41.4
30.8
49.6
37.3
D3Net [6]
72.0
30.1
37.9
M3DRef-CLIP [59]
85.3
77.2
43.8
36.8
51.9
44.7
3D-VisTA [66]
81.6
75.1
43.7
39.1
50.6
45.8
3D-LLLM (Flamingo) [22]
3D-LLM (BLIP2-(lan5)) [22]
Grounded 3D-LLM [9]
PQ3D [67]
86.7
78.3
51.5
46.2
57.0
51.2
ChatScene [24]
89.6
82.5
47.8
42.9
55.5
50.2
LLLVA-3D [65]
54.1
42.2
Video 3D-LLM [63]
88.0
78.3
50.9
45.3
58.1
51.7
3DRS (Ours)
87.4
77.9
57.0
50.8
62.9
56.1
15

---

Table Performance comparison on the validation set of Multi3DRefer [59]. ZT: zero-target, ST: single-target MT. multi-target, D: distractor.
ZT wlo
7.'w/
ST win D
ST w/
MT
ALI
Method
F1
F1
F1.0.22
F1@05
F1
F160.5
F1
F1600.5
F1.05
Fl G1.5
M3DRef-CLIP [59]
81.8
39.4
33,5
47.8
34.6
30.6
43.6
37.9
42.8
38.4
D3Net
32.5
38.6
23.3
12.2
3DJCG
94.1
66.9
26.0
16.7
26.2
26.6
Grounded DDoddDD-LLLMM 3D-LLM
45.2
40.6
PQ3D [67]
ChatScene [24]
90.3
62.6
82.9
75.9
49.1
44.5
45.7
41.1
57.1
52.4
Video 3D-LLM [63]
94.7
78.5
82.6
73.4
52.1
47.2
40.8
35.7
58.0
52.7
(Ours)
79.6
Table 9: Performance comparison the validation of ScanQA [2]. EM indicates exact match accuracy, and B-1, B-2, B-3, B-4 denote BLEU-1, -2, -3, -4, respectively.
Method
EM
B-1
B-2
B-3
B-4
ROUGE-L
METEOR
CIDEr
ScanQA [2]
21.05
30.24
20.40
15.11
10.08
33.33
13.14
64.86
3D- VisTA [66]
22.40
10.40
35.70
13.90
69.60
Oryx-34B [31]
38.00
24.60
37.30
15.00
72.30
LLaVA-Videes [60]
39.71
26.57
9.33
3.09
44.62
17.72
88.70
3D-LLM (Flamiing)) [22]
20.40
30.30
17.80
12.00
7.20
32.30
12.20
59.20
3D-LLM (BLIP2-flant5) [22]
20.50
39.30
25.20
18.40
12.00
35.70
14,50
69.40
Chat-3D [48]
29.10
6.40
28.50
11.90
53.20
NaviLLM [64]
23.00
12.50
38.40
15,40
75.90
LL3DA [8]
13.53
37.31
15.88
76.79
Scene-LLM [16]
27.20
43.60
26.80
19.10
12.00
40,00
16.60
80.00
LEO [25]
11.50
39.30
16.20
80.00
Grounded 3D-LLM [9]
13.40
72.70
ChatScenc [24]
21.62
43.20
29.06
20.57
14.31
41.56
18.00
87.70
LLaVA-3D [65]
27.00
14.50
50.10
20.70
91.70
Video 3D-LLM [63]
30.10
47.05
31.70
22.83
16.17
49.02
19.84
102.06
3DRS (Ours)
30.30
48.37
32.67
23.79
17.22
49.82
20.47
104.78
discrimination is required. These results demonstrate that 3DRS effectively leverages multi-view images for robust spatial understanding and accurate object localization.
Multi3DRefer. In Tab. 8, 3DRS achieves the best overall results on the Multi3DRefer validation set, with top F1 scores in both standard and challenging scenarios. Our method consistently outperforms previous approaches, especially in the difficult zero-target and distractor settings, demonstrating superior robustness and spatial understanding.
ScanQA. Tab. 9, 3DRS achieves the best performance on the ScanQA validation set across almost all metrics, including EM, BLEU scores, METEOR, and CIDEr, demonstrating its strong effectiveness for 3D question answering.
SQA3D. In Tab. 10, 3DRS achieves the highest scores on the SQA3D test set, outperforming all previous approaches on almost every question type as well as the overall average, which demonstrates its superior capability for 3D question answering across diverse scenarios.
Scan2cap. In Tab. 3DRS achieves the best performance the Scan2Cap validation in terms of CIDEr (C), and remains highly competitive on other metrics such as BLEU-4, METEOR, and ROUGE-L, demonstrating strong overall effectiveness for 3D captioning.
A.4 Ablation Study
Table 12 shows the effect of applying supervision to different numbers of network layers across mul- tiple 3D scene understanding tasks, including object localization, captioning, and question answering. Supervising only the last layer consistently achieves the best performance on all benchmarks, As more intermediate layers are added for supervision, the results degrade. This suggests that multi-layer supervision may over-constrain geometric features and weaken semantic representations, ultimately hindering downstream performance. Future work may explore more advanced strategies balance geometric and semantic cues.
16

---

Table 11:
Performance comparison on the
Table 10: Performance comparison on the test set of
validation set of Scan2Cap [12].
SQA3D [32].
@0..5
Method
Mathod
set
What
Ix
How
Can
Which
Others
Scan2Cap [12]
39.08
23.32
21.97
44,48
3DJCG [4]
49.48
31.03
24.22
50.80
D3Net [6]
62.64
35.68
25.72
53.90
Vidcoo [0]
42.70
56.30
47.50
$3.30
50,10
47.20
48.50
3D- TA [66]
66,90
34.00
27.10
54.30
Scone- LLM [16]
40.90
69.10
45.00
70.80
47.20
$2,30
54.20
LI.3DA [8]
65.19
36.79
25.97
55.06
LEO [25]
50.00
ChatSeone [24]]
45.40
67.00
52,00
69.50
49.90
55.00
54.60
LEO [25]
68.40
36.90
$7.80
LLaVA-3D [65]
ChatScene [24]
77.19
36.34
28.01
58.12
(63)
72.40
LLaVA-3D [65]
79.21
41.12
30.21
63.41
3DRS (Ours)
54.40
75.20
57.00
72.20
49.90
59.00
60.60
Video 3D-LLM [63]
83.77
42.43
28.87
62.34
3DRS (Ours)
86.11
41.63
28.97
62.29
Table 12: Distillation on multiple layers.
Supervision
SeanRefer
MRlee
Scan2Cap
ScanQA
SQA3D
Acc@0.25
A.c@0..5 5.5
Fl 080.5
F1
C@0.5
B-480.5
C
EM
EM
Last layer
62.9
56.1
60.4
54,9
41.6
86.1
104.8
30.3
60.6
Last layer + last 3rd layer
61.5
54.8
60.1
$4.9
41.4
101.4
29.2
60.5
Last layer last 3ed + last 5th layer
60.5
53.9
59.0
53.8
40.0
81.1
102.9
30,0
59.6
Table 13 reports the impact of different distillation loss functions, including euclidean loss, cosine loss, and their combination, across various 3D scene understanding benchmarks. The results show that all loss types yield very similar performance, indicating that the choice feature distance metric has limited influence in setting.
A.5 Qualitative Results
Visualizations Fig. 4 illustrates qualitative results of our method across three tasks: visual grounding, object captioning, and question answering.
For the visual grounding task (top two rows), the model is required to localize objects within 3D scene based on natural language descriptions. Each example shows the ground truth bounding box (blue), the result from baseline method (red), and our prediction (green). In both cases, our method's predictions match the ground truth more closely than the baseline, demonstrating improved grounding accuracy.
In the object captioning task (middle rows), the model generates descriptive captions for specific objects in the scene. The captions from the ground truth, the baseline, and our method are shown alongside their corresponding regions. We also report CIDEr scores measure caption quality. Our approach produces more accurate and detailed descriptions with significantly higher CIDEr scores compared to the baseline.
For the question answering task (bottom two rows), the model answers questions about the scene. Ground truth answers, baseline outputs, and our results are provided for each question. Red rectangles highlight the visual evidence used by our model to generate the answers. Our method provides correct answers that align with the ground truth, whereas the baseline often fails to do so.
Overall, the visualizations demonstrate that our approach consistently outperforms the baseline across all tasks, delivering more accurate grounding, richer object descriptions, and more reliable answers to visual questions.
Table 13: Performance with different distillation losses.
Supervision
ScanRefer
DRlee
Scan2Cap
ScanQA
SQA3D
Aco@0.25
Acc@0.5
F1@00.5
F1@0.5
B-400.5
Ce0.5
C
EM
EM
56.1
86.1
30.3
Cosine loss
62.2
55.5
60.4
55.2
41.8
85.9
104.5
30.1
60.7
Casine Faclidean
62.3
55.7
60.3
55.0
42.1
85.8
102.7
29.7
60.2
17

---

Figs. 5 and 6 provide visual summary of how our method performs on three challenging 3D scene understanding tasks. These tasks include identifying objects based on language, generating descriptions for specific regions, and answering spatial questions about the scene.
In the visual grounding examples at the top, the model is challenged to find the correct object in complex 3D environment based on a textual description. The comparison highlights three bounding boxes for each case: blue for the ground truth, red for the baseline, and green for our result. Our predictions consistently align with the intended targets, showing our model's ability to accurately interpret spatial and semantic cues from language.
The object captioning section in the middle presents how each model describes a highlighted object area. For each instance, the ground truth, baseline output, and our generated caption shown, along with their respective CIDEr scores. Our model's captions are both more precise and more faithful to the scene's 's content, reflected in the higher evaluation scores.
At the bottom, the question answering task demonstrates the model's reasoning abilities within a 3D) environment. The figures show the posed question, the correct answer, the baseline's 's response, and our model's s answer. Even for questions that require counting or locating objects, our approach tends to provide accurate answers, often supported by clear visual evidence in the scene.
Altogether, these qualitative results illustrate that our approach delivers more reliable scene un- derstanding across variety of tasks, outperforming the baseline in both accuracy and descriptive quality.
A.6 Broader Impacts
Positive impacts. The advancement of 3D perception in Al systems holds significant positive societal potential. Enhanced 3D understanding can benefit applications such as assistive robotics for the elderly and disabled, safer autonomous navigation, improved medical imaging, and immersive educational tools. These technologies have the capacity to improve quality of life, boost accessibility, and enable new forms of human-compunt interaction.
Negative impacts. However, the adoption of enhanced 3D perception also raises important privacy concerns, especially in surveillance and monitoring contexts where individuals' activities or envi- ronments could be reconstructed and analyzed without their consent. To address these risks, it is crucial to apply robust data anonymization methods- as blurring faces or removing identifiable features-ensure informed consent from data subjects, enforce strict access controls and data security protocols, and adhere to relevant privacy regulations to protect individual rights.
18

---

Visual Grounding
Ideetify the object according following deseription. A small blue cabinet. there is chair and bed near it.
Question:
GT:
Blue box.
Baseline: Red bux.
Ours: Groen box.
Question: Identify objoct according to folkowing description. There is rectangular white kitchen cabinet. to the right of stove.
GT: Blue box.
Raseline: Red box.
Ours: Green box.
Object Captioning
GT: This is restangalar otiomam. The oltemso in froni of the sufa.
Baseline: This round end table. to the left the conch. (CIDER:
0.01)
Ours: This is brown ottoman. froet of's sofa, (CIDER: 3.52)
GT: The cabinct is in the middlle the room. The cabinet is white box with blue rectangular top.
Raseline: This long counter. it behind a backpack, (CIDER: 0.00)
Ours: The cibinet in the center the room. the cabinet as white rectangular prism. (CIDER: 3,07)
Quesiion Answering
Question: many question simply.
Baseline: 2.
Ours:
Question: How many levels does the shelf have? Anower the question
simply.
GT:
Baseline: 6.
Ours:
Figure 4: Visualization of Results Across Different Tasks. (a) Visual Grounding: The predicted bounding box closely aligns with the ground truth. (b) Object Captioning: Our method generates accurate captions for each referred object. (c) Question Answering: The model provides precise answers, where viewed when zoomed in.
the red rectangles indicate the visual cues utilized for each response. Best
19

---

Visual Grounding
Question: couch north the round coffee table. couch Unree scals and two armrests.
GT: Blue box.
Baseline: Red bux.
Ours: Groen box.
Identify the object according following deseripsion. This is small end table with a book magazime laying on it. if you sitting in the black leveseat facing the end table, the short, bookshheff on your right.
Question:
Baseline:
Green box.
Object Captioning
GT: This brown dosk. is behind chair.
Baseline: This a blue chair. ćesk. (CIDER: 0.8247)
Ours: This is wooden desk. to the right of another desk. (CIDER: 2,968)
GT: This wooden table. is againssi a wall.
whitehoard.
chair has curved backside and four legs. (CIDER: 0.0009)
Ours: This is wooden table. it against the wall, (CIDER: 2,4132)
Quesiton Answering
Question: How many chairs are curtain? question simply.
GT: 1.
Baseline: 2.
Question: How many chairs are on the right side table? Answer the simply.
GT:
Baseline: 2.
Ours:
Figure 5: Visualization of Results Across Different Tasks. (a) Visual Grounding: The predicted bounding box closely aligns with the ground truth. (b) Object Captioning: Our method generates accurate captions for each referred object. (c) Question Answering: The model provides precise answers, where use the red rectangles indicate the visual cues utilized for each response. Best viewed when zoomed in.
20

---

Visual Grounding
Question: The grey. the on left side ofe tabli.
GT: Blue box.
Baseline: Red bux.
Ours: Groen box.
Question:
Baseline: Red
Ours: Green box.
Object Captioning
GT: This pillow the bed. It bright.
Baseline: This a bad with white sheets. Ti to the right ol'a nightstand. (CIDER: 0.2095)
Ours: This white pillow. it the bed. (CIDER: 2.7067)
GT: This red ottoman. it is nexi to round table.
(CIDER:
0,1327)
This is red oftoran, it the left of round table, (CIDER) 2,4524)
Ours:
Quesiton Answering
Question:
to plant? Answer question
simply.
GT: 2.
Baseline: 4.
Question:
Where the octagagon shape table located? Answer simply.
GT: In between 2 chairs.
Baseline: ln front of black chair.
Ours: Between 2 chairs.
Figure 6: Visualization of Results Across Different Tasks. (a) Visual Grounding: The predicted bounding box closely aligns with the ground truth. (b) Object Captioning: Our method generates accurate captions for each referred object. (c) Question Answering: The model provides precise answers, where viewed when zoomed in.
the red rectangles indicate the visual cues utilized for each response. Best
21

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's contributions and scope about 3D-aware representation learning in MLLMs for scene understanding.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
The limitation section is provided in Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not contain theoretical assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The detailed method design is given in Sec. 2, and experimental details are all provided in Sec. and App. A.2.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in the supplemental material?
Answer: [No]
Justification:
We will publicly release the code and related instructions in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Beeiii/gguides/oodeSuue www.ammmiisiooooiccic.io for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//n cts.pss.cc/pubii/gguuuu. gesdd///////eeeeee/o /CodeSubmian ss cassoonPoll. onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the necessary details are provided in Sec. 3 and App. A.2.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experimental settings are consistent with those used in prior works published at top conferences, which do not report error bars as well. Besides, due to resource constraints, we were unable to perform the multiple runs required report error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information of GPUs we use for experiments is provided in Sec. 2.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tturp:p//euriss.... wttp://ppppppppppppppeeeeeeeaeee cccpubblcc/thi ceuucieeeeeeeieeeeeeeeeeeleeee
Answer: [Yes]
Justification:
We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consider- ation due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in App. A.6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper uses all public datasets for training models and has such risk.
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers of the datasets are cited, and their licenses are provided in the reference.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We detail the training and inference settings in Sec. and App. A.2. As for the code, we will release in the near future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer:
[NA]
Justification:
We do not involve human subjects in our experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not involve human subjects our experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLM to improve the wording of the paper and implement code for visualizations.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non ata-standard components. Please refer to our LLM policy (https : httpos:///nerriie //euriiiss....// wwwoooooeneneeeeeeeaeeeeeeeeeeee /ac.cceee///////// /LLM) for what should or should not be described.
28


Reviews:
[{"review_id": "CIxEufjhcI", "reviewer": "Reviewer_QnTH", "summary": "This paper first discovers a strong positive correlation between the 3D-aware representation and downstream 3D task performance. It subsequently proposes learning better 3D-aware representation from 3D foundation model like FLARE and VGGT via knowledge distillation. On five common 3D scene understanding benchmarks, the proposed 3DRS approach shows superior performance.", "questions": "1. Why does distillation from 2D foundation models like SigLIP2 or MAE also leads to improved 3D performances as shown in Table 3? Intuitively these model should only have learnt text-aligned or low-level representation?\n2. I wonder if the proposed 3DRS method also leads to improved performance on new 3D MLLM benchmark like VSI-Bench [4], which is a video VQA dataset that requires advanced visual-spatial intelligence?\n3. In Table 5, the authors only ablate with single layer distillation. I wonder, intuitively, is it better to align with multiple layers?\n\n**References**\n\n[4] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "limitations": "1. This paper only studies QWen2-VL, but not the latest QWen2.5-VL.\n2. So far this work only studies multi-view images, which is a relatively narrow scenario. Personally I believe spatial understanding given video input is much more common and potentially impactful.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The multi-view correspondence experiment makes intuitive sense to me. Ideally, a good 3D-aware representation should extract similar features given different views of the same 3D location.\n2. This paper is relatively well-organized and easy-to-follow. I am not an expert in 3D domain, but I find reading this paper rather easy and joyful.\n3. The proposed 3DRS shows consistent and non-trivial improvement across multiple 3D scene understanding benchmark.\n\n**Weakness**\n1. The authors uses the finding of multi-view correspondence analysis to establish the superiority of 3D-aware representation, but then they turn to 3D foundation model distillation, without solid experimental proof why such approach is better. The authors are encouraged to provide further analysis on how distillation from VGGT or FLARE promotes multi-view correspondence as well as other necessary 3D properties.\n2. It is well-known in 3D detection area that knowledge distilled from a strong 3D model to a 2D model leads to better representation and improved performance [1][2][3], presumably because the spatial cue captured in 3D modality is not so well learnt in those 2D models, leading to less original contribution of this work. Also, this paper didn't cite these important related works.\n\n\n**References**\n\n[1] Chen, Zehui, et al. \"Bevdistill: Cross-modal bev distillation for multi-view 3d object detection.\" arXiv preprint arXiv:2211.09386 (2022). \n\n[2] Wang, Zeyu, et al. \"Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Bang, Geonho, et al. \"Radardistill: Boosting radar-based object detection performance via knowledge distillation from lidar features.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "was_revised": true, "final_justification": "I am delighted to see the effectiveness of the proposed method on VSI-Bench. I am also satisfied with the explanation of distilling from pure 2D vision models like SigLIP2 or MAE. I am, however, still a bit skeptical about dramatic change from the finding of multi-view correspondence analysis to 3D foundation model distillation.\n\nOverall, I agree that this is a decent work and should be accepted. I am inclined to raise my rating to 5.", "rebuttal": "Thanks for the constructive feedback and for acknowledging our ideas and experimental results. Below, we provide detailed responses to the concerns.\n\n---\n### **Weaknesses**\n\n> **Q1:** *The authors use the finding of multi-view correspondence analysis to establish the superiority of 3D-aware representation, but then they turn to 3D foundation model distillation, without solid experimental proof why such an approach is better. The authors are encouraged to provide further analysis on how distillation from VGGT or FLARE promotes multi-view correspondence as well as other necessary 3D properties.*\n\n**R:** We analyze through multiview correspondences as they are the fundamental cues for 3D geometric modeling [1]. The 3D foundation models, such as VGGT, are trained on large-scale 3D tasks, which go far beyond multi-view correspondence. These models also learn from depth maps, camera poses, point clouds, and other 3D-related tasks. Therefore, when we adopt feature distillation from such models, our method implicitly inherits rich 3D features from VGGT.\n\nSince our model cannot be directly evaluated on these 3D tasks (as doing so would require additional training heads for each specific task), we instead use the VSI-Bench [2] dataset for evaluation. VSI-Bench covers a diverse range of tasks—**object counting, absolute and relative distance estimation, object size, room size, relative direction, route planning, and appearance order**—all of which are tightly coupled with core 3D properties such as spatial relationships, relative scale, and geometric understanding. We use the training data from VG-LLM [3] and adopt LLaVA-Next-Video as the baseline.\n\nAs shown in the table below, our method achieves consistent and notable improvements across both the overall metric and individual sub-tasks, compared to the baseline and a variant using DINOv2 features. This demonstrates that our approach effectively inherits the 3D properties from the foundation model, thus promoting not only multi-view correspondence but also other critical 3D capabilities. We will include these results in our revision.\n\n**Table 1: Evaluation on VSI-Bench.** \n|        Method        | Avg. | Object Count | Absolute Distance | Object Size | Room Size | Relative Distance | Relative Direction | Route Plan | Appearance Order |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| GPT-4o              | 34.0 | 46.2         | 5.3               | 43.8        | 38.2      | 37.0              | 41.3               | 31.5       | 28.5             |\n| Gemini-1.5 Flash    | 42.1 | 49.8         | 30.8              | 53.5        | 54.4      | 37.7              | 41.0               | 31.5       | 37.8             |\n| Gemini-1.5 Pro      | 45.4 | 56.2         | 30.9              | **64.1**        | 43.6      | **51.3**              | **46.3**               | **36.0**       | 34.6             |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| Baseline            | 42.1 | 62.4         | 34.0              | 52.4        | 53.6      | 39.3              | 41.2               | 21.7       | 29.2             |\n| Baseline w/Ours (DINOv2) | 44.2 | 67.1   | 33.8              | 52.6        | 56.1      | 39.0              | 42.7               | 26.3       | 35.6             |\n| Baseline w/Ours (VGGT)   | **45.9** | **68.7**   | **34.8**              | 53.6        | **56.6**      | 40.9              | 43.2               | 30.4       | **39.2**             |\n\n> **Q2:** *It is well-known in the 3D detection area that knowledge distilled from a strong 3D model to a 2D model leads to better representation and improved performance [1][2][3], presumably because the spatial cues captured in the 3D modality are not so well learned in those 2D models, leading to less original contribution of this work. Also, this paper didn't cite these important related works.*\n\n**R:** Thank you for the suggestion. Previous works mainly focus on improving pure vision models or CLIP-style VLMs. In contrast, our work targets at enhancing the 3D representation abilities of 3D MLLMs, which remains an under-explored problem. We will discuss these relevant papers in our revision.\n\n---\n### **Questions**\n\n> **Q1:** *Why does distillation from 2D foundation models like SigLIP2 or MAE also lead to improved 3D performance as shown in Table 3? Intuitively, these models should only have learned text-aligned or low-level representations.*\n\n**R:** Although 2D foundation models like SigLIP2 or MAE are mainly trained on 2D tasks, recent studies [4,5] have shown that they can learn emergent properties beyond low-level features. During training, these models develop object awareness, an understanding of shapes, spatial relationships, and even implicit geometry, as discussed in works like DINOv2 [6] and Dino-ViT-Features [7]. As a result, some 3D-related knowledge—such as depth cues and object correspondences—is also encoded in their features. This explains why distillation from strong 2D models can bring improvements on 3D tasks.\n\n> **Q2:** *I wonder if the proposed 3DRS method also leads to improved performance on the new 3D MLLM benchmark like VSI-Bench, which is a video VQA dataset that requires advanced visual-spatial intelligence?*\n\n**R:** Following the suggestion, we evaluated our 3DRS method on the VSI-Bench benchmark. The results are reported in Table 1 above. The results show that our method achieves clear improvements in visual-spatial understanding on this challenging video VQA dataset, further demonstrating the effectiveness of our approach for advanced 3D MLLM benchmarks.\n\n> **Q3:** *In Table 5, the authors only ablate with single layer distillation. I wonder, intuitively, is it better to align with multiple layers?*\n\n**R:** We also try supervising multiple layers, as shown below. However, as the number of supervised layers increases, the performance drops. We believe this is because multi-layer supervision over-constrains geometric feature learning and weakens semantic representation learning, leading to worse downstream results. These results suggest that while single-layer supervision is most effective in our setting, future work could further investigate more sophisticated multi-layer strategies to balance both geometric and semantic cues. We will discuss these findings and insights in the revised paper.\n\n**Table 2: Distillation on multiple layers.**\n| Supervision                              | Scanrefer |      Scanrefer     | Multi3dref |    Multi3dref     | Scan2cap |      Scan2cap  | Scanqa |   Scanqa     | SQA3d |\n|------------------------------------------|-----------|-----------|------------|---------|----------|--------|--------|--------|-------|\n|                                          | Acc@0.25 | Acc@0.5   | Acc@0.25   | Acc@0.5 | C@0.5    | B-4@0.5| C      | EM     | EM    |\n| Last layer                               | **62.9**      | **56.1**      | **60.4**       | **54.9**    | **41.6**     | **86.1**   | **104.8**  | **30.3**   | **60.6**  |\n| Last layer + last 3rd layer              | 61.5      | 54.8      | 60.1       | **54.9**    | 41.4     | 84.4   | 101.4  | 29.2   | 60.5  |\n| Last layer + last 3rd + last 5th layer   | 60.5      | 53.9      | 59.0       | 53.8    | 40.0     | 81.1   | 102.9  | 30.0   | 59.6  |\n\n\n---\n### **Limitations**\n\n> **Q1:** *This paper only studies QWen2-VL, but not the latest QWen2.5-VL.*\n\n**R:** We actually study three types of MLLMs in our paper, i.e., LLaVA-Next-Video, LLaVA-One-Vision, and Qwen2.0-VL. Below, we also provide results for Qwen2.5-VL, where our method can still consistently improve the performance notably, further demonstrating its effectiveness.\n\n**Table 3: Evaluation using Qwen2.5-VL.**\n| Method                       | Scanrefer |    Scanrefer    | Multi3dref |   Multi3dref   | Scan2cap |   Scan2cap   | Scanqa |   Scanqa   | SQA3d |\n|------------------------------|-----------|-----------|------------|---------|----------|--------|--------|--------|-------|\n|                              | Acc@0.25 | Acc@0.5   | Acc@0.25   | Acc@0.5 | C@0.5    | B-4@0.5| C      | EM     | EM    |\n| Qwen2.5-VL 7B                |     58.4      |     52.5      |   59.6     | 52.9    |   80.0   | 40.1   | 100.5  | 29.2   | 59.1  |\n| Qwen2.5-VL 7B w/Ours         |   **60.0**    | **54.3**      |   **61.8**     | **54.6**    |   **82.1**   | **41.3**   | **102.8**  | **29.3**   | **60.3**  |\n\n> **Q2:** *So far this work only studies multi-view images, which is a relatively narrow scenario. Personally, I believe spatial understanding given video input is much more common and potentially impactful.*\n\n**R:** As mentioned in our response to Q1 in the weaknesses section, our method also works on VSI-Bench with video inputs, demonstrating its effectiveness beyond posed multi-view images. We will add the corresponding experimental results in the revised paper.\n\nThanks again for the insightful review.\n\n[1] Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.\n\n[2] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" CVPR 2025.\n\n[3] Zheng, Duo, et al. \"Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors.\" Arxiv 2025.\n\n[4] El Banani, Mohamed, et al. \"Probing the 3d awareness of visual foundation models.\" CVPR 2024.\n\n[5] Man, Yunze, et al. \"Lexicon3d: Probing visual foundation models for complex 3d scene understanding.\" NeurIPS 2024.\n\n[6] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" Arxiv 2023.\n\n[7] Amir, Shir, et al. \"Deep vit features as dense visual descriptors.\" Arxiv 2021."}, {"review_id": "M9Uc8gPmtH", "reviewer": "Reviewer_AJfQ", "summary": "This paper explores the 3D-awareness of Multimodal Large Language Models (MLLMs) by evaluating their consistency across multi-view image features. The authors show a positive correlation between 3D-aware representation quality and downstream scene understanding performance. They first apply a correspondence loss to improve multi-view consistency and then introduce a more powerful distillation loss from pretrained 3D foundation models, enabling MLLMs to better align their visual features with rich 3D representations. The proposed method, termed 3DRS, leads to consistent performance improvements across multiple benchmarks and model backbones without adding runtime or training complexity.", "questions": "1. It appears there is a typo in Figure 3 where “XE loss” is likely intended to be “CE loss” (Cross-Entropy). Please clarify or correct this for clarity.\n2. Table 5 suggests that applying the distillation loss to different final layers of the LLM improves downstream task performance. It would be valuable to explore the effect of applying distillation across multiple layers simultaneously, potentially capturing both low-level and high-level 3D cues. This could provide deeper insights into where 3D supervision is most effective within the model architecture.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear Motivation and Insightful Evaluation: The paper provides a well-motivated investigation into a key limitation of current MLLMs: limited 3D-awareness due to 2D pretraining. The use of multi-view correspondence as a proxy for 3D consistency is intuitive and effective.\n\n2. Simple but Effective Techniques: The proposed correspondence loss and distillation loss from 3D foundation models are easy to implement, yet significantly improve 3D feature learning.\n\n3. Strong Empirical Validation: Results show consistent and notable improvements across diverse tasks (3D captioning, grounding, QA) and popular MLLMs, including LLaVA variants and Qwen2-VL.\n\nWeaknesses:\n\n1. Limited Evaluation of Generalization Ability:\nThe generalization capability of 3DRS is not fully explored in the paper. Since the training and test data appear to come from the same or similar distributions, it's unclear how well 3DRS performs in out-of-distribution settings. Evaluating the method on unseen benchmarks such as MMScanQA [1] or OpenEQA [2] would provide stronger evidence of its robustness and broader applicability.\n\n2. Lack of Discussion on Broader Applicability:\nWhile 3DRS is shown to be effective for 3D scene understanding, the paper does not explore or discuss its potential benefits for more general-purpose tasks in vision or multimodal learning, such as image or video understanding. Clarifying whether the proposed 3D supervision could also improve performance in such broader domains would enhance the impact and relevance of the work.\n\n[1] MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations. NeurIPS 2024\n[2] OpenEQA: Embodied Question Answering in the Era of Foundation Models. CVPR 2024", "was_revised": true, "final_justification": "The proposed method is decent with great generalization ability. I'll keep my accept recommendation.", "rebuttal": "Thanks for the valuable feedback and for recognizing our work. We sincerely appreciate the constructive comments and address the concerns below.\n\n---\n### **Weaknesses**\n\n> **Q1:** *Limited Evaluation of Generalization Ability: The generalization capability of 3DRS is not fully explored in the paper. Since the training and test data appear to come from the same or similar distributions, it's unclear how well 3DRS performs in out-of-distribution settings. Evaluating the method on unseen benchmarks such as MMScanQA [1] or OpenEQA [2] would provide stronger evidence of its robustness and broader applicability.*\n\n**R:** Thank you for highlighting this important point. Following the suggestion, we conduct additional experiments on the OpenEQA benchmark. We use LLaVA-Next-Video as the baseline and uniformly sample 32 frames per video for testing. Notably, the original OpenEQA evaluation uses the `gpt-4-1106-preview` model, which is no longer available; therefore, we use the updated `gpt-4.1-2025-04-14` model for our experiments.\n\nAs shown in Table 1 below, our method continues to deliver performance improvements on this out-of-distribution dataset, which demonstrates its strong generalization capability. Specifically, distillation from VGGT brings a 2.5-point improvement over the baseline.\n\n**Table 1: Results on OpenEQA.**\n\n| Method                     | OpenEQA |\n|----------------------------|---------|\n| Baseline                   | 55.1    |\n| Baseline w/Ours (DinoV2)   | 56.1    |\n| Baseline w/Ours (VGGT)     | **57.6**    |\n\nTo further verify the robustness and broader applicability of our approach, we also evaluate our method on another challenging spatial understanding dataset (VSI-Bench [1]), which adopts videos instead of posed multiview images as inputs for evaluation. VSI-Bench covers a diverse range of tasks—**object counting, absolute and relative distance estimation, object size, room size, relative direction, route planning, and appearance order**—all of which tightly couple with core 3D properties such as spatial relationships, relative scale, and geometric understanding. This dataset includes data from ScanNet++ [2] and ARKitScenes [3], which VGGT did **not** use for training, thus mitigating the domain advantage. We adopt training data from VG-LLM [4] and use LLaVA-Next-Video as the baseline.\n\nAs shown in Table 2, our method consistently achieves higher scores across multiple sub-tasks, which reinforces its effectiveness in diverse scenarios.\n\n**Table 2: Evaluation on VSI-Bench.**\n\n|        Method        | Avg. | Object Count | Absolute Distance | Object Size | Room Size | Relative Distance | Relative Direction | Route Plan | Appearance Order |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| GPT-4o              | 34.0 | 46.2         | 5.3               | 43.8        | 38.2      | 37.0              | 41.3               | 31.5       | 28.5             |\n| Gemini-1.5 Flash    | 42.1 | 49.8         | 30.8              | 53.5        | 54.4      | 37.7              | 41.0               | 31.5       | 37.8             |\n| Gemini-1.5 Pro      | 45.4 | 56.2         | 30.9              | **64.1**        | 43.6      | **51.3**              | **46.3**               | **36.0**       | 34.6             |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| Baseline            | 42.1 | 62.4         | 34.0              | 52.4        | 53.6      | 39.3              | 41.2               | 21.7       | 29.2             |\n| Baseline w/Ours (DINOv2) | 44.2 | 67.1   | 33.8              | 52.6        | 56.1      | 39.0              | 42.7               | 26.3       | 35.6             |\n| Baseline w/Ours (VGGT)   | **45.9** | **68.7**   | **34.8**              | 53.6        | **56.6**      | 40.9              | 43.2               | 30.4       | **39.2**             |\n\nThese results provide strong evidence to show that our method not only generalizes to out-of-domain settings, but also enhances performance on demanding 3D reasoning tasks. We will add these results and further discussion in the revision.\n\n> **Q2:** *Lack of Discussion on Broader Applicability: While 3DRS is shown to be effective for 3D scene understanding, the paper does not explore or discuss its potential benefits for more general-purpose tasks in vision or multimodal learning, such as image or video understanding. Clarifying whether the proposed 3D supervision could also improve performance in such broader domains would enhance the impact and relevance of the work.*\n\n**R:** We agree that it is valuable to consider the impact of 3D supervision beyond traditional 3D scene understanding tasks.\n\nWe note that our experiments on VSI-Bench demonstrate that 3DRS brings improvements in video understanding scenarios. This result provides evidence that our approach is not limited to posed multi-view scenes but also extends to video-based tasks.\n\nMoreover, we believe that by enhancing a model’s understanding of geometry and spatial relationships, our 3D supervision has the potential to benefit a wide range of image and video understanding tasks, especially those requiring deep spatial reasoning. While our current study focuses on 3D scene understanding, we appreciate the suggestion and plan to further investigate the impact of our approach on broader vision and multimodal domains in future work. We will add the discussion in our revision.\n\n---\n### **Questions**\n> **Q1:** *It appears there is a typo in Figure 3 where “XE loss” is likely intended to be “CE loss” (Cross-Entropy). Please clarify or correct this for clarity.*\n\n**R:**  Thank you for pointing this out. Yes, it should be “CE loss” (Cross-Entropy). We will update this in the revision for clarity.\n\n> **Q2:** *Table 5 suggests that applying the distillation loss to different final layers of the LLM improves downstream task performance. It would be valuable to explore the effect of applying distillation across multiple layers simultaneously, potentially capturing both low-level and high-level 3D cues. This could provide deeper insights into where 3D supervision is most effective within the model architecture.*\n\n**R:**  Thank you for the insightful suggestion. We conduct additional experiments where we apply supervision to multiple layers simultaneously, as shown in Table 3 below. Interestingly, we observe that as the number of supervised layers increases, the performance decreases. We believe that multi-layer supervision may over-constrain the geometric features while weakening the model's semantic representations, ultimately resulting in lower performance on downstream tasks.\n\n**Table 3: Distillation on multiple layers.**\n\n| Supervision                              | Scanrefer |      Scanrefer     | Multi3dref |    Multi3dref     | Scan2cap |      Scan2cap  | Scanqa |   Scanqa     | SQA3d |\n|------------------------------------------|-----------|-----------|------------|---------|----------|--------|--------|--------|-------|\n|                                          | Acc@0.25 | Acc@0.5   | Acc@0.25   | Acc@0.5 | C@0.5    | B-4@0.5| C      | EM     | EM    |\n| Last layer                               | **62.9**      | **56.1**      | **60.4**       | **54.9**    | **41.6**     | **86.1**   | **104.8**  | **30.3**   | **60.6**  |\n| Last layer + last 3rd layer              | 61.5      | 54.8      | 60.1       | **54.9**    | 41.4     | 84.4   | 101.4  | 29.2   | 60.5  |\n| Last layer + last 3rd + last 5th layer   | 60.5      | 53.9      | 59.0       | 53.8    | 40.0     | 81.1   | 102.9  | 30.0   | 59.6  |\n\nThese results suggest that while single-layer supervision is most effective in our setting, future work can further investigate more sophisticated multi-layer strategies to balance both geometric and semantic cues. We will discuss these findings and insights in the revision.\n\nThanks again for the valuable feedback.\n\n[1] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" CVPR 2025.\n\n[2] Yeshwanth, Chandan, et al. \"Scannet++: A high-fidelity dataset of 3d indoor scenes.\" ICCV 2023.\n\n[3] Baruch, Gilad, et al. \"Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.\" Arxiv 2021.\n\n[4] Zheng, Duo, et al. \"Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors.\" Arxiv 2025."}, {"review_id": "ivmNQxtYmg", "reviewer": "Reviewer_8Yhg", "summary": "This paper proposes 3DRS, which enhances multi-modal LLMs with additional 3D supervision for 3D scene understanding. The motivation is that MLLMs inherently lack 3D awareness (especially multi-view geometry). Thus, this paper introduces a simple yet effective supervision method: VGGT or FLARE distillation. It shows they outperform 2D foundation model supervisions like DINOv2. Experiments demonstrate consistent gains across the ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D benchmarks.", "questions": "See weakness and minor parts.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. This paper provides good engineering insights by demonstrating the use of MVS (i.e. view equivariance in L43) for supervision. The method is simple yet effective, which distills the 3D-aware knowledge from 3D foundation models like VGGT.\n\n2. The experiments are conducted across various MLLMs (LLaVA-Next-Video, LLaVA-OneVision, and Qwen2-VL). Tables 3 and 4 demonstrate the performance gains from VGGT's 3D priors over 2D foundation models and direct correspondence loss.\n\n3. The motivation of 3D-awareness is well illustrated and validated in using the correspondence score in Sec. 2.1.3. Fig. 2 intuitively demonstrates how the similarity of visual embeddings influences 3D scene understanding.\n\nCons:\n\n1. The claim in L247 that 2D foundation models lack specific spatial priors doesn't seem convincing. Moreover, VGGT and FLARE are also trained on ScanNet specifically, giving them a domain advantage. I'm not sure this claim holds strong. \n\n2. The idea of \"distillation from 3D\" has been explored in prior 3D scene understanding works. However, related works [1,2,3] are missing, and there is a lack of discussion on other \"distillation from 3D\" ways. \n\n[1] 3D-to-2D Distillation for Indoor Scene Parsing. CVPR 21\n[2] Pri3D: Can 3D Priors Help 2D Representation Learning? ICCV 21\n[3] Openscene: 3d scene understanding with open vocabularies. CVPR 2023\n\nMinor:\n\n1. The motivation makes sense to me overall, but VGGT has been trained on ScanNet to reconstruct its 3D geometry, which gives it a strong domain alignment with the benchmark tasks.\n\n2. Fig. 2 primarily illustrates 3D correspondence at the voxel level. However, the following 3D tasks rely more on instance-level annotations. The similarity between the same objects is more directly related to 3D scene understanding and could better illustrate the relations.", "was_revised": true, "final_justification": "The results on VSI-Bench (out-of-domain data) effectively address my concerns. Table 2 shows that the instance-level alignment IoU indicates a positive correlation with object-level semantics.\n\nOverall, these complementary results help strengthen the claims. I am inclined to raise my rating from borderline accept to accept.", "rebuttal": "Thanks for the insightful review and for recognizing our motivation, experimental evaluation, and method design. We address the concerns below.\n\n---\n### **Weaknesses**\n\n>**Q1.** *The claim in L247 that 2D foundation models lack specific spatial priors does not seem convincing. Moreover, VGGT and FLARE are also trained on ScanNet specifically, giving them a domain advantage. I am not sure this claim holds strong.*\n\n**R:** We appreciate the thoughtful comment. To further assess the effectiveness of 3D foundation models versus 2D foundation models, we use training data from VG-LLM [1] and use LLaVA-Next-Video as a baseline to conduct experiments on the VSI-Bench dataset. VSI-Bench [2] is a challenging benchmark for spatial reasoning. This dataset includes data from ScanNet++ [3] and ARKitScenes [4], which VGGT did **not** use for training, thus mitigating the domain advantage. VSI-Bench covers a diverse range of tasks—**object counting, absolute and relative distance estimation, object size, room size, relative direction, route planning, and appearance order**—all of which tightly couple with core 3D properties such as spatial relationships, relative scale, and geometric understanding.\n\nAs shown in Table 1 below, using 3D foundation model consistently outperforms the 2D foundation model and other strong baselines on VSI-Bench. This demonstrates that geometric knowledge remains crucial for scene understanding, especially on tasks that demand spatial reasoning:\n\n**Table 1: Evaluation on VSI-Bench.**\n\n|        Method        | Avg. | Object Count | Absolute Distance | Object Size | Room Size | Relative Distance | Relative Direction | Route Plan | Appearance Order |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| GPT-4o              | 34.0 | 46.2         | 5.3               | 43.8        | 38.2      | 37.0              | 41.3               | 31.5       | 28.5             |\n| Gemini-1.5 Flash    | 42.1 | 49.8         | 30.8              | 53.5        | 54.4      | 37.7              | 41.0               | 31.5       | 37.8             |\n| Gemini-1.5 Pro      | 45.4 | 56.2         | 30.9              | **64.1**        | 43.6      | **51.3**              | **46.3**               | **36.0**       | 34.6             |\n|---------------------|------|--------------|-------------------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| Baseline            | 42.1 | 62.4         | 34.0              | 52.4        | 53.6      | 39.3              | 41.2               | 21.7       | 29.2             |\n| Baseline w/Ours (DINOv2) | 44.2 | 67.1   | 33.8              | 52.6        | 56.1      | 39.0              | 42.7               | 26.3       | 35.6             |\n| Baseline w/Ours (VGGT)   | **45.9** | **68.7**   | **34.8**              | 53.6        | **56.6**      | 40.9              | 43.2               | 30.4       | **39.2**             |\n\nThese results indicate that geometric priors provided by 3D foundation models offer advantages in spatial reasoning and scene understanding, even without the domain-specific training. We hope this addresses the concern and clarifies the role of geometry-aware models for these challenging tasks.\n\n>**Q2.** *The idea of \"distillation from 3D\" has been explored in prior 3D scene understanding works. However, related works [1,2,3] are missing, and there is a lack of discussion on other \"distillation from 3D\" ways.*\n\n**R:** Thanks for highlighting this important point. We agree that there exist works exploring \"distillation from 3D,\" which primarily focus on pure visual models or CLIP-style vision-language models. However, there is very limited exploration of leveraging 3D distillation specifically to enhance 3D representations in LLMs, which is the main focus of our work.\n\nWe will include the relevant references and a more thorough discussion of related methods in our revision.\n\n---\n### **Minor Issue**\n\n>**Q1.** *The motivation makes sense to me overall, but VGGT has been trained on ScanNet to reconstruct its 3D geometry, which gives it a strong domain alignment with the benchmark tasks.*\n\n**R:** Following this suggestion, in Table 1 above, we experiment on the VSI-Bench dataset, which includes out-of-domain data from ScanNet++[3] and ARKitScenes[4]. The consistent and notable improvements demonstrate that the 3D foundation model maintains strong performance even without domain-specific training. This highlights the model’s robust generalization ability and the critical role of geometric knowledge in scene understanding.\n\n>**Q2.** *Fig. 2 primarily illustrates 3D correspondence at the voxel level. However, the following 3D tasks rely more on instance-level annotations. The similarity between the same objects is more directly related to 3D scene understanding and could better illustrate the relations.*\n\n**R:** Thanks for the valuable suggestion. Our use of voxel-level correspondence in Fig. 2 aims to emphasize the model’s understanding of 3D geometric space, as fine-grained spatial relationships effectively reflect the model’s ability to capture geometric features and spatial positioning.\n\nWe agree that instance-level correspondence can highlight high-level semantic relationships between objects. Due to rebuttal format limitations, we are unable to include additional figures at this stage. However, we compute instance-level correspondence results based on the LLaVA-Next-Video model and present them in the tables below, which still exhibit the positive relation between downstream performance and 3D feature learning quality.\n\n**Table 2: Instance-level correspondence results**.\n\n| Dataset         | Metric      | Q1     | Q2     | Q3     | Q4     |\n|-----------------|------------|--------|--------|--------|--------|\n| Multi3D-Refer   | IOU_25_ALL | 43.07  | 48.77  | 48.88  | 54.42  |\n| Multi3D-Refer   | IOU_50_ALL | 39.45  | 44.12  | 45.32  | 50.09  |\n| ScanRefer       | IOU_25_ALL | 45.98  | 51.88  | 55.26  | 57.09  |\n| ScanRefer       | IOU_50_ALL | 40.37  | 46.39  | 49.23  | 50.80  |\n| Scan2Cap        | CIDEr      | 85.14  | 100.06 | 104.41 | 103.40 |\n| ScanQA          | CIDEr      | 97.70  | 96.17  | 105.94 | 109.34 |\n| SQA3D           | All        | 56.44  | 57.27  | 61.02  | 59.37  | \n\nThanks again for the valuable feedback.\n\n[1] Zheng, Duo, et al. \"Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors.\" Arxiv 2025.\n\n[2] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" CVPR 2025.\n\n[3] Yeshwanth, Chandan, et al. \"Scannet++: A high-fidelity dataset of 3d indoor scenes.\" ICCV 2023.\n\n[4] Baruch, Gilad, et al. \"Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.\" Arxiv 2021."}, {"review_id": "LYb2hgWo4s", "reviewer": "Reviewer_Ko1x", "summary": "This paper addresses the challenge of improving the 3D scene understanding capabilities of multimodal large language models, which are inherently pretrained on 2D data and thus lack strong 3D geometric awareness. The authors first conduct a study revealing a strong positive correlation between a model's multi-view feature consistency (a proxy for 3D awareness) and its performance on downstream 3D reasoning tasks. Motivated by this finding, they propose a framework called 3D-Aware representation supervision (3DRS). 3DRS employs knowledge distillation, using a pretrained 3D foundation model (eg, VGGT) as a \"teacher\" to provide rich 3D-aware feature targets. The MLLM is then fine-tuned with an auxiliary loss that encourages its visual features to align with these 3D-aware targets, in addition to the standard text generation loss. The authors demonstrate that this approach consistently and significantly improves performance across three different MLLM architectures (LLaVA-Next-Video, LLaVA-OneVision, Qwen2-VL) on five diverse 3D scene understanding benchmarks", "questions": "1. The paper convincingly shows that 3DRS improves performance. To better understand why, could you provide a more fine-grained analysis? For example, could you design a targeted probe or qualitative analysis to see if the model with 3DRS is better at specific geometric reasoning tasks that are not directly measured by the benchmarks, such as:\n    - Judging the relative size of two objects at different depths.\n    - Understanding occlusion relationships in a scene.\n    - Identifying support surfaces (eg, items on a single table).\n\n2. The performance of 3DRS relies heavily on the quality of the 3D foundation model teacher. Could the authors discuss the failure modes of the teacher (VGGT) and how they might propagate to the student MLLM? For instance, what happens in scenes with reflective surfaces, textureless objects, or dynamic elements where the teacher's correspondence or geometry estimates might be noisy or incorrect? \n\n3. The distillation uses a simple cosine similarity loss between the final layer features. Did you explored other knowledge distillation strategies?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is very well-motivated. The initial investigation (Sec 2.1) that establishes a quantitative link between multi-view correspondence scores and downstream task performance is very interesting. This diagnostic-first approach provides a clear and compelling rationale for the proposed approach, moving beyond simple intuition.\n\n- The experimental evaluation is good. The authors validate their method on multiple MLLM backbones and a wide range of 3D vision-language tasks (grounding, captioning, QA). The ablation study comparing 2D vs. 3D foundation models as teachers (Table 3) is interesting and strongly supports the central claim. There is consistent improvements across the board (Figure 1b, Table 2)  which indicates the effectiveness of the proposed approach.\n\n- The paper is well-written and easy to follow. Figure 1 provides an excellent high-level overview of the key results. The method is explained clearly, and the overall narrative is logical and persuasive.\n\n**Weaknesses**\n\n- The primary weakness of the paper is its limited conceptual novelty. The core proposal is to distill the features of a multi-view model to an MLLM. I guess the contribution here is to show that this actually leads to improved performance, but because the tasks are spatial in nature this could be considered obvious. \n\n- The paper shows that 3D supervision works by improving a \"correspondence score\" and downstream metrics, but it does not go deeper into what specific 3D properties are being learned. Is the MLLM learning about object permanence, relative scale, or other geometric properties? VGGT only outputs pointmaps and camera poses so there's not much information about objects, and it's not metric scale.\n\n- The performance of the proposed method is upper-bounded by the quality of the \"teacher\" 3D foundation model. The paper does not discuss the limitations of the teacher models themselves or analyze how their potential failure modes (e.g., incorrect correspondence or depth estimation) might negatively impact or mislead the student MLLM during training.", "was_revised": true, "final_justification": "This paper shows that geometric vision models can be useful for MLLMs in understanding spatial relationships. While the performance improvement isn't very large, the results show it clearly is significant. My final rating is borderline accept.", "rebuttal": "Thanks for the constructive feedback. We appreciate the recognition of our motivation, experimental results, and paper writing. We address the concerns below.\n\n---\n\n### **Weaknesses**\n\n> **Q1.** *The primary weakness of the paper is its limited conceptual novelty. The core proposal is to distill the features of a multi-view model to an MLLM. I guess the contribution here is to show that this actually leads to improved performance, but because the tasks are spatial in nature this could be considered obvious.*\n\n**R:** We agree that it is intuitive to consider distilling knowledge from 3D models to enhance 3D awareness in other models. However, in the context of modern MLLMs, it remains unclear whether these models inherently possess 3D spatial understanding due to their massive-scale pretraining. To our knowledge, there is not a systematic investigation of 3D awareness in MLLMs, and this remains an open question.\n\nOur paper first provides a thorough analysis to reveal the lack of 3D awareness in existing MLLMs, which motivates our proposed method. We then introduce a simple and effective framework to distill geometric knowledge from 3D models into MLLMs, and empirically demonstrate that this approach significantly enhances their spatial reasoning abilities. We believe our work provides both an important empirical finding and a practical solution for improving 3D scene understanding in MLLMs.\n\n> **Q2.** *The paper shows that 3D supervision works by improving a \"correspondence score\" and downstream metrics, but it does not go deeper into what specific 3D properties are being learned. Is the MLLM learning about object permanence, relative scale, or other geometric properties? VGGT only outputs pointmaps and camera poses so there's not much information about objects, and it's not metric scale.*\n\n**R:**  We begin with correspondence learning, a fundamental cue in 3D vision and multiview geometry, as it directly reflects a model’s spatial reasoning ability [1]. Correspondence estimation serves as a critical probe for evaluating the 3D awareness of MLLMs. However, our analysis shows that existing MLLMs struggle with correspondence estimation and lack robustness across many aspects of 3D understanding.\n\nBy distilling knowledge from VGGT, our approach enables models to inherit strong 3D capabilities from extensive training tasks, e.g., pointmap and depth map prediction, camera pose estimation, and point tracking. While VGGT does not provide ideal object-centric or metric-scale supervision, this represents a significant improvement over previous methods. With continued advances in 3D foundation models, we believe that metric-scale spatial awareness is within reach, and our method helps drive the broader development of 3D-aware MLLMs.\n\nTo further investigate which specific 3D properties are learned, we evaluate our method on VSI-Bench [2], a challenging benchmark designed for geometric-aware spatial reasoning with video inputs. VSI-Bench covers a diverse range of tasks—**object counting, absolute and relative distance estimation, object size, room size, relative direction, route planning, and appearance order**—all of which are tightly coupled with core 3D properties such as spatial relationships, relative scale, and geometric understanding. We follow the training protocol of VG-LLM [3] and use LLaVA-Next-Video as our baseline.\n\nAs shown in Table 1 below, our method consistently achieves higher scores across these spatially-grounded tasks compared to strong proprietary models (e.g., GPT-4o, Gemini-1.5 Pro) and our baseline. This demonstrates that our approach not only improves correspondence estimation but also significantly enhances the model’s geometric reasoning abilities, including scale estimation, object association, spatial arrangement, and relational understanding—even without explicit metric-scale supervision.\n\n**Table 1: Evaluation on VSI-Bench.**\n\n|        Method        | Avg. | Object Count | Absolute Distance | Object Size | Room Size | Relative Distance | Relative Direction | Route Plan | Appearance Order |\n|---------------------|------|--------------|----------|-------------|-----------|----|---------|------------|------------------|\n| GPT-4o              | 34.0 | 46.2         | 5.3               | 43.8        | 38.2      | 37.0              | 41.3               | 31.5       | 28.5             |\n| Gemini-1.5 Flash    | 42.1 | 49.8         | 30.8              | 53.5        | 54.4      | 37.7              | 41.0               | 31.5       | 37.8             |\n| Gemini-1.5 Pro      | 45.4 | 56.2         | 30.9              | **64.1**        | 43.6      | **51.3**              | **46.3**               | **36.0**       | 34.6             |\n|---------------------|------|--------------|---------|-------------|-----------|-------------------|--------------------|------------|------------------|\n| Baseline            | 42.1 | 62.4         | 34.0              | 52.4        | 53.6      | 39.3              | 41.2               | 21.7       | 29.2             |\n| Baseline w/Ours (DINOv2) | 44.2 | 67.1   | 33.8              | 52.6        | 56.1      | 39.0              | 42.7               | 26.3       | 35.6             |\n| Baseline w/Ours (VGGT)   | **45.9** | **68.7**   | **34.8**              | 53.6        | **56.6**      | 40.9              | 43.2               | 30.4       | **39.2**             |\n\nThese results confirm that our framework substantially improves the MLLM’s grasp of a wide range of 3D geometric properties—not just low-level correspondence—demonstrating robust and generalizable 3D spatial understanding.\n\n> **Q3.** *The performance of the proposed method is upper-bounded by the quality of the \"teacher\" 3D foundation model. The paper does not discuss the limitations of the teacher models themselves or analyze how their potential failure modes (e.g., incorrect correspondence or depth estimation) might negatively impact or mislead the student MLLM during training.*\n\n**R:** We agree that, due to the distillation-based nature of our approach, the performance of our method is upper-bounded by the quality of the teacher 3D foundation model. Any limitations or failure modes of the teacher—such as inaccurate correspondence, erroneous depth estimation, or incomplete geometric representations—can be propagated to the student MLLM and may potentially mislead it during training.\n\nWhile our experiments demonstrate consistent improvements over strong baselines, it is possible that errors or biases in the teacher's predictions can negatively impact the downstream 3D reasoning abilities of the student model. We believe that, as 3D foundation models continue to rapidly advance, this limitation becomes less pronounced over time.\n\nWe appreciate the reviewer’s suggestion and will include a dedicated discussion of this limitation in our revised version.\n\n---\n\n### **Questions**\n\n> **Q1.** *The paper convincingly shows that 3DRS improves performance. To better understand why, could you provide a more fine-grained analysis? For example, could you design a targeted probe or qualitative analysis to see if the model with 3DRS is better at specific geometric reasoning tasks that are not directly measured by the benchmarks.*\n\n**R:** Our experiments on VSI-Bench above (Table 1) provide a fine-grained analysis across diverse geometric reasoning tasks, such as distance estimation, object counting, and spatial relationships. The consistent gains with 3DRS on these sub-tasks demonstrate its effectiveness on specific geometric reasoning abilities. We will include this in the revision.\n\n> **Q2.** *The performance of 3DRS relies heavily on the quality of the 3D foundation model teacher. Could the authors discuss the failure modes of the teacher (VGGT) and how they might propagate to the student MLLM? For instance, what happens in scenes with reflective surfaces, textureless objects, or dynamic elements where the teacher's correspondence or geometry estimates might be noisy or incorrect?*\n\n**R:** As discussed above (Q3 in Weaknesses), limitations and errors in the teacher model (VGGT) can propagate to the student MLLM through distillation. This may result in suboptimal geometric reasoning in such challenging scenarios. We will add a discussion on this point.\n\n> **Q3.** *The distillation uses a simple cosine similarity loss between the final layer features. Did you explore other knowledge distillation strategies?*\n\n**R:** As shown in the table below, we try L2 loss and a combination of cosine and L2 losses. The results show that all strategies yield similar performance, indicating that the choice of feature distance metric has limited impact in our setting.\n\n**Table 2: Performance with different distillation losses.**\n| Supervision            | Scanrefer | Scanrefer | Multi3dref | Multi3dref | Scan2cap | Scan2cap | Scanqa | Scanqa | SQA3d |\n|------|------|-----------|------------|------------|----------|----------|--------|--------|-------|\n|                       | Acc@0.25  | Acc@0.5   | Acc@0.25   | Acc@0.5    | B-4@0.5  | C@0.5    | C      | EM     | EM    |\n| Cosine loss           | **62.9**      | **56.1**      | **60.4**       | 54.9       | 41.6     | **86.1**     | **104.8**  | **30.3**   | 60.6  |\n| L2 loss               | 62.2      | 55.5      | **60.4**       | **55.2**       | 41.8     | 85.9     | 104.5  | 30.1   | **60.7**  |\n| Cosine + L2           | 62.3      | 55.7      | 60.3       | 55.0       | **42.1**     | 85.8     | 102.7  | 29.7   | 60.2  |\n\nWe will include this comparison in the revised version.\n\nThanks again for the detailed and thoughtful review.\n\n[1] Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.\n\n[2] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\" CVPR 2025.\n\n[3] Zheng, Duo, et al. \"Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors.\" Arxiv 2025."}]

Author rebuttal:

Meta-review: All reviewers agreed that the problem studied is important, the findings are interesting, the proposed method is effective, and the results are promising. With additional experimental results and clarifications, the rebuttal successfully addressed the reviewers’ concerns, and all reviewers recommended acceptance. The authors are encouraged to further strengthen the camera-ready version by incorporating the reviewers’ suggestions.
Decision: Accept (poster)