Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
Authors: Duo Zheng
Shijia Huang
Yanyang Li
Liwei Wang
Venue: NeurIPS 2025 poster ()
Keywords: MLLM
3D Scene Understanding
Spatial Reasoning
Primary area: deep_learning

Abstract: Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.

Learning from Videos for 3D World: Enhancing MILMs with 3D Vision Geometry Priors
Duo Zheng", Shijia Huang", Yanyang Li, Liwei Wang
The Chinese University of Hong Kong
https://lavi-lab wwwpaaaiiiaaa.iiuub.....a 1o/V-LLLM
Abstract
Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's s-rdds-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MILMs to understand in 3D spaces directly from video data, without the need for additional 3D input. We propose novel and efficient method called the Video-3D Geometry Large Language Model (VG L.LM). Our approach utilizes 3D visual geometry encoder to extract 3D prior information from vidco sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-ard methods, and even surpasses the Gemini-115-Por in the VSI-Bench evaluations.
reason
Introduction
Rapid advancements and impressive performance of Multimodal Large Language Models (MLLMs) [2, 24, 26, 29, 32, 39, 40, 43] have driven their applications in various fields, such 3D scene understanding [11, 20, 21, 22, 35, 58, 63]. vision-language-action models [3, 25, 36, 47], embodied navigation [55, 59, 61], and learning 3D knowledge from videos [35, 55, 58].
Efforts [35, 58] have been made to improve the 3D spatial understanding capability of MLLMs by considering scenes video sequences. For example, Video-3I L.M [41, 58] injects 3D coordinates into visual features at patch level to improve 3D perception. GPT4Scene [35] leverages BEV maps [28] rendered from reconstructed 3D point clouds for global awareness. However, shared limitation of these approaches their dependence on dense 3D data input (e.g., depth maps and point maps), which often hard acquire in certain real-world scenarios. Although estimating 3D attributes directly from images is possible [38, 42]. it can introduce estimation errors and therefore degrade the performance, thus restricting their practical applicability. This naturally leads to the question: "Can MLLMs understand the 3D world directly from videos without any explicit 3D data input?"
Recent research [50] has shown that MILMS face difficulties understanding 3D geometry from encoded visual representations. This issue arises because these MILMs process video frames as separate tokens through a visual encoder, which fails to capture crucial 3D geometric information, such correspondences across frames [30]. Consequently, the MLLM backbone has to infer the
denotes equal contribution *Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

3D structure from the visual tokens to comprehend spatial relationships, which is both challenging
and ussurce-intensivee This process often requires extensive supervision [6, 14, 54] and meticulous
design to prevent issues such as catastrophic forgetting during fine-tuning [52]. These challenges highlight the critical need for methods that can incorporate 3D geometry priors into MILMMs
In this work, we propose Video-3D Geometry LLM (VG LLM), a novel framework designed to explicitly integrate 3D visual geometry priors into MLLMs. To achieve this, we introduce 3D visual geometry encoder that enriches input visual sequences with additional geometric information. Specifically. input images are processed by both conventional visual encoder and the newly integrated 3D visual geometry encoder. The features extracted by these encoders are fused at the patch level and subsequently passed to the MLLM backbone. As the 3D visual geometry encoder is pre-trained on tasks such as point map prediction on pairs or sequences of images [38, 42, 44], it embeds strong 3D perception prior knowledge and is able to capture correspondences across frames. By doing so, VG LLM can effectively incorporate 3D geometry priors into the model and become more robust to viewpoint transformations, significantly improving its spatial reasoning abilities.
Extensive experiments have been conducted on various 3D scene understanding and spatial reasoning tasks, where the model accepts video input. These 3D scene understanding tasks include 3D visual grounding [7]. 3D dense captioning [8]. and 3D video object detection [45]. For spatial reasoning tasks, we evaluate our model on VSI-Bench [50] and CV-Bench [40]. The experiments show that our fine-tuned model outperforms larger spatial-enhanced models by substantial margin. The results uncover several interesting findings: (1) Without explicit dense 3D inputs, our approach outperforms many leading 3D input-based models, underscoring its effective 31) geometric understanding. (2) By implicitly modeling inter-frame correspondences within the visual representation, our 8B model has learned strong egocentric-allocentie transformation capabilittess leading to significant improvements of precision by 11.9% and F1 by 10.7% on 3D video object detection. (3) On tasks that require complex spatial reasoning skills, i.e. VSI-Bench [50], our 4B model attains an impressive average score of 47.3%, %, surpassing even the best proprietary model, Gemini- 1.5-n [39]. Furthermore, our 8B model sets new state-of-the-art performance with a score of 50.7 %. This highlights the great utility of 3D geometry modeling in broader scenarios.
2 Related Work
Multimodal Large Language Models (MLLMs). MLLMs [2, 24, 26, 29, 32, 39, 40, 43] have achieved significant progress in 2D image and video understanding. However, recent findings [31, 50] indicate critical limitation: current MLLMs still struggle with complex visual spatial reasoning tasks. To address this challenge, some research efforts [2, 40, 43] have focused on enhancing the models' ability to perceive and process spatial relationships through improved model representation. For example, Cambrian-1 1 [40] integrates self supervuper 2D visual representations with semantically rich features, aiming to provide benefit spatial reasoning. These methods focus on enriching the representation of each individual image, neglecting the 3D geometry information inherent in the continuous frames of video. In contrast, our approach integrates 3D geometry priors from the video into the MLLM.
comprehensive understanding of visual content that could
3D Large Language Models. Recent efforts [11, 13, 18, 20, 35, 58] have focused on enabling MLLMs to better understand 3D scenes. Previous work develops comprehensive 3D scene rep- resentations by using different types of 3D data. These include point cloud features [11, 13, 49], lifted multi-view image features [18, 20, 63). treating multi-view images as video sequences [35, 58]. The most closely related approaches are Video-3D LLM [58], which integrates positional informa- tion into the visual representation, and GPT4Scene [35], which constructs BEV maps through 3D reconstruction. In contrast to these methods, our model does not require any explicit dense 3D inputs.
Spatial Reasoning. Some work [5, 6, 14] has enhanced spatial understanding through large-scale synthesized VQA datasets for improved depth estimation. However, these methods primarily focus on static images, overlooking the dynamic and relational aspects inherent in complex spatial reasoning scenarios. To address this limitation, the Visual-Spatial Intelligence Benchmark (VSI-Bench) [50] transformation
was introduced, explicitly evaluating relational reasoning and egocentric-allocentee abilities. Recognizing the scarcity of data for such complex scenarios, subsequent works like SAT [37] and SPAR [54] have proposed data generation pipelines to synthesize spatial QA datasets for supervised fine-tuning. In contrast, our model elicits 3D geometry information from the video and injects it into the model architecture, improving its spatial reasoning capabilties.
2

---

The shoe cabinet the right back.
Video LLM
Geometry
2D Visual Tokens
Tolenization
3D Visual Geometry Encoder
2D Visual Encoder
Patchify
Question: How does the relative position the shoe cabinet change observer's original viewpoint when moves to center the office chair and faces bod?
Figure I: The architecture our VG LLM. The 3D visual geometry encoder processes a sequence of images to produce globally geometry-awant visual features, while the 2D visual encoder extracts semantic-aware visual features from each individual image. @ vividly shows that the latent 3D geometry tokens able to recover the 3D scene if with dense prediction head [42].
3 Method
We aim to model the 3D visual geometry in MLLMs without relying on explicit dense 3D input. To achieve this, as illustrated Figure introduce 3D visual geometry encoder extract rich visual-geometric features from multiple images and feed them into the MLLM backbone. Section 3.1 outlines the architecture design of our model, and Section 3.2 details the training process.
3.1 Architecture
Preliminary. Given a sequence of RGB images (4)=- and a natural language question Q, conventional MILM utilizes 2D visual encoder to encode these images into image tokens TVV E
] xc first, where I; E R and p is the patch size. Then MILLM backbone accepts (TY N==1 as its input and outputs the response. In this work, we choose Qwen2.5-VI. [2] as the MLLM backbone. Note that Qwen2.5- VL additionally compresses the image tokens to reduce the computational cost. Qwen2.5-VL groups spatially adjacent 2 x patches into single image token,
resulting in smaller set of image token input to MILM backbone, TV E
3D Visual Geometry Encoder. To model 3D geometric information like inter-frame correspon- dences within input frames, we employ 3D visual geometry encoder to extract such information.
This 3D visual geometry encoder produces 3D visual geometry features T. R _ from all input images (4) jointly. 3D visual geometry models [44, 38, 42] are natural candidates for the 3D visual geometry encoder, they trained capture inter- frramf correspondences and reconstruct 3D scenes without relying on additional 3D priors. These 3D visual geometry models comprise three key components: encoder for per-image feature extraction, fusion decoder for cross-frame interaction, and task-specific prediction heads for 3D attributes. Since we focus on feature extraction, which embeds 3D geometry prior information, rather than directly outputting 3D attributes, we leverage the encoder and the fusion decoder as our 3D visual geometry encoder. Specifically, we choose VGGT [42] to extract 3D visual geometry features given its superior performance in 3D tasks.
Visual Feature Fusion. We fuse both the image tokens (T.Y") yı and the 3D visual geometry features ( M before passing them into the MILM backbone. We first transform each 7,
into T.' E x xe which has the identical shape of T.", and then generate the geometry- augmented visual features T.S G + T.V". The transformation of T.C aligns with the spatial merging strategy in Qwen2.5-VL, where we concatenate spatially adjacent 2 x 2 features in 7.C and pass them to two-layer MLP output single feature in T.G'.

---

To this end, the final visual features {-- question Q into the MLLM backbone to produce the response.
Vi=1 are concatenated with the text embeddings of the
3.2 Training
Our VG LLM offers versatile framework designed to integrate 3D vision priors into MLLMs for various 3D tasks. In this work, we demonstrate the application VG LLM to 3D scene understanding and spatial reasoning tasks.
3.2.1 Applying VG LLM to 3D Scene Understanding
To validate the efficacy of the model in understanding and reasoning about 3D scenes, we adapt it to several 3D scene understanding tasks, i.e., 3D visual grounding. 3D dense captioning, and 3D video object detection. In contrast to previous work on understanding 3D scenes, our model relies solely on RGB images as input during both training and inference, without any 3D scene data as prerequisites. For 3D dense captioning, while we utilize pre-detected 3D proposals as the input, our model itself operates solely on RGB images. We directly learn 3D scene understanding tasks through text generation, which obviates the need for task-specific heads while maintaining simple next-token prediction objective during training. The unified text generation objective allows us to mix all tasks into a combined dataset for multi-task training. The detailed data format can be found in the appendix.
Coordinate System and Representation. Since we do not utilize any ground truth information of the 3D scene, we follow VGGT [42] to employ the coordinate system of the first frame the base coordinate system. All coordinates are transformed into this base coordinate system (except for 3D visual grounding, which represents the bounding box in each frame's coordinate system). All the numbers are expressed in plain text with 2 decimal places.
3D Visual Grounding. We follow previous work [54] to formulate the 3D visual grounding task as 3D video grounding problem, which involves locating the frame index where the object appears and its 3D bounding box in the corresponding frame's 's coordinates in one feed- feed-forward pass. Unlike SPAR [54], which generates axis-aligned boxes, our method directly predicts 3D-oriented bounding boxes, making it more suitable for real-world scenarios. Specifically, given the 3D scene represented In and natural language query Q, our model is designed output both the frame index and the bounding box in the form (I, y. w, h, d, tb, A, Φ). where (z, 2) is the center coordinate, (w, h d) is the object size, and (si, 0, Φ) is the rotation angles.
by frames {/1, 12,
3D Dense Captioning. We follow the recent work [58, 63] that decomposes the 3D dense captioning task into two phases, i.e., detecting 3D object proposals with an off- the-shelf detector and generating object descriptions based on object coordinates. Specifically, given frame sequence { Iz, we prompt our model to "describe the object located at (Ir, y, x) in detail", where (ær, y. 2) is the box center of the object to be described. This setup ensures fair comparison with prior work and effectively evaluates the model's 's ability to understand positional and spatial relationships.
}
3D Video Object Detection. To investigate the capability of handling egocentric-allocente trans- formation, we set up a 3D video object detection task based on the ScanNet [15]. Unlike previous benchmarks that focus monocular multi-view detection [45], our task requires models to detect all objects throughout the video in a unified coordinate system, without relying on explicit camera parameters depth information. Since some objects aren't 't visible in the first frame, the model must track changes between frames, estimate camera movement, and convert object locations from a global view the camera's 's perspective. Specifically, given continuous video frames [2, 12. In the model is tasked to detect all objects [(b2, (1), "... .... in the form (x., y, 2, w. h, d, v,0, Φ) represents its bounding box in the unified coordinate system and e is its category.
(bm, } that appear in this video, where each b;
3.2.2 Instruction Tuning for Enhanced Spatial Reasoning.
Previous work [53, 62] has highlighted that existing pre-training and instruction tuning datasets often lack sufficient spatial-related phrases in their annotations, consequently hindering the spatial reasoning capabilities of MI.LMs. To overcome this limitation, we leverage the SPAR-7M [54] dataset for instruction tuning. SPAR-7M is comprehensive spatial reasoning dataset curated from three

---

richly annotated 3D datasets: ScanNet [15], ScanNet++ [51], and Structure3D [60]. It encompasses 33 diverse tasks, ranging from fundamental perception to mid-level viewpoint transformation and high-level scene imagination. As some work has shown that post-training on specific datasets can sometimes degrade the original performance on general benchmarks, we also incorporate a visual instruction tuning dataset, the LaaV--ieeo---- 178K's LLaVA-Hound split [57], into our training pipeline to preserve the generalization capability.
4 Experiments
In this section, we first present the implementation details of our model, followed by an evaluation of our model's performance on 3D scene understanding tasks in Section 4.1, including 3D visual grounding, 3D dense captioning, and 3D multi-view detection. Next, we provide a comprehensive comparison with state-of-the-art methods on spatial reasoning benchmarks and generic multimodal benchmarks in Section 4.2. 3D scene understanding emphasizes 3D perception, while spatial reasoning focuses on interpreting and reasoning about spatial relationships in video. We train two models for 3D scene understanding and spatial reasoning tasks separately for fair comparison. Finally, we conduct a comprehensive analysis in Section 4.3 to reveal the effectivess of our model's core aspects: the integration of 3D geometry, the feature fusion strategy, and the data composition.
Implementation and Training Details. Our models are built upon two sizes of Qwen2.5-VLL ----2B and 7B [2]. and integrated with VGGT- IB [42] as the 3D geometry encoder. We trained our models for one epoch on a mixed dataset, detailed as followed in this section, and employed Adam with a batch size of 64 and a warmup ratio of 0.03. During the warmup phase, the learning rate was gradually increased to le-5 before linearly decaying to O. In each training step, batch was randomly sampled from a single source from the mixed dataset. During training, the MLLM's s visual encoder, the integrated 3D geometry encoder, and the multimodal connector are frozen, while the MLLM backbone remains unfrozen. All experiments were conducted on H100 80G GPUs. For the 4B and 8B models, the training took 9 and 12 hours for 3D scene understanding, and 7 and 9 hours for spatial reasoning, respectively.
4.1 3D Scene Understanding Tasks
4.1.1 Setting
Datasets and Benchmarks. To evaluate the versatility of our model across diverse 3D scene understanding tasks, we employ a multi-task learning approach on a combination of datasets.
31) Visual Grounding. We leverage the ScanRefer [7] dataset, which provides 36,665 object de- scriptions paired with axis-aligned bounding boxes across 562 indoor seans. We follow SPAR [54] reformulate 3D visual grounding as 3D spatial-temporal video grounding, aiming to locate the target object's bounding box in camera coordinates along with its corresponding frame index. To determine the relevant appearance frame, we utilize the visible object annotations from Embodied- Scan [45]. We match the target 3D bounding box with those in EmbodiedScan based on their IoU and subsequently pick the optimal frame by comparing the 2D projection areas.
3D Dense Captioning. We utilize the Scan2Cap benchmark [8] for 3D dense captioning, which requires generating descriptive captions for all objects within scene. Following prior work [22, 58, 63], we use Mask3D object proposals extracted by LEO [22] and task our model with generating captions conditioned on their center coordinates. To better leverage the visual geometry, we transform all object center coordinates to the coordinates of the first captured frame.
3D Video Object Detection. For 3D video object detection, we curated a dataset from EmbodiedScan [45] consisting of consecutive frames and their corresponding visible object annotations in indoor scenes. Each sample comprises four consecutive frames sampled at I FPS, with all associated object instances transformed to the coordinate system of the initial frame. Consistent with EmbodiedScan, we divide the data into 958 training and 243 evaluation scenes. Within each scene, we randomly select 150/10 samples for training/cvaluation.
Comparison Baselines. For the tasks of 3D visual grounding and dense captioning, our evaluation includes comparisons with both task-specific expert models and general-purpose 3D models. Specifi- cally, for expert models, we compare our method against ScanRefer [7]. MVT [23]. and ViL.DRel
[10] on ScanRefer [7]. On the Scan2Cap dataset [8], we include Scan2Cap [8], 3DICG [4], D3Net [9]. and DDETR [12] for comparison. Furthermore, we compare approach with the
5

---

generalist models. including Chat-3D v2 [21], Grounded 3D-LLM [13], LL3DA [1I]E [22], LLDVA-3D [63], and Video-3D LLM [58]. For 3D video object detection, we compare our method with baseline (Qwen2.5-VI-33B [2]) that does not incorporate 3D geometry information.
Evaluation Metries. For ScanRefer [7], report the accuracy at IoU thresholds of 0.25 and 0.5. For Scan2Cap [8], we use CIDEr (C), BLEU-4 (B-4), METEOR (M), and ROUGE (R) scores. The "@0.5" suffix indicates that these metrics computed only for objects detected with an IoU of 0.5 or higher against the ground truth. Lastly, for 3D video object detection, we report the accuracy, recall, and F1 score for 20 common object classes at IoU threshold of 0.25.
4.1.2
Results on 3D Visual Grounding
3D Scene Input
0.c
Model
3D Scene Input
C000.5
R-4980.57
M6N.51
R4F0.51
ScanReler [7]
37.3
24.3
ScoehCap [S]
19.1
23.1
23.0
44.
MVT [23]
ViL.DRe [10]
3D-LLM
40.8
47.9
30.3
31.7
31.3
LESDA []]
D3Ner [9]
3D3CXI (4)
Vine&Cap DEEeappDDTRRRRC (12)
61.8
65.2
62.6
49.5
34.5
51.0
35.7
96.8
25.7
26.2
24.2
26.0
54.4
$3.9
50.8
$5.0
Char:31
35.9
30.4
Chat 3D v2 [2]]
6399
31.8
Grounded 3D-LLM
Grounded 3D LLM
70.2
35.0
ChatSeene [21]
LEO LEO [22]
72.4
38.2
27,9
58.1
LLaVA-3D [63]
54.1
42.4
Chat-Seene [2]]
77.1
36.5
Video-3D LLLM [S8]
58.1
51.7
LLaVA [63]
79.2
41.1
30.3
63.4
SPAR|[5]
K
31 (48.8)
12.4 (43.1)
Video 3D LLM (59)
80.0
40.2
28.5
61.7
LLM-4B (Ouns)
(53.5
(47.5)
VG LLM 4B (Ours) VG LLM HB (Ours)
78.6
80.9
28.6
62.4
LLM-SB (Ouns)
(50.9)
80.0
41.5
28,9
62.6
Table 1: The quantitative results on Scan-
Table 2: The performance on Scan2Cap. VG LLM
Refer. The content in "O" indicates results with proposal refinement
captions 3D object proposals using RGB cues only.
This is black chair. it pushed away from the desk.
IKE
The table is the opposite side from the refrigeratior the table brown and hollow restangle.
are black middle
black
Figure 2: Qualitative results for 3D visual grounding. The ground truth and prediction masked in blue and green, respectively. The predicted boxes are directly generated by our model without the refinement process.
In addition the directly predicted bounding boxes, we also follow SPAR [54] calculate the refined results with proposal refinement process?.
VG LLM demonstrates strong 3D grounding capabilities from monocular RGB videos. The performance on ScanRefer shown Table Our 8B model achieves an accuracy of 41.6% at an loU threshold of 0.25, significantly outperforming SPAR's 31.9% by 9.7 percentage points. Moreover, with the integration of proposal refinement technique, Acc@0.25 increases substantially to 57.6%, making our results highly competitive with state-of-the-art methods like Video-3D L.M.. This demonstrates that 3D visual grounding can be effectively approached in a video grounding manner.
Qualitative results for 3D visual grounding. Figure 2 illustrates the qualitative results the 3D visual grounding task. For visualization purposes, the predicted and ground-truth 3D bounding boxes projected onto their respective 2D image planes. Our method effectively handles spatial
For proposal refinement, we compare the predicted box against all proposals detected by Mask3D and select the with the highest loU.

---

relationships between objects (e.g., away. opposite, next to), accurately identifying the corresponding frame index and generating the 3D-oriented bounding box.
4.1.3 Results 3D Dense Captioning
Our model processes 3D object proposals generated by an off-the-shelf' 3D object detector. Once these objects are localized, the model generates descriptions using only the RGB images and the center coordinates of the detected objects, without requiring explicit depth or geometrie data as additional inputs. This approach achieves previous state-of-the of-the-art art methods such as LLaVA-3D and Video-3D LLM. These results validate the effectiveness of model for 3D dense captioning when conditioned 3D proposals.
of 80.0 C@0.5 and 41.5 B-4@0.5, comparable to
4.1.4 Results on 3D Video Object Detection
20 Common Classes
Model
chair
cabinet
table
bin
couch
bed
bathtub
toilet
Pas
Rgs
Flas
4-Frame Setting
Qwen2,5-VL-3B
37.7
10.2
35.0
23.1
39.0
64.8
32.4
68.8
32.6
27.9
30.0
Visual Geometry (VG LLM-4B)
13.1
71.2
33.5
41.7
A Improvement
+12.0
+2.9
+6.3
+16.1
+6.4
+1.1
+14.6
+9.1
+7.8
+8.2
Qwen2.5-VL-77B
41.2
11.6
30.2
68.2
36.6
34.6
32.5
Visual Geometry (VG LLM-8B)
54.0
17.1
46.5
39.8
47.0
74.1
42.1
82.5
43.4
39.6
41.2
A Improvement
+12.8
+5.5
+10.0
+9.7
+5.9
+5.9
+5.5
+13.8
+8.8
+8.6
+8.7
6-Frame Setting
Qwen2.5-VL-3B
32.8
7.8
31.3
20.9
32.2
58.8
36.5
66.1
27.8
24.1
25.7
+ Visual Geometry (VG LLMM4B)
41.6
12.4
39.8
33.1
45.0
70.2
33.8
80.6
39.7
34,0
36.4
A Improvement
+8.8
+4.6
+8.5
+12.2
+12.8
+11.4
-2.7
+14.5
+11.9
+9.9
+10.7
Qwen2,5-VL-7B
36.1
10.6
32.7
25.0
40.7
64.6
38.4
68.6
31.8
28.0
29.6
+ Visual Geometry (VG LLM-SB)
48.7
17.9
44.8
38.5
46.4
75.8
40.4
43.5
38.7
40.8
A Improvement
+12.6
+7.3
+12.1
+13.5
+11.2
+2.0
+14.6
+11.7
+10.7
+11.2
Table The results on 3D video detection. The reported object categories follow the monocular detection in EmbodiedScan, and we report the average Fl score per category.
Buseline
VG-MLLM
Baseline
VG-MLLM
Figure 3: Qualitative results for 3D video object detection.
As shown in Table 3, we follow the monocular detection of EmbodiedScan [45] to evaluate the performance for categories that are common daily life.
geometry significantly boosts cross-frame detection performance. Qwen2.5-VL. [2], upon being fine-tuned on detection data, achieves promising performance common object categories. Incorporating 3D geometry [42] yields substantial improvement across all evaluation metrics. In the 4-frame setting. this approach elevates the average Fl score for the 4B model by 8.2% (from
7

---

30.0% to 38.2%) and for the 8B model by 8.7% (from 32.5% to 41.2%). This performance gain is attributed to the model's improved ability comprehend the ggocentric-allooeene transformation between frames through the utilization of 3D geometry. Figure 3 provides qualitative results on the 3D video object detection task.
Integrating 3D geometry enhances the model's robustness to variations in frame count. Table 3 shows that our method, although trained on 4-frame sequences, maintains strong performance when evaluated longer sequences during inference. In contrast, the baseline's performance drops noticeably, underscoring the robustness of our approach to variations in the number of frames.
4.2 Spatial Reasoning Benchmarks
Obj Connt
Abs. Dist.
Obj Size
Room Size
Rel. Dist
Rel. Dir.
Romte Plan
Appr. Order
Model
Numerical Answer
Multiple- Cholle-Chhcce Answer
Proprietary Models (AP1))(APl)
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5-Ra 1.5-Flash
42.1
49.8
30.8
53.5
54.4
37.7
41.0
31.5
37.8
Gemini 1,5 Pro
45.4
56.2
30.9
64.1
43.6
51,3
46.3
36.0
34.6
Open-source Models
1L2-
34.6
23.1
28.7
48.2
39.8
36.7
30.7
29.9
39.6
Intern MMerrVLL-1105
36.0
34.9
26.9
46.5
31.8
42.1
32.2
34.0
39.6
Long V---8B
21.6
29.1
9.1
0.0
29.6
32.5
25.5
VILA-115-100
31.2
22.4
24.8
48.7
22.7
40.5
25.7
31.5
32.9
VA-7B
33.1
LeXVA-NeXT
Video-72B
40.9
48.9
22.8
57.4
35.3
42.4
36.7
35.0
48.6
LLaVA-One Vision-773
40.2
43.5
23.9
57.6
37.5
42.5
39.9
32.5
44,6
Sputial-Enhuneee Models
SAT-LLVV Video-
47.3
41.1
37.1
36.1
40.4
SPAR-8B
VG M144 (Ours)
47.3
66.0
37.8
55.2
59.2
44.6
45.6
33.5
364
VG LLM-8B (Ours)
50.7
67.9
37.7
58.6
62.0
46.6
40.7
32.4
59.2
Table The comparison with state-of-the-eee models on VSI-Bench. Spatial-Enhanen Enhanced Models are models that are specialized for spatial reasoning.
4.2.1 Setting
Datasets and Benchmarks. To fully leverage the 3D knowledge inherent in the 3D visual geometry encoder, we train our model on a dataset sampled from SPAR-7M [54] and the LLa -oundd split of the LLaVA-Video- 17KK [57]. In our work, we sample only 234K and 63K data points from SPAR-7M and the VLaoo-o split of Video--i respectively, which constitute 3% and 25% of the original datasets.
We first evaluate our method on a video spatial reasoning benchmark, VSI-Bench [50], which evaluates the egocentric-allocemn
transformation and relational reasoning capabilities of MLLMs. Then we test our model on an image spatial- related benchmark i.e., CV-Bench [40]. CV-Bench [40] assesses 2D understanding through spatial relationships and object counting, while its 3D evaluation focuses on depth ordering and relative distance perception.
Comparison Baselines. We compare our model with state-of-the-ard proprietary and open-source MLLMs (e.g., GPT-40, Gemini-1.5-Prae LLLVA-NeeT--inne We also include two spatial-enhanced MILMs, SAT-LLaVA--Vdeo-- [37] and SPAR-8B [54], which are fine-tuned on datasets targeted for spatial reasoning.
[1, 24, 26, 27, 32, 34, 39, 40, 46, 57].
Evaluation Metrics. For VSI-Bench, we adopt accuracy for multiple-choice tasks and Mean Relative Accuracy (MCA) for numerical tasks. MCA calculates the average accuracy across range of confidence thresholds, where prediction considered correct if its relative error within a specified threshold. For CV-Bench and other generic multimodal benchmarks, report accuracy.
4.2.2 Results on Spatial Reasoning Tasks
VG LLM achieves state-of-the-are performance on VSI-Bench. A detailed comparison with leading models is presented in Table 4. As shown in the table, our 4B model achieves an impressive average accuracy of 47.3%, outperforming all competitors, including the leading proprietary model, Gemini-1.5 Pro. Futhermore, our 8B model sets new state-of-ther -het--rrt- performance with an average accuracy 50.7%. These results underscore the model's strong capability in understanding and reasoning from monocular videos.
8

---

Model
(D)
Arg.
Proprietary (API)
GPT-4V [34]
GPT44 [24
64.3
74.8
83.0
73.8
69.1
73.9
Qweac 5- VL.38
VGLLM-1
Qwea2
VOLLM-30
Gemini-1. 1.5-Flshh [9]
70.9
7L8
71.4
Gemini-1.5 5.P
77.1
77.6
Benchmark
Open sounce
Volen-MMFF [
60.1
574
62.9
98.3
Mini Gemini IDD 34B
[27]
71.5
79.2
75.4
Vida-MMFF [17]
60.9
915
61.1
53.9
LLaVA-NeXT-31B [26]
73.0
74.8
73.9
BLINK [19]
46.6
48.9
53.9
51.5
Cambriza- 1-34B [40]
74.0
99.7
76.9
TangClongiooe
62.2
63.9
71.
67.5
SAT-LLaVA- Video-7B [57]
73.0
83.8
78.4
NeutAAu [48]
77.3
74.5
514
79.3
[54]
VG LLLM-4 (Ouns
71.3
87.7
79.5
Table 6: Comparison of model performance on
V--LM-88 (Ours)
72.2
91.1
81.7
generic multimodal benchmarks. To ensure fair comparison, we evaluate Qwen2.5-VI with the resolu- tion and sampling rate matched to those of VG LLM.
Table 5: The comparison with state-of-
the-art methods on CV-Bench.
Our model demonstrates generalization across distinct data sources. To access the general- ization capability of our method, we evaluate approach spatial reasoning benchmark with different data sources, CV-Bench. This benchmark constructed by repurposing the traditional CV datasets (i.e., ADE20K, COCO, and Omni3D) to vision-centric MILM benchmark. As illustrated Table 5, our method achieves the highest accuracy on 3D tasks at 91.1% 11. These results demonstrate model can also generalize well across the spatial reasoning benchmarks with -fonnn data sources.
4.2.3 Results on Generic Multimodal Benchmarks
Enhancing spatial understanding incurs negligible loss general multimodal performance. Our model integrates 3D geometry information and is fine-tuned on spatial reasoning datasets to bolster its spatial understanding capabilities. As presented in Table 6, these enhancements slightly compromise VG LLM's performance on general multimodal benchmarks when compared to the baseline model. VG LLM-4B (+1.7%). This suggests that augmenting spatial reasoning capabilities can not only preserve but also enhance performance specific multimodal tasks.
achieves improvements BLNK (+2.3%) and TempCompasswc
4.3 Analysis
4.3.1
3D Scene Understanding
Type
ScanRefer
Scan2Cap
3D Video Detection
Acce0..5
Acc600.5
CIDER@0.5
Pas
Fl
No Additional Info. (Baseline)
31.9 (49.9)
9.3 (43.8)
$8.0
36.3
32.6
27.9
30.0
Cross Altn (1 Layer)
33.7 (51.3)
10.7 (45.2)
74.7
40.1
38.0
33.6
35.4
Cross-Attn Layers)
34.4 (51.3)
10.5 (45.2)
75.7
40.2
38.5
44.0
35.4
CLL
27.7 (47.0)
(41.3)
75.7
39.9
37.1
32.5
34.4
Add (Ours)
36.4
11.8 (47.5)
78.6
40.9
41.7
35.7
38.2
Pred Camera Info.
32.1 (50.0)
9.9 (43.9)
$6.8
36.1
33.3
28.1
30.3
Depth Infu.
Pred Point Info.
31.7 (49.7)
9.6 (43.8)
67.7
38.2
33.0
28.5
30.4
(Depth Camern)
(49.8)
(43.7)
29.1
Latent 3D Genmetry (Ours)
(63.555355)
11.8 (47.5)
78.6
40.9
41.7
35.7
38.2
Table 7: Ablation study of the effects of 3D visual geometry modeling. All models are fine-tuned using the same training data and built upon Qwen2.5-VL-3B.
In this section, we conduct further analysis the feature fusion strategies and the types of added signals. To investigate the effect of feature fusion strategies, we have experimented with several feature fusion strategies, including 1) "Cross-Attn" which consists of multiple blocks of cross- attention modules and MLPs with skip connections. 2D visual tokens serve queries, while 3D visual tokens serve as keys and values. 2D positional embeddings are added both the queries and keys; 2) "Concat+ML.P first concatenates the 2D and 3D visual tokens along the feature dimension, followed by an MI.P to transform them into the text embedding space; 3) "Add" is the strategy employed in paper, which directly adds the 2D and 3D visual tokens patch level.

---

Besides the visual geometry feature adopted in our model, VGGT can also directly predict the camera poses (Pred Camera Info.), the depth maps (Pred Depth Info.), and the point maps (Pred Point Info.) of each image frame. For these predicted spatial signals, we adopt two-layer MI. to transform them and add to the corresponding 2D visual token. All models are trained and evaluated following the datasets and benchmarks in Section 4.1.
Directly adding the geometry features obtains the best results among all compared feature fusion strategies. As Table 7 illustrates, the "Cross-Attm" strategy significantly improves performance over the baseline. Moreover, increasing the number of layers to three yields even greater performance pe
gains. While "Concat+MLP" performs worse than the baseline ScanRefer, it outperforms it on Scan2Cap and 3D video object detection. Nevertheless, "Add" surpasses all other comparison methods despite its simplicity.
Incorporating point maps enhances 3D scene understanding, while camera or depth information offers no clear benefits. After fine-tuning on 3D downstream tasks, the baseline model Qwen2.5- VL shows enhanced 3D scene understanding. However, it continues to struggle with accurately localizing and describing 3D objects, as well as learning spatial transformations across different frames. As illustrated in Table 7, incorporating predicted camera or depth information alone provides no clear benefit. In contrast, the inclusion of predicted point information demonstrates significant advantage for the 3D dense captioming task, boosting the CIDEr@ 0.5 score on Scan2Cap from 58.0 67.7 and the BLEU-4@00.5 from 36.3 to 38.2. It also provides a marginal improvement in 3D video object detection. These results indicate that current MLLMs still lack adequate spatial modeling in both data curation and architectural design.
Visual geometry features are more helpful than predicted spatial information. Although the point maps predicted directly by VGGT can enhance the spatial understanding ability of Qwen2.5- VL, these signals relatively sparse and may contain errors. In contrast, directly using the visual geometry features provided by VGGT not only incorporates these spatial signals simultancously but also reduces the impact of noise on performance, thereby achieving better results.
4.3.2
Spatial Reasoning
VSI-Bench
CV-Bench (3D)
Data
Model
Obj. Count
Abs, Dist.
Objj Size
Room Size
Rel. Dist.
Rel. Dir.
Rouiic Plan
Appr. Onder
Avg
Depth
Distance
Arg
QQw-2255VL--7B
25.4
10.4
36.4
29.1
38.2
37.9
30.9
27.5
29.5
84.5
76.5
80.5
S1
Qwen2.5-VL. LBV
LLM-8B
67.9
46.6
40.7
38.9
59.2
92.3
87.1
91.1
Loee22.5VLL-78
21.3
48.3
68.5
65.5
68.1
77.5
40.2
16.7
56.6
66.8
74.5
70.7
S2
VG LLM SB
71.7
53.8
68.8
62.1
63.8
83.0
44.3
18.4
58.2
66.8
73.3
70.1
81.82
Qwen2.5 VL 7B
70.2
51.0
69.8
64.4
67.0
79.1
45.4
31.6
59.8
87.0
83.8
65.4
VGL-LMBB
71.4
83.2
32,5
92.0
90.8
Table 8: Ablation study on spatial reasoning tasks. indicates testing in zero-shot setting. S1 is the training data detailed in Section 4.2.1, and S2 refers to the VLM-3R dataset.
We then investigate the effects of fine-tuning and visual geometry in Table 8. The table illustrates that fine-tuning on our mixed dataset (SI) leads to substantial performance gain in measurement estimation (rows 1 vs. 2). In addition to the data used in this paper (S1), the VLM-3R dataset [16] (S2), which is specially curated for the task types in VSI-Bench, is also incorporated in our analysis. After incorporating the 3D geometry into the model architecture, the results show consistent gains with the different data compositions, i.e., S1. S2 and S1+S2.
Conclusion
We present a novel framework to enhance MI.LMs' 3D spatial understanding capability, which incorporates 3D visual geometry encoder to provide latent 3D geometric information given only video inputs. While being straightforward, our extensive experiments show that our model can outperform larger spatial-enhanced models on various 3D seene understanding tasks and spatial reasoning benchmarks, without relying on any explicit 3D scene input.
Acknowledgements This work is supported by National Key R&D Program of China (Project No. 2022/D0161 200, 20227D0161201). This work also supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 24200223).
10

---

References
[1] Authropic. Introducing the next generation of claude. http::/wwwwwaatarrropo...cooeeee
claude- 3-mmmly 2024. 8
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang, et Qwen2.5-vl technical report, arXiv:2502.13923, 2025. 1,
2, 6.6,7.67
[3] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. RT-2: vision-language- action models transfer web knowledge to robotic control. In CoRL, 2023.
[4] Daigang Cai, Lichen Zhao, Jing Zhang, Lu Sheng, and Dong Xu. 3djeg: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In CVPR, 2022. 5,
[5] Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models.
a406v:2406. 13642, 2024.
[6] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In CVPR, 2024.
[7] Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in RGB-D D scans using natural language. In ECCV, 2020. 2, 5, 6
[8] Dave Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X. Chang, Scan2cap: Context- aware dense captioning RGB-D scans. In CVPR, 2021. 2, 5,
[9] Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X. Chang. net: A unified speaker-listener architecture for 3d dense captioning and visual grounding. In ECCV, 2022. 5,
[10] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. In NeurlPS, 2022. 5,
[11] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LL3DA: visual interactive instruction tuning for omni-3d understanding, reasoning, and planning. In CVPR, 2024. 6
[12] Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In CVPR, 2023. 6
[13] Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang. Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. 2xXiv22405.10370 2024. 2, 6
[14] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifci Liu. Spatialrgpt: Grounded spatial reasoning in vision- languaela models. In NeurlPS, 2024. 2
[15] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias NieBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 5,5
[16] Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, et al. Vlm-3r: Vision-language models augmented with instruetion-aligned
reconstruction. arXiv preprint arXiv:2505.20279, 2025. 10
[17] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms video analysis. arXiv:2405. 200:2205..2275, 2024. 9, 22
[18] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual reasoning. In WACV, 2025.

---

[19] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang, Xudong Lin. Dan Roth, Noah A. Smith, Wei-Chi Ma. and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. In ECCV, 2024. 9, 22
[20] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In NeurIPS, 2023. 1,
2,
[21] Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In NeurIPS, 2024. 1,6
[22] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang. Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In ICML, 2024. 1, 5, 6,
[23] Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In CVPR, 2022. 6,
[24] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv:2410.212777 2024. 2, 8,9
[25] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source lision-lngguagg-attion model. In CoRL, 2024.
[26] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. 0xXiv:2408.03326 2024. 1, 2, 8,
[27] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 188/4, 2024. 8,
[28] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye- e-yeeviiww representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022. 1
[29] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: on pre-training for visual language models. CVPR, 2024. 1,
[30] Benlin Liu, Yuhao Dong Yigin Wang, Zixian Ma, Yansong Tang. Luming Tang, Yongming Rao, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondences boost spatial-temporal reasoning in multimodal language model, 2024.
[31] Benlin Liu, Yuhao Dong, Yigin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. 2rXiv:2408.00755 2024. 2
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023. 2, 8
[33] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? In ACL, 2024. 22
[34] OpenAL. GPT-4 technical report. arXiv:2303.08724 2023. 8.9
[35] Zhangyang Qi. Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with lasion-language models. arXiv:2501.01422 2025. 1,2
[36] Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. Spatialvla: Exploring spatial representations for visual-language-actio model. arXivv2501.15830 2025. 1
12

---

[37] Arijit Ray. Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Dynamic spatial aptitude training for multimodal language models, 2024. 2, 8,
[38] Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander G. Schwing, and Zhicheng Yan. Mv-dust3r4: Single-stage scene reconstruction from sparse views in seconds. arXiv:2412.06974, 2024. 2,
[39] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530, 2024. 1, 2, 8,9
[40] Peter Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu. Ziteng Wang. al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurlPS, 2024. 1,
2, 2,8,99 9, 22
[41] Haochen Wang, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiangyu Zhang, and Zhaoxiang Zhang. Ross3d: Reconstructive visual instruction tuning with 3d-awareness, 2025.
[42] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: visual geometry grounded transformer. M.Ki:25033.1665// 2025. 2, 5,
[43] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision- saiinnla model's perception of the world at any resolution. 1xxxv:2249.121999 2024. 1.2
[44] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024. 2,3
[45] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied AI. In CVPR, 2024. 2, 4. 7,22
[46] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. In NeurlPS, 2024. 8
[47] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In ICLR, 2024.
[48] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, 2021.
[49] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointlim: Empowering large language models to understand point clouds. In ECCV, 2024. 2
[50] Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Think- ing in space: How multimodal large language models see, remember, and recall spaces. arXiv:2412.14171, 2024. 1, 2, 8, 22
[51] Chandan Yeshwanth, Bheh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, 2023.
[52] Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jac Lee, and Yi Ma. Inves- tigating the catastrophic forgetting in multimodal large language models. arXiv:2309.1033 10313, 2023.
[53] Huanyu Zhang. Chengzu Li, Wenshan Wu, Shaoguang Mao, Yan xia, Ivan Vulić, Zhang Zhang, Liang Wang, Tieniu Tan, and Furu Wei. call for new recipes to enhance spatial reasoning in milms, 2025. 4
13

---

[54] Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu- Jie Yuan, Xinyue Cai, Guowei Huang, al. From flatland to space: Teaching vision-language models to perceive and reason in 3d. arXiv:2503.22976, 2025. 2, 6, 8,9
[55] Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based VLM plans the next step for vision-and-langusee navigation. In RSS, 2024.
[56] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. 22
[57] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv:2410.02713, 2024. 5, 8
(58) Duo Zheng. Shijia Huang, and Liwei Wang. Video-3d LLM:: learning position-aware video representation for 3d scene understanding. CVPR, 2025. 1, 4, 6,
[59] Duo Zheng, Shijia Huang. Lin Zhao, Yiwu Zhong. and Liwei Wang. Towards learning a generalist model for embodied navigation. In CVPR, 2024.
[60] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, 2020.
[61] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-langun navigation with large language models. AAAI, 2024.
[62] Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, Aditya Nagachan- dra, Di Chang, Dongdong Chen, Eric Xin Wang. and Achuta Kadambi. Vlm4d: Towards spatiotemporal awareness vision language models. 2025. 4
[63] Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang. and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering imms with 3d-awareness. arXi::2409.1812, 18125, 2024. 2,
4, 5, 6, 22
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state our main contributions. Our experimental results also support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have section to discuss the limitations of our work. Please refer to Appx.
E.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have included the detailed model design (Sec. 3.1), implementation details (Sec. 4) and evaluation details (Appx. A.1)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided instructions on data access and preparation, including how to access the raw, preprocessed, intermediate, and generated data. Please refer to Appx. B. The data and model weights will publicly available upon paper publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce cc/
Sublic/guides Seaidees/oooSuummieee crriiisssoPooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ stips.cc/publice hwt//uuieeesseee /CodeSubmiiee hww.simmissioooooooioioooooo onliiicyllly ic) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included detailed experimental setup in Appx. A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report the type of computing resources, memory, and time required to reproduce our results in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : //tp://///eeee http//nuripp.pee
Answer: [Yes]
Justification:
Our research conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have dedicated section to discuss the societal impact of our work. Please refer to Appx.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our research builds upon existing datasets rather than creating new ones from the internet. So our work does not pose such risks to the best our knowledge.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used in our work are commonly used datasets with open access. We have adhered to their licenses and provided citations to give them credit.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The data and model weights will be publicly available upon paper publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method in our research does not involve LLMs as any components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee wwc/ooneeeencce///000000 /LLM) for what should or should be described.
21

---

A Experimental Setting and Details
A.1 Evaluation Details.
We utilize the LMMs-Eval [56] to evaluate our method. LMMs-Eval offers flexible functionalities that support the customization of specific tasks and model interfaces, thereby standardizing the evaluation process of MLLMs. For inference, we employed greedy sampling generate model outputs.
Spatial Reasoning and General Multi-modal Benchmarks. For benchmarks already integrated into LMMs-Eval, e.g., VSI-Bench [50]. Vidco-MME [17], and TempCompass [33], we utilize their original default configurations. For benchmarks not yet implemented in LMMs-Eval. i.e., BLINK [19] and CV-Bench [40], we customize the task configurations and metrics according to their official implementations. For each input video in video-based benchmarks, we uniformly sample 32 frames from the entire duration.
3D Visual Grounding. For 3D visual grounding, the model is tasked with locating the frame index where the target object appears and the oriented bounding box within its coordinate system. Given a predicted frame index and bounding box, we first transform the bounding box to the world coordinate system. Then, the ground truth bounding box is extended to an oriented bounding box by appending (0, O, 0) represent its orientation. Finally, calculate the Intersection-ovee Enve-Union (IoU) between the predicted and ground truth bounding boxes. In the proposal refinement process, we utilize the Mask3ID-detected object proposals extracted by LEO [22], and select the proposal that has the highest IoU with the predicted box.
3D Dense Captioning. For 3D dense captioning, we follow previous work [63, 58] to decompose the task into two stages, i.e., detecting all object proposals and generating object descriptions based on object center coordinates. Specifically, we first leverage the Mask3ID-detectee object proposals extracted from LEO [22] to obtain the box's center coordinates within the coordinate system of the reference frame. This can be achieved by multiplying the coordinates by the extrinsic matrix [45]. Then, the model is asked to generate descriptions based on the coordinates through greedy sampling. Lastly, we calculate the CIDEr, BLEU, Rouge, and METEOR scores utilizing the COCO caption evaluation toolkit'.
3D Video Object Detection. For each predicted bounding box, we find its best match among the unused ground truth boxes of the same category through greedy matching. Specifically, the IoU with all unused ground truth boxes of that category and select the one with the highest IoU if it exceeds threshold (e.g.. 0.25). Once a ground truth is matched, it is marked as used and cannot be matched again. Finally, for each category, the precision, recall, and Fl score are calculated based on the number of true positives, false positives, and false negatives.
caleulate
Additionally, we provide the prompt for 3D scene understanding tasks in Table 9.
B Data preparation
All datasets used in this research are publicly available, and we will provide the details for the data preparation in this section.
SPAR-7M. We follow the official codebase to mix data and draw visual markers on the input images. Since the navigation type contains images of varying lengths, we discard annotations of this type for simplicity.
LLa
(LLaVA-Hound Split). For each input video, we sample frames with a sampling rate of 2 FPS, while constraining the total frame count between 4 and 8 through adaptive sampling.
3D Scene Understanding. For both ScanRefer and Scan2Cap, we uniformly sample 32 frames for each scene. For 3D video object detection, each entry in the training contains consecutive frames sampled at FPS.
http://///ggtaabb.ooo/ll///////a 'https//ggithbb httpe::///ithuub.co//oeeeeee cww/fudan wvg/spar
22

---

3D Visual Grounding
Xuiarr Frame-O: ImaageFFrunee 1:ane--I:: 2<<mmage>Prame-2:
Frame-2: <image> Frame-3; <image>....
Localize the first clear frame in video showing object described the text.
Text: There is a beige wooden bookshelf. placed next lo another bookshelf.
Oulput ISON dictionary with the frame index in "frame" and its 3D bounding box in "bux 3d' in the frame's coordinates.
Xagent:
json
("frame": : 12, "bbox 3d": [-0.63, -083, 2.43, 3.0. 0.59, 2.35, 2232, 1.18, 3.05])
3D Dense Captioning
Xutar: <image><<n iiaaammmmmagmemaeeeeemmm
Carefully watch video and descrihe the ohject located 1-0.89, -0.74, 2,45] in detail.
Xagent: white cabinet in the corner of the room. in the direction from the door and from the inside. it will be the left, there is small brown table on the left. side of the cabinet and smaller tahle the right side of the cabinet.
3D Video Object Detection
Xueer: <image><<magee (aaggee>imaee<immmaaaaa
Detect the 3D bounding boxes in the camera coordinate system of the first frame.
Output json list where each entry contains the object name in "lahel" and its 3D bounding box in "box_3d" The 3D bounding box format should he "k_eette, y_center, 2_center, x size, y_size, s_size, yaw, pitch, rolll).
Xaget:
json
Cllbbel": "hag", "bhox_3d" [0.0, -0.3, 1.0, 0.26, 0.26, 0.15, 1.67, 0.96, -2.98}3
1
Table The prompt for scene understanding tasks. For 3D visual grounding and 3D video object detection, the output should be in JSON format, while for 3D dense captioning, it should be natural language description.
C Detailed Results on 3D Video Object Detection
Detailed quantitative results for our 3D video object detection are presented in Table 10 and 11. As illustrated in the table, method significantly outperforms the baseline that does not utilize visual geometry across most categories. Nevertheless, we observe that detecting small objects such pillows, lamps, and backpacks remains challenging.
D More Visualization
Figure 4 shows more visualization results for 3D visual grounding. The first three examples are positive cases where our model successfully locates the 3D bounding boxes for different categories like chair, cabinet, and document organizer. While the model correctly identifies these objects, we notice that still struggles with accurately predicting their orientation. The last two examples negative cases. While the predictions might appear correct in the 2D projection, the model's inaccurate depth estimation causes some discrepancies in 3D space.
We provide more visualization results on 3D video object detection in Figure 5. From this figure, we observe that while baseline performs decently in detecting objects given video, incorporating 3D geometry significantly improves both detection precision and recall. For instance, in the first example, the baseline's prediction for the desk mismatches the ground truth, whereas our approach improves the prediction box of the desk.
23

---

Object
Qwen2.5-VL-3B
VG LLM-4B
Qwen2.5-VL-7B
VG LLM-8B
chair
32.8
41.6
36.1
48.7
pillow
11.4
15.9
12.4
18.9
cabinet
7.8
12.4
10.6
17.9
table
31.3
39.8
32.7
44.8
lamp
5.6
15.6
8.0
15.9
couch
32.2
45,0
40.7
46.4
desk
31.6
38.5
34.4
32.3
stand
20.6
41.4
31.3
47.3
bed
58.8
70.2
64.6
75.8
backpack
28.9
37.5
28.6
40.4
bathtub
36.5
33.8
38.4
45.2
ottoman
0.0
2.8
2.7
5.3
dresser
31.2
41.1
36.4
bin
20.9
33.1
25.0
38.5
toilet
66.1
80.6
68.6
83.2
refrigerator
28.9
64.3
33.2
62.0
stove
37.8
68.5
55.9
69.5
microwave
11.5
21.1
7.8
25.8
monitor
15.5
22.1
20.3
31.7
computer
4.4
3.1
5.2
11.6
Table 10: The detailed results on 3D video object detection in 6-frame setting. We report the F1 score for all categories.
Object chair pillow cabinet table
Qwen2.5-VL-3B
VG LLM-4B
Qwen2.5-VL-7B
LLM-BB
37.7
49.7
41.2
54.0
11.9
17.9
13.2
22.9
10.2
13.1
11.6
17.1
35,0
41.3
36.5
46.5
lamp
4.4
16.4
9.0
18.9
couch
39.0
44.6
41.1
47.0
desk
33.7
39.0
34.5
41.0
stand
23.3
37.9
33.7
46.1
bed
64.8
71.2
68.2
74.1
backpack
27.4
42,0
33.3
41.4
bathtub
32.4
33,5
36.6
42.1
ottoman
3,2
3.8
0.0
0.0
dresser
33.1
40.3
38.2
bin
23.1
39,2
30,2
39,8
toilet
68.8
83.4
68.7
82.5
refrigerator
32.2
57.3
38.5
53.6
stove
62.5
68.8
61.2
68.6
microwave
25.9
32.1
21.3
33.3
monitor
22.3
29.1
21.3
34.0
computer
6.8
3.8
12.0
11.4
Table 11: The detailed results on 3D video object detection in 4-frame setting. We report the Fl score for all categories.
E
Limitations
While our model is built upon 3B and 7B MLLM backbones, its capabilities remain constrained by the model's inherent capacity. Scaling up the model size could potentially enhance both fundamental abilities and generalization performance. Furthermore, this work focuses exclusively on supervised fine-tuning, whereas emerging research demonstrates that reinforeement learning can significantly enhance model reasoning capabilities. Since VGGT trained on commonly used 3D scene datasets, evaluating it benchmarks built from that same data (such as ScanNet) poses a greater risk of overestimating its performance compared evaluating it on out-of-distributiin benchmarks. We leave these for future work.
24

---

Broader Impact
VG LLM built upon Qwen2.5-VL and consequently shares many common challenges associated with multi-modal large language models (MLLMs). These challenges include hallucinations in visual understanding, inherited biases from the base models, and susceptibility to adversarial inputs. Despite these limitations, releasing VG LLM to the research community would be highly beneficial, as it could catalyze further advancements vision geometry priors.
world understanding by leveraging Video LLMs and 3D
25

---

The office chair the wooden table and bed. the office chair has curved backside and four legs.
The cabinet is in the middle of the room. the cabinet is white box with blue rectangular
A small document organizer with shelves. there is door entry near it.
There is door in the middle of the northern wall next to metal cabinct. the door is shaped like a brown
rectangle und us dark yellow colored doorknob.
Figure 4: Visualization of 3D visual grounding. The first three examples are positive, whereas the last two are negative cases.
26

---

Base Baseline
VG-MLLM
Buseline
VG-MLLM
Baseline
VG-MLLM
Baseline
VG-MLLM
Figure 5: Visualization of 3D video object detection results.
27


Reviews:
[{"review_id": "ZaIMSxZpI2", "reviewer": "Reviewer_dMC1", "summary": "- The paper proposes Video-3D Geometry LLM (VG LLM), a framework that explicitly integrates 3D visual geometry priors into Multimodal Large Language Models (MLLMs) for enhanced 3D spatial understanding and reasoning.\n- The paper conducted extensive experiments showing the effectiveness of incorporating the features from a pretrained 3D visual geometry encoder.", "questions": "- I wonder if the authors have tried plugging predicted 3D information (from off-the-shelf models) into baseline models that require 3D inputs? This would create a fair and controlled comparison between VG LLM and existing 3D-enhanced baselines", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes to incorporate the 3D features from a pretrained 3D visual geometry encoder into MLLMs for better 3D spatial understanding and reasoning.\n- The proposed model demonstrates notable improvements on a range of 3D scene understanding and spatial reasoning benchmarks without requiring explicit 3D input.\n\n**Weaknesses**\n- From an architectural standpoint,  it seems like the novelty is mainly in leveraging 3D representations from a pretrained 3D visual geometry encoder (developed in prior work) and integrating these with 2D visual tokens, which is relatively limited.\n- The paper lacks analysis or investigation of alternative fusion strategies for integrating 3D geometry and 2D visual tokens.\n- It is unclear to me how the experiments in Table 8 can be used to show that “the improvement in spatial reasoning and 3D scene understanding in MLLMs 308 is genuinely driven by visual geometry modeling”.  Given that the camera info, etc. is predicted from visual geometry features and this prediction is inherently imperfect, it is natural for the pred. XX variants underperform the original visual geometry features.", "was_revised": true, "final_justification": "After reading the other reviews and the authors' response, I still believe that the novelty of the proposed method is relatively limited and cannot be fully offset by the empirical results. However, I appreciate the authors' efforts in the rebuttal, as they have addressed most of my concerns. Therefore, I am willing to increase my score to 4.", "rebuttal": "Thanks for your time, effort, and valuable comments!\n\n* **(W1) The novelty of our method.**\n\n    We highlight our core contributions as follows:\n    1. We propose a novel framework to enhance spatial understanding in MLLMs, incorporating a 3D geometry encoder that provides latent geometry information from video inputs. Unlike previous approaches heavily dependent on large volumes of 3D data, our method efficiently processes only video sequences, thereby creating new opportunities for significantly boosting the 3D understanding and spatial reasoning capabilities of MLLMs.\n    2. While straightforward, our extensive experimental results unveil the potential of leveraging geometry models for improved 3D understanding and spatial reasoning. We pioneered a new setting for 3D scene understanding by eliminating the dependence of 3D point clouds in MLLMs and validated the superiority of our methods across tasks, with a substantial improvement of F1 of 8.3 on 3D video object detection. Moreover, our method attains an impressive average score of 46.1 on VSI-Bench, a benchmark demanding complex spatial reasoning, outperforming previous state-of-the-art methods.\n    \n\n* **(W2) The effect of feature fusion strategies.** \n\n    We have experimented with several feature fusion strategies on 3D scene understanding tasks.\n\n    1. 'cross-attention': We refer to the strategy in PAVE [1], which consists of multiple blocks of cross-attention modules and MLPs with skip connections. 2D visual tokens serve as queries, while 3D visual tokens serve as keys and values. 2D positional embeddings are added to both the queries and keys. \n    2. 'concat': This approach first concatenates the 2D and 3D visual tokens along the feature dimension, followed by an MLP to transform them into the text embedding space.\n    3. 'add': This is the strategy employed in our paper. It directly adds the 2D and 3D visual tokens at a patch level.\n\n    As the table below illustrates, the 'cross-attention' strategy significantly improved performance over the baseline. Moreover, increasing the number of layers to three yielded even greater performance gains. While 'concat' performed worse than the baseline on ScanRefer, it outperformed it on Scan2Cap and 3D video object detection. Nevertheless, 'add' surpassed all other comparison methods despite its simplicity.\n        \n    | | ScanRefer | | Scan2Cap | | 3D Video Object Detection (4frames) | | |\n    |---|---|---|---|---|---|---|---|\n    | | IoU@0.25 | IoU@0.5 | C@0.5 | B-4@0.5 | Precision@0.25 | Recall@0.25 | F1@0.25 |\n    | Baseline (w/o geometry) | 31.9 | 9.3 | 58.0 | 36.3 | 32.6 | 27.9 | 29.9 |\n    | Concat | 27.7 | 6.8 | 75.7 | 40.0 | 37.1 | 32.5 | 34.4 |\n    | Cross-attention (1 layer) | 33.7 | 10.7 | 74.9 | 40.1 | 38.0 | 33.6 | 35.4 |\n    | Cross-attention (3 layers) | 34.4 | 10.5 | 75.5 | 40.2 | 38.5 | 33.0 | 35.4 |\n    | Add | 36.4 | 11.8 | 78.6 | 40.9 | 41.7 | 35.7 | 38.2 |\n\n* **(W3) The clarification of Table 8.**\n\n    The table primarily compares model variants that incorporate different types of spatial signals.\n    1. The most crucial comparison is between our model (last row) and the baseline without any geometry information (first row). The significant improvement demonstrates the effectiveness of incorporating 3D geometry signals into MLLMs.\n    2. We agree with the reviewer that the predicted attributes (pred. XX) are inherently imperfect. The core question behind these variants was to determine the most effective way to integrate spatial signals into the MLLM: explicit, strucutred attributes (e.g., predicted camera pose, pointmaps) or implicit, dense features? This question is non-trivial. The pred. XX attributes, while structured, suffer from prediction errors. Conversely, the visual geometry features, while dense, are learned without direct supervision. Furthermore, the predicted attributes are not solely predicted from the geometry features but generated by a dense prediction transformer [2] that fuses multi-level hidden states with multi-scale DINO image features. Therefore, it was not initially obvious which form of spatial signal would be more beneficial. Our empirical results conclude that the visual geometry features make a more direct and effective contribution to the model's 3D understanding capabilities.\n\n    These two points confirm the importance of our 3D geometry modeling.\n\n* **(Q1) Baselines that incorporate 3D information from off-the-shelf models.**\n\n    Video-3D LLM [3] introduces a method for 3D understanding in MLLMs by directly injecting pointmaps into visual representations. Following this approach, and as detailed in Table 8, we integrated pointmap predictions from VGGT—the current state-of-the-art model for tasks like pointmap prediction and camera pose estimation—into our LLM, which we refer to as 'Pred Point Info.' Our experimental results clearly indicate that incorporating explicit pointmap information improves performance in tasks such as dense captioning and 3D video object detection. Nevertheless, our method significantly outperforms these comparison methods with explicit geometry information.\n\n[1] PAVE: Patching and Adapting Video Large Language Models. Liu et al. CVPR 2025.\n\n[2] Vision Transformers for Dense Prediction. Ranftl et al. ICCV 2021.\n\n[3] Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding. Zheng et al. CVPR 2025."}, {"review_id": "EXPuoj7LRU", "reviewer": "Reviewer_eRbm", "summary": "The paper aims to integrate 3D priors into the vision-language models. Specifically, the paper proposes to encode the input video images with a feed-forward 3D reconstruction model, VGGT, and enhance the original vlm features with VGGT features via a patch merging strategy. The paper tests this method on several standard 3D benchmarks on visual grounding, captioning and spatial understanding and obtain superior results than other baselines that also do not use sensor 3D information. The ablations show that the proposed strategy of using implicit 3D via features is better than using no 3d information, and also better than directly using the explicit 3D positional coordinates.", "questions": "My main concerns are regarding some details and claims like using 3D information via object detectors but claiming not using them as a major point of this work. I would recommend making the numbers of no proposal refinement as the main numbers of table-1, and re-write the description of captioning experiments to be more precise (see my review for additional details). Overall, I think besides these fixable issues, the paper is solid, and I will be happy to support its acceptance.", "limitations": "yes (but authors may consider additional limitations that I mention in my main review)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written, well-motivated and easy to follow\n- The paper shows results on standard benchmarks and uses strong baselines covering all kinds of existing methods — methods which use sensor 3D information, as well as no explicit 3d methods. The ablations are thorough and sufficient. \n- The paper tries to benchmark things in this new and important setup where we do not have access to camera pose and depths from sensors. \n\nWeaknesses:\n- “Proposal Refinement” term in Table-1 is not defined prior to its introduction. From supplementary, we know that it involves using off-the-shelf 3D object detector and finding the best match detection from it (and refers to lines 224-225 in the main paper). The results with proposal refinement are shows as main result, while the ones without are shown in brackets. I think this is misleading — the main thread in the paper is about not using camera parameters or explicit depth and thus in my opinion the main results should be about that. With proposal refinement, the results indirectly use this information through Mask3D. \n    - On a similar note, I would recommend tempering down the use of “ excellent” to describe the performance of the proposed model in Table-1. Even with proposal refinement, the results are much weaker than SOTA, and as I argue above, perhaps the more faithful results are the ones in the brackets which are lower than original ScanRefer’s numbers too. I do not intend to say that not achieving SOTA is a weakness, I appreciate the new setting, but I think it is important to convey the large gap in performance compared to situations when we indeed have access to depth and cameras from our sensors.\n- Similar issue as above exists in 3D dense captioning results. Lines 238-239 claim “despite not utilizing 3D camera parameters or explicit depth...” but indeed those numbers utilize them through the 3D detectors that the paper uses. And indeed then the claims that follow become incorrect too like: “indicating the potential of pure-vision solutions for 3d dense captioning” since the proposed method very strongly and clearly relies on 3D detectors. (Side note: “pure vision” solution is a non-standard terminology, many people will argue that using depth and camera is pure vision as well). \n    - (Minor comment): L239 saying that the proposed method surpasses prior SOTA LEO is somewhat weird since the same table contains several other methods which are better than both LEO as well as the proposed method. Perhaps consider deleting this statement.\n- It’s important to note that methods like VGGT are in fact trained on scannet. this paper studies the use of these models on benchmarks based on 3D scenes that are very much in-distribution of VGGT. Thus showing that 3D tokens help in these benchmarks compared to 2D only methods is riskier — the true test will be in videos / datasets outside VGGT’s distribution. Note: I am not expecting authors to perform such an experiment, but perhaps it would be nice to make this explicit in the limitations.", "was_revised": true, "final_justification": "I am happy to recommend acceptance for this work given the promised clarifications will be added to the camera ready.", "rebuttal": "We are really grateful for your constructive suggestions.\n\n* **(W1, Q1) The table presentation of Table 1.**\n\n    We included proposal refinement in our experiments to ensure a fair comparison with prior works like LEO [1] and Chat-Scene [2], which often leverage off-the-shelf 3D object detectors to extract proposals, either directly or indirectly. We agree with you that the table should focus on the results without using any depth or camera information, and we will revise the table presentation in the next version.\n\n* **(W1, W2, Q1) The claims about the 3D visual grounding and 3D dense captioning.**\n    Our framework could also be extended to perform 3D dense captioning by locating object bounding boxes and generating the associated object captions in a single forward pass, akin to 3D video object detection. However, this promising direction is beyond the scope of our current work and will be reserved as future research.\n    Thank you again for helping us improve the clarity and rigor of the paper. We will revise the description according to your suggestions in the next version. \n\n* **(W3) Benchmarks outside VGGT's distribution.** \n    Given that VGGT has already leveraged almost all existing 3D scene datasets, there is a scarcity of available OOD evaluation benchmarks. Therefore, we will explore the generalization of our method as future work and discuss this limitation in the revised version.\n\n[1] An Embodied Generalist Agent in 3D World. Huang et al. ICML 2024.\n\n[2] Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. Huang et al. NeurIPS 2024."}, {"review_id": "KvkJce2r7o", "reviewer": "Reviewer_MDYF", "summary": "The key innovation of this work is to learn 3D knowledge directly from videos by integrating implicit 3D geometry priors. The proposed VG LLM architecture uses two parallel encoders: a standard 2D visual encoder to extract semantic features from individual frames, and a dedicated 3D visual geometry encoder (instantiated with VGGT) that processes the video sequence to extract latent geometry-aware features. These two sets of features are fused and fed into the MLLM backbone. The authors conduct extensive experiments on a wide range of tasks, including 3D scene understanding (visual grounding, dense captioning, video object detection) and complex spatial reasoning (VSI-Bench, CV-Bench, BLINK). Notably, their 4B parameter model achieves state-of-the-art results on the challenging VSI-Bench, outperforming even proprietary models like Gemini-1.5-Pro.", "questions": "Questions\n1. Have you considered or experimented with other 3D visual geometry encoders (e.g., Dust3R)? Would the framework still be effective with a less powerful or different type of geometry encoder, or is the success fundamentally linked to the specific capabilities of VGGT?\n\n2. Could you please provide more details on the computational overhead at inference time? Specifically, what is the increase in latency and memory usage for a typical video input compared to the Qwen2.5-VL baseline?\n\n3. The paper focuses on a 4B model. What is your hypothesis on how the performance gains from VG LLM would translate to much larger base MLLMs (e.g., Llama-3 8B scale)? Is it possible that the spatial reasoning capabilities emerging in larger models would render the explicit geometry encoder less critical?\n\n4. During the training of VG LLM, are the weights of the 3D visual geometry encoder (VGGT) frozen, or are they fine-tuned along with the MLLM backbone? This is an important implementation detail for reproducibility and for understanding the interplay between the two components.\n\n5. The ablation in Table 9 shows that fine-tuning on the mixed dataset already provides a substantial boost in spatial reasoning. Could you further disentangle the contributions of the SPAR-7M dataset and the LaVA-Video-178K dataset? For instance, how does the baseline model (without the geometry encoder) perform when trained only on the LaVA-Video-178K, and how does your VG LLM perform on that same subset? This could help clarify how much of the gain comes from the data.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper's primary contribution is both simple and highly effective. Move away from the reliance on explicit, dense 3D inputs (e.g., point clouds, depth maps) and instead leverage the VGGT (CVPR Best Paper) to extract implicit 3D geometry priors from raw video.\n\n2. The performance of VG LLM is outstanding across multiple benchmarks. The state-of-the-art result on VSI-Bench, where the 4B model surpasses much larger proprietary models like Gemini-1.5-Pro, is a headline achievement and provides strong evidence for the method's effectiveness in complex spatial reasoning. The substantial gains in 3D video object detection (Table 3, +19.3 Recall, +8.3 F1) also convincingly demonstrate the model's enhanced ability to handle egocentric-allocentric transformations, a key challenge in this domain.\n\n3. The authors have been thorough in their experimental validation. The evaluation spans a diverse set of tasks, from 3D perception (grounding, captioning) to high-level reasoning. This demonstrates the general applicability and robustness of the proposed framework.\n\n\nWeaknesses:\n1. The success of VG LLM appears to be heavily tied to the performance of the VGGT [43] model, a powerful geometry encoder. The contribution might be perceived as effectively demonstrating the power of VGGT as a feature extractor for MLLMs.\n\n2. The proposed method adds a 1B parameter VGGT encoder to the 3B Qwen2.5-VL backbone. This represents a significant (~33%) increase in model size. The paper mentions the training setup (8x H800 GPUs) but does not provide a detailed analysis of the inference-time trade-offs, which is an important consideration for practical deployment.\n\n\n3. The experiments are conducted on a 4B scale model. While outperforming larger models is a clear strength, it leaves open the question of how this approach scales. Would the relative gains from the geometry encoder diminish if the base MLLM were much larger and more capable (e.g., a 7B, 34B, 70B model)? An exploration or at least a discussion of this scaling dynamic would strengthen the paper.", "was_revised": true, "final_justification": "Thank Authors for addressing my previous concerns. I maintain my original score.", "rebuttal": "Thanks for your positive comments!\n\n* **(W1, Q1) Extension to other geometry models.**\n\n    Our framework can also be extended to other geometry models. Specifically, we conducted experiments by replacing the VGGT with Pi3 [1], a powerful visual geometry model with high inference speed.\n\n    The table below gives the comparison results. As depicted in the table, both 'Pi3' and 'VGGT' outperform the baseline ('None') without any geometry encoders. 'Pi3' outperforms 'VGGT' on the 3D visual grounding (ScanRefer), but underperforms it on the 3D dense captioning (Scan2Cap) and 3D video object detection task. This is because ScanRefer requires the model to predict the frame in which an object appears and its associated bounding box, and Pi3's reference-free design makes it more effective for adapting to this task. However, for Scan2Cap and 3D video object detection, models with a fixed reference frame, such as VGGT, performs better.\n\n    |         | ScanRefer |         | Scan2Cap |         | 3D Video Object Detection (4 frames) |         |         |\n    |---------|-----------|---------|----------|---------|-------------------------------------|---------|---------|\n    |         | Acc@0.25  | Acc@0.5 | C@0.5    | B-4@0.5 | Precision@0.25                      | Recall@0.25 | F1@0.25 |\n    | None    | 31.9      | 9.3     | 58.0     | 36.3    | 32.6                                | 27.9    | 29.9    |\n    | Pi3     | 39.7      | 14.7    | 76.4     | 40.0    | 41.3                                | 35.8    | 38.2    |\n    | VGGT    | 36.4      | 11.8    | 78.6     | 40.9    | 41.7                                | 35.7    | 38.2    |\n\n* **(W2, Q2) The inference latency and memory usage of our method.**\n    \n    We compared the inference speed of our proposed method with Qwen2.5-VL-3B. The test was conducted on 3D scene understanding tasks, with 100 data samples randomly selected per task, and a maximum of 100 tokens for generation. As presented in the table below, our method incurred an average increase of 20.2% in runtime compared to the baseline. In addition, our method also imposes an additional 36% of memory usage during inference.\n\n    | | Video Object Detection (4 frame) | Video Object Detection (6 frame) | Dense Captioning (16 frame) | Visual grounding (24 frame) |\n    |-------|---|---|---|---|\n    | Qwen2.5-VL-3B | 2.36 | 2.46 | 1.11 | 2.62 |\n    | VG-LLM-4B | 2.73 (+15.6%) | 2.82 (+14.4%) | 1.41 (+27.0%) | 3.24 (+23.6%) |\n\n* **(W3, Q3) Scaling up the LLM backbone.**\n\n    Thanks for your insightful suggestions. We have extended our model to larger language models for spatial reasoning tasks. As shown in the following table, our model built upon Qwen2.5-VL-7B achieved better performance compared to its counterpart without geometry information. This demonstrates that the introduction of geometry information still brings improvements in larger MLLMs.\n\n    | | VSI-Bench | | | | CV-Bench | |\n    |---|---|---|---|---|---|---|\n    | | Abs. Dist | Obj. Size | Room Size | Route Plan | 3D Distance | 3D Depth |\n    | Qwen2.5-VL-3B | 34.7 | 56.4 | 55.0 | 32.0 | 83.5 | 75.8 |\n    | VG-LLM-4B | 37.0 | 56.6 | 60.4 | 34.5 | 88.0 | 84.0 |\n    | Qwen2.5-VL-7B | 36.3 | 55.9 | 58.9 | 28.4 | 87.2 | 87.2 |\n    | VG-LLM-8B | 38.7 | 58.5 | 63.3 | 29.9 | 92.3 | 93.0 |\n\n* **(Q4) Is the 3D geometry encoder frozen or fine-tuned?** \n\n    As detailed in Appendix A.1, we froze the 3D visual geometry encoder during training to preserve its generalization capabilities.\n\n* **(Q5) The ablation study of data composition.**\n    \n    We appreciate the reviewer's insightful question regarding our ablation study on data composition. To disentangle the contribution of SPAR and LLaVA-Hound, we trained both the baseline (Qwen2.5-VL-3B) and our model on the SPAR-234K, which utilizes the same data subset as presented in our paper.\n\n    Comparing rows 1 and 2, we can see that the introduction of geometry information brings consistent improvements across most tasks, notably yielding a 4.5% improvement in 3D Distance and a 10.5% improvement in 3D Depth. Furthermore, a comparison between Rows 2 and 4 shows that co-training on a mixture of SPAR-234K and LLaVA-Hound-64K further enhances the model's performance. This is because training exclusively on SPAR-234K can limit the model's generalization ability, whereas the addition of instruction data significantly boosts its robustness. \n        \n    | Training Data                 | Model         | Abs. Dist | Obj. Size | Room Size | Route Plan | 3D Distance | 3D Depth |\n    |-------------------------------|---------------|-----------|-----------|-----------|------------|-------------|----------|\n    | SPAR-234K                     | Qwen2.5-VL-3B | 33.5      | 57.3      | 54.0      | 33.0       | 84.8        | 77       |\n    |                               | VG-LLM        | 35.0      | 55.9      | 56.7      | 34.0       | 89.3        | 87.5     |\n    | SPAR-234K + LLaVA-Video-63K   | Qwen2.5-VL-3B | 34.7      | 56.4      | 55.0      | 32.0       | 83.5        | 75.8     |\n    |                               | VG-LLM        | 37.0      | 56.6      | 60.4      | 34.5       | 88.0        | 84.0     |\n\n[1] $\\pi^3$: Scalable Permutation-Equivariant Visual Geometry Learning. Wang et al. arXiv 2025."}, {"review_id": "LH3fUqpvWZ", "reviewer": "Reviewer_FeDL", "summary": "This paper presents VG LLM (Video-3D Geometry LLM), a novel multimodal large language model (MLLM) framework designed to enhance spatial understanding from videos without relying on explicit dense 3D inputs. The key idea is to leverage a pretrained 3D visual geometry encoder (VGGT) to extract latent 3D geometric priors from video frames. The proposed model is evaluated on a variety of 3D vision-language tasks, including 3D visual grounding, dense captioning, video object detection, and spatial reasoning benchmarks. VG LLM achieves competitive or even state-of-the-art performance compared to models that rely on explicit 3D inputs, demonstrating the effectiveness.", "questions": "Please refer to the above weaknesses.\n\nOther minor questions:\n1. Would VG LLM fail under low-quality videos where the pretrained VGGT may generalize poorly?", "limitations": "Yes, in the appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. VG LLM does not require explicit dense 3D inputs, making the model more flexible and generalizable.\n2. The performance of VG LLM is superior.\n3. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. Although VGGT demonstrates excellent performance for multi-view 3D reconstruction, its output is in a relative scale rather than metric. How does VG LLM handle this scale ambiguity without using any explicit 3D inputs?\n2. The metrics in Table 2 (C@0.5, B-4@0.5, etc.) should be explained in more detail.", "was_revised": true, "final_justification": "The idea of injecting the knowledge of a foundation 3D reconstruction model (VGGT) into MLLMs is promising, and I think the experiments have demonstrated its effectiveness. Thus, I tend to accept this paper.", "rebuttal": "Thanks for your valuable comments!\n\n* **(W1) How does our method handle the scale ambiguity?**\n    \n    While VGGT indeed provides features at a relative scale, our model effectively learns the absolute scale of objects in the real-world scenes by fine-tuning on customized datasets. Specifically, we use a training dataset encompassing tasks such as 3D visual grounding, 3D dense captioning, and 3D video object detection for 3D scene understanding. The 3D coordinates in these tasks embed the absolute scale information about common objects in the real-world scenes. Our model can learn to infer the scale of unseen objects within the scene by leveraging the relative scale provided by VGGT and the absolute scale of seen objects learned during training.\n\n* **(W2) The explanations for C@0.5 and B-4@0.5.**\n    \n    We follow the previous work [1,2] to decompose the 3D dense captioning task into two stages, i.e., detecting 3D object proposals with an off-the-shelf detector and generating object descriptions based on detected object coordinates. The evaluation metrics C@0.5, B−4@0.5, M@0.5, and R@0.5 represent the CIDEr, BLEU-4, METEOR, and ROUGE scores, respectively. The @0.5 suffix specifies that these metrics are evaluated only on detected objects with an IoU ≥0.5 against a ground-truth object.\n    \n* **(Q1) The influence of video quality.**\n\n    To investigate the impact of the low-quality videos that VGGT may generalize poorly, we conducted experiments on 3D dense captioning. Specifically, we downsampled each video frame by reducing its width to a certain pixel value (maintaining aspect ratio) and then resized it back to the original dimension before feeding it as input for VGGT. Note that the input resolution for the ViT within the MLLM was not adjusted for fair comparison.\n\n    As shown in the table, the performance degrades as the downsampling width decreases from 518 to 224, indicating an impact from accumulated errors from VGGT. However, this degradation is limited, highlighting the robustness of our methodology.\n\n    | width | CIDER@0.25 | B-4@0.25 |\n    |:-----:|:----------:|:--------:|\n    | 224px |    73.4    |   40.0   |\n    | 336px |    74.9    |   40.1   |\n    | 448px |    76.4    |   40.5   |\n    | 518px |    78.6    |   40.9   |\n\n[1] Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. Huang et al. NeurIPS 2024.\n\n[2] An Embodied Generalist Agent in 3D World. Huang et al. ICML 2024."}]

Author rebuttal:

Meta-review: All reviewers agreed that the problem under investigation is significant, the proposed approach is effective, and the results are encouraging. The rebuttal adequately addressed most of the reviewers’ concerns, leading all of them to recommend acceptance. The authors are advised to further refine the final version of the paper by incorporating the reviewers’ feedback.
Decision: Accept (poster)