Title: Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
Venue: NeurIPS 2025
Accepted: yes
Keywords: MLLM

3D Scene Understanding

Spatial Reasoning
Abstract: Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method called the Video-3D Geometry Large Language Model (VG LLM). Our approach utilizes a 3D visual geometry encoder to extract 3D prior information from video sequences. This information is then integrated with visual tokens and input into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.


Peer reviews:
comments: SUMMARY:
This paper presents VG LLM (Video-3D Geometry LLM), a novel multimodal large language model (MLLM) framework designed to enhance spatial understanding from videos without relying on explicit dense 3D inputs. The key idea is to leverage a pretrained 3D visual geometry encoder (VGGT) to extract latent 3D geometric priors from video frames. The proposed model is evaluated on a variety of 3D vision-language tasks, including 3D visual grounding, dense captioning, video object detection, and spatial reasoning benchmarks. VG LLM achieves competitive or even state-of-the-art performance compared to models that rely on explicit 3D inputs, demonstrating the effectiveness.

STRENGTHS AND WEAKNESSES:
Strengths:
1. VG LLM does not require explicit dense 3D inputs, making the model more flexible and generalizable.
2. The performance of VG LLM is superior.
3. The paper is well-written and easy to follow.

Weaknesses:
1. Although VGGT demonstrates excellent performance for multi-view 3D reconstruction, its output is in a relative scale rather than metric. How does VG LLM handle this scale ambiguity without using any explicit 3D inputs?
2. The metrics in Table 2 (C@0.5, B-4@0.5, etc.) should be explained in more detail.

QUESTIONS:
Please refer to the above weaknesses.

Other minor questions:
1. Would VG LLM fail under low-quality videos where the pretrained VGGT may generalize poorly?

LIMITATIONS:
Yes, in the appendix.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe key innovation of this work is to learn 3D knowledge directly from videos by integrating implicit 3D geometry priors. The proposed VG LLM architecture uses two parallel encoders: a standard 2D visual encoder to extract semantic features from individual frames, and a dedicated 3D visual geometry encoder (instantiated with VGGT) that processes the video sequence to extract latent geometry-aware features. These two sets of features are fused and fed into the MLLM backbone. The authors conduct extensive experiments on a wide range of tasks, including 3D scene understanding (visual grounding, dense captioning, video object detection) and complex spatial reasoning (VSI-Bench, CV-Bench, BLINK). Notably, their 4B parameter model achieves state-of-the-art results on the challenging VSI-Bench, outperforming even proprietary models like Gemini-1.5-Pro.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper's primary contribution is both simple and highly effective. Move away from the reliance on explicit, dense 3D inputs (e.g., point clouds, depth maps) and instead leverage the VGGT (CVPR Best Paper) to extract implicit 3D geometry priors from raw video.\n\n2. The performance of VG LLM is outstanding across multiple benchmarks. The state-of-the-art result on VSI-Bench, where the 4B model surpasses much larger proprietary models like Gemini-1.5-Pro, is a headline achievement and provides strong evidence for the method's effectiveness in complex spatial reasoning. The substantial gains in 3D video object detection (Table 3, +19.3 Recall, +8.3 F1) also convincingly demonstrate the model's enhanced ability to handle egocentric-allocentric transformations, a key challenge in this domain.\n\n3. The authors have been thorough in their experimental validation. The evaluation spans a diverse set of tasks, from 3D perception (grounding, captioning) to high-level reasoning. This demonstrates the general applicability and robustness of the proposed framework.\n\n\nWeaknesses:\n1. The success of VG LLM appears to be heavily tied to the performance of the VGGT [43] model, a powerful geometry encoder. The contribution might be perceived as effectively demonstrating the power of VGGT as a feature extractor for MLLMs.\n\n2. The proposed method adds a 1B parameter VGGT encoder to the 3B Qwen2.5-VL backbone. This represents a significant (~33%) increase in model size. The paper mentions the training setup (8x H800 GPUs) but does not provide a detailed analysis of the inference-time trade-offs, which is an important consideration for practical deployment.\n\n\n3. The experiments are conducted on a 4B scale model. While outperforming larger models is a clear strength, it leaves open the question of how this approach scales. Would the relative gains from the geometry encoder diminish if the base MLLM were much larger and more capable (e.g., a 7B, 34B, 70B model)? An exploration or at least a discussion of this scaling dynamic would strengthen the paper.\n\nQUESTIONS:\nQuestions\n1. Have you considered or experimented with other 3D visual geometry encoders (e.g., Dust3R)? Would the framework still be effective with a less powerful or different type of geometry encoder, or is the success fundamentally linked to the specific capabilities of VGGT?\n\n2. Could you please provide more details on the computational overhead at inference time? Specifically, what is the increase in latency and memory usage for a typical video input compared to the Qwen2.5-VL baseline?\n\n3. The paper focuses on a 4B model. What is your hypothesis on how the performance gains from VG LLM would translate to much larger base MLLMs (e.g., Llama-3 8B scale)? Is it possible that the spatial reasoning capabilities emerging in larger models would render the explicit geometry encoder less critical?\n\n4. During the training of VG LLM, are the weights of the 3D visual geometry encoder (VGGT) frozen, or are they fine-tuned along with the MLLM backbone? This is an important implementation detail for reproducibility and for understanding the interplay between the two components.\n\n5. The ablation in Table 9 shows that fine-tuning on the mixed dataset already provides a substantial boost in spatial reasoning. Could you further disentangle the contributions of the SPAR-7M dataset and the LaVA-Video-178K dataset? For instance, how does the baseline model (without the geometry encoder) perform when trained only on the LaVA-Video-178K, and how does your VG LLM perform on that same subset? This could help clarify how much of the gain comes from the data.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to integrate 3D priors into the vision-language models. Specifically, the paper proposes to encode the input video images with a feed-forward 3D reconstruction model, VGGT, and enhance the original vlm features with VGGT features via a patch merging strategy. The paper tests this method on several standard 3D benchmarks on visual grounding, captioning and spatial understanding and obtain superior results than other baselines that also do not use sensor 3D information. The ablations show that the proposed strategy of using implicit 3D via features is better than using no 3d information, and also better than directly using the explicit 3D positional coordinates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written, well-motivated and easy to follow\n- The paper shows results on standard benchmarks and uses strong baselines covering all kinds of existing methods — methods which use sensor 3D information, as well as no explicit 3d methods. The ablations are thorough and sufficient. \n- The paper tries to benchmark things in this new and important setup where we do not have access to camera pose and depths from sensors. \n\nWeaknesses:\n- “Proposal Refinement” term in Table-1 is not defined prior to its introduction. From supplementary, we know that it involves using off-the-shelf 3D object detector and finding the best match detection from it (and refers to lines 224-225 in the main paper). The results with proposal refinement are shows as main result, while the ones without are shown in brackets. I think this is misleading — the main thread in the paper is about not using camera parameters or explicit depth and thus in my opinion the main results should be about that. With proposal refinement, the results indirectly use this information through Mask3D. \n    - On a similar note, I would recommend tempering down the use of “ excellent” to describe the performance of the proposed model in Table-1. Even with proposal refinement, the results are much weaker than SOTA, and as I argue above, perhaps the more faithful results are the ones in the brackets which are lower than original ScanRefer’s numbers too. I do not intend to say that not achieving SOTA is a weakness, I appreciate the new setting, but I think it is important to convey the large gap in performance compared to situations when we indeed have access to depth and cameras from our sensors.\n- Similar issue as above exists in 3D dense captioning results. Lines 238-239 claim “despite not utilizing 3D camera parameters or explicit depth...” but indeed those numbers utilize them through the 3D detectors that the paper uses. And indeed then the claims that follow become incorrect too like: “indicating the potential of pure-vision solutions for 3d dense captioning” since the proposed method very strongly and clearly relies on 3D detectors. (Side note: “pure vision” solution is a non-standard terminology, many people will argue that using depth and camera is pure vision as well). \n    - (Minor comment): L239 saying that the proposed method surpasses prior SOTA LEO is somewhat weird since the same table contains several other methods which are better than both LEO as well as the proposed method. Perhaps consider deleting this statement.\n- It’s important to note that methods like VGGT are in fact trained on scannet. this paper studies the use of these models on benchmarks based on 3D scenes that are very much in-distribution of VGGT. Thus showing that 3D tokens help in these benchmarks compared to 2D only methods is riskier — the true test will be in videos / datasets outside VGGT’s distribution. Note: I am not expecting authors to perform such an experiment, but perhaps it would be nice to make this explicit in the limitations.\n\nQUESTIONS:\nMy main concerns are regarding some details and claims like using 3D information via object detectors but claiming not using them as a major point of this work. I would recommend making the numbers of no proposal refinement as the main numbers of table-1, and re-write the description of captioning experiments to be more precise (see my review for additional details). Overall, I think besides these fixable issues, the paper is solid, and I will be happy to support its acceptance.\n\nLIMITATIONS:\nyes (but authors may consider additional limitations that I mention in my main review)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper proposes Video-3D Geometry LLM (VG LLM), a framework that explicitly integrates 3D visual geometry priors into Multimodal Large Language Models (MLLMs) for enhanced 3D spatial understanding and reasoning.\n- The paper conducted extensive experiments showing the effectiveness of incorporating the features from a pretrained 3D visual geometry encoder.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper proposes to incorporate the 3D features from a pretrained 3D visual geometry encoder into MLLMs for better 3D spatial understanding and reasoning.\n- The proposed model demonstrates notable improvements on a range of 3D scene understanding and spatial reasoning benchmarks without requiring explicit 3D input.\n\n**Weaknesses**\n- From an architectural standpoint,  it seems like the novelty is mainly in leveraging 3D representations from a pretrained 3D visual geometry encoder (developed in prior work) and integrating these with 2D visual tokens, which is relatively limited.\n- The paper lacks analysis or investigation of alternative fusion strategies for integrating 3D geometry and 2D visual tokens.\n- It is unclear to me how the experiments in Table 8 can be used to show that “the improvement in spatial reasoning and 3D scene understanding in MLLMs 308 is genuinely driven by visual geometry modeling”.  Given that the camera info, etc. is predicted from visual geometry features and this prediction is inherently imperfect, it is natural for the pred. XX variants underperform the original visual geometry features.\n\nQUESTIONS:\n- I wonder if the authors have tried plugging predicted 3D information (from off-the-shelf models) into baseline models that require 3D inputs? This would create a fair and controlled comparison between VG LLM and existing 3D-enhanced baselines\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}