Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction–Reasoning Synergy
Authors: Haijier Chen
Bo Xu
Shoujian zhang
Haoze Liu
Jiaxuan Lin
Jingrong Wang
Venue: ICLR 2026 Oral (2026)
Keywords: video-based 3D MLLM
geometric priors
Cross-Task Adapter
Metric Depth calibration
Primary area: foundation or frontier models, including LLMs

Abstract: Recent developments in Multimodal Large Language Models (MLLMs) have significantly improved Vision–Language (VL) reasoning in 2D domains. However, extending these capabilities to 3D scene understanding remains a major challenge. Existing 3D Multimodal Large Language Models (3D-MLLMs) often depend on 3D data inputs, which limits scalability and generalization. To address this limitation, we propose Vid-LLM, a video-based 3D-MLLM that directly processes video inputs without requiring external 3D data, making it practical for real-world deployment. In our method, the geometric prior are directly used to improve the performance of the sceen perception. To integrate the geometric cues into the MLLM compactly, we design a Cross-Task Adapter (CTA) module to align the 3D geometric priors with the vision-language representations. To ensure geometric consistency and integrity, we introduce a Metric Depth Model that recovers real-scale geometry from the reconstruction outputs. Finally, the model is fine-tuned with a two-stage distillation optimization strategy, realizing fast convergence and stabilizes training. Extensive experiments across diverse benchmarks verified the effectiveness of our method on 3D Question Answering, 3D Dense Captioning and  3D Visual Grounding tasks,  demonstrating the superior multi-task capabilities.

Published as a conference paper at ICLR 2026
VID-LLM: A COMPACT VIDEO-BASED 3D MULTIMODAL LLM WITH RECONSTRUCTIO-RRESSONIII SYNERGY
Haijier Chen" Bo Xu Shoujian Zhang" Haoze Liu', Jiaxuan Lin', Jingrong Wang
School of Geodesy and Geomatics, Wuhan University
2 Hubei Luojia Laboratory
3 School of Architecture and Urban Planning, Shenzhen University
3D Captioning
3D Question Answering
Describe
center the room?
The room small office with desks ar arranged along the walls, a chairs. has compact layout around the edges..
orange sofa.
How many chalrs are in the room?
workspace
There are six chairs the room.
Describe
3D Visual Grounding
right side of the room detail.
the
The desk and rectangular, placed against the wall with monitors, books, and
right of the sofa.
[1.13, 1.64, 0.35, 0.92, 0.03, 1.16]
Figure 1: We propose Vid-LLM to achieve diverse 3D vision-language reasoning tasks using only video inputs.
ABSTRACT
Recent developments Multimodal Large Language Models (MLLMs) have sig- nificantly improved Vision-Language (VL) reasoning in 2D domains. However, extending these capabilities to 3D scene understanding remains a major challenge. Existing 3D Multimodal Large Language Models (3D-MLLMs) often depend on 3D data inputs, which limits scalability and generalization. To address this lim- itation, propose Vid-LLM, vidco-based 3D-M 3D-MLLM that directly processes video inputs without requiring external 3D data, making practical for real-world deployment. In method, the geometric prior directly used improve the performance of the sceen perception. To integrate the geometric cues into the MLLM compactly, we design Cross-Tase Adapter (CTA) module align the 3D geometric priors with the vision-langusal representations. To ensure geometric consistency and integrity, we introduce Metric Depth Model that recovers real- scale geometry from the reconstruction outputs. Finally, the model is fine-ture fine-tuned with a two-stage distillation optimization strategy, realizing fast convergence and stabilizes training. Extensive experiments across diverse benchmarks verified the effectiveness of our method on 3D Question Answering, 3D Dense Captioning and 3D Visual Grounding tasks, demonstrating the superior multi-task capabili- ties. Project page: https:://ceec. htpps:///chehaae wirehhhaaaieerr. github.ggitt.b....... 1o/Vid-LLM
Equal contribution.
Corresponding author: shjzhang wtghhang@sggwwuu.du....

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in Large Language Models (LLMs) (Vaswani et al. 2017 Radford et al. 2019
Naveed et al.) 2025) and Multimodal Large Language Models (MLLMs) (Zhang et al. 2024a
Yin et al. 2024, Wu et al. 2023) have reinforced the paradigm of language as universal inter- face, substantially improving cross-modal perception and reasoning. Extending this progress to 3D,
3D-aware Multimodal Large Language Models (3D-MLLM) (Ren ct al. 2025), which unify 3D scene understanding and vision-language reasoning under a linguistic interface. This line of work underscores the importance of grounding language in persistent 31) spatial representations (Cheng et al.] 2024a Roh et al. 2022). offering a unified pathway toward systematic scene-level reasoning.
recent research has focused
Recent studies have made substantial progress in 3D vision-language (3D VL) reasoning (Chen al. 2024c] Huang et al. 2023b), yet most approaches rely on complex 3D inputs, incurring high costs m data collection, preprocessing, and computation. Some models rely on point clouds or re- constructed scenes augmented with rendered views semantic- geomeetii features (Hong et al.) 2023a Fu et al., 2024). while others adopt simpler inputs but still depend on explicit 3D scene rep- resentations such as reconstructed objects aligned with semantic representations (Chu et al.) 2024 Huang et al. 2023a; 2024). Despite their effectiveness, these pipelines depend depth, poses, or external modules, leading to substantial data and engineering overhead as well as high memory and latency costs. This rigid input requirements and system complexity fundamentally limit the scalability and transferability of current MLLMs.
To overcome these limitations, more general solution is to enable the model directly reconstruct scene geometry from video (Leroy et al., 2024 Wang et al., 2024), thereby climinating the reliance on external depth, pose, registration modules. More importantly, reconstruction and reasoning intrinsically interdependent: geometric structures underpin semantic understanding, while semantic reasoning, turn, provides contextual priors that guide and refine geometric modeling (Cheng
ct al. 2024a Ha & Song 2022).
In this work, introduce Vid-LLM, compact model that jointly performs reconstruction and 3D vision-language reasoning from monocular video inputs, as illustrated in Fig. [ The core compo- nent of Vid-LLM a Cross-Task Adapter (CTA) that tightly couples reconstruction with reasoning, enabling intrinsic semmetry-semantic interaction with mutual reinforcement and constraint. CTA disentangles geometry-awan aware and language-aware features; the geometric stream is then processed by Global-Frama Attention backbone and specialized heads to estimate camera poses and relative depth, followed by a Metric Depth Model for real-scale calibration. The recovered 3D informa- tion is then fused with semantic features to construct 3D patches, which are fed into the LLM for spatial reasoning. Finally, a two-stage training strategy ensures convergence and improves overall performance. Extensive experiments across diverse 3D vision- language benchmarks demonstrate the performance Vid-LLM and confirm its effectiveness as a practical and scalable framework for video-based 3D multimodal reasoning.
Our main contributions are summarized as follows:
We propose Vid-LLM for versatile 3D scene understanding. The framework does not rely dense 3D inputs or prior poses, making it practical for real-world deployment.
We design a Cross Adapter to align the 3D geometry priors with VL representations, boosting the integration of 3D visual geometry priors into MLLM. A two-stage training strategy is further adopted to improve the stability and performance.
Extensive experimental evaluations are conducted on real datasets to evaluate the perfor- mance of our method. The experimental results demonstrate that our method achieves perior performance in terms of question answering, dense captioning and visual grounding. We will publish our code facilitate communication.
RELATED WORK
3D-MLLMs have achieved significant advances in 3D scene understanding, yet their reliance on explicit 3D data still limits scalability and applicability. Meanwhile, progress in 3D reconstruction
2

---

Published as a conference paper at ICLR 2026
shows that geometry be directly reconstructed from videos. Integrating such geometric priors into 3D-MLLMs represents a promising approach to enhance semantic grounding. We therefore review related work in three directions: (i) 3D-MLLMs, (ii) 3D reconstruction, and (iii) geometry priors in vision-language models.
3D-aware Multimodal Large Language Models (3D-MLLMs). 3D-MLLMs aim to unify 3D scene understanding and lanion-language reasoning within a unified linguistic interface, represent- ing an important extension of multimodal large language models (MLLMs). Existing approaches predominantly rely on explicit geometric inputs: some leverage point clouds or reconstructed scenes, often augmented with rendered views, region-level alignments, or condensed 3D feature grids to support large-scale embodied training (Hong et al.) 2023a; Chen et al. 2024b| Fu et al., [024] Huang et al. 2023b); others map 3D features to the language space and model spatial relations to enable interactive dialogue (Huang et al. 2023a 2024 Zheng et al., [025b] Huang et al. 2025
Despite their progress, these methods invariably depend on complex inputs such as point clouds, reconstructed scenes, multi-view renderings, or object-level annotations, which impose substantial burdens on data acquisition, preprocessing, and computation, thereby limiting scalability and trans- ferability. In addition, the recent approach VGLLM (Zheng al., (025a) explores video-based 3D-MLLM setting by adopting the geometry encoder of VGGT (Wang et al.] (025a) to extract 3D geometry features from video.
3D Reconstruction. 3D reconstruction has evolved from multi-view geometry pipelines to neural implicit representations and, more recently, feed-forward transformer-based architectures. Classical methods yield accurate geometry but require dense views and heavy preprocessing (Schonberger & Frahm 2016 Furukawa et al. 2015). Neural radiance fields and point-based extensions improve fidelity and efficiency but focus mainly on appearance modeling while lacking semantic reason- ing (Mildenhall al. 2021 Barron al. [022]] Kerbl et al., 2023). ). Recent forward approaches enable direct prediction of depth, pose, and point clouds from video inputs (Wang et al.) (0044) Leroy jtt al. 2024 Wang al. 2025a). Nevertheless, 3D reconstruction is still only weakly integrated into
vision langon-l research, and its role in supporting semantic reasoning remains underexplored.
Geometry Priors in Vision-Language Models. Incorporating geometry priors has become a key approach for Vision-Language Models (VLMs) to enhance spatial understanding. Along spec- trum of reliance on explicit 3D inputs, existing methods can be organized into three categories: first, explicit input injection, which introduces depth, point clouds, scene graphs as additional modalities to provide metric properties (Cai et al.) 2025) [Ceeng et al.] 2024b Guo et al.] 2023);
second, internalization at the data and training level, which leverages spatially annotated corpora or geometric distillation to embed geometry implicitly into the alignment space, enabling spatial reasoning without explicit 3D inputs inference (Chen et al.) 2024a] Peng et al.) (003);; and third, modular or prompt-based integration, which augments VLMs with lightweight modules or outputs from 3D foundation models, typically without large-scale retraining (Ma et al. 2024 Kerr al.
2023). In contrast, our approach generates geometry through video-driven reconstruction branch and achieves alignment with the semantic branch, enabling structured and reusable integration of geometry priors within video-based setting.
3 METHOD
We present Vid-LLM, video-based 3D multimodal large language model (3D-MLLM). The main components are presented in the following sections: the Cross-Task Adapter described in Sec-
tion 3.1 the reconstruction and reasoning branches detailed in Sections 3.2 and 3.3 and the
training strategy is outlined in Section 3.4. The overall architecture shown in Fig
3.1 CROSS-TASE ADAPTER
In Vid-LLM, employ DINOv2 as the shared visual encoder to extract base tokens Tiase E Nxc from the input image sequence, where N denotes the number of tokens and C is the embedding di- mension. To enhance feature effectiivnesss, we introduce a Cross-Tase Task Adapter (CTA) that aligns 3D geometry priors with vision- language (VL) representations, facilitating the integration geometric cues into multimodal reasoning.
3

---

Published as a conference paper at ICLR 2026
Crozs-Task Adapter
3D Reconstruction
Cameraa
Hexd
MLP
Frame Aliontion
Dense
Head
Vision Encoder (DINOv2)
attention
EPT
Add
bridge
attention
30 Porties
Poeine
placedin
Putches
LLMS
AcAi crange
Asking
sola
Input
30.VL Reaconing
Figure 2: Architecture of Vid-LLM. From video, a shared DINOv2 encoder produces tokens that bidirectionally fused by Cross-Task Adapter with learnable Bridge Tokens, yielding geometric and semantic streams. The reconstruction branch predicts camera poses, depth and recovers real- scale via a Metric-Bins module, while the 3D-VL branch lifts features into 3D tokens for LLM reasoning.
To adapt the shared visual representations for different branches, we employ two lightweight MLP projection heads, (-)om(-) and plang(.). which map the shared vision tokens to geometry speecifc and semantic feature spaces, respectively:
T goom geomm (Thase); Tiang Otang (Tbase)
(1)
To effectively align 3D geometry priors with vision language representations thus enhance the in- tegration of spatial cues into multimodal reasoning, introduce learnable Bridge Tokens, denoted as Tbridge E TR xxx_ Acting as shared memory units, the bridge tokens attend to geometric and semantic features separately, and the updated representation formulated as:
Torsdge Attn(Tbridge: Tfused Tfused) + Attn(Tbridges Tefusd p geom
where Attn(-) denotes standard multi-head attention operation. This operation enables bridge tokens to dynamically capture complementary information from both tasks and update their repre- sentations during training. The joint propagation geometric and semantic signals strengthens the alignment of 3D geometry priors with vision- lsiion-lnnguaee features, leading more robust cross-modal representations.
Finally, the updated bridge tokens Ty Tridge are integrated into the feature streams, yielding enhanced task-specific representations Ty geom and TV Flang" These enriched features capture complementary geo- metric and semantie cues and are subsequently passed to the reconstruction and reasoning branches. geometry-semantics semantics interaction at the fea- ture level, allowing the two streams reinforce and guide each other for more robust representa- tions.
In essence, the Cross-Tase Adapter establishes intrinsic geometry
3.2 3D RECONSTRUCTION MODEL
In the reconstruction branch of Vid-LLM, we build on recent transformer-basee architectures for end-to-end 3D reconstruction (Wang al. 2025a) to recover scene geometry from video inputs. To additionally recover real-scale information, we design a Metric Depth Model that provides robust global scale cues, enabling reconstructions with both fine structural details and metric consisteney.
Geometry Encoding and Prediction Heads. Based on the cross-task enhanced geometric features T" - together with camera tokens 7 and register tokens 7 the Global-Frame Attention back- produces an integrated geometric representation, which is then fed into two prediction heads: camera head estimating extrinsic-ettriniie parameters and DPT head that predicts the relative depth map Dr E HxW where H and W denote the image height and width, respectively.
4

---

Published as a conference paper at ICLR 2026
Geometry Teacher
Cross-Top Task Adapter
Geom Tokerd
4/2 images
D
Tokans
Dianl
Language
Stage2
Reconstruction
DINOv2
Adapter
Point Cloud
imaages
3D-VL Tasks
Jakkd Loss
Figure 3: Overview of the two-stage training strategy. Stage- I employs dual-teacher distillation to align geometry and semantics, and Stage-2 2 jointly optimizes reconstruction and 3D vision-lande lisoun-angguage tasks.
Metric Depth Model. To recover real-scale geometry, equip the DINOv2 features with DPT- style decoder that produces multi-scale depth representations. Each pixel's depth modeled using bin-based formulation, where the probability (k(k) over the k th bin and its refined center c;(k)
jointly determine the prediction as d(i)) = Ei= (k(k (k)).. we use an ordinal-aware normalization to capture relative depth ordering. To further stabilize scale, the bin centers c, are adaptively refined as (k) = Ck + (k)) where (ky(k) = T rr((F) (Fi) predicted from the decoder features F,, The resulting metric depth map Dmteric { HxW provides global scale cues that are aligned with relative predictions for real-scale reconstruction.
Real-Scale Alignment. We estimate scaling factor between the relative depth D predicted by the DPT head and the metric depth Dmetric predicted by the Metric Depth Model via weighted least squares. For each scene, 16 images are randomly sampled to compute per-image scaling factors, and their median is taken median as the final scene-level factor. This factor is then applied convert both the relative depth and the predicted camera pose into real-world units. Rather than directly using metric depth as the final output, we adopt this alignment strategy since the DPT head provides more accurate texture details compared with the Metrie Depth Model, which could also be observed from experimental results.
3.3
3D
MODEL.
In the reasoning branch Vid-LLM, -aansanee--e semantic features Ting are combined with reconstructed geometry generate dense 3D patch representations, which are then fed into the LLM for 3D question answering, grounding, and captioning tasks.
3D Patch Construction. Each 2D feature Tung (i, is back-projected into 3D using the estimated depth D, camera pose (ř, f) and intrinsics K produced by the 3D Reconstruction Model, yielding its camera-frame coordinates ji, as:
(1()))
(3)
These coordinates encoded by MIP into positional embeddings (i,jj) that match the di- mensionality the semantic features. Therefore, the final 3D patch tokens are then obtained by fusing geometry and semantic features:
(ap(i.j) Tang (i.j) + (i
(4)
This operation feeds spatial information into the semantic tokens, further enhancing the spatial awareness in the LLM.
3.4 TRAINING STRATEGY
We adopt a two-stage training strategy to utilize the shared encoder for both geometry and semantics. Stage performs dual-teacher distillation, transferring geometric priors from reconstruction model and semantic knowledge from multimodal LLM, enabling the encoder to learn both capabilities in balanced way. Stage jointly optimizes all downstream modules with 3D vision-language
5

---

Published as a conference paper at ICLR 2026
objectives, while incorporating auxiliary reconstruction losses to provide the model with sufficient reconstruction capability and ensure real-scale consistency. The overall pipeline illustrated in Fig. 3
Stage-1 Dual-Teacher Distillation. In stage I, we adopt a dual-tcacher distillation strategy to jointly train the DINO encoder and the Cross-Task Adapter, enabling the modules to quickly learn both geometric and semantic representations. The pretrained DINO encoder and CLIP encoder serve the geometry and semantic teachers in the distillation strategy, which are initialized from VGGT (Wang et al. 2025a) and LLaVA-3D (Zhu et al. 2025), respectively. The distillation loss is defined as:
Ldistau = Leat + feat + ALsc
(5)
where Lgeat, feat and Lse are the geometry loss, semantic loss and structural consistency loss, respectively. À is a balancing hyperparameter. For geometry learning, an 1.2 loss is applied to patch-level features, following DINO's patch regression strategy. For semantic learning, cosine similarity loss is applied to pooled features to align with CLIP's global semantic embedding space. The specific forms are defined follows:
I Lfeat
Σ INrmm (oomm Norm(T 1l. L L cos(Pool (a Pool tea
(6) where Norm(-) denotes L2 normalization, and Pool(-) indicates mean pooling. T Teoom and Tiang are the task-specific geometric and semantic features extracted by the Cross-Task Adapter, and serve as the student representations in the distillation strategy. Toeom and Tians are features from the DINO and CLIP encoders and serve as teacher representations. N represents the number of tokens in Tv geom-- Before computing L/fe and L/eng- we apply lightweight linear projection layers to align the features to the same embedding dimension. To maintain structural consistency, we also introduce the structural consistency loss Lec, which is defined as:
Lgn
Sat - Steall? where Satu = Zatn ZT M
stu Suma = Zreo Zio tea
(7)
Norm(TTngg) E MxC is obtained by concatenating the geometry and se- mantic tokens from the student representation. Ziea (Tom) Norm(Tlam)) is defined in the same way Zstuu but using the teacher representation. || || is the Frobenius norm, M is the total
Zetu = [Nrm geom): );
number of tokens, and C' is the embedding dimension of each token.
Stage-2 Joint Optimization. In Stage 2, further fine-tune all the modules to optimize overall performance, The joint loss is defined as:
Ljoint taskk LVI.-task LMD
where T-recon-taas task is the multi-task loss for 3D reconstruction, consisting of the camera loss, depth loss, and point map loss following Wang al. (2025a). LyI.-task supervises 3D) vision-language reasoning. including cross-entropy loss for inttruction-follwwee regression and matching losses for grounding (Zhu et al. (2025)). LMD represents the metric depth loss, combining global scale penalty and robust local refinement term, and is defined as:
tasks, along with bounding box
K
(e, b)2
LMD = 12
K
1 + ale, 6)
+ log(d, + E), where is small constant for numerical stability. = 1 ΣKK e; the mean error across all K valid pixels in the image. The K parameter > 0 controls the robustness by downhweigh large residuals.
The log-depth error is defined e; log log(d!ret (dtredd
During joint training, Lrecon-task and LMD optimize the 3D reconstruction model, LVL-tosk opti- mize the 3D vision-language model, while the shared CTA is jointly optimized by Lyecon task and Ly L-tek is worth to noticing that the loss constructed in the 3D-VL reasoning branch is not used to optimize the 3D reconstruction model, which can be seen in Fig, 2 This one-way gradient flow ensures that the CTA acts as a stable bridge for geometry-semantic interaction, enabling effective feature exchange.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation of 3D Question Answer- ing on ScanQA and SQA3D. Methods marked with * are 3D MLLM evaluated in video mode. i indicates the model consumes VGGT-generated 3D geometry. "C" stands for "CIDEr", "B-4" for "BLEU-4", "M" for "METEOR", "R" for "ROUGE", and "EM@ 1" for top-1 exact match.
Table 2: Evaluation of 3D Dense Caption- ing on Scan2Cap. The n-gram metrics for Scan2Cap are governed by loU@0.5.
Method
Scan2Cap
Ce:0.1
B-48P0S1
M680.5
ReD0.
3D-bared models
Scon2Cap
Grounde3D-
70.2
35.0
LEO
68.4
36.9
27.7
57.8
Mathed
Ct
B-4
SaanQA
MI
R
EM@!
SQA3D EM 17
LLaVA-3D
Video LLM
CharScene
77.1
84.1
36.3
42.6
28.9
28.0
29.0
63.4
30-bused models
3DRS
86.1
41.6
28.9
62.3
Scun2Cap
41.0
ScanQA
64.9
10.1
13.1
33.3
21.1
47.2
3DRS
76.4
39.5
27.3
57.1
3D-NisTA
10.4
VGLLM
78.6
40.9
28.6
62.4
3D-LLMM
69.4
12.0
14.5
35.7
20.5
LL3DA
76.8
13.5
15.9
37.3
GhoundefiD-LLLM
75.9
Chat-3D x2
77.1
7,3
16.1
40.1
21,1
Table 3: Evaluation of 3D Visual Ground-
Sccne-LIM
12.0
ChatSeene
87.7
14.3
18.0
41.6
21.6
$1.6
ing ScanRefer and Multi3DRefer.
LED
101.4
13.2
20.0
49.2
24.5
50.0
SeanRefer
MRRter
Vidko-3D
Mechod
LLaVA-3D
103.1
15.4
20.8
49.5
30.6
60.1
A
00.25 0.51
3DRS
104.8
17.2
20.5
49.8
30,3
60.6
Scunkefer
37.3
24.3
Vadeo-based models
Chas 3D v2
35,9
30.4
VILA-40B
48.2
9.9
11.4
27.3
17.2
37.2
GrondddED-LLM
48.6
44.0
44.7
400.8
IXC-2.5-7B
19.4
LLaVA-3D
49.8
LLVVA-O--78
78.2
13.3
17.2
40.5
22.9
47.6
LLaVA Video-7B
88.7
48.5
Video-3D
58.0
Umi
70.3
12.2
14.9
36.3
17.3
3DKS
62.9
56.1
60.4
54.9
ChatScene*
85.6
52.9
12.3
23.9
VGLLM
33.5
41.5
Vid-LLM (Ouns)
101.9
15.8
18.3
49.5
27.6
$7.3
3DR
55.2
51.1
52.8
48.3
ViG.LM
63.2
61.6
EXPERIMENTS
This section provides comprehensive evaluation of Vid-LLM. In Section 4.1 we introduce the experimental setup, covering training details Vid-LLM and the evaluation settings used in our experiments. In Section 4.2 we evaluate the model on 3D vision-language reasoning tasks against oftte-of-the-art methods. In Section 4.3) we compare Vid-LLM with representative joint reconstruction-reassse reasoning models to examine how different design choices affect performance when jointly executing reconstruction and 3D VI. reasoning. In Section 4.4 analyze the contributions of core modules.
we conduct ablation studies
4.1 EXPERIMENTAL SETUP
Training Details. We adopt DINOv2-L as the visual backbone, consisting of 24 Transformer layers with hidden dimension of 1024. The projection MIPs use fully connected layers with an expansion factor of and GELU activation. For data processing, we uniformly sample 32 frames per scene, resize the shorter side 518, crop the resolution that both height and width are multiples of 14. Optimization is performed using AdamW with 31 = 0.9, 32 0.999, and weight decay of 0.05, with gradient clipping at 1.0.
Evaluation Details. We follow the standard evaluation protocols and metrics defined for each dataset. For tables that summarize performance across multiple datasets (c.g.. Tables 5 and 66). the reported numbers are computed as the mean of the metrics for each benchmark. For the Scan2Cap dataset, we follow prior work (Zhu al. 2025| ZZenn al., 2025a): instance proposals are gen- erated using Mask3D (Schult al.)) 2022). and 3D coordinate tokens are constructed from the pre- dicted instance centers to enable instance-aware caption generation. For the ScanRefer dataset, Mask3D (Schult al.) 2022) serves the 3D segmentor to remain consistent with supervised base- lines (Huang et al. 22244 [204||Chen et al. 2024c: Huang et al., (202a).. For the Nr3D and Sr3D datasets, the provided segmentation annotations
adopted align with previous methods (Chen et al. 2022 HHang et al.] 2023a). For reconstruction evaluation on ScanNet. the metrics are computed in metric scale. All models trained with the same data and settings for fair comparison.

---

Published as a conference paper at ICLR 2026
4.2 3D VISION-LANUUAGE REASONING
Overview. To comprchensively assess the performance of Vid-LLM on 3D vision-languagg rea- soning tasks, we conduct experiments using widely adopted datasets covering three task categories: 3D Question Answering on ScanQA (Azuma et al., (022)) and SQA3D (Ma et al. 2022), 3D Dense
Captioning on Scan2Cap (Chen et 20.1). and 3D Visual Grounding on ScanRefer (Chen et al.,
2020), Multi3DRefer (Zhang et al. 2023). and Nr3D/Sr3D (Achlioptas et al. (020b).
Baseline. We compare Vid-LLM with a broad set of 3D-based and video-based vision-language reasoning models. 3D-based methods rely on explicit 3D scene imputs such as point clouds or reconstructed geometry, whereas video-based methods operate solely on video. 'The 3D-based baselines include ScanQA (Azuma et al., 2022). Scan2Cap (Chen al.) 202T), 3DsVisTA (Zhu
et al.
2. 3D-LLLM (Hong et 20023b). LL3DA (Chen et al. 2024b)) Grounded31D-LLLM (Chen 2024c), Chat-3D v2 (Huang et al. 2023a). LEO (Huang et al. (023b), Scene-LL.M (Fu al, ChatScene (Huang al. (222))
et al.
2024).
VA-33 (Zhu et al. 2025). Video-3D LLM (Zheng al.
2025b), and 3DRS (Huang et al. 2025), The video-based baselines include VILA-40B (Lin et al,
2024). IXC (Zhang et |l.]|||2.22 2024bb. LLa VV- (Li | 2024). LLaVA- (LLaVA Team 2024). Uni3DR2 (Chu et al. 2024). and VGLLM (Zheng et al. 2025a). To further assess the grounding additionally compare against task task-specific 3D visual grounding models,
capability of Vid-L.MM,
including ScanRefer (Chen et al.] 2020). 3D-VisTA (Zhu et al. 2023), Referlt3D (Achlioptas et al.,
2020ab, 3DVG-Trans (Zhao et al. 2021 D. MVT (Huang al. (022). ViL.DRRel (Chen [
and Scene (Jia et al. 2024). To allow comparison under the same video input setting with the most competitive 3D-based baseline in our evaluation, we additionally include 3DRS variant of 3DRS that takes VGGT--geerrnn
3D geometry as input. These comparisons collectively provide a comprehensive evaluation of Vid-LLM across 3D vision- language reasoning tasks.
Result& & Analysis. Vid-LLM consistently shows robust performance across all 3D vision- reasoning benchmarks. As shown in Tab. [ Vid-LLMM achieves the best results among video-based models on both ScanQA and SQA3D, outperforming the second- best baseline (3DRS²) by an av- crage margin of 11%. Tab. 2 further shows that Vid-LLM attains the highest performance among video-based methods on Scan2Cap; notably, its M@0.5 score (28.7) is close to that of the best- performing 3D-based model (29.0), despite not using depth or point clouds. These results col- lectively indicate that the integration of geometric cues into semantic reasoning can significantly improve perception performance. Tab. Bl and Tab. present results on ScanRefer/Multi3DReeo Nr3D/Sr3D. Vid-LLM achieves the best performance across all metrics the two grounding bench- marks, surpassing all comparable 3D-based and video-based counterparts. These results demon- strate that the proposed Cross-Task Adapter, which effectively integrates semantic and geometric information, enables effective spatial reasoning and yields substantial gains on geometry-intenss tasks such as 3D visual grounding. For qualitative analysis, we visualize the 3D grounding results on the ScanRefer dataset in Fig. E which further demonstrates the 3D visual grounding capability of Vid-LL. More qualitative visualizations for 3D VL tasks can be found in Appendix
and
4.3 COMPARISON WITH JOINT RECONSTRUCTION RECONTTRUCTIOONRRLLSNNN REASONNINGRAGGGGG MODELS
Overview, In addition to 3D vasion- reasoning, Vid-LLMM also incorporates a reconstruc- tion branch that provides geometric priors and metric-scale cues. This naturally raises the question of how different design choices for coupling reconstruction and reasoning affect performance when both tasks must be performed from a single video input. To provide comprehensive analysis, we baselines:
compare Vid-LLM with two representative categories of joint reconstruction-reasse (i) simple concatenation pipelines that feed the 3D geometry reconstructed from video into 3D multimodal LLM,, and (ii) end-to-end architectures that integrate reconstruction and 3D VI. reason- ing within a single model.
Baseline. For the concatenation baselines, we construct pipelines that feed the predicted geometry information of VGGT into LLaVA-3D (Zhu et al. 2025)) for 3D linion-l reasoning. Three variants of the concatenation pipeline are evaluated: (i) VGGT+LLaVA-3D. the direct combination of VGGT and LLaVA-3D; (ii) VGGT+LLaVA-SDi which introduces metric-scale aligmment to the which further fine-tunes VLaVA-3D using depth and camera poses predicted by VGGT as geometric supervision. For end-to-ent joint-task baselines, include 2ni3DR2 CCh al. 2024), end-to-end framework that integrates reconstruction
predictions of VGGT; (iii) VGGT+LLaVA-3DD
8

---

Published as a conference paper at ICLR 2026
Image View
Grounding
Figure 4: Qualitative results of 3D visual grounding on the ScanRefer dataset. The predicted 3D bounding boxes are visualized on point clouds reconstructed by our model.
Table 4: Evaluation of 3D Visual Ground- Table 5: Comparison with end-to-end frameworks ing on Nr3D and Sr3D, Results are reported for joint 3D reconstruction and 3D VL. reasoning. in grounding accuracy (%).
Method
ScanNeff SamQAI Scun2Gap
3D VL Tisks
SeanReler
somo))
Method
Uverall
Harcl
Ne3D
View Deep Overall
Hard!
SEAD
View
Concolemation models
VGGI-LLaVA-3i
35.1
31.6
2.7
SeanReter
ReforH33
35.6
$4.2
27.0
23.5
29.9
12.5
40.8
11.5
39.2
VGGT-LLaVA-RD
VGOT-LLaVA-3D-
0.591
41.6
48.2
50.9
45.9
47.3
27
2.7
3DVG Trans
40.8
34.8
34.8
51.4
46.9
44.6
Joint-lnsk models
MVT
55.1
49.]
56,3
64.5
58.8
58.4
Uni31K
0.50
30.2
42.7
33.6
21
VIL3DR
62.6
59.8
68.1
M-LLM--Rc
Clai-3D
63.6
68.4
71.3
Vid-LLM (Our))
45.7
53.2
59.8
Somee Veme
61.9
57.8
56.9
77.5
71.6
62.8
1 denotes introducing real scale:
Vid-ITM (Ours
65.4
57.9
61.9
77.8
72.2
63.1
denotes training LLaVA-3D with 3D geometry predicted VGGT
module with a 3D VL reasoning branch. Notably, Uni3DR² requires ground-truth camera poses as input, whereas Vid-LLM performs both tasks directly from video, making the comparison conser- vative in favor of 2 We also include VGLLM-Rec, an extension of VGLLM (Zheng et al, 2025a). To support joint reconstruction and reasoning, we reconnect the original camera and depth prediction heads of VGGT to the geometry encoder in VGLLM, enabling WGLLM-Rec to generate geometric predictions addition to performing 3D VL reasoning from video.
Result & Analysis. As shown in Tab. 5 VGGT+LLaVA-3) obtains very low 3D VL accuracy, indicating that celativ-scalle geometry cannot provide reliable cues for 3D VL reasoning. Building on this, VGGT+LLa LLA--LLLVA-3 shows clear performance gain once metric-scale alignment is ap- plied. However, even with this setting, the concatenation baseline still underperforms Vid-LLM on all 3D VI. tasks despite achieving higher reconstruction score (0.591 vs. 0.582). This indicates that simply providing accurate reconstructed geometry to a 3D-LLM is insufficient and that effee- tive geometry-semane NGGT+LLaVA-3DS yields slight drop in 3D VL accuracy compared with VGGT+LLaVA-3DD suggesting that training the 3D-LLM with noisy predicted geometry may introduce biases that de- grade the semantic representations of the framework. In contrast to concatenation pipelines, end- to-end joint-task models avoid multi-stage processing and achieve lower inference latency, reducing runtime from 2.7 s/scene to 1.6-2.1 s/scene. From Tab. 5 we can observe that Vid-LLM achieves the best overall performance among the joint models on both reconstruction and 3D VL tasks. marginally outperforms Uni in reconstruction quality on ScanNet (0.582 vs. 0.580) despite Uni3DR² having access to ground-truth camera poses. On the 3D VI. tasks, Vid-LLM demonstrates clear advantage, with an average relative improvement of 47.6% over Uni3DR2 and 12.1% over VGLLM-Rec across ScanQA. Scan2Cap, and ScanRefer. Overall, Vid-LLM delivers the fastest in- ference speed and the best 3D VL performance, while also achieving the best reconstruction quality among joint-task models. These gains stem from our integrated architecture and the Cross-Taas Task Adapter, which facilitates geometry- semanties interaction and allows the model to effectively lever- age geometric cues during joint reasoning. More experimental results about reconstruction perfor- mance can be found in Appendix A.2
interaction is necessary for reliable reasoning performance. Furthermore,

---

Published as a conference paper at ICLR 2026
Table 6: Ablation on Cross-Task Adapter (CTA) and Metric Depth (MD) Modules.
1.8
Singlessagge
Two-Stage(On-TT
Two-Sageewlo ReiaLoss)
Two- Sturgl
144
3D VL Tisks
3D Recon
Scan2Cap
ScanReferr
scannet
w/o CTA
33.7
26.6
34.1
0.102
CTA Concat-SA
38.2
41.9
39.6
0.461
1.9
CTA we Dridge (CA)
40.8
45.6
42.2
0.473
CTA Stokens
43.8
50.9
47.8
0.501
CTA Btokems
44.1
52.7
18.1
0.529
0.8
Iniokess (Ouns)
48.4
0.582
(10f)
CTA 32tokens
45.2
52.8
47.6
0.564
#Data
29.6
35.7
33.9
MD w/o Aliigmment
42.1
49.6
43.2
0.531
Figure 5: Test loss vs. data size across train- ing strategies.
(Ours)
45.7
48.4
0.582
4.4 ABLATION STUDIES
Cross-Task Adapter. To assess the contribution of the Cross-Task Adapter, we compare several configurations: removing the CTA module (w/ CTA); in the setting without bridge tokens, applying self-attention to the concatenated T Toom and Tang (CTA-Concat-SA) and performing cross-attention between the two feature sets (CTA-w/o Bridge (CA)); and finally using the full CTA with bridge tokens. We also analyze how different numbers of bridge tokens (4, 8, 16, and 32) affect model performance. As shown in Tab. [ w/o CTA leads to the lowest performance firming that a shared visual representation alone cannot jointly support reconstruction and 3D VL reasoning. Both CTA-Concat-SA and CTA-w/o Bridge(CA) show clear improvements, yet their effectiveness is limited as neither design provides a stable shared latent space for passing geometric cues to semantic features. The configurations using the full CTA module achieve clear performance improvements, since the bridge tokens provide a dedicated latent space that enables consistent geom- etry-semantic alignment and more effective cross-task interaction. Among different token counts, using 16 bridge tokens achieves the best balance between accuracy and model complexity.
both tasks, con-
Metric Depth Modules. Tab. Glaso reports ablations on the use of metric depth and our alignment strategy. Without metric depth supervision (w/ MD), scale ambiguity destroys geometric consis- tency and leads to near failure of 3D VL reasoning. Incorporating metric depth without alignment (MD w/o Alignment) preserves metric scale but produces less accurate reconstruction, which in turn limits the performance of 3D VL tasks. By contrast, combining metric depth with our scale alignment strategy effectively exploits both global scale cues and relative structural details, yielding the most accurate reconstruction and consistently more reliable results on 3D VL tasks.
Two-Stage Training. Fig. compares different training strategies assess their impact on vergence and performance. Training all modules jointly without staging (Single-stage) converges slowly and stabilizes at relatively high loss, due to gradient interference across modules. Using single teacher under the two-stage strategy (Two-Stage (eee-each)))) offers limited supervision, resulting in less effective optimization. Removing the relational loss during two-stage strategy (Two- Stage (w/o RelaLoss)) further slows convergence and reduces accuracy, highlighting the importance of enforcing cross-task consistency. In contrast, using the full two-stage strategy with dual-teacher supervision and relational consistency (Two-Stage (Ours)) yields the best training performance.
CONCLUSION
In this work, we present Vid-LLM, a video-based 3D Multimodal Large Language Model (3D- MLLM). Our compact architecture extracts geometric cues from video and feeds them into the LLM through a 3D patch construction strategy to accomplish spatial reasoning. A central component of our framework is the Cross-Task Adapter that aligns 3D geometry priors with vision-language rep- resentations. This design enhances their integration into the MLLM and improves the robustness of reasoning under uncertain geometry. With a two-stage training strategy, our model achieves greater training stability and faster convergence. Extensive experiments on 3D rision-languag benchmarks demonstrate that Vid-LLM achieves state-of-the the-hhhan art results on several benchmarks and remains com- petitive the others, while ablation studies validate the effectiveness of each component.
10

---

Published as a conference paper at ICLR 2026
6 REPRODUCIBILITY STATEMENT
We are committed to ensuring reproducibility. Part of our implementation provided in the Sup- plementary Materials. Upon acceptance, will release the complete code, datasets, and pretrained checkpoints to enable full verification of our results.
7 ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 42374016), the Major Program (JD) of Hubei Province (Grant No. 2023BAA025), and the Open Fund of Hubei Luojia Laboratory (Grant No. 260100023). The numerical calculations this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.
REFERENCES
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes, In Euro- pean conference on computer vision, pp. 422 440. Springer, 2020a.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes, In Euro- pean conference on computer vision. pp. 422 440. Springer, 2020b.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa:
3d question an- swering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19139, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 5470-5479, 2022.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXhv:2302.12288. 2023.
Wenxiao Cai, laroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang. Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. In 2025 IEEE International Conference Robotics and Automation (ICRA), pp. 9490-9498. IEEE, 2025.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221.. Springer, 2020.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522-20533 2022.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reason- ing and planning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26428 264326638, 2024b.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405. 205374055.00700 2024c.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3193-3203, 2021.

---

Published as a conference paper at ICLR 2026
An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision- lasgunlanguage models. Advances in Neural Information Processing Systems, 37:135002-135099 2024a.
An-Chieh Cheng, Hongxu Yin. Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision- languaee models. Advances in Neural Information Processing Systems, 37:135062-135000 2024b.
Tao Chu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Qiong Liu, and Jiaqi Wang. Unified scene rep- resentation and reconstruction for 3d large language models. arXiv preprint arXiv:2404.13044, 2024.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnnr. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3828-5839, 2017.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401 2024.
Yasutaka Furukawa, Carlos Hernández, al. Multi-view stereo: A tutorial. Foundations and frends@ in Computer Graphics and Vision, 9(1-2)11-44, 2015.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang. Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzii Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint mrXiv:2309.0067.. 2023.
Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. arXiv preprint arXiv:2207.11514 2022.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems, 36:20482-20499 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494 2023b.
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. CoRR, 2023a.
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:1 13991- 114017, 2024.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311. 2023b.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual ground- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15524-15533, 2022.
Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. Mllms need 3d-aware representation super- vision for scene understanding. arXiv preprint arXiv:2506.00 0/946, 2025.
Baoxiong Jia. Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling vision-language learning for grounded scene understanding. In European Conference Computer Vision, pp. 289-310. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
Jihong Ju, Ching Wei Tseng, Oleksandr Bailo, Georgi Dikov, and Mohsen Ghafoorian. Dg-recon: Depth-guided neural 3d scene reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18184-18194, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 42(4): 1,
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 19729-19739, 2023.
Johannes Kopf, Xuejian Rong, Jia-Bin Huang, Kevin Matzen, et al. Robust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2021.
Seoyoung Lee and Joonscok Lee. Posediff: Pose-conditioned multimodal diffusion model for un- bounded scene synthesis from sparse inputs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5007-5017, 2024.
Yao-Chih Lee, Kuan-Wei Tseng, Guan-Sheng Chen, and Chu-Song Chen. Globally consistent video depth and pose estimation with efficient test-time training. arXiv preprint arXiv:2208.02709, 2022.
Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference Computer Vision, pp. 71-91. Springer, 2024.
Bo Li et al. Llava-onevision: Easy visual task transfer. arXiv preprint vrXiv:2408.03326 2024.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shocybi, and Song Han. Vila: On pre-training for visual language models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-perfect cencture-from-motte with featuremetric refinement. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021.
Liman Liu, Fenghao Zhang, Wanjuan Su, Yuhang Qi, and Wenbing Tao. Geometric prior-guided self-supervised learning for multi-view stereo. Remote Sensing, 15(8):2109, 2023. doi: 10.3390/ rs 1s15082109.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and reso- lution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 120099 2022.
LLaVA Team. Llava-video: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02711. 2024.
Chenyang Ma, Kai Lu, Ta- Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. Advances in neural information processing systems, 37:68803-68832, 2024.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Zak Murez, Tarrence As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Ra- binovich. Atlas: End-to-ent 3d scene reconstruction from posed images. In European Conference Computer Vision (ECCV), 2020.
13

---

Published as a conference paper at ICLR 2026
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. ACM Transactions Intelligent Systems and Technology, 16(5):1-72, 2025.
Jia Ning, Chen Li, Zheng Zhang, Zigang Geng, Qi Dai. Kun He, and Han Hu. All in tokens: Uni- fying output space of visual tasks via soft token. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Mingyang Ou, Heng Li, Haofeng Liu, Xiaoxuan Wang, Chenlang Yi, Luoying Hao, Yan Hu, and Jiang Liu. Mvd-net: Semantic segmentation of cataract surgery using multi-view learning. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 5035-5038. IEEE, 2022.
Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3depth: Monocular depth estimation with piecewise planarity prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 815-824, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)9, 2019.
René Ranfil, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12159- 12168, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 10901-10911, 2021.
Ruilong Ren, Xinyu Zhao, Weichen Xu, Jian Cao, Xinxin Xu, and Xing Zhang. survey of language- -rouundde multimodal 3d scene understanding. Knowledge-Baud Systems, pp. 113650, 2025.
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial-language model for 3d visual grounding. In Conference on Robot Learming, pp. 1046-1056. PMLR, 2022.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Paltern Recognition (CVPR), pp. 4938 4938-4947, 2020.
Johannes L Schonberger and Jan-Michael Frahm. Structure-from mootio revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113 2016.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. arXiv preprint arXiv:2210.03105 105, 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujum Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15598-15607, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
14

---

Published as a conference paper at ICLR 2026
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306, 2025a.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa, Con- tinuous 3d perception model with persistent state. arXiv preprint arXiv:2501.12388 2025b.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Viston and Pattern Recognition, pp. 20697-20709, 2024.
Dong Wu, Zike Yan, and Hongbin Zha. Panorecon: Real-time panoptic 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21507-21518, 2024.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip Yu. Multimodal large language models: A survey. In 2023 IEEE International Conference Big Data (BigData), pp- 2247-2256. IEEE, 2023.
Guangkai Xu, Wei Yin, Hao Chen, Chunhua Shen, Kai Cheng, and Feng Zhao. Frozenrecon: Pose- free 3d scene reconstruction with frozen depth models. In 2023 IEEE/CVF International Confer- on Computer Vision (ICCV), pp. 9276-9286. IEEE, 2023.
Jianing Yang, Alexander Sax, Kevin J Liang. Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 21924 21935, 2025.
Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10371 10381,, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. National Science Review, 11(12))nwae433 2024.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm- Ilms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.1360 2024a.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320 2024b.
A versatile large vision language
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yu- jun Shen, and Gordon Wetzstein. Flare: Ferwaforwwrd geometry, appearance and camera estima- tion from uncalibrated sparse views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CV (CVPR), 2025.
Yiming Zhang. ZeMing Gong, and Angel X Chang. Multi3drefer: multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15225-152366 2023.
Grounding text description to
Lichen Zhao, Daigang Cai, Lu Sheng. and Dong Xu. towg-trrasffrmmr: Relation modeling for visual grounding point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu. Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), 2023.
Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3d world: Enhancing milms with 3d vision geometry priors. arXiv preprint arXiv:2505.2440 2025a.
15

---

Published as a conference paper at ICLR 2026
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position- aware video represen- tation for 3d scene understanding. In Proceedings the Computer Vision and Pattern Recognition Conference, pp. 8995-90066 2025b.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprini arXiv: 180::088117 2018.
Chemming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway empowering Imms with 3d capabilitiis. International Conference on Computer Vision [ICCV/19//02223-///1000000
Honolulu, Hawai' i), 2025.
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng. Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 2911-2922 2023.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
In the Appendix, we provide the following experimental results:
qualitative results and challenging cases of Vid-LLM Appendix A.I
supplementary experiments on reconstruction performance in Appendix A.2
the use of large language models (LLMs) statement in Appendix A.3
A.1 QUALITATIVE RESULTS AND CHALLENGING CASES
Reconstruction
Question & Answer
Reconstruction
Question & Answer
3D Question Answering
How many chairs there
Q: Where is the bed the room
Three,
the right the
Q: Where is the monitor?
Q: What color are the curtains.
In the of the desk against the wall.
A: Brown.
3D Dense Captioning
Q: Describe the room detail.
Q: Describe the Washbasin in detail.
A: In the center ohe meeting room, there is light-colored conference table, surrounded by black conference chairs...
A: This is brown washbasin, made entirely of light brown wood, with circular washbasin in the center of the wash...
3D Visual Grounding
There are two beds room. me the coordinates these
Q: Give me the coordinates of the light brown wooden coffee table.
A:[1.13, 1.64, 0.35, 0.92, 0.03, [1.13, (1.3,, 1.64, 0.35, 0.92, 0.03, 1.16)
A:[1.13, 1.64, 0.35, 0.92, 0.03, 1.16]
Figure 6: Qualitative Results.
Qualitative Results. Fig. presents reconstruction results on the ScanNet dataset, together with three representative 3DD vision-language (3D) VI.) tasks. Specifically, it illustrates (1) 3D Question Answering, where the model answers queries on object counts, locations, and attributes from recon- structed scenes; (2) 3D Dense Captioning, which provides detailed semantic descriptions of rooms and key objects; and (3) 3D Visual Grounding, where the model localizes target objects in 3D space according textual instructions. These qualitative results demonstrate the capability of Vid-LLM in scene reconstruction and 3D VL reasoning from video inputs.
Challenging Cases. Fig. 7 presents three challenging cases for Vid-LLM on 3D VL tasks. These scenes reflect typical failure modes of monocular video reconstruction, where limited camera view- points, reflective surfaces, or oblique viewing angles lead to incomplete geometry. As shown in examples (a) and (b), tasks that rely explicit 3D structural information, such as instance counting or estimating 3D object extents for grounding, are directly affected by incomplete or unreliable ge- ometry. In contrast, example (c) shows that Vid-LLM remains effective when sufficient 2D cues present, due to the Cross-Tane Adapter and bridge-toke token mechanism, which jointly align semantic and geometric information. These observations suggest that improving the reconstruction branch to enhance geometric fidelity could further strengthen the overall 3D VL performance, indicating a promising direction for future work.
17

---

Published as a conference paper at ICLR 2026
Input images
Reconstruction
Question Answer
How many chairs there around the dining table?
(a)
A: Five.
GT: Six.
Q: Give me the coordinates of the long wooden table.
(b)
Our box
Red
GT box Green
Q: Describe the wooden cabinet detail.
(c)
A: The long wooden cabinet many labeled drawers. Its cluttered with rolled papers, cardboard boxes, and plastic-wrapped items
Figure 7: Challenging Cases. We present one challenging case for each 3D VL task-3D question answering (a), 3D visual grounding (b), and 3D dense captioning (c). (a) Due to limited camera viewpoints in the input video, chair is only partially observed and consequently missing from the reconstruction, leading to an undercounted result. (b) The highly reflective tabletop results in unstable depth estimates and incomplete reconstruction, resulting in predicted 3D box that is smaller than the ground truth. (c) The top surface of the cabinet is not well reconstructed because appears only briefly and mostly from oblique angles, yet Vid-LLM nonetheless produces accurate caption by leveraging rich 2D semantic cues.
Table 7: Camera pose evaluation on Co3Dy2 and RealEstate10K.
Co3Dv2
RcalEEtatel1UK
Method
RRA@ 151
RTA @ 15t
mAA(30))
mAA(30)1
COLMAP4SG
(Sarlin et al.
2020
36.1
27.3
25.3
45.2
PixSIM
Lindenberger et
2021
33.7
32.9
30.1
49.4
PoseDiff
Lee & Lee
2024
80.5
79.8
66.5
48.0
DUS:3
Wang al
94.3
88.4
77.2
61.2
MAS33
Lenoy el
2024
94.6
FLARE
Zhang
95.4
83.6
78.4
75.3
3Rst3R
Yang et al.
1125
96.2
81.6
75.0
72.7
CUT3R
Wang
202Sh
95.7
84.5
73.3
77.1
VGGT
Wang ct al.
2025a
97.3
93.3
89.6
83.8
Vid-LLM (Ours)
96.8
93.4
88.5
83.1
18

---

Published as a conference paper at ICLR 2026
Table 8: Depth estimation results on the NYU Depth v2 dataset. Model outputs are in relative scale; align them to the ground-truth metric scale prior to evaluation.
Method
511 f
AbsRel
RMSE
log10 4
DPT (Ranftl et al. 2021
0.904
0.110
0.357
0.045
P3Depth (Patil cl al. 2022
0.898
0,104
0.356
0.043
SwinV2-L (Lu ct al. 2022
0.949
0.083
0.287
0.035
AiT [Ning ct al. 202.5
0.954
0.076
0.275
0.033
VPD Zhao el al. 2126
0.964
0.069
0.254
0.030
ZoeDepth (Baa cl u...
2023
0.953
0.075
0.270
0.032
DepthAnything Yang cl Bl.
2024
0.984
0.056
0.206
0.024
VGGTI Wang
2025
0.989
0.022
0.103
0.011
Vid-LLM(OOus)
0.987
0.025
0.109
0.010
Table 9: Reconstruction results on the ScanNet dataset. 1 Model outputs are in relative scale; we align them to the ground-truth metric scale prior evaluation.
Method
GT cams
Comp |
Acc
Recall
Pree
F-score
MVDNet Ou et al.
2022
0.040
0.240
0.831
0.208
0.329
GPMVS [Lu cl al. 2023
0.031
0.879
0.871
0.188
0.304
Atlas
(Muirez et al. 20201
0.062
0.128
0.732
0.382
0.499
NeuralRecon (Sun ct 2021
0.106
0.073
0.428
0.592
0.494
DG-Recon
al 2023
0.085
0.039
0.476
0.675
0,521
PanoRecon wu al.
20/41
0.089
0.064
0.530
0.656
0.584
RCVD Kopf ct al. 2021
X
0,161
0.425
0,164
0.109
0.125
GCVD
Lee al. 2U1P
X
0.175
0.278
0.178
0.146
0.147
Colmap [Schonherger & Frahm 2016
0.142
0.367
0.119
0.267
0.178
FrazenRezon [Xu al. 2023
0.092
0.085
0.436
0.336
0.410
DUS13R [Wang et al. 2024
0,243
0.179
0.284
0.657
0,387
VGGT!
Wang el al. 2025a
0.067
0.044
0.658
0.891
0,580
Vid-LLM(OOrs)
X
0.071
0.042
0.634
0.885
0.582
A.2 SUPPLEMENTARY EXPERIMENTS ON RECONSTRUCTION
Overview. Since the reconstruction branch not only provides 3D cues for the 3D-MLLM but also explicitly produces point clouds that can serve downstream applications, it is important to assess its individual performance. To evaluate the 3D modeling capability of Vid-LLM, we assess three tasks; camera pose estimation, depth prediction, and point-cloud reconstruction. Depth prediction is crucial to real-seale reconstruction, and together with poses determines the quality of the recovered point clouds. Following standard protocols, we evaluate poses Co3Dv2 (Reizenstein et al.) (221) and RealEsttae (Zhou et al.) (2018), depth on the indoor NYU Depth v2 dataset (Silberman et al., 2012), and point-cloud reconstruction on ScanNet (Dai et al., 2017), ensuring consistency with the 3D vision-language benchmarks introduced earlier. Depth and reconstruction evaluations require the model predict real-scale geometric outputs, while camera pose estimation is scale-invariant.
Baseline. For 3D reconstruction, we evaluate three subtasks: (i) camera pose estimation, where compare against geometry-based optimization methods including COLMAP+SG and PixSIM, as well as state-of-tha ahr learning- based approaches such as Dife DUSt3R, MASt3R, FLARE, Fast3R, and CUT3R; (ii) monocular depth estimation, where we benchmark against DPT, P3Depth, SwinV2-L. AiT, VPD, ZoeDepth, and Depth and (iii) point-cloud reconstruction, where we consider pose-dependent methods such as MVDNet, GPMVS, Atlas, NeuralRecon, DG-Recon, and PanoRecon, together with pose-free approaches including RCVD, GCVD, COLMAP, Frozen- Recon, and DUS13R, ranging from traditional multi-view geometry to modern learning-baat based tech- niques. We further benchmark against VGGT, which serves both high-performing reconstruc- tion model and as the reconstruction teacher in our framework.
Result & Analysis. Across the three benchmarks, Vid-LLM achieves reconstruction quality com- parable to VGGT while operating in an end-to-end real-scale setting. On camera pose estimation (Tab. 7, Vid-LLM attains results close to VGGT both Co3Dv2 and RealFistate 10K and slightly
19

---

Published as a conference paper at ICLR 2026
outperforms it on RTA @ 15 for Co3Dv2. On NYU Depth v2 and ScanNet (Tab, 8 and Tab. 9 VGGT requires an explicit allemaligmment step to match the metric ground truth, whereas Vid-LLM pre- diets real-scale depth and point clouds in end-to-end manner. In this setting, Vid-LLM achieves lower log 10 error on NYU and higher F-score and accuracy on ScanNet than VGGT, indicating that its real-scale geometry remains competitive against the teacher model. Vid-LLM also surpasses all other models that predict real-scale outputs. Overall, these results confirm that its reconstruction module delivers consistent and reliable geometric performance.
A.3 THE USE OF LARGE LANGUAGE MODELS (LLMs) STATEMENT
Parts of the language in this manuscript were polished with the assistance of Large Language Models (LLMs). The authors have carefully reviewed and verified all -Ls-ssiitted text to ensure accuracy and appropriateness. The intellectual contributions, ideas, and conclusions presented in this work are entirely those of the authors.
20


Reviews:
[{"review_id": "5j306LhoSB", "reviewer": "Reviewer_D2AN", "summary": "This paper presents Vid-LLM, a compact multimodal model that unifies 3D reconstruction and vision-language reasoning within a single framework. I really like this idea -- combining reconstruction with VL reasoning not only bridges geometry and semantics but also achieves faster inference through a more efficient architecture. The work is conceptually elegant and technically well-motivated.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The model design is well thought out, especially the Cross-Task Adapter (CTA) that enables effective geometry–semantics interaction.\n\n- The overall training pipeline, including two-stage distillation and joint optimization, is clear and coherent.\n\n- The experimental setup is solid - reconstruction and 3D-VL baselines are comprehensive, and the ablation studies are extensive enough to verify the core hypothesis and support the narrative.\n\n- Results demonstrate consistent gains in both reasoning and reconstruction efficiency, validating the proposed synergy between the two tasks.", "weaknesses": "The main baseline (VGGT + LLaVA-3D) appears to involve two independently fine-tuned components that are only concatenated during inference, rather than being jointly optimized end-to-end. This makes the comparison to the proposed method somewhat unfair, as Vid-LLM benefits from full end-to-end training while the baselines do not. It would strengthen the paper to clarify this distinction and, if possible, include a jointly optimized baseline to better isolate the benefit of the Cross-Task Adapter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4AKSugWFr6", "reviewer": "Reviewer_Zqjo", "summary": "This submission proposes Vid-LLM, a video-based 3D multimodal large language model that performs both 3D reconstruction and vision-language reasoning directly from monocular video inputs. The main contributions of this submission are\n\n1. Cross-Task Adapter (CTA) that aligns geometric priors with vision-language features, enabling joint geometry–semantic interaction.\n2. Metric Depth Model (MD) that restores real-scale 3D geometry with bin-based depth prediction and adaptive scale alignment.\n3. Two-Stage Training Strategy with dual-teacher distillation and joint optimization to improve convergence and stability.\n\nThe authors conduct extensive evaluations on 3D-QA, dense captioning, and grounding benchmarks showing consistent performance gains over 3D- and video-based baselines.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written and structured. Overall it easy to follow the technical narrative and experiments.\n- The related work section provides a comprehensive overview of 3D multimodal large language models and video-based reasoning approaches.\n- The overall design, including the Cross-Task Adapter (CTA), Metric Depth (MD) module, and the two-stage training strategy, is well motivated and technically sound. The integration of geometric and semantic cues is conceptually coherent and justified.\n- The ablation studies are thorough, demonstrating the necessity of each proposed component to the final performance.\n- The experimental results show consistent and significant improvements across multiple 3D vision-language reasoning benchmarks.", "weaknesses": "- Missing details and clarifications:\n    - In Figure 2, the predicted 3D reconstruction results are used for generating 3D position embeddings. It would be helpful to clarify whether gradients are propagated through this path when jointly training the reconstruction and reasoning modules.\n    - In Section 3.4, Equation (6), the notation $\\text{Norm}(\\cdot)$ appears ambiguous. Please clarify whether it denotes feature normalization (e.g., L2-normalization) or the 2-norm operation itself. An explicit definition would avoid confusion.\n    - In Section 3.4, Equation (6), what is the dimensionality of $T_{\\text{tea}}^{\\text{lang}}$? Is the feature-level loss$L_{\\text{feat}}^{\\text{lang}}$ averaged over N samples similar to $L_{\\text{feat}}^{\\text{geo}}$? Clarifying this would help in understanding the implementation consistency.\n- Simplification of the bridge token mechanism:\n    \n    The current Cross-Task Adapter introduces bridge tokens for cross-modal interaction. Would a simpler alternative—such as applying self-attention directly on the concatenation of $ T_{\\text{geom}} $ and $ T_{\\text{lang}} $—achieve similar performance with lower complexity? Some empirical or ablation analysis could strengthen the design justification.\n    \n- Qualitative analysis:\n    \n    More qualitative results and failure case discussions would be valuable to understand when the model fails. The authors are suggested to include them in the camera-ready version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7iAJM29om", "reviewer": "Reviewer_1jxT", "summary": "The paper introduces Vid-LLM, a video-based Multimodal Large Language Model for 3D scene understanding that works directly from video, removing the need for external 3D data. The method leverages geometric priors and incorporates them efficiently using a Cross-Task Adapter (CTA) to align 3D geometry with vision-language features. A Metric Depth Model ensures accurate real-scale geometry, and a two-stage distillation strategy improves training. Experiments show Vid-LLM excels at 3D Question Answering, Dense Captioning, and Visual Grounding, outperforming existing methods in multi-task 3D reasoning.", "questions": "1. Please incorporate comparison with recent works.\n\n2. Please add performance comparison with VGGT.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of 3D geometric information into 3D LLMs is well-motivated in this paper.\n\n2. The paper is clearly written and easy to follow.\n\n3. The experiments effectively demonstrate the improvements brought by the proposed modules.\n\n4. It is encouraging to see that the proposed method achieves competitive performance on both pose estimation and depth estimation tasks.", "weaknesses": "1. Integrating 3D foundation models into 3D MLLMs is not a novel concept. Recent works such as VG-LLM [1] and 3DRS [2], which also utilize VGGT to enhance the 3D-awareness of 3D MLLMs, are not cited or compared in this paper. Including a discussion or direct comparison with these approaches would strengthen the related work section and better contextualize the contributions of this work.\n\n2. In Tables 7 and 8, it is unclear how the proposed model's performance compares to that of the teacher model, VGGT. Providing this comparison would improve the clarity and completeness of the evaluation.\n\nReferences:\n\n[1] Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors. NeurIPS 2025.\n\n[2] MLLMs Need 3D-Aware Representation Supervision for Scene Understanding. NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)