Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
  Understanding
Authors: name: Duo Zheng; name: Shijia Huang; name: Liwei Wang
Published: 2024-11-30 14:28:53

Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has
significantly impacted various multimodal tasks. However, these models face
challenges in tasks that require spatial understanding within 3D environments.
Efforts to enhance MLLMs, such as incorporating point cloud features, have been
made, yet a considerable gap remains between the models' learned
representations and the inherent complexity of 3D scenes. This discrepancy
largely stems from the training of MLLMs on predominantly 2D data, which
restricts their effectiveness in comprehending 3D spaces. To address this
issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
for 3D scene understanding. By treating 3D scenes as dynamic videos and
incorporating 3D position encoding into these representations, our Video-3D LLM
aligns video representations with real-world spatial contexts more accurately.
Additionally, we have implemented a maximum coverage sampling technique to
optimize the balance between computational costs and performance efficiency.
Extensive experiments demonstrate that our model achieves state-of-the-art
performance on several 3D scene understanding benchmarks, including ScanRefer,
Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.