Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision
  Geometry Priors
Authors: name: Duo Zheng, name: Shijia Huang, name: Yanyang Li, name: Liwei Wang

Previous research has investigated the application of Multimodal Large
Language Models (MLLMs) in understanding 3D scenes by interpreting them as
videos. These approaches generally depend on comprehensive 3D data inputs, such
as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research,
we advance this field by enhancing the capability of MLLMs to understand and
reason in 3D spaces directly from video data, without the need for additional
3D input. We propose a novel and efficient method, the Video-3D Geometry Large
Language Model (VG LLM). Our approach employs a 3D visual geometry encoder that
extracts 3D prior information from video sequences. This information is
integrated with visual tokens and fed into the MLLM. Extensive experiments have
shown that our method has achieved substantial improvements in various tasks
related to 3D scene understanding and spatial reasoning, all directly learned
from video sources. Impressively, our 4B model, which does not rely on explicit
3D data inputs, achieves competitive results compared to existing
state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the
VSI-Bench evaluations.