LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with
  TriMap Video Diffusion
Authors: name: Fangfu Liu, name: Hao Li, name: Jiawei Chi, name: Hanyang Wang, name: Minghui Yang, name: Fudong Wang, name: Yueqi Duan

Recovering 3D structures with open-vocabulary scene understanding from 2D
images is a fundamental but daunting task. Recent developments have achieved
this by performing per-scene optimization with embedded language information.
However, they heavily rely on the calibrated dense-view reconstruction
paradigm, thereby suffering from severe rendering artifacts and implausible
semantic synthesis when limited views are available. In this paper, we
introduce a novel generative framework, coined LangScene-X, to unify and
generate 3D consistent multi-modality information for reconstruction and
understanding. Powered by the generative capability of creating more consistent
novel observations, we can build generalizable 3D language-embedded scenes from
only sparse views. Specifically, we first train a TriMap video diffusion model
that can generate appearance (RGBs), geometry (normals), and semantics
(segmentation maps) from sparse inputs through progressive knowledge
integration. Furthermore, we propose a Language Quantized Compressor (LQC),
trained on large-scale image datasets, to efficiently encode language
embeddings, enabling cross-scene generalization without per-scene retraining.
Finally, we reconstruct the language surface fields by aligning language
information onto the surface of 3D scenes, enabling open-ended language
queries. Extensive experiments on real-world data demonstrate the superiority
of our LangScene-X over state-of-the-art methods in terms of quality and
generalizability. Project Page: https://liuff19.github.io/LangScene-X.