FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D
  Scene Understanding
Authors: name: Xingxing Zuo; name: Pouya Samangouei; name: Yunwen Zhou; name: Yan Di; name: Mingyang Li
Published: 2024-01-03 20:39:02

Abstract: Precisely perceiving the geometric and semantic properties of real-world 3D
objects is crucial for the continued evolution of augmented reality and robotic
applications. To this end, we present  (), which
incorporates vision-language embeddings of foundation models into 3D Gaussian
Splatting (GS). The key contribution of this work is an efficient method to
reconstruct and represent 3D vision-language models. This is achieved by
distilling feature maps generated from image-based foundation models into those
rendered from our 3D model. To ensure high-quality rendering and fast training,
we introduce a novel scene representation by integrating strengths from both GS
and multi-resolution hash encodings (MHE). Our effective training procedure
also introduces a pixel alignment loss that makes the rendered feature distance
of same semantic entities close, following the pixel-level semantic boundaries.
Our results demonstrate remarkable multi-view semantic consistency,
facilitating diverse downstream tasks, beating state-of-the-art methods by
10.2 percent on open-vocabulary language-based object detection,
despite that we are 851times faster for inference. This research
explores the intersection of vision, language, and 3D scene representation,
paving the way for enhanced scene understanding in uncontrolled real-world
environments. We plan to release the code upon paper acceptance.