LangSplat: 3D Language Gaussian Splatting
Authors: name: Minghan Qin; name: Wanhua Li; name: Jiawei Zhou; name: Haoqian Wang; name: Hanspeter Pfister
Published: 2023-12-26 15:14:37

Abstract: Human lives in a 3D world and commonly uses natural language to interact with
a 3D scene. Modeling a 3D language field to support open-ended language queries
in 3D has gained increasing attention recently. This paper introduces
LangSplat, which constructs a 3D language field that enables precise and
efficient open-vocabulary querying within 3D spaces. Unlike existing methods
that ground CLIP language embeddings in a NeRF model, LangSplat advances the
field by utilizing a collection of 3D Gaussians, each encoding language
features distilled from CLIP, to represent the language field. By employing a
tile-based splatting technique for rendering language features, we circumvent
the costly rendering process inherent in NeRF. Instead of directly learning
CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
then learns language features on the scene-specific latent space, thereby
alleviating substantial memory demands imposed by explicit modeling. Existing
methods struggle with imprecise and vague 3D language fields, which fail to
discern clear boundaries between objects. We delve into this issue and propose
to learn hierarchical semantics using SAM, thereby eliminating the need for
extensively querying the language field across various scales and the
regularization of DINO features. Extensive experiments on open-vocabulary 3D
object localization and semantic segmentation demonstrate that LangSplat
significantly outperforms the previous state-of-the-art method LERF by a large
margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
times speedup compared to LERF at the resolution of 1440 times 1080. We
strongly recommend readers to check out our video results at
https://langsplat.github.io