Segment Any 3D Gaussians
Authors: name: Jiazhong Cen; name: Jiemin Fang; name: Chen Yang; name: Lingxi Xie; name: Xiaopeng Zhang; name: Wei Shen; name: Qi Tian
Published: 2023-12-01 17:15:24

Abstract: Interactive 3D segmentation in radiance fields is an appealing task since its
importance in 3D scene understanding and manipulation. However, existing
methods face challenges in either achieving fine-grained, multi-granularity
segmentation or contending with substantial computational overhead, inhibiting
real-time interaction. In this paper, we introduce Segment Any 3D GAussians
(SAGA), a novel 3D interactive segmentation approach that seamlessly blends a
2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent
breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D
segmentation results generated by the segmentation foundation model into 3D
Gaussian point features through well-designed contrastive training. Evaluation
on existing benchmarks demonstrates that SAGA can achieve competitive
performance with state-of-the-art methods. Moreover, SAGA achieves
multi-granularity segmentation and accommodates various prompts, including
points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation
within milliseconds, achieving nearly 1000x acceleration compared to previous
SOTA. The project page is at https://jumpat.github.io/SAGA.