SAMPart3D: Segment Any Part in 3D Objects
Authors: name: Yunhan Yang; name: Yukun Huang; name: Yuan-Chen Guo; name: Liangjun Lu; name: Xiaoyang Wu; name: Edmund Y. Lam; name: Yan-Pei Cao; name: Xihui Liu
Published: 2024-11-11 17:59:10

Abstract: 3D part segmentation is a crucial and challenging task in 3D perception,
playing a vital role in applications such as robotics, 3D generation, and 3D
editing. Recent methods harness the powerful Vision Language Models (VLMs) for
2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation.
However, these methods are limited by their reliance on text prompts, which
restricts the scalability to large-scale unlabeled datasets and the flexibility
in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable
zero-shot 3D part segmentation framework that segments any 3D object into
semantic parts at multiple granularities, without requiring predefined part
label sets as text prompts. For scalability, we use text-agnostic vision
foundation models to distill a 3D feature extraction backbone, allowing scaling
to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we
distill scale-conditioned part-aware 3D features for 3D part segmentation at
multiple granularities. Once the segmented parts are obtained from the
scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels
to each part based on the multi-view renderings. Compared to previous methods,
our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse
and handle complex, non-ordinary objects. Additionally, we contribute a new 3D
part segmentation benchmark to address the lack of diversity and complexity of
objects and parts in existing benchmarks. Experiments show that our SAMPart3D
significantly outperforms existing zero-shot 3D part segmentation methods, and
can facilitate various applications such as part-level editing and interactive
segmentation.