Point-SAM: Promptable 3D Segmentation Model for Point Clouds
Authors: Yuchen Zhou
Jiayuan Gu
Tung Yen Chiang
Fanbo Xiang
Hao Su
Venue: ICLR 2025 Poster (2025)
Keywords: 3D vision
promptable segmentation
point cloud segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The development of 2D foundation models for image segmentation has been significantly advanced by the Segment Anything Model (SAM). However, achieving similar success in 3D models remains a challenge due to issues such as non-unified data formats, poor model scalability, and the scarcity of labeled data with diverse masks. To this end, we propose a 3D promptable segmentation model Point-SAM, focusing on point clouds. We employ an efficient transformer-based architecture tailored for point clouds, extending SAM to the 3D domain. We then distill the rich knowledge from 2D SAM for Point-SAM training by introducing a data engine to generate part-level and object-level pseudo-labels at scale from 2D SAM. Our model outperforms state-of-the-art 3D segmentation models on several indoor and outdoor benchmarks and demonstrates a variety of applications, such as interactive 3D annotation and zero-shot 3D instance proposal.

Published as a conference paper at ICLR 2025
POINT-SAM: PROMPTABLE MODEL FOR POINT CLOUDS
3D
SEGMENTATION
Yuchen Zhou Jiayuan Gu
Tung Yen Chiang" Fanbo Xiang'
Hao Su
HHillbot Inc. 2ShanghaiTech University
UC San Diego
ABSTRACT
The development of 2D foundation models for image segmentation has been sig- nificantly advanced by the Segment Anything Model (SAM). However, achieving similar success in 3D models remains challenge due to issues such as non-unified data formats, poor model scalability, and the scarcity of labeled data with diverse masks. To this end, we propose 3D promptable segmentation model Point-SAM, architecture
focusing on point clouds. We employ an efficient transformer-bast tailored for point clouds, extending SAM to the 3D domain. We then distill the rich knowledge from 2D SAM for Point-SAM training by introducing a data engine to generate part-level and object-level pseudo-labels at scale from 2D SAM. Our model outperforms stae-off-the-ae 3D segmentation models on several indoor and outdoor benchmarks and demonstrates a variety of applications, such as interactive 3D annotation and zero-shot 3D instance proposal.
INTRODUCTION
The development of 2D foundation models for image segmentation has been significantly advanced by Segment Anything (Kirillov et al., (003). That pioneering work includes promptable segmentation task, a segmentation model (SAM), and a data engine for collecting a dataset (SA-1B) with over 1 billion masks. SAM shows impressive zero-shot transfcrability to image distributions and tasks. Thus, it has been widely used in many applications, c.g., segmenting foreground objects for songge-ooddiiiond 3D generation (Liu et al. 2023c ab. NeRF (Cen et al. 2023b), and robotic tasks [Wang et al. 2023 Chen et al. 2023).
Can we just lift SAM to create 3D foundation models for segmentation?? Despite few efforts (Yang et al. 2023 Xu et al. 2023) Zhou et al.) 2023b) to extend SAM to the 3D domain, those existing approaches limited to applying SAM 2D images and then lifting the results to 3D. This process is constrained by image quality, and thus is likely to fail for textureless or colorless shapes like CAD models (Lambourne al. 2021). }. Besides, it is also affected by view selection. Too few views may not adequately cover the entire shape, while too many views can significantly increase the computational burden. Moreover, it can suffer from multi-view inconsistency when results are merged from different views, since they may conflict and be impacted by occlusions. Furthermore, multi-view images only capture surface, making infeasible label internal structures, essential for annotating articulated objects (e.g. drawers in cabinet). Therefore, it is necessary to develop native 3D foundation models to address the aforementioned limitations.
However, developing native 3D foundation models, or extending SAM to the 3D domain, presents several challenges: 1) There is no unified representation for 3D shapes. 3D shapes can be represented by meshes, voxels, point clouds, implicit functions, multi-view images, while 2D images are usually represented by a dense grid of pixels. Unlike 2D images, 3D shapes can vary significantly in scale and sparsity. For example. indoor and outdoor datasets often cover different ranges and typically require different models. 2) There are no unified network architectures in the 3D domain. Due the heterogeneity of 3D data, different network architectures have been proposed for different representations, such as PointNet (Qi et al. 2017a) for point clouds and SparseConv (Graham al. 2018 for voxels. 3) It is more difficult to scale up 3D networks. 3D networks are natively more computationally costly. For instance, SAM utilizes deconvolution and
"Equal Contributions, Corresponding Authors: yuz256@uus @ uesd.edu, gujy @ @eshanghatech.ccuu

---

Published as a conference paper at ICLR 2025
Data
Model
Application
seg labels
Hetcrogenowes
Datasets
PartNet
StanDSyNN
ScanNet
Indotrionsdur
parisiofjccts
point cloud
Point-SAM
outdoor segmentation
indoor segmentation
GTipseudo
wio seg Inbrls
segmentation
ShapeNet
Objaverse
SAM
interactive part annotation
zero-shot instance proposal
data engine
prompt points
Figure 1: We propose 3D extension of SAM, named Point-SAM (Sec. 3). which predicts masks given the input point cloud and prompts. To scale up training data, we develop a data engine (Sec. generate pseudo labels with the help of SAM. The final models, trained on a mixture of datasets, are capable of handling data from various sources and producing results at multiple levels of granularity. We demonstrate the versatility and efficacy of our approach through multiple applications and downstream tasks, as detailed in Sec. 5
bilinear upsampling in its decoder, whereas there are no 3D operators for point clouds as efficient as their 2D counterparts. 4) High-quality 3D labels, especially those with diverse masks, are rare. SAM is initially trained on existing datasets with ground-truth labels of low diversity, and then used facilitate annotating more masks at different granularity (e-g., part, object, semantics) to increase label diversity. However, in the 3D domain, existing datasets contain only a small number of segmentation labels. For example, the largest dataset with part-level annotations, PartNet (Mo et al.) 2019), includes only about 26,671 shapes and 573,585 part instances.
In this work, our goal is to build 3D promptable segmentation model for point clouds, founda- tional step towards 3D foundation models. Point clouds are selected as our primary representation, because other representations can be readily converted into point clouds, and real-world data is often captured in this format. Following SAM, we address 3 critical components: task, model, and data. We focus on the 3D promptable segmentation task, which involves predicting valid segmentation masks in response any given segmentation prompt. To address this task, propose 3D extension of SAM, named Point-SA.M. We utilize a transformer-based encoder to embed the input point cloud, alongside point prompt encoder for point prompts, and a mask prompt encoder for mask prompts. Point-cloud and prompt embeddings are fed to a transformer-basee tation masks. To efficiently encode point clouds and pointwise prompts, develop novel tokenizer based on Voronoi diagram to obtain point-cloud embeddings. as input to the transformer-base encoder. Regarding data, train Point-SAM
mask decoder to predict segmen-
mixture of heterogeneous datasets, including PartNet and ScanNet (Dai et al. 2017). with both part- and object-level annotations. To expand label diversity and leverage large-scale unlabeled datasets such ShapeNet (Chang et al., 2015), we have developed data engine to generate pseudo labels with the assistance of SAM. This pipeline enables us distill knowledge from SAM, and our experiments demonstrate that these pseudo labels significantly improve zero-shot transferabilitty Our contributions include:
We develop a 3D promptable segmentation model Point-SAM, adept at processing point clouds from various sources in unified way. A novel tokenizer based Voronoi diagram is proposed to efficiently embed point clouds and dense prompts.
We propose data engine to generate pseudo labels with substantial mask diversity by dis- tilling knowledge from SAM. It is shown significantly enhance our model's performance on out-of-distribution (OOD) data.
Our experiments demonstrate the strong zero-shot transferability of our model to unseen point cloud distributions and new tasks, positioning it as a 3D foundation model.
2 RELATED WORK
Lifting 2D foundation models for 3D segmentation Despite the growing number of 3D datasets, high-quality 3D segmentation labels remain scarce. To address this, 2D foundation models trained on web-scale 2D data, such as CL.P (Radford et al. 2021). GL.P (Li et al. 2022), and SAM (Kirillov
2

---

Published as a conference paper at ICLR 2025
et al., 2023), have been leveraged. A prevalent framework involves adapting these 2D foundation
models is for 3D applications by merging results across multiple views (Liu et al.
2023d (Zhou
al. 2024 (Cen et al. 2024). SAM3D (Yang et al., (023) and SAMPro3D (Xu et al.
2023 utilize
RGB-D D images with known camera poses to lift SAM to segment 3D indoor scenes. PartSLIP (Liu
et al. 2023b Zhou et al. 2023b), dedicated to part-level segmentation, first renders multiple views of a dense point cloud. then employs GLIP and SAM to segment parts, and finally consolidates multi-view results into 3D predictions. These methods are limited by the capabilities of foundation models and the quality of multi-view rendering. Besides, they usually require complicated and slow post-processing to integrate multi-view results, which also poses challenges in maintaining multi-view consistency. Another strategy involves distilling knowledge from 2D foundation models directly into 3D models. For example, Segment3D (Huang et al., 2023) and SAL (OKep et al., 2024) both utilize SAM to generate pseudo labels given RGB images and train native 3D models on scene-level point clouds. However, these approaches can only handle surface points, making it difficult to segment internal structures that are common in part-level segmentation of articulated 3D shapes such as cabinets with drawers.
3D foundation models The development of 3D foundation models has advanced notably. Point- BERT Yu et al. 2022b) proposes self-supervised paradigm for pretraining 3D representations for point clouds. OpenShape (Liu et al. (224) and Uni3D (Zhou et al. (023a) scale up 3D representations with multi-modal contrastive learning. (Hong et al. 2023) trains 3D-based I Large Language models (3D-LLMM on collected diverse 3D-language data, utilizing 2D pretrained VLMs.. LEOO (Huang et al.) 2024). sharing similar ideas, focuses on embodied ability such as navigation and robotic manipulation. Our work concentrates segmentation, such as OpenScene (Peng et al. (202)) and OpenMask3D (Takmaz et al. 2024). these primarily address scene-level segmentation and are trained on relatively small datasets.
3D segmentation. Despite several initiatives aimed at open- world 3D
3D interactive segmentation Interactive segmentation has been explored across both 2D and 3D domains. (Kirillov et al. 2023) introduces groundbreaking project including the promptable segmentation task, the 2D foundation model (SAM), and a data engine to collect large-scale labels. In the 3D domain, InterObject3D (Kontogianni et al. 2023) and AGILE3D ( Yue et al.] 2023) share similar ideas segment point clouds while their training confined to ScanNet (Dai et al. 2017). In contrast, our model is designed to handle both object- and part-level segmentation, leveraging wide range of datasets including CAD models and real scans. Thus, our model shows greater versatility and adaptability. Besides, 3D interactive segmentation is also explored within implicit representations. SA3D (Cen et al. 2023b) enables users to achieve 3D segmentation of any target object through a single one-shot manual prompt in rendered view. SAGA (Cen al. (023a) distills SAM features into 3D Gaussian point features through contrastive training. While these methods necessitate an additional optimization process, model operates feed-forward basis and can respond within seconds, offering a more efficient solution.
3 POINT-SAM
In this section, we present Point-SAM, a promptable segmentation model for point clouds. Fig. provides overview of PAM. Inspired by SAM (Kirillov et al. 2023), Point-SAM consists of components: a point-cloud encoder, a prompt encoder, and a mask decoder. Unlike 2D models, Point- SAM addresses unique challenges related to point clouds: computation efficiency, scalability, and irregularity. We denote the input point cloud as P Nx3 and its point-wise feature F E NxD
Point-cloud encoder with Voronoi tokenizer The point-cloud encoder transforms the input point cloud into a point-cloud embedding. Inspired by recent advancements in 3D point-cloud transformers (Zhao et al. 202T: Wu et al. 2022, Yu et al. 2022a| Zhou al.] 2023a), we employ similar transformer- based encoder. Concretely, first selects a fixed number of centers C' E R.x3 using farthest point sampling (FPS), and groups the k-nearest neighbors of each center patch. A local PointNet (Qi al., 2017ap is used tokenize each patch Gpatch TR xx+xxx3+D) The resulting patch tokens Fpatch E L combined with the positional embeddings of the group centers, are processed by a Vision Transformer (Dosovitskiy et al. 202T) to generate the final point-cloud embedding Fpc E RRLD
3

---

Published as a conference paper at ICLR 2025
mas
raich R Di
mask deseder
print [N D1)
Scattor dlus
beawway fiemm
Indices
cloud
uikentacr
DI
I-NN
VIT
DI
FPS
Varool tskenivee
point
poonpt points 3)
DN. (..3
1-N8
scatier-max
(a) Network Architecture
(b) Vorenoi Tokenizer
Grouping Diagram
Figure 2: Overview of Point-SAM. (a) illustrates the overall network architecture. The model takes a point cloud along with several point prompts as inputs. Initially, the point cloud divided into patch tokens using Voronoi tokenizer. After that, the patch tokens are embedded through vanilla Vision Transformer (VIT). The token features are then fused with the mask features from the previous iteration. The two-way transformer is employed to allow interaction with the features of the prompt points. Finally, a lightweight decoder generates the mask output. (b) depicts the design the Voronoi tokenizer, where Voronoi diagram is used for grouping the high-resolution point cloud into patch tokens, instead of relying on traditional K K-nearest neighbors (KNN) methods. (c) provides a visual diagram of the grouping process within the Voronoi tokenizer.
We observe that L x is typically much larger than N, making the process tokenizing point clouds obtain Fpatch time-consuming and memory-intensive. To address this, propose novel tokenizer based on the Voronoi diagram, which strikes a balance between efficiency and effectiveness. Specifically, we group points by assigning each point its nearest center, forming Voronoi diagram where each patch corresponds to a Voronoi cell. An MLP is then used to extract pointwise features Fattc E RNN based on the relative position of each point to its nearest center. Patch tokens Fattc E RRLDD are max-pooled within each Voronoi cell via a scatter-max operator.
Prompt encoder The prompt encoder encodes various types of prompts into prompt embeddings. In this work, focus two types of prompts: points and masks. Point prompts processed similarly to SAM. Each point is associated with a binary label indicating whether it is foreground prompt. These prompts are encoded their positional encodings (Tancik et al. 2020 Fpoint E RQXD summed with learned embeddings indicating their labels. Q denotes the number of point prompts. Mask prompts are represented as pointwise logits Xmask E Nx typically derived from the model's previous predictions. These logits are concatenated with the input point cloud's 's coordinates and processed through tokenizer described in the point-cloud encoder. The resulting mask prompt embeddings F Fask TR are element-wise summed with the point-cloud embedding.
Mask decoder The mask decoder efficiently maps the point-cloud embedding, prompt embeddings, and an output token Fout 1 into segmentation mask Ymask xx We follow SAM to employ two Transformer decoder blocks that use prompt self-attention and cross-attention in two directions (prompt-to-ponl tottto----ooooud and vice versa), to update all embeddings. Different from the 2D counterpart, we upsample the updated point-cloud embedding Fpc. E IR 1.x to match input resolution by using inverse distance weighted average interpolation based on nearest neighbors (Qi et al. 2017b), followed by an MLP. The upsampled point-cloud embedding is denoted as Xpe E R XD Another MLP transforms the output token to the weight of a dynamic lincar classifier Xont E RRIDD, which calculates the mask's foreground probability at each point location outt Consistent with SAM, our model can generate multiple output masks for single point prompt by introducing multiple output tokens. Note that multi-mase maak outputs are enabled only when there is only single point prompt with mask prompts. In addition, also introduce another token Fiom e IR 2x x to predict the loU score for each mask output, where is the number of multiple mask outputs.
as Ymask Xxpe XT
4

---

Published as a conference paper at ICLR 2025
Table 1: Summary of training datasets.
PartNet- Mobility
ScanNet- block
Dataset
PartNet
ScanNet
Fusion360
ShapeNect
Overall
Number of point clouds
16442
2163
1198
24328
35000
20000
99131
Average number of masks Has ground truth labels
13
16
30
2
17
True
True
True
True
True
False
Training Mask predictions are supervised with weighted combination of focal loss (Lin et al. 2017) and dice loss (Milletari et al., 2016). in line with SAM. We simulate an interactive setup, detailed in Sec. Sc.. by sampling prompts across 7 iterations per mask. The loss for mask prediction is computed between the ground truth mask and the predictions at all iterations. More details are provided in App. A For multiple mask outputs, we follow SAM to use "hindsight" loss, where we only back-propan propppate only the minimum loss over masks. Additionally, the predicted IoU score is supervised using a mean squared error loss. For training, we randomly sample 10,000 points as input. Besides, we normalize the input point to fit within unit sphere centered at zero, to standardize the inputs. The number of patches and the patch size K are set to 512 and 64 by default.
Inference with variability significant challenge in handling 3D point clouds their irregular input structure; the number of points can vary, necessitating dynamic approach to group points into a varying number of patches with adjustable sizes. While previous point-based methods (Zhou al.] 2023a) are typically limited to processing fixed number of points, our model's flexible design allows it to handle larger point sets than those used during training, by adjusting the number of patches and the patch size. Unless otherwise specified, set the number of patches and the patch size to 2048 and 512 when the number of input points exceeds 32768. In contrast, voxelization- based methods (Yue et al. (023) struggle with such variations changing voxel resolution can significantly impact performance, the results with different voxel resolutions are shown in App. B
4 TRAINING DATASETS
Integrating existing datasets Foundation models are typically data-hungry, and the diversity of segmentation masks is crucial to support "segment anything". Thus, we use a mixture of existing datasets with ground truth segmentation labels, which summarized in Table We utilize synthetic datasets including the training split of PartNet (Mo et al.] [2019). PartNet- Morbiet-Mobility (Xiang et al. 2020), and Fusion360 (Lambourne al., 2021). Since PartNet does provide textured meshes, only keep the models that are from ShapeNet where textured meshes are available. We use all part hierarchies of PartNet. For PartNet-Mobility, we hold out 3 categories (scissors, refrigerators, and doors) not included in ShapeNet, which are used for evaluation on unseen categories. For PartNet and Fusion360, we uniformly sample 32768 points from mesh faces. For each object Mobility we render 12 views, fuse point clouds from rendered RGB-D images, and sample 32768 points from the fused point cloud using Farthest Point Sampling (FPS). For scene-level datasets, we use the training split of ScanNer200 (Dai et al.) (017)) and augment it by splitting each scene into blocks. The augmented version is denoted as Scan
Concretely, we use a 3m 33 block with stride of 1.5m. We use FPS to sample 32768 points per scene or block.
Generating pseudo labels Existing datasets lack sufficient diversity in masks. Large-scale 3D datasets like ShapeNet (Chang et al. 2015) usually do not include part-level segmentation labels. Besides, most segmentation datasets only provide exclusive labels, where each point belongs to single instance. To this end, we develop a data engine to generate pseudo labels.
Figure 3 Jillustrates the pseudo label generation process. Initially, Point-SAM is trained on the mixture of existing datasets. Next, we utilize both pre-trained Point-SAM and SAM to generate pseudo labels. Concretely, for each mesh, we render RGB-D D images at 6 fixed camera positions and fuse colored point cloud. SAM is applied to generate diverse 2D proposals for each view. For each 2D proposal, intend to find 3D proposal corresponding to it. We start from the view corresponding to the 2D proposal. A 2D prompt is randomly sampled from the 2D proposal and lifted to a 3D prompt, which prompts Point-SAM to predict 3D mask the fused point cloud. Then, we sample the next
5

---

Published as a conference paper at ICLR 2025
sample prompt
sample prompt
SAM
SAM
rendered view
2D seg mask
View
rendered view
2D seg mask
View
render sample
View 1 prompt
View 2 prompt
Point-SAM
Point-SAM
point cloud
mask prompt
33D seg mask
refined mask
Posilive prompl
Negutive prompt
noise removed by 2D prior
Figure 3: Illustration of pseudo label generation. Initially, we select one segmentation mask from the instance proposals (" ("segment everything") generated by SAM on the first view. Then, we prompt Point-SAM by lifting 2D prompt points to 3D (View 1 prompt). Subsequently, the 3D segmentation mask output by Point-SAM is refined using additional views. We first prompt SAM by projecting the 3D segmentation mask onto the second view (View 2), leveraging SAM's strong prior knowledge to revise the mask. Then, sample more 2D prompt points from the revised area by SAM, and prompt Point-SAM again by lifting these points to 3D (View 2 prompt).
2D prompt from the error region between the 2D proposal and the projection of the 3D proposal at this view. New 3D prompts and previous 3D proposal masks are fed to Point-SAM to update the 3D proposal. The process is repeated until the IoU between the 2D proposal and the projection of the 3D proposal is larger than threshold. This step ensures 3D 3D-consistent segmentation regularized by Point-SAM while retaining the diversity of SAM's predictions. We repeat the above process with few modifications at other views to refine the 3D proposal. At other view, we first sample the initial 2D prompt from the projection of previous 3D proposal, which is used to prompt SAM to generate multiple outputs. The output 2D mask with the highest IoU relative to the projection is selected as the "2D proposal" in the previous process. If the loU is lower than a threshold, the 3D proposal is discarded. Previous 3D proposal mask is used to prompt Point-SAM at each iteration. This step aids in refining the 3D masks by incorporating 2D priors from SAM through space carving. We use our data engine generate pseudo labels for 20000 shapes from ShapeNet. On average, each shape is annotated with 17 masks, offering a diversity comparable to PartNet.
EXPERIMENTS
We have conducted the experiments showing the strong zero-shot transferability and the superior efficiency of our method. Experiments are conducted on zero-shot cant-prompted segmentation (Sec.(5.1) few-shot part segmentation (B.)) and zero-shot object proposal generation (App. (.I).B.1). Furthermore, we showcase an application of 3D interactive annotation in our supplementary materials.
5.1 ZERO-SHOT
SEGMENTATION
Task and metric The task is to segment instances based on 3D point prompts. For automatic evaluation, point prompts need to be selected. We adopt the same method to simulate user clicks described in Kontogianni et al. (2023). In brief, the first point prompt is selected as the "center" of the ground truth mask, which is the point farthest from the boundary. Each subsequent point is chosen from two candidates: one from the false- calse-positive set at the farthest minimum distance the complementary set, and the other from the false-negative set selected similarly. Then, the candidate farther from the boundary is selected. See App. C for details. This evaluation protocol is commonly

---

Published as a conference paper at ICLR 2025
Table 2: Summary of evaluation datasets.
PartNet- Mobility
Property
ScanObjectNN
S3DIS
KITT1360
Replica
Overall
Number of point clouds
125
2,723
68
379
17
3312
Average number of masks Average number of points
3
9
152
10,000
157,669
522,058
40,000
123
used in prior 2D Kirillov et al. 2023 Zhang et al. 2024) and 3D (Kontogianni et al. 2023 Yue
et al.. (202) works on interactive single-object segmentation. Following those prior works, use the metric loU@k, which is the Intersection over Union (IoU) between ground truth masks and prediction given I point prompts. The metric is averaged across instances.
Datasets We evaluate on heterogeneous collection of datasets, covering both indoor and outdoor data, along with part- and object-level labels. For part-level evaluation, we use the synthetic dataset PartNet-Mobility (Xiang et al., 2020) and the real-world dataset ScanObjectNN (Uy al.) 2019). As mentioned in Sec. 4 we hold out 3 categories of PartNet-Mobility for evaluation. In the same way as the training dataset, we render 12 views for each shape, fuse a point cloud from multi-view depth images, and sample 10,000 points for evaluation. ScanObjectNN contains 2902 objects of 15 categories collected from SceneNN (Hua et al. 2016) and ScanNet (Dai et al. 2017). For scene-level evaluation, we use S3DIS (Armeni et al. 2016) and KITTI-360 (Liao et al. 2022). Specifically, we use the processed data from AGILE3D Yuue et al. 2023), which contains scans cropped around each instance. Table 2 summarizes the datasets used for evaluation.
Baselines We compare Point-SAM with a multi-view extension of SAM, named MV-SAM, and
3D interactive segmentation method, AGILE3D YYue et al. (023). Inspired by previous works (Yang al. 2023 Xu et al. 2023] Zhou al. 2023b) that lift SAM's multi-view results to 3D, we introduce MV-SAM for zero-shot point-prompted segmentation as a strong baseline. First, we render multi- view RG images from the mesh of each shape. Note that mesh rendering is needed to ensure high-quality images, which are essential for good SAM's performance. Thus, this baseline actually has access to more information than ours. Then we prompt SAM at each view with the simulated click sampled from the "center" of the error region between the SAM's prediction and 2D ground truth mask. The predictions are subsequently lifted back to the sparse point cloud (10,000 points) and merged into single mask. If point is visible from multiple views, its foreground probability is averaged. For both MV-SAM and our method, we select the most confident prediction if there are multiple outputs. AGILE3D is close to our approach, while it uses a sparse convolutional U-Net as its backbone and only trained on the real-world scans of ScanNet40. Besides, it does not normalize its input, and thus it is sensitive to object scales. To process CAD models without known physical scales, we adjust the scale of the input point cloud for AGILE3D, so that its axis- sais--a bounding box has maximum size of 5m, determined through a grid search (App. B.3)
Results Table Jableesrresents the quantitative results. Point-SAM shows superior zero-shot transferabil- ity and effectively handle data with different numbers of points as well as from different sources. Point-SAM significantly outperforms MV-SAM, especially when only few point prompts are pro- vided, while MV-SAM achieves reasonably good performance with a sufficient number of prompts. Notably, for IoU @ MV-SAM actually samples & prompts per view. It indicates that our 3D native method is more prompt-efficient Besides, it is challenging for SAM to achieve multi-view consistency without extra fine-tuning, especially with limited prompts. Moreover, Point-SAM also surpasses AGILE3D across all datasets, particularly in out-of-distributio (OOD) scenarios such as Partiie-Mobility (held-out categories) and KITTI360. underscores the strong zero-shot transferabil- ity of our method and the importance of scaling datasets. Figure 4 the qualitative comparison between Point-SAM, AGILEE3 and MV-SAM, where Point-SAM demonstrates superior quality with single prompt and significantly faster convergence compared to AGILE3D and MV-SAM.
Table 3 also compares the Voronoi tokenizer with the previous KNN tokenizer. We observe that the Voronoi tokenizer achieves comparable performance to the KNN tokenizer, while showing superior efficiency. We test the time and memory efficiency on a single Nvidia RTX-4090 GPU using point clouds from KITTI360. For each point cloud, 10 prompt points sampled. The Voronoi tokenizer increases the frames per second (FPS) by 163.4 163.4% > 13.7 shape/sec) and reduces GPU memory usage by 18.4% (3890- 3172 MB).
7

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative results for zero-shot point-prompted segmentation. The notation Voronoi indicates the use of the Voronoi tokenizer and KNN indicates the use of the KNN tokenizer, while all other settings remain unchanged.
Dataset
Method
loU@
loU@3
loU@5
loU@7
IoU@
AGILE3D
26.4
40.8
50.8
57.4
61.9
PartNet-Mobility
MV-SAM
29.3
57.0
69.7
74.3
76.9
Ours (Voronoi)
46.0
68.1
73.2
76.0
77.8
Ours (KNN)
47.9
67.7
74.2
77.0
78.6
AGILE3D
34.8
52.0
61.6
67.2
72.3
ScanObjectNN
Ours (Voronoi)
52.4
76.3
81.4
84.0
85.7
Ours (KNN)
49.4
75.3
82.0
84.8
86.3
InterObject3lD++
32.7
69.0
80.8
89.2
S3DIS
AGILE3D
58.7
77.4
83.6
86.4
88.5
Ours (Voronoi)
63.6
81.8
84.8
86.4
87.3
Ours (KNN)
47.6
78.4
86.2
89.2
90.4
InterObject3D++
3.4
11.0
19.9
40.6
KITTI360
AGILE3D
34.8
42.7
44.4
45.8
49.6
Ours (Voronoi)
52.8
71.4
81.3
83.9
85.7
Ours (KNN)
49.4
74.4
81.7
84.3
85.8
Table 4: Ablation study on training dataset. We report the loU@k metrics for zero-shot prompt- segmentation on PartNet-Mobility (held-out categories).
PartNet +ScanNet
PartNet +ShapeNet
PartNet
Training dataset
PartNet
Full
SShapeNet+ Netn
IoU@1 1
38.2
39.7
44.5
45.4
47.9
56.3
58.6
65.2
66.5
67.7
loU@5
60.6
68.8
71.8
72.6
74.2
IoU@ 10
63.5
71.9
76.2
77.5
78.6
5.2 ABLATIONS
Scaling up datasets Previous works have been limited by the size and scope of their training datasets. For example, AGILE3D (uue et al. (023) was trained solely on ScanNet (Dai et al. 2017 which includes only 1,201 scenes. As detailed Table our training dataset encompasses 100,000
point clouds, 100 times larger than ScanNet. To verify the effectiveness of scaling up training data, we conduct an ablation study on dataset size and composition. We introduce dataset variants: 1) PartNet only, 2) PartNet+ScanNee (including SartNet-ShapeNetttSeaee ScanNet. We train Point-SAi on these variants, resulting in different models. PartNet-Mobilit
3) PartNet+ShapeNet (pseudo labels), and 4)
Table Table the comparison of these models in zero-shot prompt segmentation (held-out categories). The model trained on PartNet+ScanNe surpasses the one trained solely on PartNet, although the evaluation dataset (part-level labels) has markedly different distribution from the added ScanNet (object-level labels). Moreover, the model trained on PartNettSShapeNe achieves even better performance, particularly with single prompt. Note that the loU@) metric assesses whether the model captures sufficient mask diversity, since single prompt is inherently ambiguous and the ground- ground-truth label depends on dataset bias. suggests that our pseudo labels effectively incorporate part-level knowledge distilled from SAM. Furthermore, it is observed that the zero-shot performance on -iato-d-iitributionn data consistently improves, as we utilize increasingly larger and more diverse data.
Trained on ScanNet only To ensure a fair comparison with AGILE3D, we conduct an ablation study, where Point-SAM trained exclusively ScanNet, following the same settings AGILE3D. This resulting model denoted as Point-SAM* Table preseentts the comparison among AGILE3D, Point-SAM* and the original Point-SAM. Provided similar training data, Point-SAM* significantly
8

---

Published as a conference paper at ICLR 2025
3D Scene
AGILE3D@
A
Point-SAM@I
Point-SAM@3
Ground Truth
KITTI360
Object
MVSAM@1
MVSAM@3
Point-SAM@1
Point-SAM@3
Ground Truth
PartNet-
Figure 4: Qualitative results of prompt segmentation are presented for three different settings: KITTI360 for zero-shot outdoor scene segmentation, S3DIS for indoor scene segmentation, and PartNet-Mobility for zero-shot part segmentation. We compare our results with AGILE3D on KITTI360 and S3DIS, and with MVSAM on PartNet-Mobility. Point-SAN demonstrates superior segmentation results with fewer prompt points across all three datasets. Red points represent positive prompt points, while blue points indicate negative prompt points.
outperforms AGILE3D on the OOD datasets like KITTI360 PartNer-Mobility although perform- ing worse the in-domain dataset S3DIS. We hypothesize that AGILE3D is highly optimized for ScanNet (the only dataset was trained on), and some of its design choices may lead to overfitting to this dataset, like its objective of simultaneously generating exclusive multiple objects.
To demonstrate the effectiveness of distilling knowledge from SAM, conduct an ablation study on our design the data engine. We compare our current pipeline with a baseline that uses pre-trained Point-SAM generate instance proposals as pseudo labels. Specifically, sample
Data Engine

---

Published as a conference paper at ICLR 2025
Table Ablation studies on model design and data engine.
(a) Quantitative results for Point- SAM trained on ScanNet. Point-
SAM* is trained ScanNet following the setting of AGILE3D.
(b) Ablation study on the data engine. Different models are trained on different pseudo label datasets, and evaluated on PartNee-Mobility. * means using pseudo labels generated by Point-SAM ( ("segment everything") without SAM.
Point-SAM SAM refers to our original setting.
Dataset
Method
lotr@t
IoU
1ol
ToU@ 7
Jol, 10
Pariiit-Mollity
AGILE3D
Point SAM
33.5
48.5
57.0
61.2
66.8
Point-SAN
47,9
67.7
74.2
77.0
78.6
ScanOhjNN
F3D
Point- SAM
32.0
$6.7
63.2
68.8
70.5
PartNet Mohility
PartNet
PartNet +ShapeNet
PartNet +ShapeNer*
Point-SAM
49.4
75.3
$2.0
84.8
86.3
S3DIS
AGILE3D
58.7
77.4
83.6
86.4
88.5
loU@1
38.2
39.6
44.5
Point-SAM
38.8
67.1
72.2
78.9
80.6
IoU@3
56.3
65.2
56.7
PAM
loU@5
60.6
71.8
64.6
KITT1360
AGII E3D
34.8
42.7
44.4
45.8
49.6
IoU@
63.5
76.2
Point: SAM
44.0
67.1
78.9
80.8
Point-SAM
49.4
87.1
84.3
85.8
Table 6: Sensitivity to point count. We report the IoU @k metrics for zero-shot prompt-segmentttion on S3DIS, with varying the number patches and patch size.
(#patches, patch size)
(512,64)
(512,256)
(2048,64)
(2048,256)
IoU@I @l
41.9
49.0
47.4
47.6
IoU @3
64.2
69.9
74.3
78.4
loU @5
72.2
76.4
81.7
86.2
loU
76.7
80.2
85.8
90.5
1,024 prompt points from the point cloud, using each point to prompt Point-SAM. Non-Maximum Suppression (NMS) is applied to filter out duplicate instances. T5ble(55coompaeee the models trained on pseudo labels generated by our pipeline and the baseline. Interestingly, Point-SAM even benefits from pseudo labels generated by itself. Moreover, incorporating 2D SAM plays a crucial role in improving the quality of the pseudo labels, leading to substantial boost in overall performance.
Sensitivity to Point Count Point clouds are typically irregular. When handling point clouds with more points than those used in our training, we have adjust the number of patches and the patch size accordingly. Thus, we conduct experiments to study the effect of these two hyperparameters. Table shows the qualitative results of zero-shot prompt-segmennationn on S3DIS (Armeni al.. 2016 We select S3DIS, because the average number of points for S3DIS is about 500K, 50 times larger than that our training datasets. Our results indicate that is important to increase the number of patches to accommodate larger point clouds. Enlarging the patch size is also crucial due to the different neighborhood densities compared to our training distribution.
CONCLUSION
In conclusion, our work presents significant strides towards developing a foundation model for 3D promptable segmentation using point clouds. By adopting censformer-based architecture, have successfully implemented Point-SAM, which effectively and efficiently responds to 3D point and mask prompts. Our model leverages robust training strategy across mixed datasets like PartNet and ScanNet, which has proven beneficial, especially when enhanced with pseudo labels generated through our novel pipeline that distills knowledge from SAM.
However, there are inherent limitations and challenges in our approach. The diversity and scale of the 3D datasets used still lag behind those available in 2D, posing challenge for training models that can generalize well across different 3D environments and tasks. Furthermore, the computational demands of processing large-scale 3D data and the complexity operations remain significant hurdles. Our reliance on pseudo labels, while beneficial for expanding label diversity, also introduces dependencies on the quality and variability of the 2D labels provided by SAM, which may not always capture the complex nuances of 3D structures.
developing efficient 3D-specific
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1534-1543, 2016.
Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv preprint arXiv:2312.0086/ 2023a.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems, 36:25971 1559990,, 2023b.
Jiazhong Cen, Jiemin Fang, Zanwei Zhou, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment anything in 3d with radiance fields, 2024. URL sttps::/aaxiv..rt
abs 0bs/2300 2304. 12309
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015.
Linghao Chen, Yuzhe Qin, Xiaowei Zhou, and Hao Su. Easyhec: Accurate and automatic hand-eye calibration via differentiable rendering and space exploration. IEEE Robotics and Automation Letters, 2023.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Benjamin Graham, Martin Engeleke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9224 9232, 2018.
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20499 2023.
Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 2016 2016fourth international conference on 3D vision (3DV), pp. 92-101. leee, 2016.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
Rui Huang. Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. arXiv preprint arXiv:2312. 172332 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Theodora Kontogianni, Ekin Celikkan, Siyu Tang, and Konrad Schindler. Interactive object segmen- tation in 3d point clouds. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2891-2897. IEFE, 2023.
11

---

Published as a conference paper at ICLR 2025
Joseph G. Lamboure,, Karl D.D. Willis, Pradeep Kumar Jayaraman, Aditya Sanghi, Peter Meltzer, and Hooman Shayani. Brepnet: A topological message passing system for solid models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12773-12782, June 2021.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong. Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training, In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
Yiyi Liao, Jun Xie, and Andreas Geiger. KITTI-360: novel dataset and benchmarks for urban scene understanding in 2d and 3d. Pattern Analysis and Machine Intelligence (PAMI), 2022.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:231 1,07855, 2023a.
Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, and Hao Su. Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21736-21746, 2023b.
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open- open-world understanding. Advances in Neural Information Processing Systems, 36, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298-9309, 2023c.
Yichen Liu, Benran Hu, Chi-Keung Tang, and Yu- Yi-Wing Tai. Sanerf-hq: Segment anything for nerf in high quality. arXiv preprint arXiv:23/2.01531, 2023d.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 665-571. Iece, 2016.
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 909-918, 2019.
Aljoša OSep, Tim Meinhardt, Francesco Ferroni, Neehar Peri, Deva Ramanan, and Laura Teix- Better call sal: Towards learning to segment anything in lidar. arXiv preprint arXiv:2403.13129, 2024.
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 815-824, 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference compuler vision and patlern recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. 2023.
Julian Straub, Thomas Whelan, Lingni Ma. Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon. Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv: /e0p://00077777 2019.
Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Keil,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model real- world data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1588 1597, 2019.
Ziyu Wang, Yanjie Ze, Yifei Sun, Zhecheng Yuan, and Huazhe Xu, Generalizable visual reinforcement learning with segment anything model. arXiv preprint arXiv:2312.17116 2023.
Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2:
Grouped vector attention and partition-based pooling, Advances in Neural Information Processing Systems, 35:33330-33342, 2022.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 11097-11107, 2020.
Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707, 2023.
Yunhan Yang. Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306. 03908, 2023.
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 3-,, 2016.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 19313-19322, 2022a.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jíwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling, 2022b.
Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. Agile3d: Attention guided interactive multi-object 3d segmentation. arXiv preprint arXiv:2306.00977. 2023.
13

---

Published as a conference paper at ICLR 2025
Zhuoyang Zhang, Han Cai, and Song Han. Efficientvit-sam:: Accelerated segment anything model without performance loss. arXiv preprint arXiv:2402.05008, 2024.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021.
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale, 2023a.
Kaichen Zhou, Lanqing Hong, Enze Xie, Yongxin Yang, Zhenguo Li, and Wei Zhang. Serf: Fine- grained interactive 3d segmentation and editing with radiance fields, 2024. URL https://
Yuchen Zhou, Jiayuan Gu, Xuanlin Li, Minghua Liu, Yunhao Fang, and Hao Su. Partslip++: Enhancing low-shot 3d part segmentation multi-view instance segmentation and maximum likelihood estimation. arXiv preprint arXiv:2312.0300 2023b.
14

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS
Training recipe. Point-SAr Sonnt-SAMii is trained with the AdamW optimizer. We train Point-SAM for 100k iterations. The learning rate (Ir) is set to 5e-5 after learning rate warmup. Initially, the Ir is warmed up for 3k iterations, starting at 5e-8. A step-wise scheduler with a decay factor of 0.1 is then used, with Ir reductions 60k and 90k iterations. The weight decay is set to 0.1. The training batch size for Point-SAM, utilizing ViT-g as the encoder, is set to per GPU with a gradient accumulation of 4, and it is trained on 8 NVIDIA H100 GPUs with total batch size of 128. The ViT-I version can be trained across 2 NVIDIA A 1100 GPUs, with a batch size of 16 per GPU and gradient accumulation of 4, for 50k iterations. For Point-SAM, utilizing ViT-I as the backbone, the step-wise learning rate decay milestones are set at 30k and 40k iterations.
Data augmentation. We apply several data augmentation techniques during training. For each object, we pre-sample 32,768 points before training and then perform online random sampling of 10,000 points from these 32,768 points for actual training. We apply random scale for the normalized points with scale factor of [0.8, 1.0] and a random rotation along y-axis from -880 to 180". For object point clouds we also apply a random rotation perturbation to x- and z-axis. The perturbation angles sampled from normal distribution with standard deviation (sigma) of 0.06, and then these angles are clipped to the range [0, 0.18].
B ADDITIONAL EXPERIMENTS
B. ZERO-SHOT OBJECT PROPOSALS
In this section, we evaluate Point-SAM on zero-shot object proposal generation. The ability to automatically generating masks for all possible instances is known as "segment everything" in SAM. SAM samples 64x64 point grid the image prompts, and uses Non-Maximum Suppression (NMS) based on bounding boxes to remove duplicate instances. We adapt this approach for 3D point clouds with some modifications. First, sample prompts using FPS, and then prompt Point-SAM to generate 3 masks per prompt. For post-processing. a modified version of NMS based on point-wise masks is applied.
We compare with OpenMask3D Takmaz et al. 2024) on Replica (Straub et al. 2019). OpenMask3D
utilizes class-agnostic version of Mask3l (Schult et al. 2023) trained on Scan Net200 generate object proposals. For our Po,,t-SAM, we sample 1024 prompts and set the NMS threshold to 0.3. In addition, handle the extensive point counts Replica, we downsample each scene to 100,000 points and later propagate the predictions to their nearest neighbors at the original resolution. We also adjust the number of patches and the patch size 4096 and 64 respectively. For both methods, we truncate the proposals to the top 250.
We use the average recall (AR) metric. We filter out "undefined" and "floor" categories from the ground truth labels. Table 7a shows the quantitative results. Point-SAM showcases strong performance compared to OpenMask31, which is tailored for this task, even though our model never trained on such large number of points and is zero-shot evaluated on unseen data. It highlights the robust zero-shot capabilities our method.
Table 7: Qualitative results
of zero-shot object proposal generation and few-shot part segmentation.
(a) Zero-shot object proposal genera- tion on Replica.
(b) Few-shot part segmentation on ShapeNetPart. The numbers
with * are reported by Uni3D (Zhou et al. (023a).
Method
AR25
AR50
PointBERT Uni3D(close) Uni3D(open) Ours
OpenMask3D
40.2
31.5
1 I-shot 2-shot
66.2* 71.9*
71.5
75.9*
73.9
Ours
49.2
31.5
73.8
78.2*
76.1
B.2
FEW-SHOT PART SEGMENTATION
Foundation models can be effectively fine-tuned for various tasks. In this section, we demon- strate that Point-SAM has captured good representations for part segmentation. We compare with
15

---

Published as a conference paper at ICLR 2025
PointBERT Yu al., 2022b and Uni3D Zhou et al. 2023a) close- vossbuooa..l.ry. few-shot part segmentation. We use ShapeNetPart (i et al. 2016 and report the mloUc, which is the mean IoU averaged across categories. Similar Uni3D, we adapt Point-SAM for close-vocabulary part segmentation. Specifically, we extract features from the 4th, 8th, and last layers of the ViT in our encoder and use feature propagation (Qi al., 2017b) upscale them into point-wise features, followed by an MLP to predict point-wise multi-class logits. During few-shot training, we freeze our encoder and only optimize the feature propagation layer well the MLP using cross-entropy loss. Unlike PointBERT and our method, Uni3D originally aligns point-witd woint-wise features with text features of ground truth part labels extracted by CLIP. We refer it as Uni3D (open), since it is designed for open-vocaben vocabulary part segmentation. We also evaluate its variant sharing our modification for part segmentation, denoted as Uni3D (close). Table 7b presents the results for both s-ohot and 2-shot settings. Point-SAM surpasses both PointBERT and Uni3D (close), which indicates that our approach has acquired versatile knowledge applicable to downstream tasks.
B.3 NORMALIZATION SCALE FOR AGILE3D
We conduct grid search to determine the optimal normalization scale for AGILE3D on PartNet- Mobility. Table&Sshov the effect of normalization scale for AGILE3D in zero-shot prompt segmenta- tion on PartNet-Mobility (held-out categories). We find that AGILE3D achieves its best performance with normalization scale of
Table 8: The effect of normalization scale for AGILE3D.
Normalization scale
IoU@3
loU@5
IoU@7
IoU@9
29.0
36.4
41.0
44.7
35.7
43.1
48.4
51.7
40.8
50.8
57.4
61.9
36.8
45.6
51.3
57.5
10
35.1
42.2
48.6
53.1
C PROMPT SAMPLING
Following AGILE3D Yue et al., 2023 we sample two prompt points for each instance: one from false positive points and another from false negative points. The prompts selected by identifying the foreground point that has the furthest distance to the nearest background point. Specifically, this involves computing pairwise distances from foreground points background points, determining the minimum distance to background points for each foreground point, and selecting the foreground point with the maximum of these minimum distances.
After computing the distances and selecting the candidates, we have two prompt point candidates: the point sampled from false positive points serves as negative prompt, and the point sampled from false negative points serves as positive prompt. We select the one with the furthest distance to the nearest background points as the final prompt point.
D VORONOI TOKENIZER
We shows the whole table of the Voronoi tokenizer efficiency improvement in Table 9 Voronoi-based tokenizer surpasses the KNN-based tokenizer all the datasets for both time and memory. Different with 2D SAM, the mask encoder of Point-SAM also need point cloud tokenizer, which makes the voronoi tokenizer more important.
E TRAINING DATASETS ABLATION
Table presents the ablation study on training datasets evaluated across different datasets. This experiment highlights the scaling-up effect as more data is incorporated. The results demonstrate
16

---

Published as a conference paper at ICLR 2025
Table 9: This table shows the evaluation results of the Voronoi tokenizer. As the Voronoi tok- enizer increases the time and memory efficiency significantly, it's very important for the real-world applications.
Method
FPS@1
FPS@5
@PS@ 10
Memory
PartNet-Mobility
KNN
22.3
11.7
7.3
2524
Voronoi
28.6 (+34.4%)
22.4 (+82.0%)
17.8 (+143.8%)
2078 (-21.5%)
ScanObjectNN
KNN
5.9
3.0
1.4
8548
Voronoi
7.3 (+23.7%)
4.9 (+63.3%)
2.3 (+62.0%)
6316 (-26. %))
S3DIS
KNNN
3.31
1.57
0.93
13680
Voronoi
3.52 (+34.4%)
1.72 (+8.7%)
1.04 (+11.8%)
10788 (-20.5%)
KITTI360
KNN
15.7
8.9
5.2
3890
Voronoi
21.1 (+34.4%)
16.2 (+82.0%)
13.7 (+163.4%)
3172 (-18.4%)
Table 10: This table shows the datasets ablation results evaluated on all the evaluation datasets.
PartNet +ScanNet
PartNet +ShapeNet
PartNet +ScanNet+ShapeNet
Eval Dataset
Point Number
PartNet
ScanNet
Full
loU@1
38.2
33.5
39.7
44.5
45.4
47.9
PartNet -Mobility
IoU@3
56.3
48.5
58.6
65.2
66.5
67.7
60.6
S7.0
68.8
71.8
72.6
74.2
IoU@
63.5
66.8
71.9
76.2
77.5
78.6
IoU@!
38.4
32.0
44.8
45.2
48.2
49.4
63.9
56.7
71.8
72.4
73.9
75.3
ScanOhjectNN
loU@5
66.8
63.2
78.9
80.2
81.4
82.0
loU 10
67.7
70.5
80.8
81.4
84.8
86.3
loU@!
25.6
38.8
43.5
30.5
46.2
47.6
IoU@3
48.7
65.3
73,6
54.1
75.9
78.4
S3DIS
15
63.2
71.4
83.9
67.9
85.8
86.2
IoU@ 10
67.9
80.6
88.3
76.3
90.1
90.4
IoU@
36.2
44.0
46.3
39.0
47.3
49.4
loU@]
62.9
67.1
70.5
67.8
72.1
74.4
KITT1360
IoU@5
68.9
72.2
77.9
73.5
80.4
81.7
loU @ 10
74.8
80.8
83.6
80.9
85.0
85.8
that adding data consistently enhances performance, although the extent of improvement varies depending on the type of data and the evaluation dataset. For example, adding ShapeNet leads to greater improvements on PartNet-Mobility, while adding ScanNet has a more pronounced effect on S3DIS. Furthermore, we observe that increasing the dataset size also improves performance on KITTI360, suggesting that the transferability of Point-SAM increases as the amount of training data grows.
F MORE VISUALIZATION
We provide additional qualitative results in the appendix. Figure presents qualitative results on the Waymo Open dataset. As fully out-of-distribution experiment, this figure highlights the transferability of Point-SAM, demonstrating its ability correctly segment outdoor objects such as cars and trees.
Figure shows the ground truth segmentation results alongside outcomes with varying numbers of prompt points. As Waymo is an OOD dataset, this figure demonstrates the superior transferibility of Point-SAM.
Figure 6 illustrates the few-shot segmentation results on the ShapeNet-Part dataset. The pre-trained model is used the embedding for linear probing. We randomly select one sample from each category
17

---

Published as a conference paper at ICLR 2025
ground truch
prompt
prompts
prompes
Figure 5: This figure shows the segmentation results of Waymo.
ground truth
Uni3D
Point-SAM
ground truth
Uni3D
Point-SAM
ground truth
Uni3D
Point-SAM
ground truth
Uni3D
Point-SAM
Figure 6: This figure shows the few-shot segmentation results on ShapeNet-
as the training set, then perform inference on the evaluation set to obtain the one-shot probing results. These results demonstrate the superior pre-trie pre-training quality of Point-SAM for segmentation tasks.
G EVALUATION FOR INTERIOR SEGMENTATION
Table shows the segmentation results for the StorageFurniture from PartNet-Moblite We use the Point-SA.M trained only on PartNet, and ScanNet. As shown in Table [1] MV-SAM achieves worse performance on StorageFurning than the other categories shown in Table Point-SAM achieves better performance than MV-SAM with interior segmentation masks.
We present the evaluation results for the StorageFurniture category from PartNet-Mobility: To ensure the evaluation data is not in the training dataset, we used Point-SAM trained on PartNet. ShapeNet and ScanNet.
Table 11:
IoU@1
IoU@3
IoU@5
loU@7
IoU@10
MV-SAM
26.9
51.0
63.3
65.3
67.3
Point-SAM
42.3
63.4
68.6
71.8
74.5
18

---

Published as a conference paper at ICLR 2025
Table 12: This table presents the evaluation results Replica. Following the evaluation procedure of AGILE3D, we crop the point clouds centered on the segmentation mask.
IoU@
loU@3
loU@5
IoU@7
IoU@ 10
AGILE3D
55.9
74.8
81.7
85.4
87.9
Point-SAM
58.3
79.5
86.2
90.1
91.4
H EVALUATION ON REPLICA
We present the quantitative results on Replica Table 12
We still follow the progress of evaluating S3DIS and KITTI360 to split the scene into blocks centering on the segmentation target. In this experiment, we use the Point-SAM trained on the whole training dataset with Voronoi-basee based tokenizer. Point-SAM outperforms AGILE3D the evaluation of Replica.
I MORE INTERACTIVE SEGMENTATION RESULTS
Figure shows more visualization results for complicated scenes and objects. We show both the segmentation results projected to meshes and the raw point cloud results. Figure showw the superior transferibility of Point-SAM.
19

---

Published as a conference paper at ICLR 2025
mesh results
point cloud results
mesh results
point cloud results
Figure 7: This figure presents additional visualization results of the interactive promptable segmenta- tion. All objects were sourced from Polycam and Objaverse. We show both the projection results on meshes and the raw results on point clouds.
20


Reviews:
[{"review_id": "qD4Ni6qyZS", "reviewer": "Reviewer_AfEp", "summary": "This paper addresses the problem of deploying 3D foundational models by introducing a novel 3D promptable segmentation model for point clouds, named Point-SAM. This model extends the Segment Anything Model (SAM) into the 3D domain by developing a promptable 3D architecture that utilizes both point and mask prompts. These prompts, together with the input point cloud, are processed through a transformer-based encoder-decoder architecture to generate a 3D segmentation mask.\n\nThe Point-SAM model first tokenizes the input point cloud using one of two approaches: a single set abstraction layer (based on PointNet++), which creates patch-based features centered at each patch’s centroid encoding local geometry, or a Voronoi partition-based tokenizer, where  each Voronoi cell represents the patch’s centroid, and point-wise features are max-pooled  within each cell. The Voronoi-based method reduces computational and memory costs by bypassing the need for dense point-wise feature extraction in each centroid's vicinity. Patch features are then extracted through a ViT model, and mask prompts are incorporated from previous decoder iterations. The model upscales these patch embeddings to the original point cloud resolution using 3-nn inverse distance weighting. Following this, a two-way transformer facilitates interaction between point prompts and point cloud embeddings by utilizing cross-attention operations. The final 3D proposal is derived by applying an output token processed through a MLP.\n\nPoint-SAM is trained on a diverse set of datasets that include part-level and object-level annotations across single-object and scene-level modalities. To improve generalizability on out-of-distribution data, the ShapeNet dataset is used to transfer knowledge from SAM. This is achieved by generating 2D pseudo masks that serve as prompts to a pretrained Point-SAM model, iteratively refining its 3D proposals.\n\nPoint-SAM's is evaluated on several benchmarks for the tasks of zero-shot point-prompted segmentation, zero-shot object proposal and few-shot part segmentation, where it achieves performance that is either on par with or surpasses competing methods.", "questions": "- Regarding the generation of pseudo labels, lines 265-269 state that the generated 3D proposal, using the 1st view’s random selected 2D prompt from SAM’s proposal, is projected back into the same view. This is then used to sample a 2D prompt (negative prompt) from the error region between the two (2D-3D proposal). Based on Figure 3, this is done in the next view (view 2). The first is used to sample the 2D prompt, the generated 3D proposal is used to prompt SAM with the 2nd view as input, and then the negative prompt is sampled for the generated 2nd 2D proposal. So, which of the two is the correct procedure?\n- In Table 3, the MV-SAM baseline is absent for the ScanObjectNN dataset. Additionally, it's unclear which specific approach the term “InterObject3D++” refers to (lines 391 and 395). Is this the same baseline used in AGILE3D? \n- In Figure 4, what is the difference of blue and red point prompts?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The method is a step forward towards 3D foundational models, eliminating the need of using multiple 2D views of the 3D object and 2D-3D lifting of SAM proposals at inference, while it provides the ability of refining the 3D proposals with additional prompts (as seen in supp.).\n+ Unified training strategy on 3D point clouds across several datasets, covering different modalities either at annotation or scale level (part, object masks; single object, entire scenes).\n+ Good performance on zero-shot point-prompted segmentation w.r.t. alternative methods.\n+ Knowledge distillation from 2D foundational models such as SAM during training to further enhance the generalizability of Point-SAM.", "weaknesses": "- The proposed Voronoi diagram tokenizer while it manages to lower the computational and memory cost of the overall pipeline, in many cases it fails to surpass the performance of the k-nn based tokenizer. \n- Regarding the OOD scenarios and particularly the PartNet-Mobility, the held-out categories (scissors, refrigerators, and doors) are all part of the PartNet, thus Point-SAM has seen these during training. This weakens the zero-shot transferability of the method.\n- The process of generating pseudo labels and transferring knowledge from SAM is somewhat unclear (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bi63D90btv", "reviewer": "Reviewer_b6dA", "summary": "The paper under review presents a 3D promptable segmentation model for point clouds, called Point-SAM. Similar to the architecture of Segment Anything Model(SAM), Point-SAM also contains three parts: a point-cloud encoder, a prompt encoder and a mask decoder. A Voronoi tokenizer is adopted to divide the point cloud into patch tokens. To improve the generalization of the model, multiple existing 3D segmentation datasets are included in training. Additionally, to augment part-level segmentation data, the authors utilize both pre-trained Point-SAM to obtain an initial 3D mask and then leverage SAM to refine the mask using additional views. The method demonstrates good performance on several indoor and outdoor scenes and showcases applications such as interactive 3D annotation, zero-shot 3D instance proposal and few-shot part segmentation.", "questions": "Please refer to the \"weaknesses\" section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper extends a powerful 2D segmentation anything model (SAM) into the 3D point cloud domain\n2. It introduces a novel data engine to generate multi-level pseudo-labels and augment the training data\n3. The interactive segmentation video is impressive, which illustrates its potential real-world application", "weaknesses": "1. Lack of Visualization Results: The authors provide only a limited number of visualizations in both the main paper and supplementary materials. The qualitative results presented in Figure 4 are inadequate. It is anticipated that more complex examples, such as those depicted in the supplementary videos, could be included. Moreover, several applications discussed in the paper, such as few-shot part segmentation and zero-shot object proposal generation, lack corresponding visualization results.\n\n2. In line 40, the paper asserts that \"multi-view images only capture the surface, making it infeasible to label internal structures.\" However, the paper does not include examples, such as drawers, to demonstrate the superior performance of Point-SAM in these scenarios. The authors are expected incorporate additional experiments or examples to make this claim more convincing.\n\n3. The segmentation quality does not look good: The paper does not directly compare its results with those of other 3D frameworks that merge results across multiple views, despitethe experiments on MVSAM. While it is acknowledged that most of these frameworks require post-processing, which can be time-consuming, their segmentation quality reported in those papers are superior, particularly along boundary regions, such as teh results in SAM3D and more advanced methods like Gaussian Grouping.\n\n4. Prompt Point Selection: Could the authors provide a more detailed explanation of how point prompts are selected in the experiments? In Figure 4, the points are not consistently placed, and some points are marked as negative in one method while none are negative in another. Also. the positions of points are different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQCL6qsjBD", "reviewer": "Reviewer_YF5t", "summary": "This paper addresses the 3D promptable segmentation task by examining it from three key aspects: task, model, and data. The authors propose a unified model named Point-SAM, which utilizes a point cloud representation and introduces a novel tokenizer based on Voronoi diagrams. In terms of data, they explore the generation of pseudo labels using SAM. The experimental results highlight the model's robust zero-shot transferability to unseen point cloud distributions and new tasks.", "questions": "1. Could you provide more visualizations for object-level segmentation in OOD scenarios to demonstrate the generalization of the method?\n\n2. Are there any special design considerations when training with both indoor datasets and object-level datasets together?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a clear writing logic, effectively outlining the challenge to be addressed and the three distinct perspectives of the research.\n2. The 3D segmentation task is a crucial direction in embodied intelligence, as it enables machines to understand and interact with complex environments. Moreover, scaling up to achieve 3D foundation models presents significant value.", "weaknesses": "The main consideration is the technical novelty of this paper, which leads me to feel that the overall contribution is somewhat weak.\n\n1. In terms of model design, the method introduces the Voronoi tokenizer, which is innovative. However, compared to KNN, there is no significant performance improvement; the gains are only noticeable when the number of prompt points is low.\n2. Additionally, there are many works utilizing SAM for 3D pseudo-labeling, whether in autonomous driving or in indoor settings, such as SAM3D, which also leverages multi-view consistency in 2D projections to refine results. What distinguishes the approach in this paper from previous works in terms of innovation?\n3. I have some concerns about the practicality of this method. Is it can serve as a more general tool for 3D segmentation labeling? For instance, in autonomous driving, if segmentation requires placing points on every object, it seems inefficient. The challenge with object-level distinction lies in identifying the various parts, making the placement of prompt points crucial.\n\nBesides, I would like to share some suggestions or open problems:\n\n1. I think semantic information is crucial for 3D foundation models, as segmentation can vary in granularity, and semantics often dictate the level of segmentation.\n2. Additionally, part segmentation without the need for interaction points is also valuable, such as distinguishing different parts in robotic grasping tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLbvawkV8j", "reviewer": "Reviewer_nzz5", "summary": "The paper adheres to the philosophy of the Segment Anything Model by proposing Point-SAM, a 3D promptable segmentation model specifically designed for point clouds. It primarily makes two contributions: (1) a scalable and efficient transformer-based architecture that facilitates 3D promptable segmentation; and (2) a data engine that distills knowledge from the Segment Anything Model to generate pseudo labels. Experiments conducted on various indoor and outdoor datasets have demonstrated its zero-shot transferability and significant application potential.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors develop a data engine to generate pseudo labels, providing sufficient diverse masks for datasets lacking ground truth. This allows Point-SAM to be trained with more heterogeneous datasets.\n- Following the design philosophy of SAM, the implementation on 3D point clouds is shown through experiments to endow Point-SAM with zero-shot capabilities.\n- The Voronoi tokenizer achieves comparable performance to the KNN tokenizer, while showing superior efficiency in KITTI360 dataset.", "weaknesses": "- The challenges presented in the introduction are not well sovled by the proposed method. For instance, the paper mentions \"There is no unified representation for 3D shapes,\" yet does not provide a unified representation and only designs for point cloud representation. \n- While the authors have made efforts to train Point-SAM on a diverse set of datasets, the reliance on synthetic datasets and the dominance of indoor scenes may limit the model's generalizability to outdoor and more varied environments. The paper would benefit from an evaluation on a broader range of datasets, particularly those capturing diverse outdoor scenes such as Waymo Open Dataset [1] and nuScenes [2].\n- Although the paper claims efficiency improvements with the Voronoi tokenizer and tests it on KITTI360 with 10 prompt points, a comprehensive analysis of the computational costs is lacking. The authors should provide more detailed benchmarks comparing Point-SAM with other methods in terms of both speed and memory usage.\n\n[1] Sun, Pei, et al. \"Scalability in perception for autonomous driving: Waymo open dataset.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Caesar, Holger, et al. \"nuscenes: A multimodal dataset for autonomous driving.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUMvizaUKj", "reviewer": "Reviewer_y8Cn", "summary": "This paper presents a method for achieving generalizable 3D point segmentation by leveraging prior knowledge from the 2D Segment Anything framework.", "questions": "Could you clarify the difference between the proposed pseudo label generation method and \"SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance Fields\"?\n\nAdditionally, the proposed Voronoi approach shows only minimal improvement compared to KNN. Could you clarify the efficiency comparison between these two?\n\nBut overall, it is a good work.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Propose a better tokenizer and show reasonable performance improvement compared with baseline.", "weaknesses": "Missing references:\n\n\"Segment Anything 3D\"\n\"SANeRF-HQ: Segment Anything for NeRF in High Quality\"\n\"SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance Fields\"\n\"Segment Anything in 3D with Radiance Fields\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AV9iVgqBl7", "reviewer": "Reviewer_e1ZC", "summary": "This paper is mainly built to tackle 3D promptable segmentation task. The authors propose a segmentation model called Point-SAM, which can take 3D point prompts and mask prompts as input prompt, alongside a Voronoi tokenizer is used to fuse dense prompts. This model is trained on two phases: 1. existing well-labeled datasets; 2. unlabeled datasets, whose pseudo labels are generated by pre-trained Point-SAM from phase 1 and SAM, in order to distilling knowledge from SAM. They compare their method with baselines on zero-shot segmentation task, and showing much better performances.", "questions": "Besides the questions in weakness, I also have some other questions:\n1. The authors listed Replica as an evaluation datasets in Table 2, but only compared with OpenMask3D in appendix but not with other baselines. Why do you make this choice? Combined with what mentioned in Weakness 1 and Table 6, is this due to your model is very sensitive and not so well-defined for large scale data?\n2. According to Figure 4, for KITTI360 and S3DIS datasets, are the red and blue balls the visualization of prompt points? If so, why the prompts to Point-SAM and AGILE3D are not consistent? Do you use the same prompt in your main experiments?\nI am open to raising my score if the authors can thoroughly address the weaknesses and questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Unlike many 2D-to-3D segmentation lifting work, this paper chooses to distill knowledge from SAM in training phase instead of aggregating in very evaluation, making it more efficient and not requiring running SAM for several times during inference. The shows great performance on choosing datasets, showing great performance on zero-show segmentation task due to distilling and better model design.", "weaknesses": "The main contribution of this paper can be divided into two folds: better model design and novel data engine to distill SAM instead of aggregating. So the paper should demonstrate the effectivity for both these two parts. However, the experiments are not strong enough:\n1. This model choose to compare with AGILE3D, a straightforward baseline. In main experiments, the experiment is not fair, since the model proposed actually trained on much larger datasets than the baseline. In order to have a fair comparison, they also trained the model with the same training set as baseline in Table 6.a. However, this method only show apparent better performance on 2 out of 4 datasets. The authors argue it may due to that the baseline is overfitting to these kinds of data, but it’s assertion is not strong enough, because these two datasets both tend to include scenes with much more points than the other two. At least a cross analysis ablation, i.e. train A test B and train B test A.\n2. The baselines are not thorough and strong enough. It is strange why the baseline is not trained on the same level of data. And the multi-view SAM aggregation baseline is not strong enough. The authors make up one, MV-SAM. However, there are plenty of such multi-view aggregating baselines, such as SAM-guided Graph Cut, SAI3D, Open3DIS, OpenIns3D.\n3. In Table 4, evaluating only on PartNet-Mobility is not sufficient. From the table, we observe that adding ScanNet is less beneficial than adding ShapeNet. Further explanation or additional experiments are needed to clarify this difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)