Point-SAM: Promptable 3D Segmentation Model for Point Clouds

## Abstract

Abstract The development of 2D foundation models for image segmentation has been significantly advanced by the Segment Anything Model (SAM). However, achieving similar success in 3D models remains a challenge due to issues such as non-unified data formats, lightweight models, and the scarcity of labeled data with diverse masks. To this end, we propose a 3D promptable segmentation model ( Point-SAM ) focusing on point clouds. Our approach utilizes a transformer-based method, extending SAM to the 3D domain. We leverage part-level and object-level annotations and introduce a data engine to generate pseudo labels from SAM, thereby distilling 2D knowledge into our 3D model. Our model outperforms state-of-the-art models on several indoor and outdoor benchmarks and demonstrates a variety of applications, such as 3D annotation. Codes and demo can be found at https://github.com/zyc00/Point-SAM .

## 1 Introduction

The development of 2D foundation models for image segmentation has been significantly advanced by *Segment Anything* []. That pioneering work includes a promptable segmentation task, a segmentation model (SAM), and a data engine for collecting a dataset (SA-1B) with over 1 billion masks. SAM shows impressive zero-shot transferability to new image distributions and tasks. Thus, it has been widely used in many applications, e.g., segmenting foreground objects for image-conditioned 3D generation [, ], NeRF [], and robotic tasks [, ].

*Can we just lift SAM to create 3D foundation models for segmentation?*
Despite a few efforts [, , ] to extend SAM to the 3D domain, those existing approaches are limited to applying SAM on 2D images and then lifting the results to 3D. This process is constrained by image quality, and thus is likely to fail for textureless or colorless shapes like CAD models [].
Besides, it is also affected by view selection. Too few views may not adequately cover the entire shape, while too many views can significantly increase the computational burden.
Moreover, it can suffer from multi-view inconsistency when results are merged from different views, since they may conflict and be impacted by occlusions.
Furthermore, multi-view images only capture surface, making it infeasible to label internal structures, essential for annotating articulated objects (e.g. drawers in a cabinet).
Therefore, it is necessary to develop native 3D foundation models to address the aforementioned limitations.

Figure: Figure 1: We address 3 critical components for 3D promptable segmentation. First, we propose a 3D extension of SAM [], named Point-SAM, which predicts masks given the input point cloud and prompts. To scale up training data, we develop a data engine to generate pseudo labels, where SAM is employed to generate initial diverse mask proposals, and iteratively refine them with both SAM and our model pre-trained on existing datasets. The final models, trained on a mixture of datasets, are capable of handling data from various sources and producing results at multiple levels of granularity. We demonstrate the versatility and efficacy of our approach through multiple applications and downstream tasks, as detailed in Sec. 5.
Refer to caption: extracted/5687031/figures/teaser-4.jpg

However, developing native 3D foundation models, or extending SAM to the 3D domain, presents several challenges:
1) There is no unified representation for 3D shapes. 3D shapes can be represented by meshes, voxels, point clouds, implicit functions, or multi-view images. Unlike 2D images, 3D shapes can vary significantly in scale and sparsity. For example, indoor and outdoor datasets often cover different ranges and typically require different models.
2) There are no unified network architectures in the 3D domain. Due to the heterogeneity of 3D data, different network architectures have been proposed for different representations, such as PointNet [] for point clouds and SparseConv [] for voxels.
3) It is more difficult to scale up 3D networks. 3D networks are natively more computationally costly. For instance, SAM utilizes deconvolution and bilinear upsampling in its decoder, whereas there are no 3D operators for point clouds as efficient as their 2D counterparts.
4) High-quality 3D labels, especially those with diverse masks, are rare. SAM is initially trained on existing datasets with ground-truth labels of low diversity, and then used to facilitate annotating more masks at different granularity (e.g., part, object, semantics) to increase label diversity. However, in the 3D domain, existing datasets contain only a small number of segmentation labels. For example, the largest dataset with part-level annotations, PartNet [], includes only about 26,671 shapes and 573,585 part instances.

In this work, our goal is to build a 3D promptable segmentation model for point clouds, as a foundational step towards 3D foundation models. Point clouds are selected as our primary representation, because other representations can be readily converted into point clouds, and real-world data is often captured in this format.
Following SAM, we address 3 critical components: task, model, and data.
We focus on the 3D promptable segmentation task, which involves predicting valid segmentation masks in response to any given segmentation prompt.
To address the task, we propose a 3D extension of SAM, named Point-SAM. We utilize a transformer-based encoder to embed the input point cloud, alongside a point prompt encoder for point prompts, and a mask prompt encoder for mask prompts. Point-cloud and prompt embeddings are fed to a transformer-based mask decoder to predict segmentation masks.
Regarding data, we train Point-SAM on a mixture of heterogeneous datasets, including PartNet and ScanNet [], with both part- and object-level annotations. To expand label diversity and leverage large-scale unlabeled datasets such as ShapeNet [], we have developed a data engine to generate pseudo labels with the assistance of SAM. This pipeline enables us to distill knowledge from SAM, and our experiments demonstrate that these pseudo labels significantly improve zero-shot transferability.
Our contributions include:

- •
We develop a 3D foundation model Point-SAM for the promptable segmentation task on point clouds, adept at processing point clouds from various sources in a unified way.
- •
We propose a data engine to generate pseudo labels with substantial mask diversity by distilling knowledge from SAM. It is shown to significantly enhance our model’s performance on out-of-distribution (OOD) data.
- •
We manage to scale up our model and dataset for training in 3D segmentation. Our experiments highlight our model’s strong zero-shot transferability to unseen point-cloud distributions and new tasks.

## 2 Related Work

#### Lifting 2D foundation models for 3D segmentation

Despite the growing number of 3D datasets, high-quality 3D segmentation labels remain scarce. To address this, 2D foundation models trained on web-scale 2D data, such as CLIP [], GLIP [], and SAM [], have been leveraged. A prevalent framework involves adapting these 2D foundation models for 3D applications by merging results across multiple views. SAM3D [] and SAMPro3D [] utilize RGB-D images with known camera poses to lift SAM to segment 3D indoor scenes. PartSLIP [, ], dedicated to part-level segmentation, first renders multiple views of a dense point cloud, then employs GLIP and SAM to segment parts, and finally consolidates multi-view results into 3D predictions.
These methods are limited by the capabilities of 2D foundation models and the quality of multi-view rendering. Besides, they usually require complicated and slow post-processing to integrate multi-view results, which also poses challenges in maintaining multi-view consistency.
Another strategy involves distilling knowledge from 2D foundation models directly into 3D models. For example, Segment3D [] and SAL [] both utilize SAM to generate pseudo labels given RGB images and train native 3D models on scene-level point clouds. However, these approaches can only handle surface points, making it difficult to segment internal structures that are common in part-level segmentation of articulated 3D shapes such as cabinets with drawers.

#### 3D foundation models

The development of 3D foundation models has advanced significantly. PointBERT [] proposes a self-supervised paradigm for pretraining 3D representations for point clouds. OpenShape [] and Uni3D [] scale up 3D representations with multi-modal contrastive learning. [] trains 3D-based Large Language models (3D-LLM) on collected diverse 3D-language data, utilizing 2D pretrained VLMs. LEO [], sharing similar ideas, focuses on embodied ability such as navigation and robotic manipulation. Our work concentrates on 3D segmentation. Despite several initiatives aimed at open-world 3D segmentation, such as OpenScene [] and OpenMask3D [], these primarily address scene-level segmentation and are trained on relatively small datasets.

#### 3D interactive segmentation

Interactive segmentation has been explored across both 2D and 3D domains. [] introduces a groundbreaking project including the promptable segmentation task, the 2D foundation model (SAM), and a data engine to collect large-scale labels. In the 3D domain, InterObject3D [] and AGILE3D [] share similar ideas to segment point clouds while their training is confined to ScanNet [].
In contrast, our model is designed to handle both object- and part-level segmentation, leveraging a wide range of datasets including CAD models and real scans. Thus, our model shows greater versatility and adaptability.
Besides, 3D interactive segmentation is also explored within implicit representations. SA3D [] enables users to achieve 3D segmentation of any target object through a single one-shot manual prompt in a rendered view. SAGA [] distills SAM features into 3D Gaussian point features through contrastive training. While these methods necessitate an additional optimization process, requiring at least a few minutes, our model operates on a feed-forward basis and can respond within seconds, offering a more efficient solution.

## 3 Point-SAM

Figure: Figure 2: Overview of Point-SAM. Our model, Point-SAM takes a point cloud as input. First, we sample a fixed number of centers using Farthest Point Sampling (FPS), and group the k-nearest neighbors of each center into a patch. Patch features are extracted by a PointNet [], and fed into a ViT to obtain the point-cloud embedding. Point and mask prompts are encoded by the prompt encoder to obtain prompt embeddings. A transformer-based decoder processes the point cloud and prompt embeddings and outputs point-wise masks.
Refer to caption: extracted/5687031/figures/network-10.jpg

In this section, we present Point-SAM, a promptable segmentation model for point clouds. Fig. 2 provides an overview of Point-SAM. Inspired by SAM [], Point-SAM consists of 3 components: a point-cloud encoder, a prompt encoder, and a mask decoder.
Unlike 2D models, Point-SAM addresses unique challenges related to point clouds: computation efficiency, scalability, and irregularity. We denote the input point cloud as $P\in\mathbb{R}^{N\times 3}$ and its point-wise feature as $F\in\mathbb{R}^{N\times D}$.

#### Point-cloud encoder

The point-cloud encoder transforms the input point cloud into the point-cloud embedding. Motivated by the expressibility and scalability of 2D vision transformers [] and recent advancements in 3D point-cloud transformers [, , ], we base our encoder on Uni3D [], a scalable 3D foundation model. Concretely, we first select a fixed number of centers $C\in\mathbb{R}^{L\times 3}$ using farthest point sampling (FPS), and group the k-nearest neighbors of each center as a patch. Features of each patch are extracted using a PointNet []. The above steps constitutes the *patch embedding* module, which generates patches with features given a point cloud. These patch features $F_{patch}\in\mathbb{R}^{L\times D_{1}}$, combined with the positional embeddings of the patch centers, are processed by a pre-trained transformer from Uni3D to generate the final point-cloud embedding $F_{pc}\in\mathbb{R}^{L\times D_{2}}$. Point-SAM can be directly scaled up by scaling up this transformer. We include two versions: Point-SAM-Large and Point-SAM-Giant, corresponding to Uni3D-Large(307M) and Uni3D-Giant(1B).

#### Prompt encoder

The prompt encoder encodes various types of prompts into prompt embeddings. In this work, we focus on two types of prompts: points and masks. Point prompts are processed similarly to SAM. Each point is associated with a binary label indicating whether it is a foreground prompt. These prompts are encoded to their positional encodings [] $F_{point}\in\mathbb{R}^{Q\times D_{2}}$, summed with learned embeddings indicating their labels. $Q$ denotes the number of point prompts.
Mask prompts are represented as dense point-wise logits $X_{mask}\in\mathbb{R}^{N\times 1}$, typically derived from the model’s previous predictions. These logits are concatenated with the input point cloud’s coordinates and processed through a mask encoder that mirrors the patch embedding module in the point-cloud encoder. The resulting mask prompt embeddings $F_{mask}\in\mathbb{R}^{L\times D_{2}}$ are element-wise summed with the point-cloud embedding.

#### Mask Decoder

The mask decoder efficiently maps the point-cloud embedding, prompt embeddings, and an output token $F_{out}\in\mathbb{R}^{1\times D_{2}}$ into a segmentation mask $Y_{mask}\in\mathbb{R}^{N\times 1}$. Due to the irregular nature of point clouds, our mask decoder significantly differs from its 2D counterparts. We follow SAM to employ two Transformer decoder blocks that use prompt self-attention and cross-attention in two directions (prompt-to-point-cloud and vice versa), to update all embeddings. We upsample the updated point-cloud embedding $F_{pc}\in\mathbb{R}^{L\times D_{2}}$ to match the input resolution by using inverse distance weighted average interpolation based on 3 nearest neighbors [], followed by an MLP. We denote the upsampled point-cloud embedding as $X_{pc}\in\mathbb{R}^{N\times D_{4}}$. Another MLP transforms the output token to the weight of a dynamic linear classifier $X_{out}\in\mathbb{R}^{1\times D_{4}}$, which calculates the mask’s foreground probability at each point location as $Y_{mask}=X_{pc}\cdot X_{out}^{T}$.
Consistent with SAM, our model can generate multiple output masks for a single point prompt by introducing multiple output tokens. Note that multi-mask outputs are enabled only when there is only a single point prompt with no mask prompts.
In addition, we also introduce another token $F_{iou}\in\mathbb{R}^{M\times D_{2}}$ to predict the IoU score for each mask output, where $M$ is the number of multiple mask outputs.

#### Training

Mask predictions are supervised with a weighted combination of focal loss [] and dice loss [], in line with SAM. We simulate an interactive setup by sampling prompts across 7 iterations per mask. The loss for mask prediction is computed between the ground truth mask and the predictions at all iterations. The interactive setup is detailed in Sec. 5.1. More details are provided in Appendix. A.
For multiple mask outputs, we follow SAM to use a “hindsight” loss, where we only back-propagate only the minimum loss over masks. Additionally, the predicted IoU score is supervised using a mean squared error loss.
For training, we randomly sample 10,000 points as input. Besides, we normalize the input point to fit within a unit sphere centered at zero, to standardize the inputs. The number of patches $L$ and the patch size $K$ are set to 512 and 64 by default.

#### Inference with variability

A significant challenge in handling 3D point clouds is their irregular input structure; the number of points can vary, necessitating a dynamic approach to group points into a varying number of patches with adjustable sizes. While previous point-based methods [] are typically limited to processing a fixed number of points, our model’s flexible design allows it to handle larger point sets than those used during training, by adjusting the number of patches and the patch size.
Unless otherwise specified, we set the number of patches and the patch size to 2048 and 512 when the number of input points exceeds 32768.
In contrast, voxelization-based methods [] struggle with such variations as changing voxel resolution can significantly impact performance, the results with different voxel resolutions are shown in Appendix. B

## 4 Training Datasets

**Table 1: Summary of training datasets.**
| Dataset | PartNet | PartNet-<br>Mobility | ScanNet | ScanNet-<br>block | Fusion360 | ShapeNet | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Number of point clouds | 16442 | 2163 | 1198 | 24328 | 35000 | 20000 | 99131 |
| Average number of masks | 13 | 16 | 30 | 12 | 2 | 17 | 10 |
| Has ground truth labels | True | True | True | True | True | False | - |

#### Integrating existing datasets

Foundation models are typically data-hungry, and the diversity of segmentation masks is crucial to support “segment anything”. Thus, we use a mixture of existing datasets with ground truth segmentation labels, which are summarized in Table 1. We utilize synthetic datasets including the training split of PartNet [], PartNet-Mobility [], and Fusion360 [].
Since PartNet does not provide textured meshes, we only keep the models that are from ShapeNet where textured meshes are available. We use all part hierarchies of PartNet.
For PartNet-Mobility, we hold out 3 categories (scissors, refrigerators, and doors) not included in ShapeNet, which are used for evaluation on unseen categories.
For PartNet and Fusion360, we uniformly sample 32768 points from mesh faces. For each object in PartNet-Mobility, we render 12 views, fuse point clouds from rendered RGB-D images, and sample 32768 points from the fused point cloud using Farthest Point Sampling (FPS).
For scene-level datasets, we use the training split of ScanNet200 [] and augment it by splitting each scene into blocks. The augmented version is denoted as ScanNet-Block. Concretely, we use a 3m$\times$3m block with a stride of 1.5m. We use FPS to sample 32768 points per scene or block.

#### Generating pseudo labels

Existing datasets lack sufficient diversity in masks. Large-scale 3D datasets like ShapeNet [] usually do not include part-level segmentation labels. Besides, most segmentation datasets only provide exclusive labels, where each point belongs to a single instance. To this end, we develop a data engine to generate pseudo labels.

Initially, Point-SAM is trained on the mixture of existing datasets. Next, we utilize both pre-trained Point-SAM and SAM to generate pseudo labels.
Concretely, for each mesh, we render RGB-D images at 6 fixed camera positions and fuse a colored point cloud. SAM is applied to generate diverse 2D proposals for each view. For each 2D proposal, we intend to find a 3D proposal corresponding to it. We start from the view corresponding to the 2D proposal. A 2D prompt is randomly sampled from the 2D proposal and lifted to a 3D prompt, which prompts Point-SAM to predict a 3D mask on the fused point cloud. Then, we sample the next 2D prompt from the error region between the 2D proposal and the projection of the 3D proposal at this view. New 3D prompts and previous 3D proposal masks are fed to Point-SAM to update the 3D proposal. The process is repeated until the IoU between the 2D proposal and the projection of the 3D proposal is larger than a threshold. This step ensures 3D-consistent segmentation regularized by Point-SAM while retaining the diversity of SAM’s predictions.
We repeat the above process with a few modifications at other views to refine the 3D proposal. At other view, we first sample the initial 2D prompt from the projection of previous 3D proposal, which is used to prompt SAM to generate multiple outputs. The output 2D mask with the highest IoU relative to the projection is selected as the “2D proposal” in the previous process. If the IoU is lower than a threshold, the 3D proposal is discarded. Previous 3D proposal mask is used to prompt Point-SAM at each iteration. This step aids in refining the 3D masks by incorporating 2D priors from SAM through space carving.
We use our data engine to generate pseudo labels for 20000 shapes from ShapeNet. On average, each shape is annotated with 17 masks, offering a diversity comparable to PartNet. Figure 3 illustrates the pseudo label generation process and the refinement of the 3D segmentation mask from the initial view using an additional view.

Figure: Figure 3: This figure illustrates the process of generating pseudo labels. Initially, we select one segmentation mask from the output masks generated by SAM’s "everything" function on the first view (View 1). Then, we prompt Point-SAM by lifting prompt points from 2D to 3D (View 1 prompt). Subsequently, the 3D segmentation mask proposed by Point-SAM is refined using additional views. We begin by prompting 2D SAM by projecting the 3D segmentation mask onto the second view (View 2), leveraging SAM’s strong prior knowledge to revise the mask. Finally, we add more prompt points by sampling from the revised area by SAM and prompt Point-SAM again by lifting these points to 3D (View 2 prompt).
Refer to caption: extracted/5687031/figures/pseudo-label-3.jpg

## 5 Experiments

In this section, we present the experiments showing the strong zero-shot transferability of our method. We showcase an application of 3D interactive annotation in our supplementary materials.

### 5.1 Zero-Shot Point-Prompted Segmentation

#### Task and metric

The task is to segment objects based on 3D point prompts. For automatic evaluation, point prompts need to be selected. We adopt the same method to simulate user clicks described in []. In brief, the first point prompt is selected as the “center” of the ground truth mask, which is the point farthest from the boundary.
Each subsequent point is chosen from two candidates: one from the false-positive set at the farthest minimum distance to the complementary set, and the other from the false-negative set selected in a similar way. Then, the candidate farther from the boundary is selected, see Appendix. C.
This evaluation protocol is commonly used in prior 2D [, ] and 3D [, ] works on interactive single-object segmentation.
Following [, ], we use the metric IoU@k, which is the Intersection over Union (IoU) between ground truth masks and prediction given $k$ point prompts. The metric is averaged across instances.

#### Datasets

We evaluate on a heterogeneous collection of datasets, covering both indoor and outdoor data, along with part- and object-level labels.
For part-level evaluation, we use the synthetic dataset PartNet-Mobility [] and the real-world dataset ScanObjectNN []. As mentioned in Sec. 4, we hold out 3 categories of PartNet-Mobility for evaluation. In the same way as the training dataset, we render 12 views for each shape, fuse a point cloud from multi-view depth images, and sample 10,000 points for evaluation.
ScanObjectNN contains 2902 objects of 15 categories collected from SceneNN [] and ScanNet [].
For scene-level evaluation, we use S3DIS [] and KITTI-360 []. Specifically, we use the processed data from AGILE3D [], which contains scans cropped around each instance.
Table 2 summarizes the datasets used for evaluation.

**Table 2: Summary of evaluation datasets.**
| Property | PartNet-<br>Mobility | ScanObjectNN | S3DIS | KITTI360 | Replica | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Number of point clouds | 125 | 2,723 | 68 | 379 | 17 | 3312 |
| Average number of masks | 6 | 3 | 33 | 9 | 152 | 5 |
| Average number of points | 10,000 | 157,669 | 522,058 | 40,000 | 1,308,124 | - |

#### Baselines

We compare Point-SAM with a multi-view extension of SAM, named MV-SAM, and a 3D interactive segmentation method, AGILE3D [].
Inspired by previous works [, , ] that lift SAM’s multi-view results to 3D, we introduce MV-SAM for zero-shot point-prompted segmentation as a strong baseline.
First, we render multi-view RGB-D images from the mesh of each shape. Note that mesh rendering is needed to ensure high-quality images, which are essential for good SAM’s performance. Thus, this baseline actually has access to more information than ours.
Then we prompt SAM at each view with the simulated click sampled from the “center” of the error region between the SAM’s prediction and 2D ground truth mask.
The predictions are subsequently lifted back to the sparse point cloud (10,000 points) and merged into a single mask. If a point is visible from multiple views, its foreground probability is averaged.
For both MV-SAM and our method, we select the most confident prediction if there are multiple outputs.
AGILE3D is close to our approach, while it uses a sparse convolutional U-Net as its backbone and is only trained on the real-world scans of ScanNet40.
Besides, it does not normalize its input, and thus it is sensitive to object scales.
To process CAD models without known physical scales, we adjust the scale of the input point cloud for AGILE3D, so that its axis-aligned bounding box has a maximum size of 5m, determined through a grid search.

**Table 3: Quantitative results of zero-shot point-prompted segmentation.**
| Dataset | Method | IoU@1 | IoU@3 | IoU@5 | IoU@7 | IoU@10 |
| --- | --- | --- | --- | --- | --- | --- |
| PartNet-Mobility | AGILE3D | 26.4 | 40.8 | 50.8 | 57.4 | 61.9 |
| MV-SAM | 29.3 | 57.0 | 69.7 | 74.3 | 76.9 |  |
| Ours (large) | 47.9 | 67.7 | 74.2 | 77.0 | 78.6 |  |
|  | Ours (giant) | 48.5 | 68.2 | 73.7 | 76.8 | 79.3 |
| ScanObjectNN | AGILE3D | 34.8 | 52.0 | 61.6 | 67.2 | 72.3 |
| Ours (large) | 49.4 | 75.3 | 82.0 | 84.8 | 86.3 |  |
|  | Ours (giant) | 50.2 | 76.8 | 83.0 | 85.9 | 87.0 |
| S3DIS | InterObject3D++ | 32.7 | 69.0 | 80.8 | - | 89.2 |
| AGILE3D | 58.7 | 77.4 | 83.6 | 86.4 | 88.5 |  |
|  | Ours (large) | 47.6 | 78.4 | 86.2 | 89.2 | 90.4 |
|  | Ours (giant) | 61.3 | 81.5 | 86.7 | 88.7 | 90.5 |
| KITTI360 | InterObject3D++ | 3.4 | 11.0 | 19.9 | - | 40.6 |
| AGILE3D | 34.8 | 42.7 | 44.4 | 45.8 | 49.6 |  |
|  | Ours (large) | 49.4 | 74.4 | 81.7 | 84.3 | 85.8 |
|  | Ours (giant) | 58.8 | 74.1 | 78.6 | 80.5 | 81.3 |

Figure: Figure 4: Qualitative results of prompt segmentation are presented for three different settings: KITTI360 for zero-shot outdoor scene segmentation, S3DIS for indoor scene segmentation, and PartNet-Mobility for zero-shot part segmentation. We compare our results with AGILE3D on KITTI360 and S3DIS, and with MVSAM on PartNet-Mobility. Point-SAM demonstrates superior segmentation results with fewer prompt points across all three datasets.
Refer to caption: extracted/5687031/figures/main-experiment-9.jpeg

#### Results

Table LABEL:tab:quantitative-zero-shot-point-prompt-seg presents the quantitative results. Point-SAM shows superior zero-shot transferability and effectively handle data with different numbers of points as well as from different sources.
Point-SAM significantly outperforms MV-SAM, especially when only few point prompts are provided, while MV-SAM achieves reasonably good performance with a sufficient number of prompts. Notably, for IoU@k, MV-SAM actually samples $k$ prompts per view. It indicates that our 3D native method is more prompt-efficient. Besides, it is challenging for SAM to achieve multi-view consistency without extra fine-tuning, especially with limited prompts.
Moreover, Point-SAM also surpasses AGILE3D across all datasets, particularly in out-of-distribution (OOD) scenarios such as PartNet-Mobility (held-out categories) and KITTI360. It underscores the strong zero-shot transferability of our method and the importance of scaling datasets. Figure  4 shows the qualitative comparison between Point-SAM, AGILE3D and MV-SAM, where Point-SAM demonstrates superior quality with a single prompt and significantly faster convergence compared to AGILE3D and MV-SAM.

Figure 4 presents the qualitative results. We compare our method with AGILE3D on KITTI360 and S3DIS, and with MVSAM on PartNet-Mobility. Despite neither Point-SAM nor AGILE3D being trained on outdoor data, the results on KITTI360 demonstrate that Point-SAM can accurately annotate an outdoor object using only three prompt points. The S3DIS results further confirm that Point-SAM achieves superior performance with fewer prompt points. For PartNet-Mobility, we show that Point-SAM effectively handles small parts, such as door handles, by leveraging geometry information, whereas MVSAM is misled by RGB data rendered from texture.

### 5.2 Zero-shot Object Proposals

In this section, we evaluate Point-SAM on zero-shot object proposal generation. The ability to automatically generating masks for all possible instances is known as “segment everything” in SAM. SAM samples a 64x64 point grid on the image as prompts, and uses non-maximum-suppression (NMS) based on bounding boxes to remove duplicate instances.
We adapt this approach for 3D point clouds with some modifications. First, we sample prompts using FPS, and then prompt Point-SAM to generate 3 masks per prompt. For post-processing, a modified version of NMS based on point-wise masks is applied.

We compare with OpenMask3D [] on Replica []. OpenMask3D utilizes a class-agnostic version of Mask3D [] trained on ScanNet200 to generate object proposals.
For our Point-SAM, we sample 1024 prompts and set the NMS threshold to 0.3. In addition, to handle the extensive point counts in Replica, we downsample each scene to 100,000 points and later propagate the predictions to their nearest neighbors at the original resolution.
We also adjust the number of patches and the patch size to 4096 and 64 respectively. For both methods, we truncate the proposals to the top 250.

We use the average recall (AR) metric. We filter out “undefined” and “floor” categories from the ground truth labels. Table 4(a) shows the quantitative results. Point-SAM showcases strong performance compared to OpenMask3D, which is tailored for this task, even though our model is never trained on such a large number of points and is zero-shot evaluated on unseen data. It highlights the robust zero-shot capabilities of our method.

**Table 4: Qualitative results of zero-shot object proposal generation and few-shot part segmentation.**
| Method | AR25 | AR50 |
| --- | --- | --- |
| OpenMask3D | 40.2 | 31.5 |
| Ours | 49.2 | 31.5 |

### 5.3 Few-Shot Part Segmentation

Foundation models can be effectively fine-tuned for various tasks. In this section, we demonstrate that Point-SAM has captured good representations for part segmentation. We compare with PointBERT [] and Uni3D [] on close-vocabulary, few-shot part segmentation. We use ShapeNetPart [] and report the mIoUC, which is the mean IoU averaged across categories. Similar to Uni3D, we adapt Point-SAM for close-vocabulary part segmentation. Specifically, we extract features from the 4th, 8th, and last layers of the ViT in our encoder and use feature propagation [] to upscale them into point-wise features, followed by an MLP to predict point-wise multi-class logits. During few-shot training, we freeze our encoder and only optimize the feature propagation layer as well as the MLP using cross-entropy loss. Unlike PointBERT and our method, Uni3D originally aligns point-wise features with text features of ground truth part labels extracted by CLIP. We refer to it as Uni3D (open), since it is designed for open-vocabulary part segmentation. We also evaluate its variant sharing our modification for close-vocabulary part segmentation, denoted as Uni3D (close).
Table 4(b) presents the results for both 1-shot and 2-shot settings. Point-SAM surpasses both PointBERT and Uni3D (close), which indicates that our approach has acquired versatile knowledge applicable to downstream tasks.

### 5.4 Ablations

#### Scaling up datasets

Previous works have been limited by the size and scope of their training datasets. For example, AGILE3D [] was trained solely on ScanNet [], which includes only 1,201 scenes. As detailed in Table 1, our training dataset encompasses 100,000 point clouds, 100 times larger than ScanNet.
To verify the effectiveness of scaling up training data, we conduct an ablation study on dataset size and composition. We introduce 4 dataset variants: 1) PartNet only, 2) PartNet+ScanNet (including ScanNet-Block), 3) PartNet+ShapeNet (pseudo labels), and 4) PartNet+ShapeNet+ScanNet.
We train Point-SAM-Large on these variants, resulting in different models. Table 5 shows the comparison of these models in zero-shot prompt segmentation on PartNet-Mobility (held-out categories).
The model trained on PartNet+ScanNet surpasses the one trained solely on PartNet, although the evaluation dataset (part-level labels) has a markedly different distribution from the added ScanNet (object-level labels).
Moreover, the model trained on PartNet+ShapeNet achieves even better performance, particularly with a single prompt. Note that the IoU@1 metric assesses whether the model captures sufficient mask diversity, since a single prompt is inherently ambiguous and the ground-truth label depends on dataset bias.
It suggests that our pseudo labels effectively incorporate part-level knowledge distilled from SAM.
Furthermore, it is observed that the zero-shot performance on out-of-distribution data consistently improves, as we utilize increasingly larger and more diverse data.

**Table 5: Ablation study on training dataset. We report the IoU@k metrics for zero-shot prompt-segmentation on PartNet-Mobility (held-out categories).**
| Training dataset | PartNet | PartNet<br>+ScanNet | PartNet<br>+ShapeNet | PartNet<br>+ShapeNet+ScanNet | Full |
| --- | --- | --- | --- | --- | --- |
| IoU@1 | 38.2 | 39.7 | 44.5 | 45.4 | 47.9 |
| IoU@3 | 56.3 | 58.6 | 65.2 | 66.5 | 67.7 |
| IoU@5 | 60.6 | 68.8 | 71.8 | 72.6 | 74.2 |
| IoU@10 | 63.5 | 71.9 | 76.2 | 77.5 | 78.6 |

#### Sensitivity to Point Count

As discussed in Sec. 3, point clouds are typically irregular. When handling point clouds with more points than those used in our training, we have to adjust the number of patches and the patch size accordingly. Thus, we conduct experiments to study the effect of these two hyperparameters. Table 6 shows the qualitative results of zero-shot prompt-segmentation on S3DIS []. We select S3DIS, because the average number of points for S3DIS is about 500K, 50 times larger than that of our training datasets. Our results indicate that it is important to increase the number of patches to accommodate larger point clouds. Enlarging the patch size is also crucial due to the different neighborhood densities compared to our training distribution.

**Table 6: Sensitivity to point count. We report the IoU@k metrics for zero-shot prompt-segmentation on S3DIS.**
| (#patches, patch size) | (512,64) | (512,256) | (2048,64) | (2048,256) |
| --- | --- | --- | --- | --- |
| IoU@1 | 41.9 | 49.0 | 47.4 | 47.6 |
| IoU@3 | 64.2 | 69.9 | 74.3 | 78.4 |
| IoU@5 | 72.2 | 76.4 | 81.7 | 86.2 |
| IoU@10 | 76.7 | 80.2 | 85.8 | 90.5 |

## 6 Conclusion

In conclusion, our work presents significant strides towards developing a foundational model for 3D promptable segmentation using point clouds. By adopting a transformer-based architecture, we have successfully implemented Point-SAM, which effectively responds to 3D point and mask prompts. Our model leverages a robust training strategy across mixed datasets like PartNet and ScanNet, which has proven beneficial, especially when enhanced with pseudo labels generated through our novel pipeline that distills knowledge from SAM.

However, there are inherent limitations and challenges in our approach. The diversity and scale of the 3D datasets used still lag behind those available in 2D, posing a challenge for training models that can generalize well across different 3D environments and tasks. Furthermore, the computational demands of processing large-scale 3D data and the complexity of developing efficient 3D-specific operations remain significant hurdles. Our reliance on pseudo labels, while beneficial for expanding label diversity, also introduces dependencies on the quality and variability of the 2D labels provided by SAM, which may not always capture the complex nuances of 3D structures.

Overall, our contributions lay a strong foundation for future advancements in 3D segmentation, which is crucial for a wide range of applications from augmented reality to autonomous navigation.

## Appendix A Training Details

Training recipe. Point-SAM is trained with the AdamW optimizer. We train Point-SAM for 100k iterations. The learning rate (lr) is set to 5e-5 after learning rate warmup. Initially, the lr is warmed up for 3k iterations, starting at 5e-8. A step-wise lr scheduler with a decay factor of 0.1 is then used, with lr reductions at 60k and 90k iterations. The weight decay is set to 0.1. The training batch size for Point-SAM, utilizing ViT-g as the encoder, is set to 4 per GPU with a gradient accumulation of 4, and it is trained on 8 NVIDIA H100 GPUs with a total batch size of 128. The ViT-l version can be trained across 2 NVIDIA A100 GPUs, with a batch size of 16 per GPU and gradient accumulation of 4, for 50k iterations. For Point-SAM utilizing ViT-l as the backbone, the step-wise learning rate decay milestones are set at 30k and 40k iterations.

Data augmentation. We apply several data augmentation techniques during training. For each object, we pre-sample 32,768 points before training and then perform online random sampling of 10,000 points from these 32,768 points for actual training. We apply a random scale for the normalized points with a scale factor of [0.8, 1.0] and a random rotation along y-axis from $-180^{\circ}$ to $180^{\circ}$. For object point clouds we also apply a random rotation perturbation to x- and z-axis. The perturbation angles are sampled from a normal distribution with a standard deviation (sigma) of 0.06, and then these angles are clipped to the range [0, 0.18].

## Appendix B Additional Experiments

We conducted a grid search to determine the optimal normalization scale for AGILE3D on PartNet-Mobility. Table 7 shows the effect of normalization scale for AGILE3D in zero-shot prompt segmentation on PartNet-Mobility (held-out categories). We found that AGILE3D achieves its best performance with a normalization scale of 5.

**Table 7: The effect of normalization scale for AGILE3D.**
| Normalization scale | IoU@3 | IoU@5 | IoU@7 | IoU@9 |
| --- | --- | --- | --- | --- |
| 1 | 29.0 | 36.4 | 41.0 | 44.7 |
| 3 | 35.7 | 43.1 | 48.4 | 51.7 |
| 5 | 40.8 | 50.8 | 57.4 | 61.9 |
| 7 | 36.8 | 45.6 | 51.3 | 57.5 |
| 10 | 35.1 | 42.2 | 48.6 | 53.1 |

## Appendix C Prompt Sampling

Following AGILE3D, we sample two prompt points for each instance: one from false positive points and another from false negative points. The prompts are selected by identifying the foreground point that has the furthest distance to the nearest background point. Specifically, this involves computing pairwise distances from foreground points to background points, determining the minimum distance to background points for each foreground point, and selecting the foreground point with the maximum of these minimum distances. To optimize performance and conserve GPU memory, we have also implemented a CUDA operation for these steps.

After computing the distances and selecting the candidates, we have two prompt point candidates: the point sampled from false positive points serves as a negative prompt, and the point sampled from false negative points serves as a positive prompt. We select the one with the furthest distance to the nearest background points as the final prompt point.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Regarding the generation of pseudo labels, lines 265-269 state that the generated 3D proposal, using the 1st view’s random selected 2D prompt from SAM’s proposal, is projected back into the same view. This is then used to sample a 2D prompt (negative prompt) from the error region between the two (2D-3D proposal). Based on Figure 3, this is done in the next view (view 2). The first is used to sample the 2D prompt, the generated 3D proposal is used to prompt SAM with the 2nd view as input, and then the negative prompt is sampled for the generated 2nd 2D proposal. So, which of the two is the correct procedure?
- In Table 3, the MV-SAM baseline is absent for the ScanObjectNN dataset. Additionally, it's unclear which specific approach the term “InterObject3D++” refers to (lines 391 and 395). Is this the same baseline used in AGILE3D? 
- In Figure 4, what is the difference of blue and red point prompts?
soundness: 3
strengths: + The method is a step forward towards 3D foundational models, eliminating the need of using multiple 2D views of the 3D object and 2D-3D lifting of SAM proposals at inference, while it provides the ability of refining the 3D proposals with additional prompts (as seen in supp.).
+ Unified training strategy on 3D point clouds across several datasets, covering different modalities either at annotation or scale level (part, object masks; single object, entire scenes).
+ Good performance on zero-shot point-prompted segmentation w.r.t. alternative methods.
+ Knowledge distillation from 2D foundational models such as SAM during training to further enhance the generalizability of Point-SAM.
summary: This paper addresses the problem of deploying 3D foundational models by introducing a novel 3D promptable segmentation model for point clouds, named Point-SAM. This model extends the Segment Anything Model (SAM) into the 3D domain by developing a promptable 3D architecture that utilizes both point and mask prompts. These prompts, together with the input point cloud, are processed through a transformer-based encoder-decoder architecture to generate a 3D segmentation mask.

The Point-SAM model first tokenizes the input point cloud using one of two approaches: a single set abstraction layer (based on PointNet++), which creates patch-based features centered at each patch’s centroid encoding local geometry, or a Voronoi partition-based tokenizer, where  each Voronoi cell represents the patch’s centroid, and point-wise features are max-pooled  within each cell. The Voronoi-based method reduces computational and memory costs by bypassing the need for dense point-wise feature extraction in each centroid's vicinity. Patch features are then extracted through a ViT model, and mask prompts are incorporated from previous decoder iterations. The model upscales these patch embeddings to the original point cloud resolution using 3-nn inverse distance weighting. Following this, a two-way transformer facilitates interaction between point prompts and point cloud embeddings by utilizing cross-attention operations. The final 3D proposal is derived by applying an output token processed through a MLP.

Point-SAM is trained on a diverse set of datasets that include part-level and object-level annotations across single-object and scene-level modalities. To improve generalizability on out-of-distribution data, the ShapeNet dataset is used to transfer knowledge from SAM. This is achieved by generating 2D pseudo masks that serve as prompts to a pretrained Point-SAM model, iteratively refining its 3D proposals.

Point-SAM's is evaluated on several benchmarks for the tasks of zero-shot point-prompted segmentation, zero-shot object proposal and few-shot part segmentation, where it achieves performance that is either on par with or surpasses competing methods.
weaknesses: - The proposed Voronoi diagram tokenizer while it manages to lower the computational and memory cost of the overall pipeline, in many cases it fails to surpass the performance of the k-nn based tokenizer. 
- Regarding the OOD scenarios and particularly the PartNet-Mobility, the held-out categories (scissors, refrigerators, and doors) are all part of the PartNet, thus Point-SAM has seen these during training. This weakens the zero-shot transferability of the method.
- The process of generating pseudo labels and transferring knowledge from SAM is somewhat unclear (see Questions).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the "weaknesses" section
soundness: 3
strengths: 1. The paper extends a powerful 2D segmentation anything model (SAM) into the 3D point cloud domain
2. It introduces a novel data engine to generate multi-level pseudo-labels and augment the training data
3. The interactive segmentation video is impressive, which illustrates its potential real-world application
summary: The paper under review presents a 3D promptable segmentation model for point clouds, called Point-SAM. Similar to the architecture of Segment Anything Model(SAM), Point-SAM also contains three parts: a point-cloud encoder, a prompt encoder and a mask decoder. A Voronoi tokenizer is adopted to divide the point cloud into patch tokens. To improve the generalization of the model, multiple existing 3D segmentation datasets are included in training. Additionally, to augment part-level segmentation data, the authors utilize both pre-trained Point-SAM to obtain an initial 3D mask and then leverage SAM to refine the mask using additional views. The method demonstrates good performance on several indoor and outdoor scenes and showcases applications such as interactive 3D annotation, zero-shot 3D instance proposal and few-shot part segmentation.
weaknesses: 1. Lack of Visualization Results: The authors provide only a limited number of visualizations in both the main paper and supplementary materials. The qualitative results presented in Figure 4 are inadequate. It is anticipated that more complex examples, such as those depicted in the supplementary videos, could be included. Moreover, several applications discussed in the paper, such as few-shot part segmentation and zero-shot object proposal generation, lack corresponding visualization results.

2. In line 40, the paper asserts that "multi-view images only capture the surface, making it infeasible to label internal structures." However, the paper does not include examples, such as drawers, to demonstrate the superior performance of Point-SAM in these scenarios. The authors are expected incorporate additional experiments or examples to make this claim more convincing.

3. The segmentation quality does not look good: The paper does not directly compare its results with those of other 3D frameworks that merge results across multiple views, despitethe experiments on MVSAM. While it is acknowledged that most of these frameworks require post-processing, which can be time-consuming, their segmentation quality reported in those papers are superior, particularly along boundary regions, such as teh results in SAM3D and more advanced methods like Gaussian Grouping.

4. Prompt Point Selection: Could the authors provide a more detailed explanation of how point prompts are selected in the experiments? In Figure 4, the points are not consistently placed, and some points are marked as negative in one method while none are negative in another. Also. the positions of points are different.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you provide more visualizations for object-level segmentation in OOD scenarios to demonstrate the generalization of the method?

2. Are there any special design considerations when training with both indoor datasets and object-level datasets together?
soundness: 3
strengths: 1. The paper presents a clear writing logic, effectively outlining the challenge to be addressed and the three distinct perspectives of the research.
2. The 3D segmentation task is a crucial direction in embodied intelligence, as it enables machines to understand and interact with complex environments. Moreover, scaling up to achieve 3D foundation models presents significant value.
summary: This paper addresses the 3D promptable segmentation task by examining it from three key aspects: task, model, and data. The authors propose a unified model named Point-SAM, which utilizes a point cloud representation and introduces a novel tokenizer based on Voronoi diagrams. In terms of data, they explore the generation of pseudo labels using SAM. The experimental results highlight the model's robust zero-shot transferability to unseen point cloud distributions and new tasks.
weaknesses: The main consideration is the technical novelty of this paper, which leads me to feel that the overall contribution is somewhat weak.

1. In terms of model design, the method introduces the Voronoi tokenizer, which is innovative. However, compared to KNN, there is no significant performance improvement; the gains are only noticeable when the number of prompt points is low.
2. Additionally, there are many works utilizing SAM for 3D pseudo-labeling, whether in autonomous driving or in indoor settings, such as SAM3D, which also leverages multi-view consistency in 2D projections to refine results. What distinguishes the approach in this paper from previous works in terms of innovation?
3. I have some concerns about the practicality of this method. Is it can serve as a more general tool for 3D segmentation labeling? For instance, in autonomous driving, if segmentation requires placing points on every object, it seems inefficient. The challenge with object-level distinction lies in identifying the various parts, making the placement of prompt points crucial.

Besides, I would like to share some suggestions or open problems:

1. I think semantic information is crucial for 3D foundation models, as segmentation can vary in granularity, and semantics often dictate the level of segmentation.
2. Additionally, part segmentation without the need for interaction points is also valuable, such as distinguishing different parts in robotic grasping tasks.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weaknesses section.
soundness: 3
strengths: - The authors develop a data engine to generate pseudo labels, providing sufficient diverse masks for datasets lacking ground truth. This allows Point-SAM to be trained with more heterogeneous datasets.
- Following the design philosophy of SAM, the implementation on 3D point clouds is shown through experiments to endow Point-SAM with zero-shot capabilities.
- The Voronoi tokenizer achieves comparable performance to the KNN tokenizer, while showing superior efficiency in KITTI360 dataset.
summary: The paper adheres to the philosophy of the Segment Anything Model by proposing Point-SAM, a 3D promptable segmentation model specifically designed for point clouds. It primarily makes two contributions: (1) a scalable and efficient transformer-based architecture that facilitates 3D promptable segmentation; and (2) a data engine that distills knowledge from the Segment Anything Model to generate pseudo labels. Experiments conducted on various indoor and outdoor datasets have demonstrated its zero-shot transferability and significant application potential.
weaknesses: - The challenges presented in the introduction are not well sovled by the proposed method. For instance, the paper mentions "There is no unified representation for 3D shapes," yet does not provide a unified representation and only designs for point cloud representation. 
- While the authors have made efforts to train Point-SAM on a diverse set of datasets, the reliance on synthetic datasets and the dominance of indoor scenes may limit the model's generalizability to outdoor and more varied environments. The paper would benefit from an evaluation on a broader range of datasets, particularly those capturing diverse outdoor scenes such as Waymo Open Dataset [1] and nuScenes [2].
- Although the paper claims efficiency improvements with the Voronoi tokenizer and tests it on KITTI360 with 10 prompt points, a comprehensive analysis of the computational costs is lacking. The authors should provide more detailed benchmarks comparing Point-SAM with other methods in terms of both speed and memory usage.

[1] Sun, Pei, et al. "Scalability in perception for autonomous driving: Waymo open dataset." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Caesar, Holger, et al. "nuscenes: A multimodal dataset for autonomous driving." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Could you clarify the difference between the proposed pseudo label generation method and "SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance Fields"?

Additionally, the proposed Voronoi approach shows only minimal improvement compared to KNN. Could you clarify the efficiency comparison between these two?

But overall, it is a good work.
soundness: 3
strengths: Propose a better tokenizer and show reasonable performance improvement compared with baseline.
summary: This paper presents a method for achieving generalizable 3D point segmentation by leveraging prior knowledge from the 2D Segment Anything framework.
weaknesses: Missing references:

"Segment Anything 3D"
"SANeRF-HQ: Segment Anything for NeRF in High Quality"
"SERF: Fine-Grained Interactive 3D Segmentation and Editing with Radiance Fields"
"Segment Anything in 3D with Radiance Fields"

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the questions in weakness, I also have some other questions:
1. The authors listed Replica as an evaluation datasets in Table 2, but only compared with OpenMask3D in appendix but not with other baselines. Why do you make this choice? Combined with what mentioned in Weakness 1 and Table 6, is this due to your model is very sensitive and not so well-defined for large scale data?
2. According to Figure 4, for KITTI360 and S3DIS datasets, are the red and blue balls the visualization of prompt points? If so, why the prompts to Point-SAM and AGILE3D are not consistent? Do you use the same prompt in your main experiments?
I am open to raising my score if the authors can thoroughly address the weaknesses and questions.
soundness: 2
strengths: Unlike many 2D-to-3D segmentation lifting work, this paper chooses to distill knowledge from SAM in training phase instead of aggregating in very evaluation, making it more efficient and not requiring running SAM for several times during inference. The shows great performance on choosing datasets, showing great performance on zero-show segmentation task due to distilling and better model design.
summary: This paper is mainly built to tackle 3D promptable segmentation task. The authors propose a segmentation model called Point-SAM, which can take 3D point prompts and mask prompts as input prompt, alongside a Voronoi tokenizer is used to fuse dense prompts. This model is trained on two phases: 1. existing well-labeled datasets; 2. unlabeled datasets, whose pseudo labels are generated by pre-trained Point-SAM from phase 1 and SAM, in order to distilling knowledge from SAM. They compare their method with baselines on zero-shot segmentation task, and showing much better performances.
weaknesses: The main contribution of this paper can be divided into two folds: better model design and novel data engine to distill SAM instead of aggregating. So the paper should demonstrate the effectivity for both these two parts. However, the experiments are not strong enough:
1. This model choose to compare with AGILE3D, a straightforward baseline. In main experiments, the experiment is not fair, since the model proposed actually trained on much larger datasets than the baseline. In order to have a fair comparison, they also trained the model with the same training set as baseline in Table 6.a. However, this method only show apparent better performance on 2 out of 4 datasets. The authors argue it may due to that the baseline is overfitting to these kinds of data, but it’s assertion is not strong enough, because these two datasets both tend to include scenes with much more points than the other two. At least a cross analysis ablation, i.e. train A test B and train B test A.
2. The baselines are not thorough and strong enough. It is strange why the baseline is not trained on the same level of data. And the multi-view SAM aggregation baseline is not strong enough. The authors make up one, MV-SAM. However, there are plenty of such multi-view aggregating baselines, such as SAM-guided Graph Cut, SAI3D, Open3DIS, OpenIns3D.
3. In Table 4, evaluating only on PartNet-Mobility is not sufficient. From the table, we observe that adding ScanNet is less beneficial than adding ShapeNet. Further explanation or additional experiments are needed to clarify this difference.

Meta-review: This paper receives all positive ratings of 8,8,6,6,6,6. The AC follows the recommendation of the reviewers to accept the paper. All reviewers think that the work on 3D promptable segmenetation model of Point-SAM is useful for the community and the proposed method is good. The weaknesses mentioned by the reviewers are mainly clarifications and are well-addressed in the rebuttal and discussion phases.

Decision: Accept (Poster)