PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models

## Abstract

Abstract Recent success of vision foundation models have shown promising performance for the 2D perception tasks. However, it is difficult to train a 3D foundation network directly due to the limited dataset and it remains under explored whether existing foundation models can be lifted to 3D space seamlessly. In this paper, we present PointSeg, a novel training-free paradigm that leverages off-the-shelf vision foundation models to address 3D scene perception tasks. PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to align their corresponding pixels across frames. Concretely, we design a two-branch prompts learning structure to construct the 3D point-box prompts pairs, combining with the bidirectional matching strategy for accurate point and proposal prompts generation. Then, we perform the iterative post-refinement adaptively when cooperated with different vision foundation models. Moreover, we design a affinity-aware merging algorithm to improve the final ensemble masks. PointSeg demonstrates impressive segmentation performance across various datasets, all without training. Specifically, our approach significantly surpasses the state-of-the-art specialist training-free model by 16.3 % percent \% % , 14.9 % percent \% % , and 15 % percent \% % mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top of that, PointSeg can incorporate with various foundation models and even surpasses the specialist training-based methods by 5.6 % percent \% % -8 % percent \% % mAP across various datasets, serving as an effective generalist model.

## 1 Introduction

3D scene segmentation plays a vital role in many applications, such as autonomous driving, augmented reality and room navigation. To tackle the challenges in 3D scene segmentation, most of the previous methods (Kundu et al., ; Jiang et al., ; Rozenberszki et al., ; Kolodiazhnyi et al., ; Liang et al., ; Schult et al., ; Sun et al., ; Vu et al., ) are supervised and heavily rely on precise 3D annotations, which means that they lack the zero-shot capability. Despite recent efforts (Takmaz et al., ; Huang et al., ; Yin et al., ; He et al., ) have attempted to explore the zero-shot 3D scene understanding, these approaches either require 3D mask pre-trained networks or domain-specific data training. Consequently, the ability of domain transfer to unfamiliar 3D scenes continues to pose significant challenges.

Looking around in the 2D realm, vision foundation models (VFMs) (Radford et al., ; Jia et al., ; Oquab et al., ; He et al., ) have exploded in growth, attributed to the availability of large-scale datasets and computational resources. And they have demonstrated exceptional generalization capabilities in zero-shot scenarios, along with multifunctional interactivity when combined with human feedback. Most recently, Segment Anything Model (SAM) (Kirillov et al., ) has managed to attain remarkable performance in class-agnostic segmentation by training on the SA-1B dataset. Then it triggers a series of applications in various tasks and improvements in various aspects (Zhang et al., ; Zou et al., ; Xiong et al., ; Zhao et al., ; Zhang et al., ). Inspired by this, a natural idea is to also train a foundation model in 3D space. However, this has been hindered by the limited scale of 3D data and the high cost of 3D data collection and annotation (Goyal et al., ; Chang et al., ). Considering this, we ask: *Is it possible to explore the use of VFMs to effectively tackle a broad spectrum of 3D perception tasks without training, e.g., 3D scene segmentation ?*

Following this paradigm, some works have made some early attempts. One line focuses on segmenting 2D frame accurately with different scene deconstruction strategies (Yang et al., ; Yin et al., ; Guo et al., ). Another line tries to learn high-quality 3D points to prompt the SAM by using the projection from 3D to 2D (Xu et al., ). Though effective, none of these methods essentially acknowledge the facts of 3D scene segmentation in three challenging aspects: (i) 3D prompts are naturally prior to the one in the 2D space, which should be carefully designed rather than a simple projection, (ii) the initial segmentation mask from multiple views might include rough edges and isolated background noises, (iii) local adjacent frames maintain the global consensus, which might be overlooked during the merging process.

To address these challenges, we present PointSeg, a novel perception framework that effectively incorporates different foundation models for tackling the 3D scene segmentation task without training. The key idea behind PointSeg is to learn accurate 3D point-box prompts pairs to enforce the off-the-shelf foundation models and fully unleash their potential in 3D scene segmentation tasks with three effective components. First, we construct a two-branch prompts learning structure to acquire the 3D point prompts and 3D box prompts respectively. The 3D point prompts are derived from localization abilities of PointLLM (Xu et al., ) to provide more explicit prompts in the form of points and 3D box prompts come from the 3D detectors (Shen et al., ; Wu et al., ). Considering that the naive prompts could result in fragmented false-positive masks caused by matching outliers, we propose the bidirectional matching strategy for the generation of accurate point-box prompt pairs. Furthermore, when incorporated with different 2D vision segmentation foundation models, such as SAM 2 (Ravi et al., ), our approach involves the iterative post-refinement to eliminate the coarse boundaries and isolated instances of background noise adaptively. Finally, with the primary aim of segmenting all points within the 3D scene, we employ the affinity-aware merging algorithm to capture pairwise similarity scores based on the 2D masks generated by the vision segmentation foundation models.

Comprehensive experiments on ScanNet (Dai et al., ), ScanNet++ (Yeshwanth et al., ), and KITTI-360 (Liao et al., ) demonstrate the superior generalization of the proposed PointSeg, surpassing previous specialist training-free model by 14.9$\%$-16.3$\%$ mAP and specialist training-based methods by 5.6$\%$-8$\%$ mAP across different datasets, all without training on domain-specific data. Remarkably, our zero-shot approach yields superior results in comparison to fully-supervised PointSAM (Zhou et al., ) trained on synthetic datasets, thereby emphasizing the effectiveness of PointSeg in the segmentation of intricate 3D scene. Moreover, we incorporate different foundation models, *i.e.*, SAM 2 (Ravi et al., ), SAM (Kirillov et al., ), FastSAM (Zhao et al., ), MobileSAM (Zhang et al., ), and EfficientSAM (Xiong et al., ), into our pipeline, and the performance gain shows that enhancements on 2D images can be seamlessly translated to improve 3D results. We summarize the contributions of our paper as follows:

- •
We present PointSeg, a novel framework for exploring the potential of leveraging various vision foundation models in tackling 3D scene segmentation task, without training or finetuning with 3D data.
- •
We design PointSeg as a two-branch prompts learning structure, equipped with three key components, *i.e.*, bidirectional matching based prompts generation, iterative post-refinement and affinity-aware merging, which can effectively unleash the ability of vision foundation
models to improve the 3D segmentation quality.
- •
PointSeg outperforms previous specialist training-based and training-free methods on 3D segmentation task by a large margin, which demonstrates the impressive performance and powerful generalization when incorporated with various foundation models.

## 2 Related Work

#### Closed-set 3D Segmentation.

Considering the point clouds in 3D space, 3D semantic segmentation task aims to predict a specific category towards the given point (Graham et al., ; Hu et al., ; Kundu et al., ; Li et al., ; Qi et al., ; Rozenberszki et al., ; Wang et al., ; Xu et al., ; Wang et al., ; Zhang et al., ). 3D instance segmentation task broadens this concept by pinpointing separate entities within the same semantic class and bestowing unique masks upon each object instance (Choy et al., ; Fan et al., ; Han et al., ; Hou et al., ; Engelmann et al., ; Hui et al., ; Jiang et al., ; Kolodiazhnyi et al., ; Liang et al., ; Schult et al., ; Sun et al., ; Vu et al., ; Lahoud et al., ; Yang et al., ). Among them, Mask3D (Schult et al., ) designs a transformer-based network to build the 3D segmentation network and achieves the state-of-the-art performance. Although Mask3D has made significant progress, like previous supervised learning methods, it still necessitates a substantial volume of 3D annotated data for network training. This limitation impedes the method’s generalization to open-world scenarios that include new objects from unseen categories. Furthermore, the collection of annotated 3D data is not only costly but sometimes unfeasible due to privacy concerns. Our framework, however, aspires to directly leverage the intrinsic zero-shot potential of SAM for 3D scene segmentation, thereby negating the necessity for further model training.

#### Open-set 3D Segmentation.

Inspired by the success of 2D open-vocabulary segmentation methods (Ghiasi et al., ; Liang et al., ), a series of works (Ding et al., ; Huang et al., ; Takmaz et al., ; Peng et al., ; He et al., ; Jiang et al., ) have dived to explore the potential of 3D open-vocabulary scene understanding. OpenMask3D (Takmaz et al., ) predicts 3D instance masks with the per-mask feature representations, which can be used for querying instances based on open-vocabulary concepts. OpenIns3D (Huang et al., ) employs a Mask-Snap-Lookup scheme to learn class-agnostic mask proposals and generate synthetic scene-level images at multiple scales. On the other hand, the interpolation capabilities of NeRFs (Mildenhall et al., ) are applied to integrate language with the CLIP feature by LERF (Kerr et al., ) and DFF (Kobayashi et al., ). And OR-NeRF (Yin et al., ) empowers users to segment an object by clicking and subsequently eliminate it from the scene. Although they have achieved encouraging instance segmentation results on indoor scenes with objects similar to the training data, these methods demonstrate a failure in complex scenes with fine-grained objects. In this study, we eliminate the reliance on a pre-trained 3D mask proposal network and instead focus directly on how to leverage the segmentation results of SAM to generate fine-grained 3D masks for 3D scenes.

#### Segment Anything Model in 3D.

The emergence of the Segment Anything Model (Kirillov et al., ; Ravi et al., ) have triggered a revolution in the field of 2D segmentation. Having been trained on the extraordinary SA-1B dataset, SAM has garnered a vast amount of knowledge, equipping it to effectively segment unfamiliar images without additional training. Followed by SAM, several works (Zhang et al., ; Zou et al., ; Xiong et al., ; Zhao et al., ; Zhang et al., ; Liu et al., ) have attempted to accelerate or customize the original SAM from different aspects. Recognizing the exceptional capabilities of SAM, various recent research initiatives are working diligently to incorporate SAM into 3D scene segmentation task. Several works (Yang et al., ; Yin et al., ; Guo et al., ) attempt to segment each frame individually or constructs a graph based on the superpoints to lift the segmentation results to 3D space.
However, these methods designate pixel prompts that are specific to each frame but do not synchronize across frames. This causes inconsistencies in segmentation across frames and produces substandard 3D segmentation results. Different from these 2D-to-3D lifting methods, SAMPro3D (Xu et al., ) attempts to locate 3D points in scenes as 3D prompts to align their projected pixel prompts across frames. Albeit effective, simply connecting 3D points to 2D space through projection is still too rough for complex scenes. In this paper, we propose a two-branch prompts learning structure towards accurate 3D prompts generation.

Figure: Figure 2: Overview of our proposed PointSeg. Our framework requires no training and aims to segment anything in 3D scene via three distinct stages: Bidirectional Matching based Prompts Generation, Iterative 2D Mask Generation, and Affinity-aware 3D Mask Refinement.
Refer to caption: x2.png

## 3 Method

As illustrated in Figure 2, we build a training-free 3D scene segmentation framework based on the off-the-shelf foundation models. Our PointSeg consists of three parts:
1) Bidirectional Matching based Prompts Generation (BMP) (Section 3.1). Given the reconstructed scene point cloud $\mathcal{P}$ $=\{\mathbf{p}\}$ together with a set of posed RGB-D images $\{I_{m}\}_{m=1}^{M}$, PointSeg first employs a two-branch prompts learning structure to acquire the 3D mask $\mathbf{m}_{3D}$ and box $\mathbf{b}_{3D}$prompts, respectively. And the final point-box prompts pairs are obtained by the further bidirectional matching. These prompts serve as inputs to 2D vision foundation models, such as SAM 2 (Ravi et al., ), after aligning with the pixels in 2D images.
2) Iterative 2D Mask Generation (Section 3.2). Then, we perform Iterative Post-refinement (IPR) to enable the generation of mask proposals adaptively. 3) Affinity-aware 3D Mask Refinement (Section 3.3). We calculate the affinity scores between the points generated in the point-box pairs and the mask proposals, followed by the Affinity-aware Merging (AM) (see Algorithm 2) to obtain the final 3D segmentation masks.

### 3.1 Bidirectional Matching based Prompts Generation

Towards the generation of accurate 3D prompts, we dive into the exploration of the intrinsic characteristics of 3D data and design a two-branch prompts learning structure. The central concept of our approach entails identifying 3D points and boxes within scenes, serving as inherent 3D prompts, and aligning their projected pixel prompts across various frames. This ensures consistency across frames, both in terms of pixel prompts and the masks predicted by the segmenter.

#### Two-branch Prompts Generation.

Inspired by the strong ability of semantic understanding of some works (Xu et al., ; Zhang et al., ; Zhu et al., ) in 3D scene, we first employ PointLLM (Xu et al., ) for localization and rough segmentation in the upper branch in BMP of Figure 2. Given the point cloud $\mathcal{P}$ $\in\mathbb{R}^{N\times 3}$, we also apply realistic projection to generate the different $S$ views, using the zero-initialized 3D grid $G\in\mathbb{R}^{H\times W\times D}$, where $H$/$W$ denote the spatial resolutions and $D$ represents the depth dimension vertical to the view plane. For each view, the normalized 3D coordinates of the input point cloud $\mathbf{p}=(x,y,z)$ in a voxel in the grid can be denoted as

$$ $G(\lceil sHx\rceil,\lceil sWy\rceil,\lceil Dz\rceil)=z,$ (1) $$

where $s\in(0,1]$ is the scale factor to adjust the projected shape size. Following PointCLIPv2, we further apply the quantize, densify, smooth, and squeeze operations to obtain the projected depth maps $V=\{v_{i}\}_{i=1}^{S}$. For the textual input, we utilize the large-scale language models (Brown et al., ) to obtain a series of 3D-specific descriptions. After feeding the depth maps and texts into their respective encoders, we can obtain the dense visual features $\{f_{i}\}_{i=1}^{S}$ where $f_{i}\in\mathbb{R}^{H\times W\times C}$ and the text feature $f_{t}\in\mathbb{R}^{K\times C}$. Then, we segment different parts of the shape on multi-view depth maps by dense alignment for each view $i$ and average the back-projected logits of different views into the 3D space, formulated as:

$$ $f_{m}=average(Proj^{-1}(f_{i}\cdot f_{t}^{T})),$ (2) $$

where $f_{m}$ is the segmentation logits in 3D space.

Apart from the point-level prompts in the 3D mask, we intend to inquire into how to generate dense region-level prompts to fully unleash the advantages of 3D prompts in the another branch. To enhance the ability to segment regions accurately in the subsequent segmenters, we exploit the localization abilities of 3D detector to provide more explicit prompts in the form of bounding boxes. The point cloud $\mathcal{P}$ is taken as input by the frozen 3D detectors to generate the 3D bounding box $(x,y,z,w,h,l)$ for each category with corresponding proposal features $f_{b}$, which can be represented as

$$ $f_{b}=Det(\mathcal{P}).$ (3) $$

Figure: Figure 3: Illustration of the proposed bidirectional matching, which consists of three steps: Forward matching, Reserve matching, and Filter and Resize.
Refer to caption: x3.png

#### Bidirectional Matching.

Given the coarse segmentation mask and the bounding box, we can already conduct the alignment to 2D images. However, the naive prompts often result in inaccurate and fragmented outcomes, riddled with numerous outliers. Therefore, we design the bidirectional matching strategy to put constraints on the point and box for high quality promptable segmentation.

Considering the extracted features $f_{m}$ and $f_{b}$, we compute the region-wise similarity between the two features to discovery the best matching locations

$$ $<f_{m}^{i},f_{b}^{j}>=\frac{f_{m}^{i}\cdot f_{b}^{j}}{\|f_{m}^{i}\|\cdot\|f_{b }^{j}\|},$ (4) $$

where $<f_{m}^{i},f_{b}^{j}>$ denotes the cosine similarity between $i$-th mask feature and $j$-th box feature. Ideally, the matched regions should have the highest similarity. Then as illustrated in Figure 3, we propose to eliminate the matching outliers based on the similarity scores in three steps:

- •
First, we compute the forward similarity $<P_{m},f_{b}>$ between the points in the mask $P_{m}$ and the box $f_{b}$. Using this score, the bipartite matching is performed to acquire the forward matched points $P_{b}^{\rightarrow}$ within the box.
- •
Then, we perform reverse matching between the matched points $P_{b}^{\rightarrow}$ and $f_{m}$ to obtain the reverse matched points $P_{m}^{\leftarrow}$ within the mask using the reverse similarity $<f_{m},P_{b}^{\rightarrow}>$.
- •
Finally, we resize the box according to the points in the forward sets if the corresponding reverse points are not within mask, denoted as $\hat{P}_{b}=\{\mathbf{p}_{b}^{i}\in P_{b}^{\rightarrow}|\mathbf{p}_{m}^{i}\ in
\ \mathbf{m}_{3D}\}$. Similarly, we adjust points in the mask by filtering out the points in the reverse set if the corresponding forward points are not within the box, denoted as $\hat{P}_{m}=\{\mathbf{p}_{m}^{i}\in P_{m}^{\leftarrow}|\mathbf{p}_{b}^{i}\ in
\ \mathbf{b}_{3D}\}$.

In this way, we can form the point-box pairs with a new set of points in the mask and a different box with new size, which are feed into the further segmentation module.

### 3.2 Iterative 2D Mask Generation

Conditioned on the reorganized point-box pairs, we then make the alignment between the 3D prompts pairs and the 2D images. In particular, given a point $\mathbf{p}$ in the prompts pairs with the camera intrinsic matrix $I_{i}$ and world-to-camera extrinsic matrix $E_{i}$, the corresponding pixel projection $\mathbf{x}$ can be calculated by

$$ $\mathbf{x}=(u,v)=I_{i}\cdot E_{i}\cdot\tilde{\mathbf{p}},$ (5) $$

where $\tilde{\mathbf{p}}$ is the homogeneous coordinates of $\mathbf{p}$. Similarly, the corresponding projected box across images can be denoted as $\mathbf{b}=(u,v,h,w)$.

The vanilla segmentation model, such as SAM 2 (Ravi et al., ), accepts various inputs such as pixel coordinates, bounding boxes or masks and predict the segmentation area associated with each prompt. Hence, we feed the projected 2D point-box pairs calculated before into the foundation segmenters.

Figure: Algorithm 1 Iterative Post-refinement

Figure: Figure 4: Different Iteration strategies. Without adaptive iteration, the segmentation results can be sensitive to the number of fixed iteration steps. But the performance of adaptive iteration is proved to be more effective.
Refer to caption: x4.png

#### Iterative Post-refinement.

Through the above operation, we can obtain 2D segmentation mask on all frames from the decoder, which however, might include rough edges and isolated background noises. For further refinement, we iteratively feed the mask back into the decoder $Dec_{M}$ for the adaptive post-processing. As illustrated in Algorithm 1, we first obtain the 2D mask $M_{0}$ by feeding the 2D point-box pairs into the SAM-based decoder. Then we prompt the decoder additionally with this mask along with these projected prompt pairs to obtain the next mask. And the initial value $\Delta$ to record change is set to infinity. In each subsequent iteration, we calculate the change ratio between the two adjacent masks and compare it with our predefined threshold $\vartheta$, which is set to 5$\%$ by default. We repeat this iterative process until the change value falls below the threshold adaptively. It is worth noting that we have also tried the method of fixed iteration steps, but this adaptive iteration method have proved to be more effective which is shown in Figure 4 and Table 4.3.

### 3.3 Affinity-aware 3D Mask Refinement

After previous procedures, we have obtained the final set of 2D segmentation masks across frames. With the ultimate goal of segmenting all points in the 3D scene, we employ the affinity-aware merging algorithm to generate the final 3D masks.

#### Affinity-aware Merging.

Based on the $i$-th projected point-box pair on the $m$-th image and their 2D image segmentation mask, we compute the normalized distribution of the mask labels, denoted as $\mathbf{d}_{i,m}$. The affinity score between two projected points in the $m$-th image can be computed by the cosine similarity between the two distributions, which can be represented as:

$$ $A_{i,j}^{m}=\frac{\mathbf{d}_{i,m}\cdot\mathbf{d}_{j,m}}{\|\mathbf{d}_{i,m}\| \cdot\|\mathbf{d}_{j,m}\|}.$ (6) $$

The final affinity score across different images can be computed by the weighted-sum:

$$ $A_{i,j}=\frac{\sum_{m=1}^{M}\alpha_{i,j}^{m}\cdot A_{i,j}^{m}}{\sum_{m=1}^{M} \alpha_{i,j}^{m}},$ (7) $$

where $\alpha_{i,j}^{m}\in(0,1)$ denotes whether it is visible in the images.

Further, we utilize the designed affinity-aware merging algorithm to gradually merge 3D masks using the computed affinity matrix. As illustrated in Algorithm 2, the algorithm works on an affinity matrix representing the affinity scores between pairs of points. The goal is to assign labels to these points based on their affinities.

We start by initializing labels and an identifier. It then iterates over each point. If a point hasn’t been labeled yet, it gets the current identifier. The algorithm then checks the point’s neighbors. If a neighbor is unlabeled, it calculates an affinity score between the set of already labeled points and the current neighbor. If this score surpasses a certain threshold, the neighbor is labeled with the current identifier, the identifier is incremented, and the algorithm continues with this neighbor as the current point. The region-point merging is similar to Equation 7, which is computed as the weighted average between the current point and the points inside the region. The whole process repeats until all points have been labeled, effectively grouping points based on their mutual affinities.

Table: Table 1: Results of 3D segmentation on ScanNet, ScanNet++, and KITTI-360 datasets. We report the mAP and AP scores on the three datasets. Best results are highlighted in bold.

Table: Table 2: Results of integrating with different segmentation models on ScanNet, ScanNet++, and KITTI-360 datasets.

## 4 Experiment

### 4.1 Setup

#### Baselines.

We compare our approach with both training-based and training-free baselines. For training-based comparison, we select the state-of-the-art transformer-based
method PointSAM (Zhou et al., ). For training-free methods, we compare against the 2D-to-3D lifting methods (Yang et al., ; Xu et al., ) and the 3D-to-2D projection methods (Xu et al., ) respectively. For Implementation, we apply the V-DETR (Shen et al., ) and VirConv (Wu et al., ) as the indoor and outdoor 3D detector.

Figure: Table 3: Ablations results of different 3D point models.

### 4.2 Main Results

#### Comparisons with the state-of-the-art Methods.

We compare the segmentation results on ScanNet, ScanNet++, and KITTI-360 datasets, covering both indoor and outdoor scenes. As shown in Table 1, comparing to previous training-free methods, our PointSeg obtains 16.3$\%$ mAP, 18$\%$ AP50, and 16.4$\%$ AP25 performance gains on ScanNet. On the more challenging indoor dataset ScanNet++, our method still obtains 14.9$\%$ mAP, 15.4$\%$ AP50, and 15.6$\%$ AP25 improvements. Furthermore, when evaluating the performance of our PointSeg on outdoor KITTI-360, our method still surpasses corresponding zero-shot method by 15$\%$ mAP, 16.1$\%$ AP50, and 16.9$\%$ AP25, respectively. In this regard, our PointSeg demonstrates superior generalization ability to complex 3D scenarios.

Notably, when compared to previous training-based methods, PointSeg outperforms PointSAM (Zhou et al., ) by 5.6$\%$-8$\%$, 7.2$\%$-11.1$\%$, 7.9$\%$-10.3$\%$ in terms of mAP, AP50, and AP25 across various datasets. This further demonstrates the robustness and effectiveness of our approach in 3D segmentation task.

#### Qualitative Results.

The representative quantitative segmentation results of our proposed PointSeg on the three datasets are shown in Figure 1 and Figure A1. We also present the quantitative results of the state-of-the-art training-based methods PointSAM (Zhou et al., ) and training-free method SAMPro3D (Xu et al., ). We can observe that PointSAM and SAMPro3D often mistakenly segment an object into two different objects and exhibit poor performance in segmenting relative objects lacked spatial structure. Our PointSeg can handle complex scenes and is capable of generating clean segments on objects of small size, further underscoring the effectiveness of our approach.

### 4.3 Ablation Study

In this section, we conduct extensive ablation studies on ScanNet to show the effectiveness of each component and design. Unless otherwise specified, SAM (Kirillov et al., ) is used as 2D segmentation foundation model for ablation studies by default.

Different Foundation Models.
*i)* Apart the basic segmentation foundation model SAM 2 (Ravi et al., ), we also integrate different segmentation foundation model, *i.e.*, SAM (Kirillov et al., ), MobileSAM (Zhang et al., ), FastSAM (Zhao et al., ), and EfficientSAM (Xiong et al., ), into our framework. As shown in Table 2, PointSeg demonstrates consistent performance improvements among different datasets, where the original SAM 2-based result performs best. This is consistent with the relative results of these methods in 2D segmentation. *ii)* In Table 4.1, we change the model for localization of point prompts. The results demonstrate that more accurate points for 3D prompts can indeed contribute to the performance gain. *iii)* In Table 4.1, the improvement of different 3D detectors will also bring about the improvement of the performance of our method. These results suggest that our framework can serve as a foundational structure, capable of integrating a variety of fundamental models. And the enhancements observed in these models can be smoothly translated to 3D space, thereby augmenting the overall performance.

Main Components.
Further, we explore the effects of bidirectional matching based prompts generation (BMP), iterative post-refinement (IPR) and affinity-aware merging (AM). We illustrate the importance of different components by removing some parts and keeping all the others unchanged. The baseline setting is to use points and boxes independently as 3D prompts. And the masks from the 2D segmentation model decoder are used to merge the final 3D masks, without any post-refinement. The merging strategy is same as the adjacent frame merging in  (Yang et al., ). The results of components ablations are shown in Table 4.3. We observe that using the iterative post-refinement strategy leads to a noticeable increase in performance, which demonstrates the necessity of refining the initial 2D masks.
The performance degradation caused by the absence of bidirectional matching proves that the constraints between the point and box prompts can indeed help to generate the accurate point-box pairs. And the performance drop without the affinity-aware merging shows that the affinity score is indeed useful to link the point and the masks.

Figure: Table 5: Ablations of main components in our framework.

Figure: Table 7: Ablation of bidirectional matching. *no* means no matching.

Iterative Post-refinement.
As mentioned before, when performing the post-refinement during the initial 2D mask generation, we have tried different strategies of fixed numbers method and adaptive iteration method. As shown in Table 4.3, the six rows in the middle represent the method to use a fixed number of iterations and the last row is the adaptive iteration. We can observe that as the number of iterations increases, the corresponding AP value also becomes higher compared to no iteration, which shows that the obtained mask is also more accurate. In the method with a fixed number of iterations, the results reach the highest in the third iteration, but are still lower than those in the adaptive iteration method. This largely illustrates the effectiveness of the iterative post-processing method in generating more accurate masks, and also reveals that the adaptive iteration is more beneficial.

Figure: Table 10: Ablation of mask change ratio.

Different Matching Strategy.
To validate the effect of different matching method, we explore the effects of the forward matching and the reverse matching of the proposed bidirectional matching, as shown in Table 4.3. Without the guidance from the respective point and box, the naive prompts contain many invalid points and regions, which provide negative prompts for the following segmentation models. Our bidirectional matching improves the performance of forward and reverse matching by 5.4$\%$ and 3.7 $\%$, which show the effectiveness of the proposed bidirectional matching strategy.

Figure: Table 11: Inference Speed.

Affinity-aware Merging.
In the final mask merging stage, we have also tried other merging algorithm. As shown in Table 4.3, we compare our affinity-aware merging (AM) with (a) bidirectional merging (BM) from (Yang et al., ), (b) pure merging (PM) without affinity scores, which is simplified from our approach and (c) prompt ID based merging (IDM) from (Xu et al., ). With other inferior merging method, the performance drops dramatically which shows the superiority of our proposed affinity-aware merging algorithm in solving the mask merging problems in 3D scene.

3D Point Models.
As shown in Table 4.3, the missing of PointLLM/3D detector causes the performance drop and only the matching of combination performs best.

Mask Change Ratio.
In the iterative post-refinement module, we set a mask change ratio threshold $\vartheta$ as a condition for stopping the iteration. Here, We show the effect of different ratios on the results. As shown in Table 10, the overall results perform best when mask change ratio is set as 5$\%$.

Inference Speed.
As shown in Table 11, we test the running efficiency on NVIDIA V100 GPU and the detailed FPS of each module. The FLOPs of our method is 4.2G.

## 5 Conclusion

In this paper, we present PointSeg, a novel training-free framework integrating off-the-shelf vision foundation models for solving 3D scene segmentation tasks. The key idea is to learn accurate 3D point-box prompts pairs to enforce the off-the-shelf foundation models. Combining the three universal components, *i.e.*, bidirectional matching based prompts generation, iterative post-refinement and affinity-aware merging, PointSeg can effectively unleash the ability of various foundation models. Extensive experiments on both indoor and outdoor datasets demonstrate that PointSeg outperforms prior unsupervised methods and even surpass fully-supervised by a large margin, which reveals the superiority of our model in 3D scene understanding task.

## Appendix

## Appendix A Datasets and Metrics.

To validate the effectiveness of our proposed PointSeg, we conduct extensive experiments on three popular public benchmarks: ScanNet (Dai et al., ), ScanNet++ (Yeshwanth et al., ), and KITTI-360 (Liao et al., ). ScanNet provides RGBD images and 3D meshes of 1613 indoor scenes. ScanNet++ is a recently released indoor dataset with more detailed segmentation masks, serving as a more challenging benchmark for 3D scenarios. It contains 280 indoor scenes with high-fidelity geometry and high-resolution RGB images. KITTI-360 is a substantial outdoor dataset that includes 300 suburban scenes, which comprises 320k images and 100k laser scans. We evaluate ours segmentation performance with the widely-used Average Precision (AP) score. Following  (Schult et al., ; Dai et al., ), we report AP with thresholds of 50$\%$ and 25$\%$ (denoted as AP50 and AP25) as well as AP averaged with IoU thresholds from 50$\%$ to 95$\%$ with a step size of 5$\%$ (mAP).

Figure: Figure A1: More qualitative results comparison on ScanNet, ScanNet++ and KITTI-360 datasets with comparison to the training-based method PointSAM (Zhou et al., ) and the training-free method SAMPro3D (Xu et al., ).
Refer to caption: x5.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses that the writing is not clear. I have many difficulties in understanding the proposed method.
soundness: 3
strengths: Extensive experiments are conducted to demonstrate the effectiveness of the proposed method. The results are promising.  The paper is well-organized. It clearly reflects the significant effort put into building a complete pipeline and carefully designing and conducting the experiments.
summary: This paper introduces PointSeg, a training-free framework designed to leverage existing 2D vision foundation models for 3D scene segmentation tasks. PointSeg achieves this by generating precise 3D prompts, allowing it to match corresponding pixels across frames and segment objects in 3D space effectively. The approach is composed of a two-branch prompt-learning structure, bidirectional matching for point and proposal prompts, adaptive post-refinement with foundation models, and an affinity-aware merging algorithm for enhanced mask accuracy. PointSeg demonstrates substantial improvements over existing training-free models, surpassing state-of-the-art performance on datasets like ScanNet and KITTI-360  and even outperforming some training-based models.
weaknesses: 1. Though the organization of the paper is okay, I think the method is not easy to follow. First, the writing is not clear enough. For example, the definition in Equation (1) is weird. How to understand the ‘=z’?
2. How to understand Equation (4)? What is Det() in Equation (3)? Why can the proposal feature $f_b$ and the segmentation logits $f_m$ be used to calculate cosine similarity? This part is confusing.
3. The Affinity-aware Merging algorithm seems to be a depth-first algorithm similar to the union-find set. I think Alg.2 is confusing. In Line 12 of Alg.2, if $A_{R, p_j} > \tau$, then the same id is assigned to this point j. However, after that, I guess it should consider the neighbors of j and check whether they belong to the same group of j. And that is why we have i <- j . However, the following id <- id + 1 makes it impossible. I wonder if this is a mistake in my understanding or if the algorithm is written wrong.
4. I cannot get the information that tried to be conveyed by Fig. 3. The effect of the bidirectional filtering is not clear in the figure. More detailed captions are required.
5. Though the proposed method is demonstrated to be effective, the whole pipeline is too complex. Though the pipeline is training-free, 3 pre-trained models are required. Thus, their mistakes and shortcomings are also inherited by the proposed method. Even worse, before the segmentation begins, text prompts are required for computing the segmentation logits. This implies the user must know the categories of the objects in the scene. In contrast, similar methods that only use SAM for segmentation do not have such limitations.
6. The hybrid pipeline also leads the concerns about the segmentation efficiency. As shown in Table 11, the inference speed is very slow. A time consumption comparison with the other existing methods is required.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1) Experimental setups：The paper does not provide a detailed explanation of how hyperparameters for each module in the model (such as iteration count, thresholds, etc.) are set. The absence of this information may affect the reproducibility of the results.
2) Theoretical validation: While the paper presents some mathematical derivations, further elaboration and detail would enhance the reader's understanding of the model's operational mechanisms. For instance, the absence of a detailed analysis of the interrelationships among key components may affect the depth and comprehensiveness of the theoretical explanations. Providing additional derivations or clarifications on how these components interact and contribute to the overall performance of the model would strengthen the theoretical foundation of the paper and improve its clarity for the audience.
3) Generalization performance: The paper's testing only covers the ScanNet, ScanNet++, and KITTI-360 datasets, lacking validation on a more diverse range of scenes. Furthermore, what is the model's generalization performance on point cloud data collected from real-world sources, such as data obtained from mobile devices or drones?
soundness: 3
strengths: PointSeg introduces a novel and efficient approach to 3D segmentation that expands the possibilities for leveraging 2D foundation models in new domains, and its contributions are both methodologically sound and practically valuable. Explicitly,
1) PointSeg pioneers a framework that performs 3D scene segmentation without requiring training or fine-tuning on 3D data, showcasing the untapped potential of visual foundation models in 3D tasks.
2) The dual-branch structure, incorporating bidirectional matching, iterative refinement, and affinity-aware merging, maximizes the capabilities of foundation models, delivering high-quality 3D segmentation results through a versatile prompt-based mechanism.
3) PointSeg achieves outstanding performance on 3D segmentation tasks, surpassing both training-based and training-free methods, and demonstrates strong adaptability across different foundation models, highlighting its broad applicability and effectiveness.
summary: This paper introduces PointSeg, a novel framework for 3D scene segmentation that operates without the need for training, leveraging pre-trained 2D foundation models. PointSeg employs a unique two-branch prompt generation architecture and a bidirectional matching strategy to project 3D point cloud data into meaningful 2D cues. These cues, aligned with pre-trained 2D models, provide spatially accurate segmentation in 3D spaces. The authors evaluate PointSeg on various datasets, demonstrating its efficacy in achieving high segmentation accuracy across different indoor and outdoor scenes. The primary contributions of the paper are as follows:
i) A Training-Free 3D Segmentation Framework: PointSeg introduces a framework that bypasses the need for 3D model training by leveraging pre-trained 2D models, making it computationally efficient and adaptable to new environments.
ii) Bidirectional Matching-Based Prompt Generation: The proposed two-branch architecture generates both point and box prompts, which are refined through bidirectional matching, ensuring high segmentation accuracy.
iii) Comprehensive Evaluation: The paper presents extensive experimental results, comparing PointSeg’s performance with baseline models and demonstrating its competitive accuracy and adaptability across multiple 3D segmentation tasks.
Overall, PointSeg represents a significant step toward making 3D scene segmentation more accessible and efficient by eliminating the need for training and instead capitalizing on foundation models.
weaknesses: 1) Novelty: The framework heavily relies on the quality and appropriateness of the selected pre-trained 2D models. If these models do not generalize well to the specific characteristics of 3D scenes, it may hinder PointSeg's performance. The paper could benefit from discussing potential limitations arising from this dependency.
2)  Interaction: Although the three key components (bidirectional matching, iterative refinement, and affinity-aware merging) are introduced, the paper could provide more in-depth explanations and analyses of how each component contributes to overall performance. A more detailed exploration of their interactions and potential trade-offs would enhance the understanding of their roles within the framework.
3) Generalization: The paper only tests on the ScanNet, ScanNet++, and KITTI-360 datasets, lacking validation on a more diverse range of scenes, such as more complex 3D environments or dynamic scenarios (e.g., Waymo, nuScenes). This limitation may affect the universality of the conclusions drawn.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: The questions can be seen under "weakness." Additionally, there is another issue as follows:

1. How long does it take on average to perform inference on a single scene, including the time spent on various large models?
soundness: 3
strengths: 1. PointSeg does not require additional training, making it efficient and easy to deploy across different tasks and datasets.

2. It utilizes off-the-shelf 2D vision foundation models, alleviating the challenge of limited 3D datasets and reducing the need to build a 3D model from scratch.

3. The proposed components are useful based on the ablation.
summary: The paper presents PointSeg, a training-free method that uses existing 2D vision models for 3D scene segmentation. PointSeg aligns 3D prompts with pixels across frames and employs a two-branch structure with adaptive refinement and merging algorithms. It outperforms state-of-the-art training-free models on several 3D datasets and even surpasses some training-based methods, demonstrating its effectiveness as a generalist model for 3D segmentation.
weaknesses: 1. It is a more engineering-oriented work that lacks novelty, mainly combining various large models rather than being a research-focused work.

2. The paper lacks some relevant references. It uses 2D models to segment 3D scenes, as do the following works. However, this paper does not provide a comparative discussion with them:
[1] GOV-NeSF: Generalizable Open-Vocabulary Neural Semantic Fields. Yunsong Wang, etc. CVPR, 2024.
[2] Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D. Mukund Varma T, etc. CVPR, 2024
[3] GNeSF: Generalizable Neural Semantic Fields. Hanlin Chen, etc, NeurIPS, 2023.

3. The paper is unclear in certain aspects. For example, is the scale factor a hyperparameter or a learnable parameter? If it is a hyperparameter, what is the set value?

4. The paper lacks implementation details.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: Please see weaknesses.
soundness: 2
strengths: 1. The method introduces a novel approach by combining detection bounding boxes with point prompts to guide SAM, ensuring 3D consistency in prompts.
2. The reported quantitative results outperform current SOTA methods.
summary: The paper presents PointSeg, a training-free approach that leverages foundational vision models for 3D scene segmentation. PointSeg uses pretrained models to generate point and bounding box prompts, which are then used to prompt the SAM model for 2D segmentation masks. Through Iterative Post-Refinement, SAM is repeatedly prompted to improve 2D mask accuracy, while Affinity-aware Merging integrates the refined 2D masks into 3D. PointSeg is evaluated on both indoor and outdoor datasets, with extensive ablation studies conducted to validate its design choices.
weaknesses: 1. The description of the Two-Branch Prompt Generation process lacks clarity. Could you clarify how PointLLM is used for rough localization? What specific input prompts are used for PointLLM? Which large-scale language models are used to generate the text input, and with what prompts? Can you provide details on the models used to obtain dense visual and text features?
2. The notation $K$ in line 248 needs clarification. Could you please define the notation $K$ used in line 248? If it refers to the number of classes, how is this determined? Is $K$ an input parameter for the algorithm or derived in some way?
3. The SOTA comparisons are unclear. Can you provide a detailed description of how the comparisons with SOTA models were conducted? Specifically, what prompts were used for promptable segmentation models like PointSAM? If PointSeg's generated prompts were used for other models, please explain this process.
4. The experiment setup is unclear. Could you provide a step-by-step explanation of how the produced masks were aligned with ground truth masks for AP calculation? Given that PointSeg does not generate semantic labels, what method was used to match the segmented instances with ground truth instances?