SA3DIP: Segment Any 3D Instance with Potential 3D Priors
Authors: Xi Yang
Xu Gu
Xingyilang Yin
Xinbo Gao
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Open Vocalbulary
3D Instance Segmentation
Foundation Model
3D Detection
Primary area: machine_vision

Abstract: The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

SA3DIP: Segment Any 3D Instance with Potential 3D Priors
Xi Yang', Xu Gu', Xingyilang Yin'; Xinbo Gao2 Xidian University, "Chongqing University of Posts and Telecommunications yangxexidian edu. cn, fryangu, y yxyl) xidian edu, cn, gaoxbecqupt. edu. cn
Abstract
The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based estimated normal from spatial coordinates, resulting in under-segmeenation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's 's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide further merging process. Furthermore, notice considerable portion low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D alass-agnostic instance segmentation. Experimental evaluations various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2- INS dataset are available HERE.
Introduction
3D instance segmentation fundamental task pivotal to 3D understanding across various domains such as autonomous driving, robotics navigation, and virtual reality applications. State-of-the- art methods [1, 2] are predominantly supervised and rely heavily on precise 3D annotations for training, thus constraining their applications in open-world scenes. Compared to scarce 3D labeled data, the acquisition and annotation of 2D images are more convenient. Recently, 2D foundation models [3-6] trained on large-scale annotated 2D data show impressive performance and strong generalization capabilities in zero-shot scenarios. Recent efforts have sought to leverage Segment Anything Model (SAM) by lifting its class-agnostic 2D segmentation results to 3D tasks [7-10]. Specifically, some methods [7, 8] propose pipeline that decomposes the 3D scene into geometric primitives and leverages 2D multi-view masks from SAM to calculate pairwise similarity scores as merging guidance. Further well-designed algorithms or Graph Neural Networks (GNNs) are included to ensure multi-view consistency.
CCorresponiigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Clean 3D point cloud
Posed images
Superior superpoints
Superior segmentation
INPUR
Ours
Under-segmentation
O segmentttooo
Part-level masks
Error accumulation
Inferior superpoints
Final segmentation
Figure 1: Comparison of our SA3DIP with other methods. Methods like SAI3D (bottom) fail to distinguish instances with similar normals when computing superpoints, which accumulate the final segmentation. Moreover, the part-level 2D segmentation transfers to 3D space, resulting in over-segmented 3D instances. We present a novel pipeline segmenting any 3D instances, which overcomes the limitations by exploiting additional 3D priors, specifically by incorporating both geometric and textural prior on superpoints computing, and supplementing 3D space constraint provided 3D prior by utilizing 3D detector.
However, the geometric rudimentary vee-semmentation initialization impedes their ability to group superpoints on points with highly similar normals, such as boards on walls and books on tabletops. As shown in Fig. bottom left, the blackboard and the wall are wrongly allocated within the same superpoint using previous methods. Owing to the coarse-to-fine pattern of the pipeline, errors at this stage propagate to subsequent stages, which the sophisticated merging algorithms fail to rectify. Furthermore, current approaches heavily rely on 2D foundation models and design algorithms or GNNs within 2D space, neglecting the inherent 3D priors of the data. Part-level segmentation in the generated 2D masks by SAM transfers to 3D space and leads to over-segmented 3D instances. As illustrated in Fig. bottom right, the sofa and chairs are segmented at the part level 2D space, causing wer-seemmntttiion in the final results. These limitations primarily stem from the under- exploitation of 3D priors: (1) Complete point cloud data encompasses only spatial coordinates but also color channels; (2) Constraints provided by 3D space prior to the merging process cannot be neglected.
In this paper, we present SA3DDI (Segment Any 3D Instance with potential 3D Priors), novel method for segmenting high-quality 3D instances. Specifically, we observe that distinct instances with similar normals often exhibit different colors. Therefore, we incorporate both geometric and textural priors generate finer-grained complementary primitives. As shown in Fig. top right, our method identifies the boundary between the blackboard and the wall clearly. In this way, the initial errors are minimized, which reduces error accumulation in the subsequent process. Moreover, we exploit the 3D prior at the merging stage to provide constraints on the over- segmented 3D instances, which is implemented by incorporating 3D detector. This additional 3D prior enables rectification on the over-segmented 3D instances, while preserving the capability in handling fine-grained objects. Therefore, the sofa and chairs maintain their integrity in 3D space by our approach, which is illustrated in Fig. top right. Additionally, we notice that the widely-used benchmark, ScanNet [1 1], contains

---

considerable portion of low- cow-quality ground truth annotations for instance segmentation, which leads biases in assessing model performance. Thus, we propose ScanNerV2-INS, point-level enhanced version tailored for 3D class-agnostic instance segmentation. The revised dataset contains fewer incomplete labels and fewer missing instances, which better showcases real-world scenarios.
Our contributions are three-fold: (I) We present SA3DIP, a novel pipeline for segmenting any 3D instances by exploiting potential 3D priors, which includes incorporating both geometric and color priors on computing 3D superpoints, and introducing constraints provided by 3D prior at the merging stage: (2) We propose a point-level enhanced version of ScanNetV2, specifically for 3D aaassaa instance segmentation by rectifying incomplete annotations and incorporating more instances; (3) Extensive experiments are conducted on ScanNetV2, ScanNetV2-INg and ScanNet++ [12] datasets, and the competitive results demonstrate the effectiveness and robustness of our method.
2 Related Work
Close-set 3D segmentation. 3D semantic segmentation aims to classify each point into a specific semantic class [13-21]. 3D instance segmentation, on the other hand, assigns unique masks to each distinet instance within the same semantic category [2, 22-26]. Prior research, categorized as Grouping-baad based [23, 27, 28], Kerne--base [29], and Transformer-bar
[2, 30, 31] methods, has primarily relied on labeled datasets a supervised manner. Mask3D [2] proposes the first Transformer-bius based model for 3D semantic instance segmentation that uses instance queries and Transformer decoders. Spherical Mask [1] achieves state-of tate-of-the-art 3D instance segmentation perfor- mance on the ScanNetV2 dataset by leveraging novel coarse-to-fine approach [32, 33] based on spherical representation. However, they all necessitate a significant corpus of annotated 3D data for network training, which is financially burdensome and poses challenges for extending the method to open-world scenarios featuring novel objects from unobserved categories.
Open-set 3D segmentation. 2D foundation models [3-6] have exhibited remarkable efficacy across various tasks. Training on the SA-1B dataset with 11 million images and 1.1 billion masks, Segment Anything model (SAM) serves as a cornerstone for image segmentation, allowing strong zero-shot transfer ability and diverse prompts such as points, boxes, and texts to generate high-quality segmentation masks. Inspired by the generalization capabilities foundation models, certain works 34-37] explore the feasibility of bridging the gap between 2D and 3D, enabling various open- vocabulary 3D tasks. OpenScene [38] and OpenMask3D [25] both rely on transferring knowledge from CLIP, where the former infers CLIP features for each 3D point and classifies them embeddings of class labels, and the latter uses additional pre-trained 3D segmentation model to produce class- agnostic 3D proposals and classifies them based on CL.P scores. SAM3D [10] pioneers the extension of SAM into 3D perception by transferring segmentation information from 2D images to 3D space. SAMPro3D [9] attempts to locate 3D points natural prompts to align their projected pixel prompts across frames, while its segmentation quality heavily relies on the accuracy of 2D segmentation results. Several other works [7, 8] follow the idea that segments each frame individually and devises a merging algorithm or graph neural network (GNN) to guide the merging process of pre-segmented superpoints. However, the limited use of 3D priors restricts the performance in both superpoints computing and region growth, which results in substandard 3D segmentation. In this paper, we present SA3DIP, which incorporates more priors and constraints minimize error accumulation and over-segmentation, by thorough exploitation of 3D priors.
3 Methodology
3.1 SA3DIP
Overview. Our overall pipeline is shown in Fig. 2. Given point cloud P E N and corre- 1 which denote the RGB, depth images, camera intrinsic and extrinsic parameters, respectively, the masks of 3D instances in the scene are desired as output. First, we generate complementary 3D primitives of the given point cloud via performing 3D over- segmentation both geometric and textural priors. Then, we construct the superpoint graph by treating the 3D primitives and their relations as nodes and edges of the graph, respectively. Leveraging the 2D masks generated by 2D foundation segmentators like SAM, create the affinity matrix which
sponding 2D data {Im, I Em,
3

---

A.
Complementary Primitives G Generation
C.
Region grow ing
Piner-grained primitives (0)
Over-sag geumetry Xi
Over-arg
Instance-aware refinement
texture
ffinity
Merging criteria
Fuclidean dis.
Dynamic
Region growing
Sceme Graph Construction
Input
Graph nodes
Poontelood ge
detector
AfBlett metris
3D baxes (bb
Primary say.
Edge weights
Superpoint graph
Instance-awaa awaar refinement
Posed RGB-D Trames
Refined
foundation segmeentator
Primitive-mask matching
Outpas
Posed iine
MT. D...D..N. Pare
Figure 2: Overall pipeline. Our approach first integrates both geometric and textural priors for grouping 3D primitives (step A). Corresponding posed masks are generated using SAM. An affinity matrix is then computed based these 2D-3D results serving edge weights (step B). Region growing and instance-aware refinement are conducted on the constructed scene graph, utilizing 3D box constraint to address over-segmentttion while maintaining the fine-grained outcomes (step C).
contains the node features and edge weights. Finally, we perform affinity- and distance-and aware region growing to merge the 3D primitives. Further merging is introduced by considering the supplemental prior from 3D space, which is implemented using a 3D detector.
Complementary primitives generation. Following the graph-cut algorithm in [39], we calculate complementary primitives by employing over-segmentation on both geometric and textural priors. Previous methods [7, 8] only consider the geometry information during the primitive generation process. As shown in the middle example in Fig. 2-A, under-segmentattio occurs in regions where the door, wall, and board exhibit similar normals. Errors at this initial stage propagate and accumulate, adversely affecting the final segmentation. In contrast, we propose to incorporate additional textural prior, illustrated in the right example in Fig. 2-A. which leads to finer- finer-grained primitives. Specifically, for 3D scene P. we first treat each point Pa E P as node v, and calculate the edge weights w(Va, 16)) for each pair of nodes v2 and tb. We begin by estimating the normal n. for all Pa using corresponding 3D coordinates fa- Next, we extract the additional color information Ca. which previous methods fail exploit. Note that the combination of fa E R and ca R represents the complete point Pa E IR. We compute the cosine similarity between normals n, and na, and normalized Euclidean distance between colors c, and cb. The final edge weights w(t., vs) are obtained by a weighted sum these two dissimilarities:
3
w(Va, 0%) = Wn.
n2 n2
+ We
Σ (Cak Cba) Am
n.
Subsequently, cluster points to the finer-grained primitives w( wr(,,, v)).
(U:) Ny based each pair of
Scene graph construction. As shown in Fig. 2-B, we follow the paradigm to construct superpoint graph for the given scene. The generated primitives serve as nodes and the affinity scores obtained through a matching algorithm serve as edge weights. Specifically, we first obtain the 2D projection U.T E #x Wx x2 the i-th 3D primitive U, E & N. x3 on the m-th image by utilizing the common pinhole camera matrix:
(U 1)T Km Em (U,, T)
(2)

---

We feed the m-th RGB-image Im into the 2D foundation segmentator, e.g. SAM, to obtain its masks S..- The primitive-mask matching algorithm is then performed on 2D projections U!"" and the masks Sm. for computing affinity scores. To be specific, we calculate normalized histogram vector e to collect the 2D masks in S.. covered by rendered U?" since multiple labels may be covered due to the ambiguity or inaccuracy in 2D masks. The affinity score between i- and j-th superpoints on the m-th frame is obtained by computing the cosine similarity of their histogram vectors:
m J.
| |
Traversing all M images yields all affinity scores between U, and U.. However, primitives may not be visible in all frames, which leads to invalid affinity scores. To address this, we apply visibility- filter on the obtained scores. The visibility V.T E (0,1) 1) is defined as the ratio of the visible point number of U, on m-frame to the total point number of that in the scene. Thus, the final affinity score As is calculated in weighted-sum manner
ΣM (2 Aij
(4)
Vm 1 7E
where the weight 7i is calculated as the product of VTT and VTT Iteratively processing through all superpoints and 2D frames yields the adjacency matrix A E NoxNo Thus, the superpoint graph of the scene is constructed with primitives { as nodes and adjacency matrix A as edge weights.
Region growing and instance-aware refine- ment. We perform affinity- and distance-aware region growing on the constructed graph. Previ- ous methods inherit the part-level segmentation tendency from 2D masks output by SAM, often leading over-segmentation in 3D space. For instance, the chair of the primary segmentation shown in Fig. 2-C is segmented as two distinct parts. To address this issue, we propose to ex- ploit supplemental prior from 3D space by the integration of a 3D detector [40-42] for further merging. As shown in Fig. 2-Output, the con- straint provided by additional 3D prior rectifies the over-segmented instances while preserving the capability to handle detailed objects.
Algorithm Instance-aware refinement
I: Input:
bounding boxes (bb;) } Je in ascend- ing order of volume, primary 3D segmenta- tion masks E R.,, threshold 82
2: Output: Updated 3D segmentation /'
3: O 0' +-
4: L +- [max instance ID in [)
Nos do 5: for each bb; in {bb,)/mm
6:
O, < - bb;
7:
for each instance ID O2 in O₂ do
points O
8:
0: points in ("=O7]
9:
if a7 {L' = 0 } + L
82 then
10:
11:
end if
12:
end for
At the primary merging stage, we incorporate not only the affinity scores At, but also the Euclidean distances i between nodes U, and Uj. thus to
13:
L t- L 1
14:
end for
introduce certain level of global awareness. Dynamic thresholds 5, N: are applied to reduce the initial erroneous merges and subsequent error accumulation. Specifically, we multiply A.g with decay factor ep to get the merging confident score dæ.j between U, and Uz:
1
6.. = ED Aij
A
(5)
Dij
where ep is the reciprocal of the Euclidean distance. Then we compare the confident score 8G,1 with the threshold d1 judge whether to merge the nodes U; and Uj. Therefore, the primary segmentation results are obtained through iterating all pairs of nodes N, times.
We further introduce supplemental prior from 3D space by employing a detection-bare instance- aware refinement. As shown in Algorithm 1, we gather all points O within the bounding box bb, and assess the proportion ratio a; exceeds a specified threshold 82, it indicates high confidence that the points with instance ID O represent a portion of an over-segmented instance. We assign a new label to all points that exceed the threshold, thereby rectifying the over-segmented instance. However, there a possibility that smaller objects which are in close proximity to situated on larger objects are false-corrected. To
of points belong to instance ID O in O2 to that of the entire scene P. If the
5

---

a
Clean Stene
Official Ground-truth
Revise Ground-truth
Scares Object Count Count Distribution
No. Sceniss
No. of Objeect (Greater than)
Figure 3: Overview of our proposed ScanNerV2-INS. We present the new benchmark for 3D class-agnostic instance segmentation, which rectifies incomplete annotations and incorporates more instances based on ScanNetV2. Row (a) shows the comparison before and after revision, and row (b) illustrates the object counts per scene between the two benchmarks.
mitigate this issue, we opt to pre-sort the bounding boxes based on size. In this way, the corrections performed in descending order of bounding box size ensure that smaller objects retain their independence in the final output.
3.2 ScanNetV2-INS
ScanNetV2 has served as a standard benchmark for evaluating model performance. However, it includes a notable proportion of low-quality ground truth labels, potentially leading to misleading results. To address this issue, introduce a refined version of the dataset, termed ScanNetV2-INS, wherein annotations are enhanced at the point level.
Imperfection in vanilla ScanNetV2. The original ScanNetV2 exhibits imperfections in its ground truth annotations, primarily manifesting in two aspects. Firstly, certain obvious instances remain unmarked. For example, illustrated in Fig. 3-a top row, the board the wall and papers the desk are neglected. Secondly, some instances are incompletely annotated. For instance, doors and boards in Fig. 3-a bottom row that clearly visible in clean point clouds feature large areas of black (indicating "unlabeled") in the annotations. The prevalence of these significantly impacts the accuracy of evaluation metrics and leads to erroneous estimations of model performance. Thus, corrective measures are imperative.
Revision of ScanNetV2. With the aid of recently released annotation tool AGILE3D proposed in [43], we perform point-level updates on the ground truth annotations for all 312 scenes in the validation efficiently. The revision primarily addresses in Fig. 3-a right column. Firstly, we re-label the instances where the ground truth was obscured by unlabeled black points, such as the door and boards. Secondly, assign class-agnostic labels to certain instances that were clearly discernible to the human perception but were not originally annotated in the ground truth, such the papers on the desk and the poster the wall.
aforementioned deficiencies, as shown

---

Table 1: Instance number within varying point range of ScanNetV2 and ScanNetV2-INS dataset.
Point number of the instance
Dataset
<500
500-1000
1000-2000
2000-5000
5000-10000
>10000
V2
252
452
1119 1366
1690
567
284
ScanNetV2-INS
692
748
1873
626
291
Statistic analysis and limitation of ScanNetV2-
Table 2: Instance count of ScanNetV2 and ScanNetV2-INS dataset.
INS. Fig. 3-b demonstrates how many scenes hold more than (10, 20, .... 100) instances. In Tab. I we show the number of instances with vary- ing point counts within specified ranges for datasets. ScanNetV2-INS dataset features more smaller objects, which requires the model to have finer-grained instance perception capabilities. As
Instance Count
Dataset
Min
Max
Avg
Total
ScanNerV2 ScanNetV2-INS
47
14
4364
54
17
5596
result, the instance count of our new dataset, shown in Tab. 2, significantly increases. Therefore, better reflects and poses more challenges on the model performance. However, our dataset consists of only the revised version of 312 scenes in the validation set, focusing the evaluation use of 3D class- aansstgnosticc instance segmentation in the context of no-training methods.
Experiments
In this section, quantitatively evaluate SA3DIP ScanNet series (including the vanilla ScanNetV2 [11], our ScanNerV2-INS, and more challenging ScanNet++ [12]). Matterport3l) [44] and Replica [45] datasets demonstrate its effectiveness and robustness in 3D instance segmentation. Qualitative visualizations for ScanNet series datasets are also provided for a more intuitive comparison with other methods.
4.1 Experiment settings
Datasets. ScanNet [11] integrates a comprehensive array of 2D and 3D data sourced from indoor environments, facilitated by iPad application in tandem with depth sensors. This dataset includes RGB and depth images, along with 3D point cloud data, all meticulously annotated with semantic and instance labels. It encompasses extensive collection of over 2.5 million views derived from more than 1500 scans. In contrast, ScanNet++ [12] represents recently introduced indoor dataset exhibiting a similar composition ScanNet but boasting higher-resolution 3D geometry and more detailed data annotations. ScanNet++ data is captured using advanced equipment, including the Faro Focus Premium laser scanner, iPhone 13 Pro, and DSLR camera equipped with fisheye lens. Our proposed ScanNet-INS encompasses revised version of all 312 validation scenes in ScanNetV2. while maintaining consistency with ScanNetV2 in terms of data and label format. It provides more accurate metric and fairer comparison between methods.
Parameter settings. We conduct all experiments on a single RTX4090. The weights for geometry and texture used in the complementary primitives generation are set w,, = 0.96 and w. = 0.04. This is because texture prior such RGB values are not robust enough when being used solely due to lighting conditions, reflections, shadows, and noise collected by sensors. We conduct detailed ablation study on the choice of the two weights in the later section. The threshold 5, the region growing is empirically set (0.9, 0.8, 0.7, 0.6, 0.5] for ScanNetV2 and Scan [0.9, 0.8, 0.7] for ScanNet++, and the threshold & instance-aware refinement is set as 0.75 experimentally.
Evaluation metrics. We evaluate the quantitative results with the widely used Average Precision score. Following [2, 7, 8, 25], we report AP with thresholds of 25% and 50% (denoted as A. AP)))) and averaged over all overlaps between [50% and 95%] at 5% steps (mAP). Since the 2D foundation segmentation model produces class-agnostic masks, we ignore semantic class labels in the evaluation and consider only the accuracy of the instance masks themselves.
7

---

Table 3: Class-agnostic 3D instance segmentation comparison on ScanNetV2, ScanNetV2-INS, and ScanNet++ datasets.
ScanNetV2
ScanNetV2-INS
ScanNet++
Method
mAP
AP30 30
AP25
MAP
AP50
AP
mAP
AP50
AP2 25
Closed-vocabulary
Mask3D [2]
31.1
44.9
58.0
29.1
43,9
56.3
9.9
17.3
25.8
Open-vocabulary
Felzenszwalb [39]
5.0
12.7
38.9
2.8
6.5
24.0
4.1
9.2
25.3
SAM3D [10]
12.4
28.7
57.4
12.5
28.9
57.8
1.1
4.5
15.4
(w/o ensemble)
SAM3D [10]
20.1
33.3
52.1
20.0
33.2
52.2
7.2
14.2
29.4
(w/ ensemble)
SAM-grap [7]
24.1
40.3
65.9
23.1
39.5
64.1
12.9
25.3
43.6
SAI3D [8]
30.8
50.5
70.6
28.9
49.2
69.7
17.1
31.1
49.5
SAMPro3D [9]
33.7
56.2
75.3
32.5
54.8
73.4
18.9
33.7
51.6
SA3DIP (ours)
41.6
64.6
81.3
36.1
58.6
76.3
21.4
36.4
53.6
Baselines. We compare our approach with both closed-vocabulary and open-vocabulary methods. Mask3D [2] trained on ScanNetV2 serves the closed-vocabulary baseline. Recent methods based on leveraging 2D foundation models, including SAM3D [10] (with and without ensemble process), SAM-graph [7], SAI3D [8], and SAMPro3D [9] compared open-vocabulary methods. In addition, we compare with the traditional point grouping method proposed by Felzenszwalb [39].
4.2 Results on ScanNet series
Tab. shows the quantitative results our approach comparison with other methods on ScanNetV2,
ScanNetV2-INS, and ScanNet++ datasets. Our method achieves the best performance among all three datasets, showing effectiveness of our approach. Specifically, our SA3DIP outperform 7.9% mAP, 8.4% A. and 6.0% AP on Sc, 3.6% mAP, 3.8% AP 50. and 2.9% AP on ScanNetV2-INS On the challenging ScanNet++, our method still obtains 2.5% mAP, 2.7% 5. and 2.0% AP gain. Note that all methods except SAM3D experience drop in precision on ScanNetV2-INS dataset compared with the vanilla ScanNetV2. This indicates that our proposed ScanNetV2-INS poses
challenges in identifying fine-grained objects and yields fairer metrics. SAM3D, due to its limited segmentation capability, tends to produce a significantly higher number of instances than the actual objects in the scene. Consequently, its metrics do not show noticeable changes on finer-grained SCIINe
We also present qualitative results in Fig. 4. The visual comparison further proves the effectiveness of our method. As shown in the first two rows Fig. 4, our method maintains better instance awareness and is capable of identifying the tables 3D primitives, our approach the only one able to segment the door out from the wall, as shown in the third row of Fig. 4. This showcases the significance of exploiting the potential 3D priors.
whole. Moreover, by utilizing more accurate
4.3 Ablation studies
We conduct detailed ablation studies on prior with varying weights w,, and w.- We report the metrics in Tab. 4. We assigned several weights for geometry and texture test their contribution. Specifically, we conduct one with configuration of w,, 0.4 and w. = 0.6 which yields the similar number of 3D primitives the primitives used in SAM3D. SAI3D and others, for fair comparison.
can be observed that texture prior are not robust enough when being used solely due to the influence of shadows, reflection and thus to exploit it while minimizing its negative impact. The experiments show that the setting with wn, 0.96 and w, 0.04 suits best for approach.
Therefore, choose to assign less weight to the textural prior,
However, it is noticed that incorporating only complementary primitives yields slight drop in average precision both datasets. It is related to the definition of the metric AP (ratio of correctly

---

Scan SranNet22
StanNot++
Input
SAMED
SAMPro3D
SAL3D
Ours
Ground Truth
Figure 4: Visual comparison between our method with SAM3D [10], SAMPro3D [9], and SAI3D [8] on ScanNetV2, ScanNetV2-INS, and ScanNet++ dataset. Among all datasets, method shows the most robust and accurate segmentation.
Table 4: Ablation studies on prior we exploit with varying weights.
ScanNetV2
ScanNetV2-INS
W/
We
3D Space Prior
mAP
AP
AP25
mAP
AP 30
AP25 25
0
X
30.8
50.5
70.6
28.9
49.2
69.7
0
10.4
18.1
32.5
9.5
17.0
31.1
0.4
0.6
X
27.3
47.4
69.8
25.6
46.3
69.4
0.96
0.04
29.3
49.2
70.5
27.4
48.3
70.4
40.8
63.6
80.7
35.9
57.8
75.4
0
1
V
12.7
22.1
37.2
11.0
19.7
34.1
0.4
0.6
39.1
62.7
80.2
33.5
56.3
75.0
0.96
0.04
41.6
64.6
81.3
36.1
58.6
76.3
identified instances to the total number of identified instance). This AP metric is more in favor of under-segmentation rather than over-segmentation. since the former yields high precision and fewer false positive cases, while the latter gives fewer precision and higher recall.
4.4 More experiments
We conduct further experiments and corresponding ablation studies on both Matterport3D [44] and Replica [45] dataset to test the robustness and generalization ability of our method. Matterport3D dataset contains 194,400 RGB-D images of 90 building-scale indoor scenes and exhibits more view changes on its 2D frames compared to ScanNet. Replica dataset incorporates 18 highly photo-realistic 3D indoor scene reconstructions with dense geometry, high resolution and dynamic range textures. Our method clearly gives better quantitative results, as shown in Tab. 5.

---

Table 5: Experiments on Matterport3D and Replica datasets.
Wp
We
3D Space Prior
AP
AP25
AP
Matterport3D dataset [44]
OpenMask3D [25]
/
/
x
15.3
28.3
43.3
OVIR-3D [37]
/
X
6.6
15.6
28.3
SAM3D w/ ensemble [10]
0
10.1
19.4
36.1
SAI3D [8]
0
X
18.9
35.6
56.5
Ablation of ours
Ours #1
0
19.8
36.6
56.2
Ours #2
0,9
0.1
X
18.1
35.7
62.3
Ours #3
0.9
0.1
20.6
38.3
61.0
Replica dataset [45]
SAM3D w/o ensemble [10]
/
11.9
22.9
38.4
SAM3D w/ ensemble [10]
/
X
12.4
20.0
32.0
X
SAMPro3D [9]
0
13.1
25.2
44.7
SAI3D [8]
0
X
20.4
30.7
42.9
Ablation of ours
Ours #1
0
21.0
31.5
43.6
Ours #2
0.9
0.1
X
20.8
32.8
46.2
Ours
0.9
0.1
22.6
34.2
47.1
4.5 Limitations
Due to the trade-off between efficiency and accuracy, we choose to compute 3D superpoints based on only 3D priors. This yields extremely short time of execution within few seconds, while it may lead to an overwhelming number of superpoints which introduces challenges process. Moreover, for high-resolution point clouds with vivid light and shade effects, the superpoints generated based on geometric and texture is not enough yet. One approach is to design a more sophisticated pre-segmentation model with semantic awareness. Besides, though constraints provided by 3D prior are introduced, the affinity matrix based on 2D masking still relies heavily on the accuracy of 2D foundation segmentators. Designing more robust merging algorithm or better leveraging various 2D foundation models shows promise in the future.
the merging
Conclusion
In this paper, we introduce a novel method for segmenting any 3D instances by exploiting the potential 3D priors. The key idea is to incorporate more 3D priors into the 2D foundation model guided pipeline and leverage not only knowledge transferred from 2D space but also features in 3D space. We first generate complementary 3D superpoint primitives based both geometric and textural priors to reduce the initial errors that accumulate in subsequent procedures. Then introduce supplemental constraints from the 3D space by using 3D detector. Along with the constructed affinity matrix by using 2D masks, the region growing and refinement process is performed the 3D primitives. Furthermore, truth labels and supplement additional instances for 3D alass-ag instance segmentation, which produces unbiased metrics comparing different methods. Experimental evaluations ScanNetV2, ScanNet V2-INS, and ScanNet++ datasets demonstrate the effectiveness of our approach. We believe that we pioneer at exploiting the importance of 3D priors in the 2D foundation model guided pipeline, and it should draw attention toward future research that methods trying to extend 2D foundation models into 3D space should overlook the role of inherent 3D priors.
propose ScanNetV2-INS with complete ground
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant 62372348, Grant 62441601, Grant 62176195, Grant U21 A2152 14; part by the Key Research and Development Program of Shaanxi under Grant
in part by Shaanxi
10

---

Outstanding Youth Science Fund Project under Grant 2023-JC-1Q-53; in part by the Fundamental Research Funds for the Central Universities under Grant QTZX24080.
References
[1] Sangyun Shin, Kaichen Zhou, Madhu Vankadari, Andrew Markham, and Niki Trigoni. Spherical mask: Coarse-to-fine 3d point cloud instance segmentation with spherical representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[2] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In Proceedings of the International Conference Robotics and Automation, pages 8216 8223, 2023.
[3] Alexander Kirillov, Eric Mintun, Nikhila Ravi. Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, pages 8748-8763, 2021.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
2021.
[6] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2023.
[7] Haoyu Guo, He Zhu, Sida Peng. Yuang Wang, Yujun Shen. Ruizhen Hu, and Xiaowei Zhou. Sam- Sam-guided graph cut for 3d instance segmentation. arXiv preprint arXiv:2312.08 12.0372, 2023.
[8] Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen- Or, Jingwei Huang, and Baoquan Chen. Sai3d: Segment any instance in 3d scenes. arXiv preprint arXiv:2312. 1557, 2023.
[9] Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17700 2023.
[10] Yunhan Yang, Xiaoyang Wu, Tong Hc, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arX.::2306.03990 2023.
[11] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Confer- Computer Vision and Pattern Recognition, 2017.
[12] Chandan Yeshwanth, Yueh-Cheng Liu, Mathhia Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings the International Conference on Computer Vision, 2023.
[13] Charles R Qi. Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017.
[14] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems, volume 30, 2017.
[15] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9224-9232, 2018
[16] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3 3173-3182, 2021.
[17] Guocheng Qian, Yuchen Li, Houwen Peng. Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems, volume 35, pages 23192-23204, 2022.
11

---

[18] Xingyilang Yin, Xi Yang, Liangchen Liu, Nannan Wang, and Xinbo Gao. Point deformable network with enhanced normal embedding for point cloud analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pages 6738-6746, 2024.
[19] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr and Vladlen Koltun. Point transformer. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 16259-16268, 2021.
[20] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-bass pooling, In Advances in Neural Information Processing Systems, volume 35, pages 33330-33333 33342, 2022.
[21] Xiaoyang Wu, Li Jiang, Peng-Shuat Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035, 2023.
[22] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 3075-3084, 2019.
[23] Zhihao Liang, Zhihao Li, Songeen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2783-2799 2292, 2021.
[24] Jiahao Sun, Chunmei Qing. Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2393-2201, 2023.
[25] Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Marc Pollefeys, Federico Tombari, and Francis Engel- mann. Openmask3d: Open-vocabulary 3d instance segmentation. In Advances in Neural Information Processing Systems, volume 36, 2024.
[26] Hugues Thomas, Charles R Qi, J.an-EEmmnnn Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Lconidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411 1-6420, 2019.
[27] Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15467-15477,, 2021.
[28] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 4867-4877478 2020.
[29] Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, and Weicai Zhong. 3d instances 1d kemels. In European Conference Computer Vision, pages 235-252, 2022.
[30] Xin Lai, Yuhui Yuan, Ruihang Chu, Yukang Chen, Han Hu, and Jiaya Jia. Mask-attention-fiug free transformer for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3693-3703, 2023.
[31] Jiahao Lu, Jiacheng Deng, Chuxin Wang, Jianfeng He, and Tianzhu Zhang. Query refinement transformer for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18516-18526, 2023.
[32] Ji Hou, Angela Dai, and Matthias Nicßner. 3d-sis: 3d semantic instance segmentation of rgh-d scans. In Proced Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 4421-4433 2019.
[33] Maksim Kolodiazhnyi, Anna Vorontsova, Anton Komushin, and Danila Rukhovich. Top-down beats bottom- up in 3d instance segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3566-3574, 2024.
[34] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. In Advances in Neural Information Processing Systems, volume 36, 2024.
[35] Runyu Ding, Jihan Yang. Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabule vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7010-7019, 2023.
12

---

[36] Jihan Yang, Runyu Ding. Weipeng Deng, Zhe Wang, and Xiaojuan Qi. Regionple: Regional point-language contrastive learning for open-world 3d scene understanding, arXiv preprint arXiv:2304.00962. 2023.
[37] Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, and Kostas Bekris. Ovir-3:: Open- vocabulary 3d instance retrieval without training 3d data. In Proceedings the Conference Robot Learning, pages 1610-1620, 2023.
[38] Songyou Peng. Kyle Genova, Chiyu Jiang. Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, al. Openscene: scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 815- 924,,,2222 2023.
[39] Pedro F Felzenszwalb Daniel Huttenlocher. Efficient graph-based image segmentation. International Journal of Computer Vision, 59:167-181, 2004.
[40] Yichao Shen, Zigang Geng. Yuhui Yuan, Yutong Lin, Liu, Chunyu Wang. Han Hu, Nanning Zheng, and Baining Guo. V-detr: Detr with vertex relative position encoding for 3d object detection. In International Conference on Learning Representations, 2023.
[41] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shang- hang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1190 1190-1199, 1199, 2023.
[42] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross- moss-modal alignment for open-vocaan apen-vocabulary 3d object detection. In Advances in Neural Information Processing Systems, volume 36, 2024.
[43] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann. Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. Agile3d: Attention guided interactive multi-object 3d segmentation. In International Conference on Learning Representations, 2024.
[44] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint 17090066779 2017.
[45] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica datasct: A digital replica of indoor spaces. arXiv preprint Xiv: 05797, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The research topic of our work is 3D instance segmentation by exploit addi- tional 3D priors. In the introduction, we have clearly presented the existing problems in this field and outlined our contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitation and promising solutions of our method in Section 4.4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
There is no theory assumptions or theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We present all the information needed reproduce the results. The formulas and experimental settings are listed Section 3.1 and 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and newly proposed dateset will be available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttpssccccpbccccccceececcccccccc /ttp:/eeee//eoeemmiiiieee /Cooeeeooiissseeeeee
cesssonnooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental settings are listed in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The overall pipeline described in the paper requires training, thus the experimental results are consistent for each time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resource used deseribed in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwuriis..... oc hwwpollii//hiissuuiiioooo
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact at the end of the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research conducts experiment widely- uued datasets. Therefore, it poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the codes and data used in this paper at the reference section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The proposed method and dataset is well documented as Section 3.1, 3.2, and 4. 1.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
19


Reviews:
[{"review_id": "57RVTCh661", "reviewer": "Reviewer_oUe8", "summary": "The paper introduces SA3DIP, a novel method for 3D instance segmentation that leverages both geometric and textural priors to enhance the accuracy of segmentation tasks. The goal is to improve open-world 3D instance segmentation by addressing the limitations of current methods, which often result in under-segmentation and over-segmentation due to the limited use of 3D priors.\n\nSA3DIP integrates both geometric and textural priors to generate finer-grained 3D primitives, reducing initial errors that accumulate in subsequent processes. ItIncorporates constraints from a 3D detector during the merging process to rectify over-segmented instances, maintaining the integrity of objects in 3D space. It also introduces a revised version of the ScanNetV2 dataset, termed ScanNetV2-INS, with enhanced ground truth labels for more accurate and fair evaluations of 3D class-agnostic instance segmentation methods.\nFinally, extensive experiments on ScanNetV2, ScanNetV2-INS, and ScanNet++ datasets demonstrate the effectiveness and robustness of SA3DIP, achieving significant improvements in segmentation accuracy over existing methods.", "questions": "What are the main assumptions made in workflow and reasons as well as rationale for key design choices ?", "limitations": "It is not clear what level of demarkation of 3d segmentation is considered. Can it work for subparts within a scene such as within a chair. A level of detail seems to be one of the main deficiencies. Additionally it is not clear what the complexity of the pipeline is in terms of computation. Only one scene is shown here. What would be metrics of how much improvement across the dataset can be - an estimate would be good across a randomly chosen dataset.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Enhanced 3D Instance Segmentation Pipeline:\n   - The SA3DIP pipeline incorporates both geometric and color priors to generate complementary 3D primitives.\n   - Introduces a 3D detector to provide additional constraints during the merging process, addressing over-segmentation issues.\n\nScanNetV2-INS Dataset:\n   - A revised version of the ScanNetV2 dataset with improved annotations, providing a more accurate benchmark for evaluating 3D instance segmentation methods.\n   - Rectifies incomplete annotations and incorporates additional instances to better reflect real-world scenarios.\n\nRobust Performance:\n   - Demonstrated superior performance in 3D instance segmentation through extensive experiments on multiple datasets.\n   - Achieved competitive results, significantly outperforming existing methods in terms of mAP (mean Average Precision), AP50, and AP25 scores.\n\nThe SA3DIP method addresses the limitations of previous approaches by fully exploiting the potential of 3D priors, leading to more accurate and reliable 3D instance segmentation results. The improvement of the prior dataset is cleverly done and the overall architecture is sound. The dataset is going to be valuable to the community for future work.", "weaknesses": "1. Obfuscation of Feature Contributions:\n   - The enhancement metrics contributions by individual features are not clearly delineated. Ablation studies can be performed more meticulously to attribute the contribution of each feature individually.\n\n2. Super Primitives Definition:\n   - A more precise and comprehensive definition of super primitives could be provided to enhance understanding and reproducibility.\n\n3. Progressive Region Refinement Examples:\n   - Including examples of progressive region refinement could illustrate the process and its effectiveness more clearly.\n\n4. 2D to 3D Space Integration:\n   - While the method backprojects 3D space metrics into 2D space, the potential of lifting 2D space into the 3D object space was not fully explored. This approach could be considered and discussed to justify the design choices made.\n\nThese points highlight areas where the methodology can be further refined and expanded to provide clearer insights and potentially improve performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed feedback. \n\n**W1: Ablation studies on each feature individually**\\\nWe have conducted ablation studies on each feature we intend to exploit (e.g., geometry, texture and 3D space prior). The results are provided in the following tables. APs with subscript **1** stand for ScanNetV2 dataset, and **2** for ScanNetV2-INS. We assigned several weights for geometry and texture to test their contribution. Specifically, we conduct the config with $W_n$=0.4 and $W_c$=0.6 which yields the similar number of 3D primitives as the primitives used in SAM3D, SAI3D and others, for a fair comparison. The experiments show that the config with $W_n$=0.96 and $W_c$=0.04 suits best for our approach. More experiment on Matterport3D is shown below as well, and experiments on Replica is in **Tab.2 of the global rebuttal PDF.** \n| Geometry | $W_n$  | Texture | $W_c$  | 3D Space Prior | mAP_1      | AP25_1      | AP50_1      | mAP_2  | AP25_2      |  AP50_2      |\n|----------|------|---------|------|----------------|----------------|-------|-------|----------------|-------|-------|\n| √        | 1    | ×       | 0    | ×              | 30.8           | 50.5  | 70.6  | 28.9           | 49.2  | 69.7  |\n| ×        | 0    | √       | 1    | ×              | 10.4           | 18.1  | 32.5  | 9.5            | 17.0  | 31.1  |\n| √        | 0.4  | √       | 0.6  | ×              | 27.3           | 47.4  | 69.8  | 25.6           | 46.3  | 69.4  |\n| √        | 0.96 | √       | 0.04 | ×              | 29.3           | 49.2  | 70.5  | 27.4           | 48.3  | 70.4  |\n| √        | 1    | ×       | 0    | √              | 40.8           | 63.6  | 80.7  | 35.9           | 57.8  | 75.4  |\n| ×        | 0    | √       | 1    | √              | 12.7           | 22.1  | 37.2  | 11.0           | 19.7  | 34.1  |\n| √        | 0.4  | √       | 0.6  | √              | 39.1           | 62.7  | 80.2  | 33.5           | 56.3  | 75.0  |\n| √        | 0.96 | √       | 0.04 | √              | **41.6**       | **64.6** | **81.3** | **36.1**       | **58.6** | **76.3** |\n\n|   Matterport3D      | $W_n$  | $W_c$ | 3D Space Prior | mAP  | AP25 | AP50 |\n|---------|--------|--------|--------|--------|--------|--------|\n| OpenMask3D|     /|    /  | / |   15.3   |   28.3   | 43.3   |\n| OVIR-3D |   /    |   /   | / |   6.6    |   15.6    |   28.3   |\n| SAM3D |   1    |   0   | / |   10.1    |  19.4    |   36.1   |\n| SAI3D |   1    |   0   | / |   18.9    |   35.6    |   56.5   |\n| **Ablation of ours** |      |    |  |     |       |      |\n| Ours #1|  1   |   0   | yes |    19.8  |   36.6   | 56.2   |\n| Ours #2 |   0.9    |   0.1   | / |   18.1   |   35.7    |   **62.3**   |\n| Ours #3 |   0.9    |   0.1   | yes |   **20.6**    |   **38.3**    |   61.0   |\n\nIt can be observed that texture prior are not robust enough when being used solely due to influence on shadows, reflection and so on. Thus, we adopt the complementary primitives module, exploiting both texture and geometry. A greater weight for geometry and less for texture yield a good initial state for the subsequent merging and refinement. It can be seen in the last row that with 3D space prior, the performance exceeds that using only geometry prior.\n\n**W2: Super Primitives Definition**\\\nIn the context of our paper, 3D superpoints/primitives refer to clusters of 3D points. The points within the same group exhibit homogeneity on certain attributes. The purpose of generating primitives rather than using raw points is to introduce prior knowledge (such as geometry and texture attributes we use) and reduce computational complexity to the subsequent process.\n\n**W3: Progressive Region Refinement Examples**\\\nIn Fig. 2.C of the paper we have provided one visual example that proves the effectiveness of our refinement process. We showcase the detailed visualization at each stages in the region growing and refinement process in **Fig. 1 bottom of the global rebuttal PDF document.**\n\n**W4: 2D to 3D Space Integration**\\\nDue to inaccuracies in camera poses, the point clouds generated from posed RGB-D images are not perfectly aligned in 3D space. This misalignment can introduce noise when lifting 2D results into 3D space, as seen in methods like SAM3D. In contrast, projecting 3D mesh-sampled point clouds into 2D space experiences less influence from such inaccuracies. We showcase a example of SAM3D in **Fig. 2 of global rebuttal PDF document.**, which uses only 2D to 3D projection. It is clear that the 2D to 3D projection introduces a lot of noise, and this explains why recent methods, including ours, tend to explore 3D to 2D projection.\n\n**Q: Main assumptions & reasons for key design choices**\\\nIn general, we have made two major assumptions in our pipeline:\n1) At the Complementary primitives generation stage, it is assumed that points belong to the same semantic or instance labels share similar inherent attributes (geometry, texture and so on).\n2) At the instance-aware refinement stage, it is assumed that the points inside the 3D bounding boxes have high confidence to belong to the same instance.\n\nThe reasons for the key design choices are elaborated as follows:\n1) **Attributes and weights for primitive generation**: In the first place it is based on the observation that distinct instances with similar normal often exhibit different color. We believe it is a promising solution to the under-segmented primitives. While afterward we chose to assign greater weight to the geometric prior and less weight to the texture prior, as explained also in W1, due to the variations introduced by lighting conditions, shadows and so on.\n2) **SAM variant for 2D segmentation**: We use instance-level segmentation function of Semantic-SAM for ScanNetV2 dataset since not so many scenes contain a large amount of small objects, and we opt to SAM-HQ for ScanNet++ dataset due to its high-resolution scenes along with detailed objects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LnPtcmj4xw", "reviewer": "Reviewer_sUBY", "summary": "This study introduces SA3DIP, a novel 3D instance segmentation model based on SAM. SA3DIP leverages texture priors from point cloud color channels to generate complementary primitives and incorporates 3D spatial priors when merging 2D masks by integrating a 3D detector. These enhancements enable SA3DIP to generate superior superpoints and mitigate the over-segmentation problem found in previous SAM-based 3D instance segmentation methods.", "questions": "More comprehensive experiments are needed to validate the effectiveness of the Complementary Primitives Generation module. Additionally, the authors should further discuss the motivations for using color value similarities as texture priors.", "limitations": "The authors adequately addressed the limitations in section 4.4.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Using SAM to extract 2D masks from RGB-D frames and merge them into a final 3D segmentation result is common in 3D OV segmentation methods. However, few previous methods are geometry-aware during the merging process, highlighting the significance of SA3DIP's incorporation of 3D priors.\n\n(2) Constraints from 3D spatial priors substantially improve performance on both the ScanNetV2 and ScanNetV2-INS datasets.\n\n(3) The low quality of ScanNetV2 ground-truth segmentation results has been a persistent problem. A 3D segmentation dataset with more accurate ground truth, like ScanNetV2-INS, is demanding.", "weaknesses": "(1) Using RGB values only as texture prior are not robust enough due to their susceptibility to variations caused by lighting conditions, shadows, reflections, and object materials.\n\n(2) The ablation study shows that the performance gain from Complementary 3D superpoint primitives is not significant compared to other modules in SA3DIP.\n\n(3) This paper reports only the class-agnostic instance segmentation results of SA3DIP. However, the previous benchmark method (SAI3D [1]) also uses semantic instance segmentation as a typical evaluation metric.\n\n[1] Yin, Y., Liu, Y., Xiao, Y., Cohen-Or, D., Huang, J. and Chen, B., 2024. Sai3d: Segment any instance in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3292-3302).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed feedback.\n\n**W1/Q: RGB values as texture prior are not robust enough / Motivations for using color values / More experiments**\\\nIn the first place, our motivation is that we found **distinct instances with similar normal often exhibit different color.** Meanwhile, the commonly used 3D indoor datasets have only xyz and rgb attributes for their 3D points. Thus, we opt to explore RGB values as texture prior.\\\nIt is true, as concerned also by other reviewers, that texture prior such as RGB values are not robust enough when being used solely. We have conducted further experiments about the priors and their weights, shown in **Tab. 1 of the global rebuttal PDF document.** In our approach we assign less weight to the textural prior, thus to exploit it while minimizing its negative impact. For ScanNetV2, we set the geometric weights ($W_n$) as 0.96 and texture weights ($W_c$) as 0.04 (Line 214). Under this setting complementary primitives generation module yields a good initial state for the subsequent merging and refinement. We have conducted more experiments on Matterport3D (table below) and Replica dataset **(Tab. 2 of the global rebuttal PDF document).**\n|   Matterport3D      | $W_n$  | $W_c$ | 3D Space Prior | mAP  | AP25 | AP50 |\n|---------|--------|--------|--------|--------|--------|--------|\n| OpenMask3D|     /|    /  | / |   15.3   |   28.3   | 43.3   |\n| OVIR-3D |   /    |   /   | / |   6.6    |   15.6    |   28.3   |\n| SAM3D |   1    |   0   | / |   10.1    |  19.4    |   36.1   |\n| SAI3D |   1    |   0   | / |   18.9    |   35.6    |   56.5   |\n| **Ablation of ours** |      |    |  |     |       |      |\n| Ours #1|  1   |   0   | yes |    19.8  |   36.6   | 56.2   |\n| Ours #2 |   0.9    |   0.1   | / |   18.1   |   35.7    |   **62.3**   |\n| Ours #3 |   0.9    |   0.1   | yes |   **20.6**    |   **38.3**    |   61.0   |\n\nIt can be seen in the last row of **three tables** that, our performances on all three datasets exceed that using only geometry prior. We will include this discussion in the final version of our paper.\n\n**W2: Non-significant gain of complementary 3D superpoint primitives module**\\\nIn our approach, we designed the complementary 3D superpoint primitives module for alleviating the existing problem about under-segmented 3D primitives based only on geometry and subsequent error accumulation. \\\nFor the non-significant gain of this module, it is related to the definition of the metric AP (ratio of correctly identified instances to the total number of identified instance). **This AP metric is more in favor of under-segmentation rather than over-segmentation, since the former introduces relatively fewer false instance.** To further illustrate this, we randomly choose two scenes (scene0011_00 & scene0644_00) and conduct random 10%, 20%, 30% over/under segmentation tests based on their GT instances. Results are averaged from three experiments and shown below. APs with subscript **O** stand for over-segmentation results, and **U** for under-segmentation.\n|         | mAP_O  | AP25_O | AP50_O | mAP_U  | AP25_U | AP50_U |\n|---------|--------|--------|--------|--------|--------|--------|\n| **Scene0011_00**|     |      |  |      |      |    |\n| 10% |   87.2    |   87.2   | 98.1 |   92.4    |   96.2    |   96.2   |\n| 20% |   71.8    |   71.8   | 98.1 |   84.5    |   92.3    |   92.3   |\n| 30% |   62.3    |   62.3   | 98.1 |   85.4    |   92.3    |   92.3   |\n| **Scene0644_00**|     |      |  |      |      |    |\n| 10% |   88.1    |   88.1   | 96.7 |   97.6    |   98.8    |   98.8   |\n| 20% |   73.6    |   73.6   | 98.3 |   86.9    |   92.9    |   92.9   |\n| 30% |   60.3    |   60.3   | 98.3 |   69.8    |   82.1    |   82.1   |\n\nIt can be observed that under whatever percentage the under-segmentation seems to give higher APs, due to **its high precision and fewer false positive cases.** While for over-segmentation, it gives **fewer precision and higher recall.** This is consistent with our results, which give finer-grained (20% more in average, shown in the table below) primitives when exploiting both geometry and texture but slightly lower APs. We show the number of instances in comparison with SAI3D in the table.\n|   Count      | Min | Max | Avg || Min | Max | Avg |\n|---------|--------|--------|--------|--------|--------|--------|--------|\n| **Primitive**|       |      |  |   **Final result**    |      |  |\n| SAI3D|   159    |   3905    | 1068 | SAI3D|   6    |   258    | 59 |\n| ours  | 243 | 3989| 1272 |ours  | 5 | 159| 45 |\n\nHowever, our final results when experiencing whole pipeline (after adding 3D space prior) **reverse the minor drop and produce an extra gain at around 1% compared to adding only 3D detection.** The counts for instances decrease as well. The superior performance and decreased instance number indicate that our results achieve both high recall and high precision. It proves that he combining of finer-grained (and slightly over-segmented) primitives and instance-aware refinement during merging process (merge those over-segmented primitives) provides a thorough solution. \n\n\n**W3: Semantic instance segmentation**\\\nWe have conducted the semantic instance segmentation on ScanNet200 dataset, better aligning our capability with previous benchmark method (such as SAI3D). The results are provided in the following table. Our method clearly results in better performance than other methods on AP, head and tail.\n\n|   Method     | AP | AP50 | AP25 | Head(AP) | Common(AP) | Tail(AP) |\n|---------|--------|--------|--------|--------|--------|--------|\n|**Closed mask**|||||||\n| OpenMask3D|   15.4  |   19.9    | 23.1 |17.1|14.1|14.9|\n|**Open-vocab. mask**|||||||\n| OVIR-3D|   9.3   |   18.7    | **25.0** |9.8|9.4|8.5|\n| SAM3D|   9.8   |   15.2    | 20.7 |9.2|8.3|12.3|\n| SAI3D|   12.7   |   18.8    | 24.1 |12.1|10.4|16.2|\n| Ours|  **13.5**   | **20.4** |*24.8*|**14.9**|**11.6**|**16.9**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "54Lk1eV5tn", "reviewer": "Reviewer_3tui", "summary": "The paper proposes a pipeline to perform open-vocabulary 3D instance segmentation of scenes, incorporating geometric and RGB information. The method is based on constructing a super-points graph, which is then refined by SAM and a 3D detector (V-DETR). Also, the paper provides an enhanced version of ScanNetV2, correcting and extending the annotations. The method outperforms other SAM based baselines.", "questions": "Adding on the observations reported in the previous section:\n1) The method requires posed images. Is this a requirement also for the competitors?\n2) Images are often difficult to parse: e.g., Figure 2 contains many small images with several colors, and its flow is not linear. I would suggest providing a more schematic overview with larger figures and including in the appendix this detailed version.", "limitations": "The method briefly discusses the proposed method's main limitation but not the dataset.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Given the popularity of ScanNetV2, releasing a more curated and fine-grained annotation seems a useful contribution. The proposed method is a careful combination of components, in a cascade of steps that seems effective.", "weaknesses": "- As a non-expert in 3D scene segmentations, I find it difficult to understand the novelty of the proposed approach, whereas other works also rely on SAM. From my understanding, the key difference seems to be the exploitation of 3D prior to incorporating the 3D classifier, producing the main impact in instance awareness by constraint. If this is the case, I am unsure about the significance of the technical contribution, and I would ask for further clarification on this.\n- The paper does not provide enough documentation or analysis of the new annotations for ScanNetV2, which is critical since it is one of the core contributions of the paper. Figure 3 is insufficient to understand the quantitative statistics of the performed effort. On the new dataset, the methods perform worst, which could suggest that the new labels are more difficult/detailed, but more evidence is required to confirm this. To prove the dataset's usefulness, I would suggest comparing methods trained on ScanNetV2 and ScanNetV2-INS and incorporating them in the paper more statistics (e.g., the difference in the number of categories, how many instances for each of these, ...).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed feedback.\n\n**W1: Novelty of the proposed approach**\\\nIt is true that other methods, including ours, rely on SAM, and **this heavy reliance of 2D segmentation results is exactly what we were trying to avoid by introducing the 3D prior.** In Fig. 1 of our paper, we demonstrate the existing problem of the previous methods (like SAI3D and SAMPro3D), which inherit the over-segmented 2D masks to the final 3D instance segmentation. \\\nThus, our method is problem-oriented, trying to **alleviate two major defects** existing in the pipelines of the current works through 3D priors.\n1) The under-segmented 3D primitives and subsequent error accumulation\n2) The part-level over-segmented tendency of the 2D foundation segmentator.\n\nWe therefore design two modules accordingly. First, we use the complementary primitives generation module to generate more accurate and finer-grained 3D primitives to avoid any error accumulation. Second, we introduce the 3D space prior for providing instance-aware constraint, which we choose to implement using a 3D detection. The combination of both modules works great at solving the aforementioned problems.\n\n**W2/Limitation: Documentation or analysis of the new annotations for ScanNetV2**\\\nWe need to claim that all methods of this type (such as SAMPro3D, SAI3D, ours and so on) are zero-shot ones, which means they utilize the generalization capability of 2D foundation model and do not depend on training. Thus, the ScanNetV2-INS dataset we propose is simply of a evaluation use, since their output instance segmentation results do not change when switching the ground truth file. We will elaborate the details about ScanNetV2-INS dataset as follows:\n1) **Meaning**: There are two major deficiencies in the original ScanNetV2 dataset: missing instances and incomplete instance masks. To solve these, we assigned new class-agnostic labels to the missing ones and re-labelled the incomplete instance masks.\n\n2) **Tasks**: Following the recent methods of utilizing 2D foundation model to perform zero-shot 3D class-agnostic instance segmentation, our ScanNetV2-INS dataset also focus on this specified task and for evaluation only.\n3) **Format**: We generate a txt file (e.g., sceneid.txt) for each scene in the original ScanNetV2 validation set. The txt file contains N separate lines corresponding to N points in the scene, where the number of each line is calculated as: Number = instance id (refers to n-th object in the scene) + 1000 * semantic label id (refers to scannetv2-labels.combined.tsv). The newly labelled instances were assigned a semantic label id of 41 since we focus on class-agnostic segmentation.\n4) **Limitation**: Our dataset focus on the evaluation use of 3D class-agnostic instance segmentation, due to the recent trend of utilizing 2D foundation segmentator to perform zero-shot 3D class-agnostic instance segmentation and the high expense of annotation on 3D data.\n5) **More statistics**: In Fig. 3 of our paper, we demonstrate how many scenes with more than (10, 20, ..., 100) instances in ScanNetV2 and ScanNetV2-INS. Our proposed dataset clearly provides larger number of scenes with more instances. We will give more statistics below. The first table below illustrates the instance count of the original ScanNetV2 and ScanNetV2-INS. It can be seen that our ScanNetV2-INS dataset incorporates more instances. In the second table, we show the number of instances with varying point counts within specified ranges for two datasets, ScanNetV2 and ScanNetV2-INS. ScanNetV2-INS dataset features more smaller objects, which requires the model to have finer-grained instance perception capabilities. The statics from these two tables could, to some extent, explain why the ScanNetV2-INS dataset is more challenging than ScanNetV2.\n\n    |   Instance Count      | Min | Max | Avg | Total |\n    |---------|--------|--------|--------| ------ |\n    | ScanNetV2|    2   |   47   |  14| 4364 |\n    | ScanNetV2-INS|   2    |   54    | 17 | 5596 |\n\n\n    |   Point # Per Instance      | <500 | 500-1000 | 1000-2000 | 2000-5000 | 5000-10000 | >10000 |\n    |---------|--------|--------|--------|--------|--------|--------|\n    | ScanNetV2|    252   |   452   |  1119|1690|567|284|\n    | ScanNetV2-INS|   692    |   748    | 1366 |1873|626|291|\n\n**Q1: Requirement of the posed images**\\\nYes, posed images are also required by other competitors. As we clarified in W2/Limitation above, this type of methods (such as SAMPro3D, SAI3D, ours and so on) are zero-shot ones, which means they **utilize the generalization capability of 2D foundation model**, since to date there is no 3D foundation model due to the limited 3D labelled data. The correspondence between 2D and 3D space is the focus of these methods. Thus, posed images are essential for acting as a bridge between 2D foundation models and 3D space. We will include this discussion in the final version of our paper.\n\n**Q2: Further figure clarification**\\\nIn general, the whole pipeline shown in Fig. 2 of our paper can be separated in three parts. Step A: 3D primitives generation exploiting both geometric and textural priors. Step B: scene graph construction where the primitives serve as nodes and affinity matrix of 3D primitives guided by 2D masks generated using 2D segmentators as edge weights. Step C: Region growing and instance-aware refinement on the constructed scene graph. **We provide a more straightforward version with larger images in Fig. 1 of global rebuttal PDF document.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jssBChXbKA", "reviewer": "Reviewer_cYA9", "summary": "The paper \"SA3DIP: Segment Any 3D Instance with Potential 3D Priors\" presents a novel method for 3D instance segmentation by incorporating geometric and textural priors to generate 3D primitives. It addresses the limitations of existing methods that rely heavily on 2D foundation models, leading to under-segmentation and over-segmentation issues. The proposed approach includes a 3D detector for refining the segmentation process and introduces a revised version of the ScanNetV2 dataset, ScanNetV2-INS, with enhanced ground truth annotations. Experimental results on various datasets demonstrate the effectiveness and robustness of the SA3DIP method.", "questions": "1. In Section 3.1, the author mentions 'histogram vectors,' but the definition of these vectors is unclear in the article. It is not specified how these vectors are derived or what their dimensions are. Additionally, it is unclear why they can effectively represent the features of each superpoint for calculating the affinity score. Clarification on these points is necessary to understand their role and significance in the methodology.\n2. In Table 1, SAMPro3D is the method that is numerically closest to our work. Why is there no visual comparison in Figure 4?", "limitations": "The authors have addressed the limitations of their work. They highlight that using only 3D priors results in short execution times but can lead to an excessive number of superpoints, complicating the merging process.  Furthermore, they acknowledge that the affinity matrix based on 2D masking relies heavily on the accuracy of 2D foundational segmenters and suggest that a more robust merging algorithm or better utilization of various 2D foundational models could be promising future directions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strength of the paper lies in its innovative approach to integrating both geometric and textural priors for 3D instance segmentation, which significantly reduces initial errors and enhances the overall segmentation quality. The introduction of a 3D detector for refining segmentation further strengthens the method by addressing over-segmentation issues. Additionally, the revised ScanNetV2-INS dataset provides a more accurate benchmark for evaluating 3D segmentation models, contributing valuable data to the research community. The experimental results across multiple challenging datasets convincingly demonstrate the robustness and effectiveness of the proposed method.", "weaknesses": "Despite its strengths, the paper has certain weaknesses. Firstly, it lacks sufficient innovation, as the framework closely resembles SAI3D [1], with the primary difference being the addition of a 3D detector. The entire Scene Graph Construction part is exactly the same as SAI3D. Furthermore, as shown in the ablation study, the overall performance improvement of the model heavily depends on the pre-trained 3D detector, which diminishes the originality and contribution of this paper.\n \n[1] Yin, Yingda, et al. \"Sai3d: Segment any instance in 3d scenes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed feedback.\n\n**W: Innovation of the proposed approach**\\\nIn Fig. 1 of our paper, we demonstrate the existing problem of previous methods, which use under-segmented 3D primitives and inherit the over-segmented 2D masks to the final 3D instance segmentation. We believe our method is problem-oriented, trying to **alleviate these two major defects.**\\\nWe therefore design two modules accordingly: \n1) Complementary primitives generation module to generate more accurate and finer-grained 3D primitives to avoid any error accumulation. \n2) Introducing the 3D space prior for providing instance-aware constraint, which was implemented by a 3D detector. \n\nThe combination of both modules works great at solving the aforementioned problems.\n\n\n**W: Overall performance gain analysis**\\\nIt is true that from ablation study the performance gain seems to heavily depend on 3D detector. However, we believe that the performance gain of the complementary primitives module **is not as minor as it looks**. We think it is related to the definition of the metric AP (ratio of correctly identified instances to the total number of identified instance), **which is more in favor of under-segmentation rather than over-segmentation, since the former introduces relatively fewer false instance.** We randomly choose two scenes (scene0011_00 & scene0644_00) and conduct random 10%, 20%, 30% over/under segmentation tests based on their GT instances. Results are averaged from three experiments and shown in the table. APs with subscript **O** stand for over-segmentation results, and **U** for under-segmentation.\n|         | mAP_O  | AP25_O | AP50_O | mAP_U  | AP25_U | AP50_U |\n|---------|--------|--------|--------|--------|--------|--------|\n| **Scene0011_00**|     |      |  |      |      |    |\n| 10% |   87.2    |   87.2   | 98.1 |   92.4    |   96.2    |   96.2   |\n| 20% |   71.8    |   71.8   | 98.1 |   84.5    |   92.3    |   92.3   |\n| 30% |   62.3    |   62.3   | 98.1 |   85.4    |   92.3    |   92.3   |\n| **Scene0644_00**|     |      |  |      |      |    |\n| 10% |   88.1    |   88.1   | 96.7 |   97.6    |   98.8    |   98.8   |\n| 20% |   73.6    |   73.6   | 98.3 |   86.9    |   92.9    |   92.9   |\n| 30% |   60.3    |   60.3   | 98.3 |   69.8    |   82.1    |   82.1   |\n\nIt can be observed that under whatever percentage the under-segmentation seems to give higher APs, due to **its high precision and fewer false positive cases.** While for over-segmentation, it gives **fewer precision and higher recall.** This is consistent with our results, which give finer-grained (20% more in average, shown in the table below) primitives but slightly lower APs. We show the number of instances in comparison with SAI3D in the table.\n|   Count      | Min | Max | Avg || Min | Max | Avg |\n|---------|--------|--------|--------|--------|--------|--------|--------|\n| **Primitive**|       |      |  |   **Final result**    |      |  |\n| SAI3D|   159    |   3905    | 1068 | SAI3D|   6    |   258    | 59 |\n| ours  | 243 | 3989| 1272 |ours  | 5 | 159| 45 |\n\nUpon closer look in the ablation study in our paper, adding only our complementary 3D superpoint primitives module even slight deters the performance, due to the AP metric. However, our final results after experiencing whole pipeline (after adding 3D space prior) **reverse the minor drop and produce an extra gain at around 1% compared to adding only 3D detection.** The counts for instances decrease as well. The superior performance and decreased instance number indicate that our results achieve both high recall and high precision. It proves that he combining of finer-grained (and slightly over-segmented) primitives and instance-aware refinement (merge those over-segmented primitives) provides a thorough solution. \n\n**Q1: Histogram vectors clarification**\\\nHistogram vector represents **a distribution of 2D mask ids** which are covered by the projection of a 3D primitive. It reflects which 2D mask has the correspondence with the current 3D primitive. The histogram vectors are then used to calculate the affinity (similarity) matrix for each pairs of 3D primitives, **indicating the likelihood that two primitives belong to the same object (the same 2D mask).**\\\nIn the implement of codes, assume that for a scene there are M 2D images and N 3D points. For the m-th image, we project the N points to the image using the corresponding pose and camera intrinsic. We record the 2D mask label for every 3D point according to which pixel it is projected onto (0 for invisible points in the image). After this we get a (N, M) matrix, indicating the 2D mask label for every point in every 2D view. \\\nNext, we load all K 3D primitives in the scene. Again for the m-th image, we record the 2D mask labels each primitive covers according to the (N, M) matrix we get earlier, since multiple labels may be covered by one single primitive due to the ambiguity or inaccuracy in 2D masks. We maintain a normalized matrix of size (K, V) for each view, where V refers to the 2D mask counts in the view. **This matrix of size (K, V) is actually the histogram vector for all primitives in the m-th view.** \n|         | 2D mask #1 | 2D mask #2 | ...... | 2D mask #V |\n|---------|--------|--------|--------|--------|\n| **Primitive #1**|   0    |   0.5    | ...... |   0    |\n| **Primitive #2**|   0.3    |   0.6    | ...... |   0    |\n| **......**  | ...... | ...... | ...... | ...... |\n| **Primitive #K**|   0.1    |   0    | ...... |   0.6    |\n\nIn the end, we use the (K, V) matrix for cosine similarity calculation to obtain the affinity matrix of size (K, K) which represents the likelihood of each pair of primitives.\n\n\n\n**Q2: Visual comparison of SAMPro3D**\\\nWe showcase the comparison in **Fig. 3 of global rebuttal PDF.** Our method clearly alleviates the over-segmented instances appear in SAMPro3D, which is consistent with our initial purpose of introducing the 3D instance-aware space prior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable feedback, we appreciate their detailed suggestions. We reply to each reviewer’s questions and concerns in the individual responses, and we have added tables and figures in the attached rebuttal PDF, which we reference and explain in the responses.

We would like to emphasize that our work is **problem-oriented, motivated by the commonly existed defects in SAM3D, SAMPro3D, SAI3D and so on.** We therefore design two modules accordingly: 
1) Complementary primitives generation module to generate more accurate and finer-grained 3D primitives to avoid any error accumulation. 
2) Introducing the 3D space prior for providing instance-aware constraint, which was implemented by a 3D detector.
The visualization of Fig. 4 in our paper demonstrates that the two purposes have been achieved and indeed the problems could be alleviated through our approach.

Here, we also would like to provide an overview of the material in the attached document:

**Figure 1** – Re-drew schematic overview of the proposed pipeline, **in response to Reviewer 3tui** who found it difficult to follow the original Fig. 2 in our paper. The top blocks give a simplified flow of our method, and under which is the detailed version with images. We also show a full progressive merging and refinement process in the yellow dashed box at the bottom, **in response to Reviewer oUe8** who required progressive region refinement examples.

**Figure 2** – Visual examples from SAM3D, **in response to the discussion with Reviewer oUe8** about the potential of 2D to 3D space integration.

**Figure 3** – Visual comparison with SAMPro3D, **in response to Reviewer cYA9**. This visualization, along with Fig. 4 in our paper, fully demonstrates the effectiveness of our approach at alleviating the error accumulation caused by under-segmented 3D primitives and over-segmented 3D instance due to the transferred knowledge from the part-level masks of 2D foundation segmentation model.

**Table 1** - Detailed ablation study on prior we intend to explore. We hope this could ease the concern by most reviewers about the effectiveness of texture prior or the complementary primitive generation module.

**Table 2** – More experiment and ablation on **Replica** dataset. This further proves the robustness and generalization capability of our approach.

Meta-review: This paper received all positive scores (Accept, Weak Accept, Weak Accept, and Borderline Aceept) from 4 reviewers. These reviewers are mainly satisfied with the responses from the authors during the rebuttal phrase. The authors are encouraged to include these important clarifications into the final paper when preparing the camera ready version.
Decision: Accept (spotlight)