Learning Superpoint Graph Cut for 3D Instance Segmentation
Authors: Le Hui
Linghua Tang
Yaqi Shen
Jin Xie
Jian Yang
Venue: NeurIPS 2022 Accept (2022)

Abstract: 3D instance segmentation is a challenging task due to the complex local geometric structures of objects in point clouds. In this paper, we propose a learning-based superpoint graph cut method that explicitly learns the local geometric structures of the point cloud for 3D instance segmentation. Specifically, we first oversegment the raw point clouds into superpoints and construct the superpoint graph. Then, we propose an edge score prediction network to predict the edge scores of the superpoint graph, where the similarity vectors of two adjacent nodes learned through cross-graph attention in the coordinate and feature spaces are used for regressing edge scores. By forcing two adjacent nodes of the same instance to be close to the instance center in the coordinate and feature spaces, we formulate a geometry-aware edge loss to train the edge score prediction network. Finally, we develop a superpoint graph cut network that employs the learned edge scores and the predicted semantic classes of nodes to generate instances, where bilateral graph attention is proposed to extract discriminative features on both the coordinate and feature spaces for predicting semantic labels and scores of instances. Extensive experiments on two challenging datasets, ScanNet v2 and S3DIS, show that our method achieves new state-of-the-art performance on 3D instance segmentation.

Learning Superpoint Graph Cut for 3D Instance Segmentation
Le Huit, Linghua Tang', Yaqi Shen, Jin Xie", Jian Yang" PCA Lab, Nanjing University of Science and Technology, China tanglinghua, syq, esjxie, csjyang) @njust. edu.
Abstract
3D instance segmentation a challenging task due to the complex local geometric structures of objects in point clouds. In this paper. we propose a learning- -based superpoint graph cut method that explicitly learns the local geometric structures of the point cloud for 3D instance segmentation. Specifically, we first oversegment the raw point clouds into superpoints and construct the superpoint graph. Then, we propose an edge score prediction network to predict the edge scores of the superpoint graph, where the similarity vectors of two adjacent nodes learned through cross-graph attention in the coordinate and feature spaces are used for regressing edge scores. By foreing two adjacent nodes of the same instance be close to the instance center in the coordinate and feature spaces, we formulate a geometry-ave edge loss to train the edge score prediction network. Finally, we develop superpoint graph cut network that employs the learned edge scores and the predicted semantic classes of nodes to generate instances, where bilateral graph attention is proposed to extract discriminative features on both the coordinate and feature spaces for predicting semantic labels and scores of instances. Extensive experiments two challenging datasets, ScanNet v2 and S3DIS, show that our method achieves new state-of th-the-ar art performance on 3D instance segmentation. Code is available at athhtps::// http://gttttbbtt/ee http//u/..oooo/pphhnagaaeeeeeeee GraphCut
Introduction
In recent years, with the development of 3D sensors, such LiDAR and Kinect camera, various 3D computer vision tasks have been receiving more and more attention. 3D instance segmentation is fundamental task 3D scene understanding and has been widely used kinds of applications such as self-driving cars, virtual reality, and robotic navigation. Although recent progress in 3D instance segmentation encouraging, it is still a challenging task due irregularities and context uncertainties of 3D points in 3D scenes with complex geometric structures.
Many efforts have been dedicated 3D instance segmentation and achieved promising performance. These methods can be mainly divided into two categories: detection-based methods [44] 45) and
clustering-basee methods [40 [8]. Among detection-based methods, 3D-BoNet [44 first detects the 3D bounding boxes and then employs a mask prediction network to predict the object mask for 3D instance segmentation. However. for objects with complex geometric structures, detection- based methods [45] cannot obtain accurate 3D bounding boxes, thereby degrading the instance segmentation performance. The clustering based method SGPN 44) clusters 3D points based
Equal Contribations, "Corresponding authors.
Le Hui, Linghua Tang, Yaqi Shen, Jin Xie, and Jian Yang are with PCA Lab, Key Lab of Intelligent Pereeption and Systems for High-Ding High-Dimenssinal Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Sccurity, School of Computer Science and Engineering, Nanjing University of Science and Technology, China.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

on semantic segmentation to generate instances. Unlike SGPN, Jiang et al. [8 developed an offset branch to cluster points based on semantic predictions in dual coordinate spaces, including original and shifted coordinate spaces. Besides, some follow-up methods utilize tree structures [25] hierarchical aggregation 3 and soft semantic segmentation [37 to boost the performance of 31> instance segmentation. However, most of these clustering-base baanedd---dddd methods rely on center offsets and semantics to segment instances, which cannot effectively capture the geometric context information of point clouds. Therefore, the performance of instance segmentation is usually limited by objects with complex geometric structures in point clouds.
In this paper, we propose local geometric structures of point clouds to segment 3D instances. Specifically, we construct the superpoint graph to learn the geometric context similarities of superpoints and convert the instance segmentation into a binary classification of edges. Our method consists of an edge score prediction network to predict edge scores and superpoint graph cut network to generate instances. In our method, we oversegment the raw point clouds into superpoints and construct the superpoint graph by linking the k-nearest superpoints in the coordinate space. In the edge score prediction network, we first perform cross-graph attention on the local neighborhoods of two adjacent nodes to extract local geometric features for measuring the similarity of the nodes. Then, based on the learned similarity vectors from the coordinate and feature spaces, we adopt an edge score branch to predict the edge scores. In addition, we propose geometry by forcing the adjacent nodes of the same instance to be close to the instance center in both the coordinate and feature spaces. In the superpoint graph cut network, combined with semantic classes of the nodes to cut the edges for forming object proposals. The proposals are obtained by applying the breadth-first-st sirst- algorithm on the superpoint graph to aggregate nodes in the same connected component. In each proposal, we apply bilateral graph attention aggregate local geometric features to extract discriminative features for predicting classes and scores of proposals. Furthermore, we adopt mask learning branch to filter the low-confidence superpoints within the proposal to generate instance.
tatriing-bssed superpoint graph cut method that explicitly learns the
edge loss to train the edge score prediction network
use the learned edge scores
In summary. we present an edge score prediction network that learns the local geometric features of adjacent nodes for producing edge scores. To train it, we propose a geometry-aware edge loss to keep the instance compact in the coordinate and feature space simultancously. We present a superpoint graph cut network that extracts discriminative instance features to generate accurate instances by using bilateral graph attention in the coordinate and feature spaces. Extensive experiments on the ScanNet v2 (7) and S3DIS [1 datasets show that method achieves new state-of-the-art performance on 3D instance segmentation. On the online test set of ScanNet v2, our method achieves the performance of 55.2% in terms of mAP, which is 4.6% higher than the current best results (25). For S3DIS, our method outperforms the current best results 37 over 2% in terms of mAP.
2 Related Work
3D semantic segmentation. Extracting features from irregular 3D point clouds is crucial for 3D semantic segmentation. Qi et al. [30] first proposed PointNet to learn point-wise features from a point set for semantic segmentation through the multi-layer perceptron network. Following it, many 4 2 have been proposed to improve semantic segmentation performance.
efforts [24 34 46
15
47
14] design various local feature aggregation strategies to extract discriminative wise features for semantic segmentation. Inspired by successful 2D convolution networks, view-based methods [23] 35|
Early point-based methods 36
43
19 project the point cloud into multiple regular 2D views, where the regular 2D convolution is applied to extract features. In addition to view- first voxelize the point cloud into regular 3D grids and then apply 3D convolution to extract local features of point clouds. In order to capture the local geometric structures of point clouds, graph-based methods graph point clouds and utilize graph convolution to aggregate local geometric information for semantic segmentation.
17
based methods, volumetric-basee methods [27 39 10
6
42
38 5 16 construct the
3D instance segmentation. 3D instance segmentation is more challenging task, which further needs to identify each instance. Current methods can be roughly grouped into two classes: detection-baser methods and clustering- based methods.
2

---

Edge Score Prediction Network
Superpoint Graph Cut Network
pount cloud
graph embodd
R-NN groph
Graph Alteatioo
instanee resullss
coum edges
C
Cren-Graphi Anvation
similaily
wdys sooes
superpoint graph
shilled
J-NN graph
Proposal Algershes
graph cut reauts
fodert
condarespee
ccordamac
Ninoral spait
Cross-Graph Atfeution
Bilateral Graph Attention
The framework of our learning-based superpoint graph cut method. Given the superpoint graph, we first extract superpoint features (omitted in the figure). Then, we construct an edge score prediction network to extract edge embeddings from the coordinate and feature spaces for predicting edge scores. Finally, based on the learned edge scores, we develop superpoint graph cut network to generate accurate instances.
Figure I:
Detection- methods 13] 26 first detect 3D bounding boxes of each object in point clouds, and then apply mask prediction network each box predict the object mask for 3D instance segmentation. In [44], a 3D instance segmentation framework dubbed 3D-BoNet is proposed, which directly regresses the 3D bounding boxes for all instances and predicts point-level masks for each instance. Yi ef al. [45] proposed generative shape proposal network that generates proposals by reconstructing shapes from noisy observations in a scene for 3D instance segmentation. In addition, using both geometry and RGB inputs, [13] develops joint 2D-3D feature learning network that combines the 2D and 3D features regress 3D object bounding boxes and predict instance masks.
Clustering-based methods usually use point similarity 40), semantic maps 11 20], or geometric shifts [18 3
cluster 3D points into object instances. A similarity group proposal network was proposed in 40 to cluster points by learning point-wise similarity for generating instances. 229 proposes multi-task learning framework that simultaneously learns semantic classes and high-diimensional embeddings of 3D points to cluster the points into object instances. In [41], segmentation framework is introduced to learn semantic-aware point-wise instance embedding for associatively segmenting instances and semantics of point clouds. Han et al. []]] proposed an occupancy-awee aware method predict the number of occupied voxels for each instance. PointGroup [88] clusters points by using predicted point-wise center offset vectors and point-wise semantic labels. The follow-up method [3] adopts a hierarchical aggregation strategy for 3D instance segmentation, which first performs point aggregation to cluster points into preliminarily sets and then performs set aggregation cluster into instances. Lately, Vu al. [37] proposed soft grouping strategy to mitigate the problem of semantic prediction eтrors by associating each point with multiple classes, yielding significant performance gains in 3D instance segmentation. In addition, a semantic superpoint tree network, called SSTNet, is proposed in 255 for segmenting point clouds in instances. first groups superpoints with similar semantic features to build binary tree and then generates instances tree traversal and splitting. To make the network more efficient, a dynamic convolution network combined with small Transformer network is constructed to propose a lightweight 3D instance segmentation method [12]
25
33
Method
An overview of our learning- superpoint graph cut method is illustrated in Figure Based on the superpoint graph, the edge score prediction network (Sec. (31) extracts edge embeddings from the coordinate and feature spaces for predicting edge scores. After that, the superpoint graph cut network (Sec. 3.2 generates accurate object instances by learning discriminative instance features to
3

---

predict classes and scores of instances. Finally, in Sec. 3.3 we describe how to train our method and inference instances from point clouds.
3.1 Edge Score Prediction Network
Given a raw point cloud, we oversegment it into superpoints and construct the superpoint graph g= (V. F),, where V represents the node set of superpoints and E. represents the edge set. Since the superpoint representation is coarser than the point representation, learning features directly from the superpoint representation cannot effectively capture the local geometric structures of point clouds. Therefore, we apply submanifold sparse convolution [10] on the point cloud to extract point-level features and the point-level features to initialize superpoint-level features by average pooling. After that, we apply edge-conditioned convolution 32 to extract superpoint features, denoted as F E RVV where C' is the feature dimension.
3.1.1 Edge Feature Embedding
Once we obtain superpoint features, the edge score prediction network learns edge embeddings to predict edge scores for segmenting instances. Given the adjacent nodes (u, v) E E, it is desired that the learned edge embedding instance. To tackle this problem, we applies cross-grapp graph attention to the superpoint graph in double spaces (the coordinate and feature spaces) for learning superpoint similarities. The learned similarity vectors of nodes u and v are used to form the edge embedding for predicting edge scores.
effectively identify whether nodes and belong to the same
Edge embedding in coordinate space. To characterize the similarity of nodes and e, we first shift them toward the corresponding instance centroids in the coordinate space. Here, multi-layer preceptron (MLP) network encodes to produce V offset vectors O (01.....ovvi E R Vix3 Give the original superpoint coordinates X = ( vvi E R the shifted superpoint coordinates X (511 - $ v are obtained by X = X + O. In this way, the geometric distance of nodes belonging to different instances will be increased, so that the discrimination of the superpoints will be enhanced. After that, based on the shifted coordinate space, for node u, we leverage its k-nearest superpoints (i.e., N,,) to construct the local k--N graph G., Similarly, we can obtain the graph G, for node v. Then, we perform cross-graph attention across G,, and G, to characterize the similarity of nodes through the learned feature vectors, as shown in Figure [ Taking the node as an example, the weight of cross pooss-graphhoo attention is defined as:
auii = MLP(d; d.), Vi N, UN.
(1)
where å; and f. are the shifted coordinates. Note that enumerates the all 2* neighbors across two graphs. Therefore, the final output feature vector can be formulated as:
hn
dn, * MLP((:) + b;
KEN.UUG.
where auss is the weight au,i after softmax and b; is a learnable bias. The learned feature vector h. E RC can characterize the geometry similarity by adaptively learning the geometric differences on two graphs. In the same way, we can obtain feature vector h, for another node We combine the feature vectors h, and h, as the edge embedding in the coordinate space: ea, = =hh,, h.].
Edge embedding in feature space. In addition to the coordinate space, we also consider the feature space to extract discriminative edge embeddings. Firstly, a MLP network encodes F to produce the initial feature embedding Z E RDD By pushing feature embeddings of instances away from each other, we enlarge the gap between different instances in the feature space. Given a pair of nodes (u, v) e E, we construct the k-NN graphs G and G2 in the feature space, respectively. In this way, it is expected that each graph can aggregate superpoints within the same instance. Then, we execute poos-graph attention across G, and G, to characterize the similarity of nodes in the feature space. Finally, we can obtain the feature vectors h. E RC and h, EC. If and v belong the same instance, they share the similar k-NN graph in the feature space, so that the learned feature vectors h and he are similar to each other. Here, we also combine the feature vectors to obtain the edge embedding in the feature space: ē., [ h.).

---

coordinate space
feature space
W
2Z
P
Ov
(A)
instance centroid
mean instance embedding
Figure 2: The details of our gcometry-awarr edge loss. u and v in graphs (A) and (C) belong to the same instance, and and in graphs (B) and (D) belong to different instances.
Edge score prediction. After obtaining edge embeddings in the coordinate and feature spaces, we utilize simple MLP network to generate the edge score, which is defined
au * eu,, du,)l)
(3)
where [ indicates the concatenation operation, a denotes the sigmoid function, and du,v rep- resents the geometric distance between the nodes and in the shifted coordinate space. In the experiment, if the edge score Gg.v > 0.5, it means that the edge between nodes u and v should be cut from the superpoint graph. We use the binary cross-entropy loss Ledge minimize the edge score.
3.1.2 Geometry-Aware Edge Loss
To train the edge score prediction network, we employ the geometric structures the superpoint graph to form geometry-awan edge loss, shown in Figure
Specifically, given the nodes u, and their corresponding instance centroids C, and C.s we draw nodes toward their instance centroids by minimizing the L2 distance du,c and du.c. Furthermore, when (u, v) E belong to the same instance, it is expected that they can collaboratively shift to the same instance centroid by minimizing the area of triangle Auve. While (u, v) E E belong to different instances, is expected that they can collaboratively shift to their own instance centroids e, and co by minimizing the area of triangles Aunie, and Aurc,. The area constraint in the coordinate space written as:
1 | cull2 + || - coll2 + 2/ (1(c- I.) x (c F.. (T,,, v)
Larea
E
(U,))EE
(4)
+([(c, - 2,) x (e, - 2,)1 1c., - f.)) x (f,, - 2.) (1 I(u, v)))
where I(u, v)) is the indicator function, and T(u, r) equals to 1 u and v belong to the same instance, and 0, otherwise. Note that "x" represents the outer product of vectors for computing the area of triangles. For nodes u and v from the same instance, u and u are simultaneously close to the common instance centroid. Therefore, they are pulled close each other the coordinate space, which is helpful to group u and v into the same instance. For nodes u and v from different instances, and respectively close to the corresponding instance centroids. Thus, they are pushed away in the coordinate space, which is helpful to divide u and v into two different instances.
Likewise, expect the nodes in the same instance to be compact in the feature space by constraining
their feature embeddings. For (u, v) E E belong to the same instance, we draw embeddings of and toward the mean embedding of the instance, and also pull them to each other. For (u, v) E E belong to different instances, we push embeddings of u and v away from each other. In addition, the instances pushed away from each other by increasing the distance of their mean embedding of instances. Thus, the constraint in the feature space is written as:
1
Ljeat
(I zul12 513 [||=, gull2 -,,2 [[,,
E
(a.P)EE
(5)
+ +[2.3
2211212 [28 - llg.. g. (111 I(u,v))
where 2, D and 2, E RDD are the feature embeddings. Note that gn. R and go RD indicate the mean feature embeddings of the instances that and belong respectively. The
5

---

Algorithm Proposal Generation A Algorithm
Input: node semantic scores = {si..... Siv1 8; E R for i = 1,..., |V], J. N is the number of
classes; semantic threshold 0; edge scores A {au,e} IR E|x1, au,e indicates the score of edge which connects nodes and tı; Output: proposals I = (.,...., Im
m is the number of proposals.
1: initialize empty instance set I
2: for i= to N do
3: if i is valid class (excluding wall, floor) then
4: initialize an array f (visited) of length V with all zeros
5: for 1 [V| do
6:
if fo = 0 and st >0 0 then initialize empty queue Q initialize an empty set I
7:
8:
9:
f. = (-pushBack(e)) ; add v to while Q is not empty do
10:
h = Q.popFront()
12:
for each k E {k: 0,,1 < 0.5] do
13:
if f
and st. > 0 then
add I to (uushBack((k)) add k
fk :
15:
return I
thresholds 5 andd 3 are set to be 0.1 and 1.5 to ensure that the inter-instance distance is higher than the intra-instance. Finally, the geometry- edge loss defined as:
Loco Larea + feat Ledge
(6)
3.2
Superpoint Graph Cut Network
3.2.1 Proposal Generation via Superpoint Graph Cut
Given the edge score A {au,.} R| we propose a proposal generation algorithm to generate candidate proposals by simultaneously employing the learned edge scores and the predicted semantic classes of nodes (i.e., superpoints). Specifically, in order to mitigate semantic prediction errors, we follow (37) and adopt soft threshold 0 to associate the nodes with muitiple classes. Given semantic number of classes, if s > 0, the v-th superpoint can be associated with the i-th class. In this way, for the i-th class, the superpoint on the i-th class index is higher than Then, on the superpoint graph, for the edge (u, v)) E, if nodes C; and C, the edge (u, u) will be preserved, otherwise the edge will be deleted. In other words, we remove the edge between two superpoint nodes with different semantics. After that, for the preserved edges (u, v) the superpoint graph, utilize the edge score av.v to determine whether the edge should be cut from the superpoint graph. In the experiment, the threshold for cutting the edge is superpoint graph. Finally, we apply the breadth- sirattt-ritt--rcrch algorithm on the superpoint graph to aggregate nodes in the same connected component for generating proposals for the i-th class. this way, we can generate proposals for N classes by iterating through N classes. The details are shown in Algorithm
scores of superpoints S {81,.... 8|V1 | 8; E R for = 1,..., [V]], where N
slice superpoint subset C₁ on the superpoint graph, where the semantic score of
0.5. If the edge score is higher than 0.5, the edge will be from the
3.2.2 Bilateral Graph Attention for Proposal Embedding
As we obtain proposals I = (I,...,, Im from the point cloud, we propose bilateral graph attention extract proposal embeddings for generating instances by applying the attention mechanism both the coordinate and feature spaces. Specifically, given the i-th proposal, we first compute proposal centroid e; by averaging the shifted superpoint coordinates. Then, we adopt the inverse distance weighted average of the corresponding superpoints interpolate the embedding of the proposal

---

centroid, which is formulated as:
f(a)) Ejel, (y(c) Ejen, vy(cs)
1
| - c
where I, represents the superpoints within the i-th proposal and xy indicates the original coordinates of superpoints. Note that * indicates the Hadamard product, which outputs the element-wise production of two vectors. After obtaining the coordinate o; and embedding S; for the i-th proposal, we then link the superpoints to proposal centroid for constructing the k-NN graph. To extract discriminative embedding of the proposal, we develop bilateral graph attention to achieve this. The bilateral weight was between the superpoint j E 4 and the i-th proposal is formulated as:
wa = off ff) (3)),
(8)
where o(:,) RC x RC + R and p(..)) R2 x + RC are two mapping functions implemented
by MLP networks. (ff,, = (W, (5)) encodes the difference between the superpoint and proposal centroid in the feature space, while ((;, ReLU(W, (a: xzy)) encodes the difference between the superpoint and proposal centroid in the coordinate space. Thus, Way E RC captures the channel-wise relationship between the superpoint and proposal in the coordinate and feature spaces. We use the softmax function obtain normalized weight whi across the proposal 4, which is written as:
exp( (wi))
KEI, (xp(
Finally, we sum the weighted superpoint embeddings to obtain the proposal embedding, which is given by:
f. = =
jEl: w. f,
(10)
After obtaining the proposal embedding, we adopt classification head and a score head to predict the class and score of the proposal 4. In addition, we use superpoint mask head to predict the superpoint score for masking the low-confidence superpoints within the proposal. Note that by using the superpoint mask head, we can generate the instance from the candidate proposal. According to these three heads, we use the cross-cntropy as the classification loss Cote, the binary cross-entropy as the score loss Lxcores and the mean squared error the mask loss Lmask to form the instance loss Lins = Lds + Lscore + Lmaas for training the superpoint graph cut network.
3.3 Training and Inference
In the training process, the whole framework is optimized by joint loss, which is defined as:
Ljoint Lsem + Lgeo Lins
(11)
where Ceem is the conventional cross-entropy loss for semantic scores, Cyeo is the geometry-awar edge loss for edge scores, and Lins is the instance loss for instance classification, score prediction, and superpoint mask prediction. In the inference process, our method directly outputs instances after forward pass of the network. Note that non-maximum suppression is not necessary for method.
4 Experiments
4.1 Experimental Settings
Datasets. We conduct experiments on two benchmark datasets, ScanNet v2 7 and S3DIS [.. The ScanNet v2 dataset contains 1,613 3D scenes, which are split into 1,201 training, 312 validation, and 100 test scenes, respectively. The results of instance segmentation are evaluated on 18 object categories. We report the results on validation and hidden test set. The ablation study is conducted on the validation set. The S3DIS dataset has 272 3D scans in 6 different areas with 13 object classes. The instance segmentation is evaluated in all classes. We report Area 5 and 6-fold cross- tross-validation results, respectively.
Evaluation metrics. Following the ScanNet v2 official protocol, we use the mean average precision as the evaluation metric for both ScanNet v2 and S3DIS. The mean average precision with IoU
7

---

Table I: Instance segmentation results on the ScanNet v2 hidden test set. The reported results are from the ScanNet benchmark on 19/5/2022.
bathub
bookshe
counter
cuttin
picture
s.curaain
window
bed
chair
desk
other
fridge
sOfa
rable
toilet
Method
AP
3D-SIS
13
16.1
40.7
15.5
6.8
4.3
34.6
0.1
13.4
0.5
888
10.6
3.7
13.5
32
288
33.9
11.6
46.6
9.3
GSPN
15.8
35.6
17.3
11.3
35.9
1.2
2.3
134
12.3
0.8
8.9
14.9
11.7
22.1
12.
56.3
3D-MPA
8
35.5
45.7
48.4
29.9
27.7
59.1
4.7
33.2
21.
21.7
27.8
19.3
41.3
41.0
19.5
57.4
35.2
84.9
21.3
PointGroup
18
40.7
63.9
49.6
41.5
24.3
64.5
2.1
57.0
11.4
21.1
35.9
21.7
42.8
66.0
25.6
56.2
34.1
86.0
29.1
SSTNet
A5
50.6
73.8
54,9
49.7
31.6
69.3
17.8
37.7
19.8
33.0
46.3
57.6
51.5
85.7
494
63.7
45.7
94.3
29.0
45.7
70.4
56.1
67.3
4.6
42.6
28.8
45.4
56.3
43,4
88.9
34.4
SoftGroup
x
50.4
66.7
57.9
37.2
38.1
69.4
7.2
67.7
30.3
38.7
53.1
31.9
58.2
75.4
31.8
64.3
49.2
90.7
38.8
GraphCut (ours)
55.2
100
61.1
43.8
39.2
71.4
13.9
59.8
32.7
38.9
51.0
59.8
42.7
75.4
46.3
76.1
58.8
90.3
32,9
thresholds of 50% and 25% denoted Ao and AP25, respectively. Also, AP denotes the mean average precision with the IoU threshold from 50% to 95% with step size of 5%. Additionally, coverage (mCov), mean weighted coverage
following existing methods 41 3 37), we use (mWCov), mean precision (mPrec), and mean recall (mRec) for S3DIS evaluation.
Implementation details. Our model is trained on single TITAN RTX GPU. We use the Adam optimizer with a base learning rate of 0.001 for the network training, which is scheduled by a cosine annealing, The voxel size is set to 0.02m. A graph-based segmentation method [9] and SSP+SPG [1] are used to generate superpoints for ScanNet scene and S3DIS room, respectively, At training time, we limit the maximum number of points in scene to 250k and crop the excess randomly. Due to the high point density of S3DIS, we randomly downsample its 1/4 points before cropping. At inference, the whole scene is fed into the network without downsampling and cropping. and use the statistical average instance radius of the specific class to refine
Note that we follow 3 37 the instances.
4.2 Benchmarking Results
ScanNet v2. We compare our model with recent state-of-the-art methods the unreleased test set of ScanNet v2. Table reports the results on the leaderboard of the official testing server] It can be observed that method achieves the highest performance in terms of AP. The results the leaderboard can demonstrate the effectiveness of our method for 3D instance segmentation.
Moreover, we evaluate our method on the val- idation set of ScanNet v2. From the results in Table one can observe that the proposed Grap- Cut can achieve better results. method brings 2.8% gains for the metric AP and 1.5% gains for the metric AP to the second-best methods. In addition, we provide the visualization results of our GraphCut and SoftGroup [37] in Figure We the red rect- angular boxes to show the differences between them. It can be observed that method can generate good instances with clear boundaries for objects clustered together, such chairs. SoftGroup relies on point grouping by using offset-shifted point coordinates, which cannot make full use of local geometric information of point clouds. Since our method can fully utilize the local geometric information of point clouds by con- structing an edge score prediction network and superpoint graph network, our method achieves better results than SoftGroup on these clustered objects.
Table 2: Instance segmentation results the Scan- Net v2 validation set.
particular,
Method
AP
AP30
AP25
GSPN [45]
19.3
37.8
53.4
3D-MPA
35.3
51.9
72.4
PointGroup
18
34.8
56.9
71.3
SSTNet 25
49.4
64.3
74.0
HAIS 3
43.5
64.1
75.6
SoftGroup 37
46.0
67.6
78.9
GraphCut (ours)
52.2
69.1
79.3
S3DIS. In Table we list the results of Area 5 and 6-fold cross- validation S3DIS. Regarding the evaluation of Area 5, our method can outperform all compared methods. It is worth noting that our model improves SoftGroup by 2.5% in terms of AP. For the 6-fold cross-validation of S3DIS, method is superior to the state-of-thh--n methods on most metrics.
http://kaldrr, in.tum. dem.e//ccnnnet benchmark/semantt instance 3d.php'mmetric-ap

---

Input
Instance GT
GruphCut (ours)
SoftGroup
Figure 3: Visualization results of our GraphCut and SoftGroup 37
the ScanNet v2 validation set.
Table Instance segmentation results on S3DIS dataset.
Setting
Method
AP
AP30
mCov
mWCov
mPrecso
mRecso
SGPN
32.7
35.5
36.0
28.7
ASIS
44.6
47.8
55.3
42.4
PointGroup 18
57.8
61.9
62.1
Area 5
SSTNet [25
42.7
59.3
65.5
64.2
HAIS 3
64.3
66.0
71.1
65.0
SoftGroup
37
51.6
66.1
66.1
68.0
73.6
66.6
GraphCut (ours)
54.1
66.4
67.5
68.7
74.7
67.8
SGPN 40
37.9
40.8
38.2
31.2
PartNet 28
56.4
43.4
ASIS 41
51.2
55.1
63.6
47.5
3D-BoNet 444
65.6
47.7
OccuSeg 11
72.8
60.3
6-fold
GICN [26]
68.5
50.8
PointGrropp [18
64.0
69.6
69.2
SSTNet 225
54.1
67.8
73.5
73.4
HAIS 3
67.0
70.4
73.2
69.4
SoftGroup
37
54.4
68.9
69.3
71.7
75.3
69.8
GraphCut (ours)
56.3
68.2
72.8
75.0
74.4
73.7
4.3 Ablation Studies and Analysis
Different à in edge score prediction network. In our edge score prediction network, we learn the similarity from the local k-NN graphs of adjacent nodes to identify whether they belong the same instance. Here, we study the impact of different & on the instance segmentation performance. We select & E {0, 2, 4, 8, 16). Notably, k=0 means that we only concatenate two adjacent superpoint features as edge embedding. The results of AP, AP and AP25 are 52.0%, 68.8%, 78. 78% (k=2), 52.2%, 69.1%, 79.3% (k=4)), 51.4%, 68.1%. 79.1% 1% (k=8), and 50.8%, 67.7%, 79.1% (A=16), respectively. Since k=4 achieves the best results, set k=4 in experiment.
Effectiveness of edge feature embedding. To verify the effectiveness of our edge feature embedding, we consider three cases: (1) Only with edge embedding in coordinate space (dubbed as "Coordinate"). (2) Only with edge embedding in feature space (dubbed as "Feature") (3) Only with embeddings of adjacent nodes as edge embedding. From the instance segmentation results on the ScanNet v2 validation set listed in Table the best performance achieved with the combination of edge embeddings in both the coordinate and feature spaces. In the edge feature embedding, employing both geometry and feature embeddings of point clouds can improve the performance of the instance segmentation of point clouds.

---

Table 4: Ablation study on the ScanNet v2 vali- dation set for edge feature embedding.
Table 5: Ablation study on the ScanNet v2 vali- dation for geometry-aware edge loss.
Coordinate Feature AP
AP30 30
AP25 25
Larea Dreat
AP
AP
AP 25
50.1
66.6
77.4
47.8
65.1
75.3
50.9
68.0
78.3
51.7
68.6
78.8
51.5
68.8
78.5
51.2
67.8
77.8
52.2
69.1
79.3
/
/
52.2
69.1
79.3
Ablation study on geometry-aware edge loss. Here, we conduct the experiments on the ScanNet edge loss, Specifically, we also consider three ablations: (I) Only with area constraint in the coordinate space (i.e., "Carea") "), (2) Only with instance constraint in the feature space (i.e.. " "Leeat" "), (3) Only with binary cross-entropy loss, i.c., Ledge. The results are listed in Table 5 It can be observed that the geometry constraints bring substantial gains our method. By using the area constraint, it is easier draw the nodes of the instance toward the instance center, making the boundary between different instances clearer.
v2 validation to verify the effectiveness the propose geometry
Effectiveness of bilateral graph attention. In order validate the effectiveness of the proposed bilateral graph attention, we replace the bilateral graph attention with simple MLP network followed by max-pooling and conduct experiments the ScanNet v2 validation set. The results of AP, APs0- and AP25 are 49.9%, 66.8%, 77.3% (MLP network), and 52.2%, 69.1%, 79.3% (our bilateral graph attention), respectively. Without our designed bilateral graph attention, the performance drops lot. This is because the bilateral graph attention can adaptively aggregate the information of superpoints in the same instance, which is more reasonable than the simple max max-pooling operation for instance embedding.
5 Conclusion
In this paper, we proposed learning-based superpoint graph cut method for 3D instance segmentation, which prunes the edges off the superpoint graph for generating instances. Specifically, we proposed an edge score prediction network with cross-graph attention in the coordinate and feature spaces to capture local geometric information of two adjacent nodes and predict the edge scores. A geometry- aware edge loss was proposed to train the edge score prediction network, which encourages two adjacent nodes in the same instance be close the instance center in both the coordinate and feature spaces. Based on the learned edge scores, a superpoint graph cut network was developed to irrelevant edges for instance generation. For the generated instances, we further adopted bilateral graph attention to predict semantic classes and scores instances. Extensive experiments on ScanNet v2 and S3DIS benchmarks show that our method achieves new state-of-the-ard performance on 3D instance segmentation.
Acknowledgments
The authors would like thank reviewers for their detailed comments and instructive suggestions. This work was supported by the National Science Fund of China (Grant Nos. U1713208, 61876084).
10

---

References
[1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and
Silvio Savarese. 3D semantic parsing of large-scale indoor spaces. In CVPR, 2016.
[2] Yikai Bian, Le Hui, Jianjun Qian, and Jin Xie. Unsupervised domain adaptation for point cloud semantic segmentation via graph matching. In IROS, 2022.
[3] Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3D instance segmentation. In ICCV, 2021.
[4] Mingmei Cheng, Le Hui, Jin Xie, and Jian Yang. SSPC-Net: Semi-supervised semantic 3D point cloud segmentation network. In AAAI, 2021.
[5] Mingmei Cheng, Le Hui, Jin Xie, Jian Yang, and Hui Kong. Cascaded non-local neural network for point cloud semantic segmentation. In IROS, 2020.
[6] Christopher Choy, JouYo Gwak, and Silvio Savarese. 4D spatio-temporal convnets:
Minkowski convolutional neural networks. In CVPR, 2019.
[7] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017.
[8] Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias NieBner. 3D-MPA: Multi proposal aggregation for 3D semantic instance segmentation. In CVPR, 2020.
[9] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. International journal of computer vision, 2004.
[10] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
[11] Lei Han, Tian Zheng, Lan Xu, and Lu Fang. OccuSeg: Occupancy-aware 3D instance segmen- tation. In CVPR, 2020.
[12] Tong He, Chunhua Shen, and Anton van den Hengel. DyCo3D: Robust instance segmentation of 3D point clouds through dynamic convolution. CVPR, 2021.
[13] Hou, Angela Dai, and Matthias Nießner. 3D-SIS: 3D semantic instance segmentation of RGB-D scans. In CVPR, 2019.
[14] Qingyong Hu, Bo Yang. Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. RandLA-Net: Efficient semantic segmentation of large-scale point clouds. In CVPR, 2020.
[15) Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-lan Tai. JSEnet: Joint semantic segmentation and edge detection network for 3D point clouds. In ECCV, 2020.
[16] Le Hui, Jia Yuan, Mingmei Cheng, Jin Xie, Xiaoya Zhang, and Jian Yang. Superpoint network for point cloud oversegmentation. In ICCV, 2021.
[17] Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3D scene understanding. In ICCVW, 2019.
[18] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. PointGroup: Dual-set point grouping for 3D instance segmentation. In CVPR, 2020.
[19] Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3D semantic segmentation. In ECCV, 2020.
[20] Jean Lahoud, Bernard Ghanem, Mare Pollefeys, and Martin R Oswald. 3D instance segmenta- tion via multi-task metric learning. In ICCV, 2019.
[21] Loic Landrieu and Mohamed Boussaha. Point cloud oversegmentation with graph-structured deep metric learning. In CVPR, 2019.
11

---

[22] Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, 2018.
[23] Felix Järemo Lawin, Martin Danelljan, Patrik Tosteberg, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, Deep projective 3D semantic segmentation. In CA/P. Springer, 2017.
[24] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution
mathcal Xtrrnsfforedd points. In NIPS, 2018.
[25] Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3D scenes using semantic superpoint tree networks. In ICCV, 2021.
[26] Shih-Hung Liu, Shang- Yi Yu, Shao- Chi Wu, Hwann-Tzong Chen, and Tyng-Luh Liu. Learning gaussian instance segmentation in point clouds. arXiv preprint arXiv:2007.0955 009060,,,0 2020.
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[28] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. PartNet: large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, 2019.
[29] Quang- Hieu Pham, Thanh Nguyen, Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. JSIS3D: Joint comantic-instane segmentation of 3D point clouds with multi-task pointwise networks and multi-value conditional random fields. CVPR, 2019.
[30] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In CVPR, 2017.
[31] Charles R Qi. Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical feature learning on point in a metric space. In NeurIPS, 2017.
[32] Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks graphs. In CVPR, 2017.
[33] Linghua Tang, Le Hui, and Jin Xie. Learning inter-superpoint affinity for weakly supervised 3D instance segmentation. In ACCV, 2022.
[34] Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. ArXiv, 2022.
[35] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian- Yi Zhou. Tangent convolutions for dense prediction in 3D. In CVPR, 2018.
[36] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. KPConv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
[37] Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, and Chang D. Yoo. SoftGroup for 3D instance segmentation on 3D point clouds. In CVPR, 2022.
[38] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. In CVPR, 2019.
[39] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun- Yu Sun, and Xin Tong. O-CNN: Octree-based convolutional neural networks for 3D shape analysis. TOG, 2017.
[40] Weiyue Wang, Ronald Yu, Qiangui Huang, and UIrich Neumann. SGPN; Similarity group proposal network for 3D point cloud instance segmentation. In CVPR, 2018.
[41] Xinlong Wang, Shu Liu, Xiaoyong Shen, Chunhua Shen, and Jiaya Jia. Associatively segment- ing instances and semantics in point clouds. In CVPR, 2019.
12

---

[42] Yue Wang, Yongbin Sun, Ziwei Liu. Sanjay E Sarma. Michael M Bronstein, and Justin M Solomon. Dynamic graph emn for learning on point clouds. TOG, 2019.
[43] Wenxuan Wu, Zhongang Qi, and Li Fuxin. PointConv: Deep convolutional networks on 3D point clouds. In CVPR, 2019.
[44] Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3D instance segmentation point clouds. In NeurIPS, 2019.
[45] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J Guibas. GSPN: Generative shape proposal network for 3d instance segmentation in point cloud. In CVPR, 2019.
[46] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. PointWeb: Enhancing local neighbor- hood features for point cloud processing. In CVPR, 2019.
[47] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun, Point Transformer. In ICCV, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer to the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Please refer to the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer to the experimental part of the main paper.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer experimental part of the main paper.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'le rou're
using/curating? [No]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "SGwHAGuw0MM", "reviewer": "Reviewer_nE6M", "summary": "This is a good paper with new idea and solid experiments.", "questions": "1.Line 133, the authors design a MLP to predict the offset vectors. Is there any constrain on the MLP? Could you please add a figure (in the supp.) to show the shifted points or superpoints? Could you please add one experiment that without the offset vectors?\n2.Line 172, for nodes u and v, if they are from different instances, the triangle areas are also  minimized, just as they are from the same instance. Please explain it in detail.", "limitations": "Please refer to the Questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: 1. The idea is novel. 2. The experiments are sufficient. 3. The results are good. \nWeaknesses: Refer to the Questions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S8KUFJrfeb", "reviewer": "Reviewer_TEiW", "summary": "This paper proposes a learnable superpoint-based graph cut method to explicitly learn the geometric structures for 3D point cloud instance problem.  Experiments on benchmark datasets show that the proposed method outperforms current state-of-the-art methods.", "questions": "Using attention to capture the similarities between points or nodes in graph is not a completely novel approach, there have already been some existing works [1] in 3D point cloud proposed before. What are the main differences of the author’s attention-based approach? It is suggested to append this mention in the Related Work section.\n\n[1] Wang, Lei, et al. \"Graph attention convolution for point cloud semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n", "limitations": "What I’m more concerned about is the runtime efficiency on the proposed learnable graph cut method. For deploying the proposed framework in the autonomous driving application as mentioned in the Introduction section, a short response time is desired. From the supplemental, however, the runtime efficiency comparison only is performed on indoor datasets (S3DIS, ScanNet2). Outdoor dataset specific to autonomous driving such as SemanticKITTI should be the more suitable one for evaluation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe proposed approach which dynamically prunes the unrelated edges for better instance segmentation is relatively new. The paper presents a detailed description on the main building blocks (i) edge score prediction network and ii) superpoint graph cut network. And the ablation analysis validates the efficacy of the proposed network structure and geometry-aware edge loss.\n\nWeaknesses:\nStrictly speaking, the proposed approach is not completely learnable, since the initial superpoint graph oversegmented from point cloud is determined empirically. What is the sensitivity analysis on the different configurations on the initial superpoint graph?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FFCIIZNBGfy", "reviewer": "Reviewer_n9iC", "summary": "This work tackles the task of 3D semantic instance segmentation by exploiting the local geometry. For that purpose, a graph is defined over geometric super-points. Then a series of graph cut networks, edge scoring functions and attention mechanisms is used to extract, classify and score the final instance mask. Experiments on S3DIS A5/6fCV and ScanNet test outperform prior published methods. Ablation studies on ScanNet validation show the effect of the individual model components.", "questions": "**Question**\n\nThe method introduces some new triangle losses, but it does not help? Table 3 in the supplementary seems to suggest that the L_area hurts the performance. However, Table 5 in the main paper suggests the opposite conclusion. What is the correct answer?\n\n**Minor suggestions**\n\n- l.133/149 - repeat meaning of F.\n- Fig.1 - add the losses to the model figure (now it’s a bit unclear where they are applied), similarly add the used notation into the Figure - \n- l.184 Parameter study for delta and beta?\n- l.191 Where do the semantic classes come from?\n- l.201 Parameter study?\n- l.230 How is the scalar score obtained from the cross entropy loss / per-?\n- l.235 Where is the semantic loss applied?\n- l.278 Highlighting that the model produces clear boundaries is not that insightful - this is clear from the fact that the model is based on an over-segmentation which always produces clear boundaries.\n- Training details: is the model trained in the train and val split for the test set submission?\n", "limitations": "Limitations and negative societal impact are adequately discussed in the rebuttal. In particular, the limitations include sensitivity to the long-tail problem, i.e, refrigerators which appear sparsely in the training data and expose a large intra-class variety. The papers also suggest potential solutions such as mining context information or data augmentation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nThe proposed approach is original in terms of the model architecture which is inspired by traditional graph cut approaches and achieves state-of-the-art 3D instance segmentation scores on two popular indoor scenes datasets (ScanNet, S3DIS).\n\n**Weaknesses**\n\nThe model itself consists of numerous components that are not always very clearly explained and/or motivated. For example, l.122 mentions sparse convolutions, but it is not very clear if they are applied on points or on superpoints. Then additional edge-conditioned convolutions are applied. What is the motivation for two types of convolutions, wouldn’t one type of convolutions be enough? This could be further motivated and also evaluated to show that it actually improves performance. Overall, the model consists of numerous components which increases complexity and raises the question if a similar performance could be achieved with a simpler model, allowing the community to draw more general conclusions. At this stage, it is an interesting model that performs very well but I’m unsure about the significance to the community since it is unclear which conclusions to draw that can push the field forward.\n\nThe approach also relies on a large number of hyper-parameters (l.184, l.192, l.201, l.292 …), requiring manual tuning. The paper contains parameter studies only for a subset of them e.g. k in k-NN. From the paper it is unclear if the same hyperparameters are used for both datasets and how sensitive the approach is to the parameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept