OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds
Authors: Ziyang Song
Bo Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D object segmentation
point cloud analysis
unsupervised learning
scene flow

Abstract: In this paper, we study the problem of 3D object segmentation from raw point clouds. Unlike all existing methods which usually require a large amount of human annotations for full supervision, we propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass, without needing any type of human annotations. The key to our approach is to fully leverage the dynamic motion patterns over sequential point clouds as supervision signals to automatically discover rigid objects. Our method consists of three major components, 1) the object segmentation network to directly estimate multi-object masks from a single point cloud frame, 2) the auxiliary self-supervised scene flow estimator, and 3) our core object geometry consistency component. By carefully designing a series of loss functions, we effectively take into account the multi-object rigid consistency and the object shape invariance in both temporal and spatial scales. This allows our method to truly discover the object geometry even in the absence of annotations. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation and general object segmentation in both indoor and the challenging outdoor scenarios. 

OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds
Ziyang Song
Bo Yang
vLAR Group, The Hong Kong Polytechnic University ziyang. song@connect. polyu. bo. bo.yangdpolyu. edu.y..eedddu..
Abstract
In this paper, we study the problem of 3D object segmentation from raw point clouds. Unlike all existing methods which usually require large amount of human annotations for full supervision, we propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass, without needing any type of human annotations. The key to our approach is to fully leverage the dynamic motion patterns over sequential point clouds as supervision signals to automatically discover rigid objects. Our method consists of three major components, I) the object segmentation network to directly estimate multi-object masks from single point cloud frame, 2) the auxiliary self-supervised scene flow estimator, and 3) our core object geometry consistency component. By carefully designing series of loss functions, we effectively take into account the multi-object rigid consistency and the object shape invariance in both temporal and spatial scales. This allows our method to truly discover the object geometry even in the absence of annotations. We extensively evaluate demonstrating the superior performance for object part instance segmentation and general object segmentation in both indoor and the challenging outdoor scenarios. Our code and data are available at https: ////t/ttt/////////////////hhghht con/vLAR-aa Sacin-grrupDDDCC
method five datasets,
Introduction
Identifying 3D objects from point clouds is vital for machines to tackle high-level tasks such as autonomous planning and manipulation in real-world scenarios. Inspired by the seminal work PointNet [56]. a plethora of sophisticated models [70; 76; 37] have been developed to accurately detect and segment individual objects from the sparse and irregular point clouds. Although these methods have achieved excellent performance on wide range of public datasets, they primarily rely on huge amount of human annotations for full supervision. However, is extremely costly fully annotate every objects in point clouds due to the irregularity of data format.
Very recently, few works start to address 3D object segmentation in the absence of human notations. By analysing 3D scene flows from sequential point clouds, Jiang et al. [28] apply the conventional subspace clustering optimization technique identify moving objects from raw point cloud sequences. With the self-supervised learning of 3D scene flow, SL.M [3] is the first learning- based work to showcase that the set of moving points can be effectively learned the stationary background. Fundamentally, their design principle shares the key spirit of Gestalt theory [73; 69] developed exactly 100 years ago: the raw sensory data with similar motion likely to be organized into single object. This is indeed true in the real world where solid objects usually have strong correlation in rigid motions. However, these methods cannot learn simultaneously segment multiple interested 3D objects from single point cloud in one go.
object against
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Object
Dynamic rigid consistency
SuperSupe Scene Flow Estimator
Segmentation Network
Object shape smoothness Object shape invariance
Single point cloud
Estimated object masks
Estimated scene flows
pair of point clouds
Figure 1: The general workflow and components of our framework.
Motivated by the potential of motion dynamics, this paper aims to design general neural framework to simultaneously segment multiple 3D objects, without requiring any human annotations but the inherent object dynamics in training. To achieve this, naive approach is to train neural network to directly cluster motion vectors into groups from sequential point clouds, which is widely known as motion segmentation [77; 78]. However, such design requires that the input data points are sequential in both training and testing phases, and the trained model cannot infer objects from a single point cloud. Fundamentally, this is because the learned motion segmentation strategies simply cluster similar motion vectors instead of discriminating object geometries, and therefore such design is not general enough for real-world applications.
In this regard, we design a new pipeline which takes a single point cloud as input and directly estimates multiple object masks in a single forward pass. Without needing any human annotations, our pipeline instead leverages the underlying dynamics of sequential point clouds as supervision signals. In particular, as shown in Figure 1, our architecture consists of three major components: 1) an object segmentation network to extract per-pointt features and estimate all object masks from a single point cloud, indicated by the orange block; 2) auxiliary self-supervised network estimate per-point motion vectors from a pair of point clouds, as indicated by the green block; 3) a series of loss functions to fully utilize the motion dynamics to supervise the object segmentation backbone, as indicated by the blue block. For the first two components, it is actually flexible to adopt any of existing neural feature extractors [57] and self-supervised motion estimators [34]. Nevertheless, the third component is particularly challenging to design, primarily because we need to take into account not only the consistency of diverse dynamics of multiple objects in a sequence, but also the invariance of object geometry irregardless of different moving patterns.
To tackle this challenge, we introduce three key losses to end-to-end train our object segmentation network from scratch: 1) a multi-object dynamic rigid consistency loss, which aims to evaluate how coherently all estimated object masks (shapes) can fit the motion via rigid transformations; 2) an object shape smoothness prior, which regularizes all points of each estimated object to be spatially continual instead of fragmented; 3) an object shape invariance loss, which drives multiple estimated masks of particular object to be invariant given different (augmented) rigid transformations. These losses together force all estimated objects' geometry to be consistent and represented by high-quality masks, purely from raw 3D point clouds without any human annotations. Our method is called OGC and our contributions are:
We introduce the first unsupervised multi-object segmentation pipeline on single point cloud frames, without needing any human annotations training or multiple frames as input.
We design a set of geometry consistency based losses to fully leverage the object rigid dynamics and shape invariance as effective supervision signals.
We demonstrate promising object segmentation performance on five datasets, showing significantly better results than classical clustering and optimization baselines.
Difference from Scene Flow Estimation: We do not aim to design a new scene flow estimation method such as [74; 34; 20; 71]. Instead, we use unsupervised learning based per-point scene flow as supervision signals for single-fran frame multi-object segmentation.
Difference from Motion Segmentation: We neither aim to segment motion vectors such as [71; 63] which require multiple successive frames as input in both training and testing. Instead, our network directly estimates object masks from single frames, and therefore is more flexible and general.
Scope: This paper does not intend to replace fully-supervised approaches because the never-moving objects are unlikely to be discovered due to the lack of supervision signals. In addition, estimating object categories or non-rigid objects such as articulated buses and semi-truck with trailers also out of the scope of this paper.
2

---

PaintNat+
0
ph
0%
0 2
P
p
p
pt
Weighted Kabsch
dynamic
p
M
Hungatan
M
Transfomer
0 0¹, P
l smooth
Envariant
Sell-supervised
bbject segmentation network
Object geometry consistency losses
scene flow network
Figure 2: Components of our pipeline. The object segmentation network consists of PointNet++ and
Transformer decoders. FlowStep3D adopted as the self-supervised scene flow network.
2 Related Works
Fully-supervised 3D Object Segmentation: To identify 3D objects from point clouds, existing fully-
supervised solutions can be divided as 1) bounding box based object detection methods [84; 37; 60] 2) mask based instance segmentation pipelines [70; 76; 68]. Thanks to the dense human annotations and the well developed backbones including projection- based [38; 8; 37], point- based [57; 65; 26] and voxel-based [22; 12] feature extractors, these methods achieve impressive performance on both indoor and outdoor datasets. However, manually labelling every object in large-scale point clouds is costly. To alleviate this burden, we aim to pioneer 3D object segmentation without human labels.
3D Scene Flow Estimation and Motion Segmentation: Given sequential point clouds, per-por per-point 3D motion vectors, also known as scene flow, can be accurately estimated. Early works focus on superssue scene flow estimation [42; 4; 24; 43; 55; 53; 72], whereas recent methods start to explore self-supervised motion estimation [50; 74; 34; 46; 82; 39]. Taking the scene flow as input, number of works [80; 27; 64; 3] aim to group similar motion vectors, and then obtain bounding boxes or masks only for dynamic objects. Although achieving encouraging results, they either rely on ground truth segmentation for supervision or can only segment simple foreground and background objects, without being able to simultaneously segment multiple objects. In this paper, we leverage the successful self-super scene flow estimator our auxiliary neural network to provide valuable supervision signals, so that multiple objects can be identified in single forward pass.
Unsupervised 2D Object Segmentation: Inspired by the early work AIR [16], a large number of generative models have been proposed to discover objects from single images without needing human annotations, including MONet [6], IODINE [23], Slot-Att [44], etc.. These methods further extended to segment objects from video frames [35; 25; 49; 29; 85; 15]. However, as investigated by the recent work [79], all these approaches can only process simple synthetic datasets, and cannot discover objects from complex real-world images yet. It is still elusive to apply these ideas on 3D point clouds where 3D objects far more complicated and diverse in terms of geometry.
2D Scene Flow Estimation and Motion Segmentation: Given image sequences, pixel-level scene flow, also known optical flow, have been extensively studied in literature [18; 83]. The estimated flow field can be further grouped as objects [61; 62; 10; 45; 77; 41; 32]. Drawing insights from these works, this paper aims to segment multiple diverse objects in the complex 3D space.
3 OGC
3.1
Overview
As shown in Figure 2, given single point cloud P with N points as input, i.e., P' E Nx3 where each point only has a location (x, = without color for simplicity, the object segmentation network extracts per-por per-point features and directly reasons set of object masks, denoted as O' E (0, 1) 1)N where K is predefined number of objects that is large enough for a specific dataset. In particular, firstly adopt PointNet++ [57] to extract the per-point local features. Then we employ Transformer decoders [67] to attend to the point features and yield all object masks in parallel. The whole architecture can be regarded
3D extension of the recent MaskFormer [9] which shows excellent performance in object segmentation in 2D images. Thanks to the powerful Transformer module, each inferred object mask is effectively modeled over the entire point cloud. Implementation details are in Appendix A.1
In the meantime, have the corresponding sequence of point clouds for supervision, denoted as (P', p ......... For simplicity, we only use the first two frames {P', pp+1]] and feed them into
3

---

the auxiliary self-supervised scene flow network, obtaining satisfactory motion vectors for every point in the first point cloud frame, denoted M' E Nx3, where each motion vector represents point displacement {Ar, Ay, Az). Among the existing self-supervised scene flow methods, we choose the recent FlowStep3I [34] which shows excellent scene flow estimation in multiple datasets. Implementation details are in Appendix A. To train the object segmentation network from scratch, the key component is the supervision mechanism as discussed below.
3.2 Object Geometry Consistency Losses
Given the input point cloud P' and its output object masks O' and motion M we introduce the following objectives to satisfy the geometry consisteney on both frames p and (P' + M'). Note that, the masks O' are meaningless at the very beginning and need to be optimized appropriately.
(1) Geometry Consistency over Dynamic Object Transformations
From time + 1, the rigid objects in point cloud frame p' usually exhibit different dynamic transformations which can be described by matrices belonging to SE((3 group. For the hath object, we firstly retrieve its (soft) binary mask O and then feed the tuple {P', p' + M' O } into the differentiable Brighted-Kabsdde algorithm [31: 21], estimating its transformation matrix Tx E R
In order to drive all raw object masks to be more and more accurate, so as to fully explain the corresponding motion patterns within all masks, the following dynamic rigid loss is designed to minimize the discrepancy of per-point scene flow between time and t + 1 for each point in P
N
K
Edgramic
Σ Onk O k=1
P P.n))
(Pn, +mn)
(1)
N
==
where Onk E (0, 1) represents the probability of being assigned to the ttth object for specific nh point ps, and m, R represents the motion of P,, The operation applys the rigid transformation to the point. Intuitively, one inferred object mask includes two sets of points with two different moving directions, the transformed point cloud in higher errors. Therefore, Edynamic can push all object masks to fit the dynamic and diverse motion patterns. However, here arises critical issue: single rigid object may be assigned to multiple masks, Le. oversegmentation. We alleviate this issue by a simple smoothness regularizer discussed below.
only favor one moving direction, thereby resulting
We observe that such rigid constraint concept is also applied in recent scene flow estimation method [20]. However, their objective is to push the scene flow to be consistent given object masks (estimated by DBSCAN clustering), while our objective is to learn high-quality masks from given flows.
(2) Geometry Smoothness Regularization
The primary reason why a single object may be oversegmented is the lack of spatial connectivity between individual points. However, our common observation is that physically neighbouring points usually belong to a single object. In this regard, we simply introduce a geometry smoothness regularizer. Particularly, for specific n 3ID point P, in the point cloud Pt, we firstly search H points from its neighbourhood using either KNN or spherical querying methods, and then force their mask assignments to be consistent with the center point p,. Mathmatically, is defined as:
N
H
smooth
d(o,, Omn))
===1
h=1
where on E (0, 1) represents the object assignment of center point Pa. and On, E (0, 1)x represents the object assignment of its th neighbouring point. The distance function d() is flexible to choose L1/ L2 or more aggressive cross-entropy function.
Note that, such local smoothness prior successfully used for scene flow estimation [42; 34]. Here, we instead demonstrate its effectiveness for object segmentation.
(3) Geometry Invariance over Scene Transformations
With the above geometry constraints designed in (1)(2), the shapes of dynamic objects can be reasonably segmented. However, the learned object geometry may not be general enough. For example, moving car can be well segmented, yet another similar parked car may not be discovered. To this end, we introduce an object geometry invariance constraint as follows:

---

Firstly, given P", we apply two transformations to get augmented point clouds pd and P Secondly, we feed pa and p into object segmentation network, obtaining two sets of object masks 0 and O" Because the per-point locations in two point clouds are transformed differently, the position sensitive PointNet++ [57] features generate two different sets of masks. Thirdly, we leverage the Hungarian algorithm [36] one-one match the individual masks in O and O according to the object pair-wise IoU scores. Basically, this is to address the issue that there no fixed order for predicted object masks from the two augmented point clouds.
At last, we reorder the masks O to align with O", and design the invariance loss as follows.
N
imporiant
d 6%)
N
where 6% and , are the reordered object assignments of the two augmented point clouds for the nth point. The distance function d() is flexible to use L1, L2 or cross-entropy Ultimately, this loss drives the estimated object masks be invariant with different views of input point clouds.
Notably, unlike existing sulf-super learning [11] which usually uses invariance prior for better latent representations, here we aim to generalize the segmentation strategy to similar yet static objects.
3.3 Iterative Optimization of Object Segmentation and Motion Estimation
With the designed geometry consistency loss functions, the object segmentation network is optimized from scratch by the combined loss: Eung Edynamic + Esmooth + fiveariant- For efficiency, the auxiliary self-supervised scene flow network FlowStep3D [34] is independently trained by own losses until convergence. Intuitively, with better and better object masks estimated, the estimated scene flow is also expected to be improved further if we use the masks properly. To this end, we propose the following Algorithm I to iteratively improve object segmentation and motion estimation.
Algo Algorithm Iterative optimization of object segmentation and scene flow estimation. Assume the whole train split has S point cloud pairs: ((P', ((P',P(+)), ... .......... pP+1)) )s).
Stage O: Initial scene flow estimation.
Independently and fully train the self-supervised scene flow network on the whole training data split, and obtain reasonable scene flow estimations: {(P', pi+1 M.)....(( for number of iteration rounds R do Stage 1: Object segmentation optimization.
... (P', pi+1 M )s)
Train the object segmentation network using leeg for total E epochs on the whole training ... ppt1, p s)
split: ((P', "p',,,
Estimate reasonable object masks: {(P', pt+ 100t1,0'.o 00 p ... (P', (P*,pt+1, 0 )s)
Stage 2: Scene flow improvement.
For each pair of data (P' (P',pt+1,
ppt+l MM O', O²+1), by drawing insights from the classical
ICP [2]. propose object-aware ICP algorithm to estimate new scene flow M' for point cloud P'.
Update the new scene flow for next round training:
{(Pt, p(+1 p ...MM....PPPP ..',,'",,,'''''
+ [(P', p-tI,M'1 ptt1, M
Empirically, setting the total number of rounds be 2 or 3 has good trade off between accuracy and training efficiency. Due to the limited space, details of object-aware ICP algorithm are in Appendix A.2. We exclude the invariance loss Inwvariant from object segmentation optimization stage in the early rounds so that the networks can focus on moving objects in training and produce better scene flows, and then add Innvariont back the final round. Detailed analysis is Appendix A.5.
4 Experiments
Our method is evaluated on four different application scenarios: 1) part instance segmentation of articulated objects on SAPIEN dataset [75], 2) object segmentation of indoor scenes on our own synthetic dataset, 3) object segmentation of real-world outdoor scenes on KITTI-SF dataset [48], and 4) object segmentation on the sparse yet large-scale LiDAR based KITTI-Det [19] and SemanticK/TTI [S] datasets. For evaluation metrics, we follow [51] and report the F1-score, Precision, and Recall
5

---

Table 1: Quantitative results of our method and baselines on the SAPIEN dataset.
APT
PQT
Fif
Pret
Reet
mloUt
RIf
PointNet++ [57]
51.2
65.0
MeteorNet [43]
45.7
60.0
Supervised Methods
DeepPart [80]
53.0
67.0
MBS [27]
67.3
77.0
OGC sup
66.1
48.7
62.0
54.6
71.7
66.8
77.1
Unsupervised
TrajAffn [52]
6.2
14.7
22.0
16.3
34.0
45.7
60.1
Motion Segmentation
SSC [51]
9.5
20.4
28.2
20.9
43.5
50.6
65.9
WirdLinkage [30]
17.4
26.8
40.1
36.9
43.9
49.4
62.2
Unsupervised
DBSCAN [17]
6.3
13.4
20.4
13.9
37.9
34.2
51.4
Methods
OGC(Ours) 55.6
50.6
65.1
65.0
65.2
60.9
73.4
with an IoU threshold of 0.5. In addition, we report the Average Precision (AP) score following COCO dataset [40] and the Panoptic Quality (PQ) score defined in [33]. The mean Intersection over Union (mloU) score and the Rand Index (RI) score implemented in [27] are also included. Note that, all metries are computed in a class-agnostic manner.
4.1 Evaluation on SAPIEN Dataset
The SAPIEN dataset [75] provides 720 simulated articulated objects with part instance level anno- tations. Each object has 4 sequential scans. The part instances have different articulating (moving) states. We follow [27] to use the training data generated from [81]. particular, there are 82092 pairs of point clouds for training, 2880 single point cloud frames for testing. Each point cloud is downsampled to 512 points in both training and testing.
Since there is no existing unsupervised method for multi-object segmentation on 3D point clouds, we firstly implement two classical clustering methods: WardLinkage [30] and DBSCAN [17] to directly group 3D points from single point clouds into objects. Secondly, we implement two classical motion segmentation methods: Traj Affn [52] and SSC [51]. Note that, these two methods take the same estimated scene flows of FlowStep3D
input, while method uses the estimated scene flows during training only, but takes single point clouds as input during testing. In addition, we also include the excellent results of several fully-superviser methods (PointNet++ [57]. MeteorNet [43], DeepPart [80]) reported in MBS [27]. Their experimental details can be found in MBS [27]. Lastly, we train object segmentation network using single point clouds with full annotations, denoted as OGC All implementation details are in Appendix A.4.
Analysis: As shown in Table 1, our OGC surpasses the classical clustering based and motion segmentation methods by large margins on all metrics, showing the advantage of our method fully leveraging both the motion patterns and various types of geometry consistency. Compared with the fully supervised baselines, our method only inferior to the strong MBS [27] and OGC However, observe that OGC actually shows a higher precision score than OGCsups primarily because our method tends to learn better objectness thanks to a combination of motion pattern and smoothness constraints and avoid dividing single object into pieces. Figure 3 shows qualitative results.
4.2 Evaluation on OGC-DR OGC-DRS- Datasets
We further evaluate our method to segment objects in indoor 3D scenes. Considering that the existing dataset BlyingTTings33 [47] tends to have unrealistically cluttered scenes with severely fragmented objects and is originally introduced for scene flow estimation, turn synthesize a new dynamic room dataset, called OGC-DR, that suits both scene flow estimation and object segmentation. In particular, follow [54] to randomly place 4 ~ 8 objects belonging to 7 classes of ShapeNet [7] (chair, table, lamp, sofa, cabinet, bench, display into each room. In total, we create 3750, 250, and 1000 indoor rooms (scenes) for vraining/validationn/
splits. each scene, we create rigid dynamics by applying continuous random transformations to each object and record 4 sequential frames for evaluation. Each point cloud frame is downsampled to 2048 points. Note that, we follow [13) to split different object instances for train/val/ten sets.

---

Table 2: Quantitative results of our method and baselines on our OGC-DR/OGC-DRSV dataset.
APM
PQT
Fl1
Pret
Reet
mloUf
RIF
Supervised Mothod
OGC,up
90.7 86.3
82.67 78.8
87.6/85.0
83.7/8222
92.0/ 88.0
89.2/83.9
97.7
Unsupervised
TrijAffi [52]
42.6/
39.3
46.7/43.8
57.8/548
69.6/630
49.4/8844
46.8/55.9
80. 80.1/77.7
Motion Sogmentation
SSC [51]
74.57 70.3
79.2/75.4
842/8155
92.
77.3/74.7
91.5/91. 74.67708
WardL [30]
12.57 69.8
"/T0771666
323 7305
535/513
13.67 71.
543/995 66.766722
Unsupervised
Methoos
DBSCAN [17)
73.9/71.9
76.0/ 76.3
81.6/888
85.8/79.1
77.8/84.8
74.7 80.1
91.5/93.5
OGC(Ours)
92.3/ 86.8
85.1/77.0
89.4/8399
85.6/77.7
93.6/91.2
90.8/ 84.8
97.8/ /9.4
Based on OGC-DR dataset, we collect single depth scans every time step on the mesh models to generate another dataset, called Single- View OGC-DR (OGC-DRSV). All object point clouds in OGC-DRSV are severely incomplete due to self- and/or mutual occlusions, resulting in the new dataset significantly more challenging than OGC-DE DR. Each point cloud frame in OGC-DRSV is also downsampled 2048 points. More details of these datasets are in Appendix A.3.
Analysis: As shown in Table 2, our method outperforms all classical unsupervised methods including the clustering based and the motion segmentation based methods on OGC-DR. Since the synthetic rooms in OGC-DR all have complete 3D objects, and the generated point cloud sequences are of high quality. Therefore, our OGC even surpasses the supervised OGC This shows that the rigid dynamic motions can indeed provide sufficient supervision signals to identify objects. On OGC- DRSV, method still achieves superior performance and demonstrates robustness to incomplete point clouds, although the scores are slightly lower than that on the full point cloud dataset OGC-DR (AP: 86.8 92.3). Figure 3 shows qualitative results.
4.3 Evaluation on KITTI Scene Flow Dataset
We additionally evaluate our method on the challenging real-world outdoor KITTI Scene Flow (KITTI-SF) dataset. Officially, KITTI-SF dataset [48] consists of 200 (training) pairs of point clouds from real-world traffic scenes and an online hidden test for scene flow estimation. In our experiment, we train our pipeline the first 100 pairs of point clouds, and then test on the remaining 100 pairs (200 single point clouds). We observe that in the 100 training pairs, the moving objects are only cars and trucks. Therefore, in the testing phase, we only keep the human annotations [1] of cars and trucks in every single frame to compute the scores. All other objects are treated as part of background. Note that, the whole background is not ignored, but counted one object in our evaluation, and the cars and trucks can be static or moving. We find KITTI-SF is too challeging for the classical unsupervised methods, due the extreme imbalance of 3D points between objects and background. Besides, the background and objects in KITTI-SF are always connected because of the Earth's 's gravity, while cestering--aseed WardLinkage and DBSCAN favor spatially separated objects. Therefore, we leverage the prior about ground planes in KITTI-SF to assist these methods. We detect and specially handle the ground planes, leaving above-ground points only for these methods to handle. Implementation details are in Appendix A.4.
As shown in Table 3, our method obtains superior segmentation scores on the KITTI- SF dataset, being very close to our fully-supervised counterpart OGC excellence of our method on real-world scenes. Figure shows qualitative results.
Analysis:
sup. This demonstrates the
Table 3: Quantitative results of our method and baselines on the KITTI-SF dataset.
APt
PQt
Flf
Pret
Reet
mloUt
RIt
Supervised Method
OGC snp
62.4
52.7
65.1
63.4
67.0
67.3
95.0
Unsupervised
TrajAffn [52]
24.0
30.2
43.2
37.6
50.8
48.1
58.5
Motion Segmentation
SSC [51]
12.5
20.4
28.4
22.8
37.6
41.5
48.9
WardLinkage [30]
25.0
16.3
22.9
13.7
69.8
66.5
"44.9
Unsupervised
DBSCAN [17]
13.4
22.8
32.6
26.7
42.0
42.6
55.3
Methods
OGC(Ours)
54.4
42.4
52.4
47.3
58.8
63.7
93.6
4.4 Generalization to KITTI Detection and SemanticKITTI Datasets
Given our well trained model on KITTI-SF in Section 4.3, we directly test on the popular KITTI 3D Object Detection (KITTI-Det) [19] and SemanticKITTI [5] benchmarks. Unlike the saereo-based
7

---

point clouds in KITTI-SF, point clouds in these two datasets are collected by LiDAR sensors and thus more sparse.
KITTI-Det officially has 3712 point cloud frames for training, 3769 for validation. We only keep the ground truth object masks obtained from bounding boxes for the car category in each frame. All
other objects are treated as part of background. For comparison, we download the official pretrained models of three fully-supervised methods PointRCNN [59], PV-RCNN [58] and VoxeI-RCNN [14] to directly test on the validation split using the same settings as ours. In addition, we use the well trained sup on KITTI-SF to directly test for comparison, denoted as OGC* sup. We also train OGC the training split (3712 frames) from scratch and test it on the remaining 3769 frames using the same evaluation settings, denoted as OGC.up-
SemanticKITTT officially has 11 sequences with annotations for training and another 11 sequences for online hidden test. We only keep ground truth objects of car and truck categories. The total 11 training sequences (23201 point cloud frames) are used for testing. Compared with KITTI-Det, SemanticKIITI holds 6 more testing frames and covers more diverse scenes. Following the official split in [5], we also report the results on: i) sequences 00~~07 and 09~10 (19130 frames), and ii) the sequence 08 (4071 frames) separately.
Table 4: Quantitative results on KITTI-Det (* denotes the model trained on KITTI-SF).
AP1
PQf
F1
Pret
Rect
mloUt
RIt
PointRCNN [59]
95.7
80.1
88.9
81.3
98.0
91.4
97.2
PV-RCNN [58]
95.4
77.3
84.4
73.7
98.8
92.7
97.1
Supervised Methods
VoXeI-RCNN [14]
95.8
79.6
87.3
78.1
98.9
92.6
97.3
OGCaup sup
80.0
68.5
78.3
72.7
84.8
84.0
96.9
OGC* sup
51.4
41.0
49.1
43.7
56.0
66.2
91.0
Unsupervised Method
(Grr()
40.5
30.9
37.0
30.8
46.5
60.6
86.4
Analysis: As shown in Tables 4&5, our method can directly generalize 3D object seg- mentation on sparse LiDAR point clouds with satisfactory results, also being close to the fupey-sueer counter- part OGC* sup- It is under- standable that the other three cully-supervised models have a clear advantage over ours on KITTI-Det, because they are fully supervised and trained on the KITTI-Det training split (3712 frames) while ours does not. We hope that our method can serve the first baseline and inspire more advanced unsupervised methods in the future to close the gap. Figure 3 shows qualitative results.
Table 5: Quantitative results on SemanticKITTI (* denotes the model
trained on KITTI-SF).
Sequences
Methods APt PQ1 FIf Pre Reet mloUt RI1
OGC*
53.8 41.3 48.1 40.1 60.0
68.3
90.0
00~~10
(GCr() 42.6 30.2 35,3 28.2
47.3
60.3
86.0
00~07
OGC sup 55.3 41.8 48.4 40.1
61.1
69.9
90.3
09~10
(Gur)) 43.6 30.5 35.5 28.1
48.2
62.1
86.3
OGC sup 49.4 39.2 46.6 40.0
55.8
60.3
88.3
08
(Gurs) 38.6 29.1 34.7 28.6 44.0
51.8
84.3
4.5
Ablation Study
(1) Geometry Consistency Losses: To validate the choice of our design, we firstly conduct three groups of ablative experiments on the SAPIEN dataset [75]: 1) only remove the dynamic rigid loss Edynomie. 2) only remove the smooth- ness loss Esmooth and 3) only remove the invariance loss Lineariant- As shown in Table 6, combining the proposed three losses together gives the highest segmentation scores. Basi- cally, the dynamic rigid loss serves to discriminate multiple objects from different motion patterns. Without it, the network tends to assign all points to a single object as the shortcut to minimize the other two losses. However, we observe in Table 6 that without Edgnamics the network still works to some extend. This because the synthetic SAPIEN dataset tends to have a number of point cloud frames with only 2 or 3 objects, thus assigning all points to single object can still get plausible
Table 6: Ablation studies about loss designs on SAPIEN.
AP1 PQT F1f Pret Rect
mloUt RI
w/o ddgnamic
35,4 35.3 54.1 91.1
38.5
28.6
52.7
w/o Esmooth
21.8 18.5 26.9 19.1 45.4
52.4
63.7
w/o E Lineariant
48.9 46.1 61.3 61.9
60.7
57.9
70.3
Full OGC 55.6 50.6 65.1 65.0
65.2
60.9
73.4
8

---

scores, This issue is further validated by conducting additional ablation experiments on curated SAPIEN dataset. More details are in Appendix A.5.
In addition, we evaluate the robustness of our object segmentation method with regard to different types of motion estimations, and different hyperparameter and design choices of our smoothness loss Esmooth More results are in Appendix A.5.
(2) Iterative Optimization Algorithm: We also conduct ablative experiments to validate the effectiveness of our proposed Algorithm/ 1. We set the number of iter- ative rounds R as 21. As shown in Table 7, after 2 rounds, satisfactory segmentation results
Table 7: Iterative optimization on SAPIEN.
Object Segmentation
#R
AP1
PQ
FIf
Pret
Rect
mloUt
RI
1
45.9
47.7
62.3
60.2
64.5
60.2
72.3
2
55.6
50.6
65.1
65.0
65.2
60.9
73.4
be achieved, although we expect better results after more rounds with longer training time. indeed fully leverage the mutual benefits between object segmentation and motion estimation.
3
56.3
50.7
65.4
65.1
65.8
61.1
73.7
This shows that our iterative optimization algorithm can
4.6 Pushing the Boundaries of Unsupervised Scene Flow Estimation and Segmentation
In addition to the improvement of ob- ject segmentation from our iterative op- timization algorithm, the scene flow esti- mation can be naturally further improved from estimated object masks as well. Given our well trained model on the KITTI-SF dataset in Section 4.3, we use the estimated object masks to further im- prove the scene flow estimation. As shown in Table 8, following the exact evaluation settings of FlowStep3D [34], our method, not surprisingly, significantly boosts the scene flow accuracy, surpassing the state-of-the-un unsupervised FlowStep3D [34] and other baselines in all metries.
Table & Scene flow estimation on the KITTI-SF dataset.
EPE3D4 AccSt
AccRt Outlier.]
Ego-motion [66] 41.54
22.09
37.21
80.96
PointPWC-Net [74] 25.49
23.79
49.57
68.63
FlowStep3D [34] 10.21
70.80
83.94
24.56
OGC(Ours) 6.72
80.16
89.08
22.56
In fact, our object segmentation back- bone is also flexible to take the scene flow as input instead of point zyz to seg- ment objects. This is commonly called motion segmentation. We replace the single point clouds by (estimated) scene flow vectors as our network inputs, and train the network from scratch using the same settings on the KITTI-SF dataset. As shown in Table 9, we can see that our network can still achieve superior results regardless of the modality of inputs, demonstrating the generality of our framework.
Table 9: Motion ws Points based segmentation on KITT-sF
Input APt PQf F11
Pret
Rect
mloU
RIt
scene flow 47.3 41.2 50.2
50.9
49.6
56.0
89.3
point cloud 54.4 42.4 52.4
47.3
58.8
63.7
93.6
5 Conclusion
In this paper, we demonstrate for the first time that 3D objects can be accurately segmented using an unsupervised method from raw point clouds. Unlike the existing approaches which usually rely on a large amount of human annotations of every 3D object for training networks, we instead turn to leverage the diverse motion patterns over sequential point clouds supervision signals to automatically discover the objectness from single point clouds. A series of loss functions are designed to preserve the object geometry consistency spatial and temporal scales. Extensive experiments over multiple datasets including the extremely challenging outdoor scenes demonstrate the effecttivness of our method.
Broader Impact: The proposed OGC learns 3D objects from raw point clouds without requiring human annotations for supervision. We showcase the effectiveness for some basic applications includ- ing object part instance segmentation, indoor object segmentation and outdoor vehicle identification. We also believe that our method can be general for other domains such as AR/VR.
Acknovledgements: This work was partially supported by Shenzhen Science and Technology Innovation Commission (JCY1220103321120003)

---

ThiAffn
SSC
WardLinkage
DBSCAN
OGCsup
OGC (GC(Ours)
GT
SAPIEN
OGC-DR
3GC-DRSV DRSV
KITTI-S
TrejAffn
WardI inkage
OGCcup
GT
SSC
DBSCAN
OGC (Ours)
GT
KITTI-Dtt
OGC(Ours)
GT
SenanticrrT1
OGC(Our))
Figure 3: Qualitative results on various datasets. More qualitative results can be found in Appendix
A.7 and video demo: https: / //tp//yytuuuuu/ be/dZBjvKWJ4K
10

---

References
[1] H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother. Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes. ICV, 126:961-972, 2018. 7
[2] K. S. Arun, S. Huang, and S. D. Blostein. Least-Squares Fitting of Two 3-D Point Sets. TPAMI, (5)6998
700, 1987. 5, 17, 19
[3] S. A. Baur, D. J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger. SLIM: Self Supervisp LiDAR Scene Flow and Motion Segmentation. ICCV, 2021. 1,3
[4] A. Behl, D. Paschalidou, S. Donné, and A. Geiger. PointFlowNet: Learning Representations for 3D Scene
Flow Estimation from Point Clouds. CVPR, 2019.
[5] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. ICCV, 2019. 5,7,8
[6] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. MONet: Unsupervised Scene Decomposition and Representation. arXiv: /900:/130.,,0,,,,, 2019.
[7] A. X. Chang, T. Funkhouser, I.. Guihas, P. Hanrahan, Q. Huang, Z. Li, Savarese, M. Savva, S. Song, H. Su, J. Xiao, I.. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. arXiv, 2015. 6
[8] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia. Multi- View 3D Object Detection Network for Autonomous Driving. CVPR, 2017.
191 B. Cheng, A. G. Schwing, and A. Kirillov. Per-Pixel Classification is Not All You Need for Semantic Segmentation. NeurIPS, 2021. 3
[10] J. Cheng, Y.-H. Tsai, S. Wang, and M.-H. Yang. SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. ICCV, 2017. 3
[11] J. H. Cho, U. Mall, K. Bala, and B. Hariharan. PICIE: Unsupervised Semantic Segmentation using Invariance and Equivariance Clustering. CVPR, 2021. 5
[12] C. Choy, J. Gwak, and S. Savarese. 4D Temporll ConvNets: Minkowski Convolutional Neural Networks. CVPR, 2019. 3
[13] C. B. Choy. D. Xu, J. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV, 2016.
[14] J. Deng, S. Shi, P. Li, W. Zhou, Y. Zhang, and H. Li. Voxel R-CNNN: Towards High Performance Voxel-based 3D Object Detection. AAAI, 2021. 8
[15] D. Ding, F. Hill, A. Santoro, M. Reynolds, and M. Botvinick, Attention over learned object embeddings enables complex visual reasoning. NeurlPS, 2021.
[16] S. M. A. Eslami, N. Heess, T. Weber, Y. Tassa, K. Kavukcuoglu, and G. E. Hinton. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. NIPS, 2016.
[17] M. Ester, H.-P. Kriegel. J. Sander, and X. Xu. A density-basee algorithm for discovering clusters in large spatial databases with noise. KDD, 1996. 6, 7, 19
[18] D. Fortun, P. Bouthemy, and C. Kervrann. Optical flow modeling and computation: A survey. CVIU, 134(1-21), 2015. 2015. 3
[19] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. CVPR, 2012. 5.7
[20] Z. Gojeic. O. Litany, A. Wieser, L. J. Guibas, and T. Birdal. Weakly Supervised Learning of Rigid 3D Scene Flow. CVPR, 2021. 2.4
[21] Z. Gojcic, C. Zhou, J. D. Wegner, L. J. Guibas, and T. Birdal. Learning multiview 3D point cloud registration. CVPR, 2020. 4, 17
[22] B. Graham, M. Engelcke, and L. van der Maaten. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR, 2018. 3
[23] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran. L. Matthey, M. Botvinick, and A. Lerchner. Multi-object representation learning with iterative variational inference. ICML, 2019. 3

---

[24] X. Gu. Y. Wang, C. Wu, J. Lee, and P. Wang. HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds. CVPR, 2019. 3
[25] J. T. Hsieh, B. Liu, D. A. Huang, L. Fei-Fei, and J. C. Niebles, Learning to decompose and disentangle representations for video prediction. NIPS, 2018. 3
[26] Q. Hu, B. Yang. L. Xic, S. Rosa, Y. Guo, Z. Wang. N. Trigoni, and A. Markham. RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds. CVPR, 2020.
[27] J. Huang. H. Wang. T. Birdal, M. Sung, F. Arrigoni, S. M. Hu, and L. Guibas. MulriBodySSnc: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization. CVPR, 2021. 3, 6
[28] C. Jiang, D. P. Paudel, D. Fofi, Y. Fougerolle, and C. Demonceaux. Moving Object Detection by 3D Flow Field Analysis. 777S, 1254)1195--196,, 2021.
[29] J. Jiang, S. Janghorbani, G. D. Melo, and S. Ahn. SCALOR: Generative World Models with Scalable Object Representations. JCLR, 2020. 3
[30] J. H. W. Hierarchical Grouping to Optimize an Objective Function, Journal of the American Statistical Association, 58:236-244 1963. 6, 7, 19
[31] W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32(5):922-922 1976. 4, 17
[32] T. Kipf, G. F. Elsayed. A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff. Conditional Object-Centric Learning from Video. ICLR, 2022. 3
[33] A. Kirillov, K. He, R. B. Girshick, C. Rother, and P. Dollár. Panoptic segmentation. CVPR, 2019.
[34] Y. Kittenplon, Y. C. Eldar, and Raviv, FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation. CVPR, 2021. 2, 3, 4, 5, 9, 16, 17
[35] A. R. Kosiorek, H. Kim, I. Posner, and Y. W. Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. NeurlPS, 2018. 3
[36] H. W. Kuhn. The Hungarian Method for the assignment problem. Naval Research Logistics Quarterly, 211-2):3399, 1955. 5, 17
[37] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. PointPillars: Fast Encoders for Object Delection from Point Clouds. CVPR, 2019. 1,3
[38] B. Li, T. Zhang, and T. Xia. Vehicle Detection from 3D Lidar Using Fully Convolutional Network. RSS, 2016. 3
[39] R. Li, G. Lin, and L. Xie. Self PPint-FFoww Self Self-Super Scene Flow Estimation from Point Clouds with Optimal Transport and Random Walk. CVPR, 2021. 3
[40] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. ECCV, 2014.
[41] R. Liu, Z. Wu, S. X. Yu, and S. Lin. The Emergence of Objectness: Learning Sero-Shot Segmentation from Videos. NeurlPS, 2021. 3
[42] X. Liu, C. R. Qi, and L. Guibas. FlowNer3D: Learning Scene Flow in 3D Point Clouds. CVPR, 2019. 3,
[43] X. Liu, M. Yan, and Bohg. MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences. ICCV, 2019. 3,6
[44] F. Locatello, D. Weissenhorn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-Centric Learning with Slot Attention. NeurIPS, 2020. 3
[45] X. Lu, W. Wang, J. Shen, Y.W. Tai, J. Crandall, and S. C. H. Hoi. Learning Video Object Segmentation From Unlabeled Videos. CVPR, 2020. 3
[46] C. Luo, X. Yang, and A. Yuille. Self-Supervised Pillar Motion Learning for Autonomous Driving. CVPR, 2021. 3
[47] N. Mayer, E. 1lg. H. Philip, D. Cremers, A. Dosovitskiy, and T. Brox. A Large Dataset to Train Convolu- tional Networks for Disparity, Optical Flow. and Scene Flow Estimation. CVPR, 2016. 6
12

---

[48] M. Menze, A. Geiger, and T. Mpi. Object Scene Flow for Autonomous Vehicles, CVPR, 2015. 5.7
[49] M. Minderer, C. Sun, R. Villegas, F. Cole, K. Murphy, and H. Lee. Unsupervised Learning of Object Structure and Dynamics from Videos. NewrIPS, 2019. 3
[50] H. Mittal, B. Okorn, and D. Held. Just go with the flow: Self-supervised scene flow estimation. CVPR, 2020. 3
[51] U. M. Nunes and Y. Demiris. 3D motion segmentation of articulated rigid bodies based on RGB-D data. BMVC, 2018. 5, 6, 7, 19
[52] P. Ochs, J. Malik, and T. Brox. Segmentation of moving objects by long term video analysis. TPAMI, 16(6):1187-1200)
2014. 6, 7, 19
[53] B. Ouyang and D. Raviv. Occlusion guided scene flow estimation on 3D point clouds. CVPR, 2021.
[54] S. Peng. M. Niemeyer, I.. Mescheder, M. Pollefeys, and A. Geiger. Convolutional Occupancy Networks. RCCV, 2020. 6,
[55] G. Puy, A. Boulch, and R. Marlet. FLOT: Scene Flow Point Clouds Guided by Optimal Transport. ECCV, 2020. 3
[56] C. R. Qi. H. Su, K. Mo. and L. J. Guibas. PointNet: Deep Learning Point Sets for 3D Classification and Segmentation. CVPR, 2017.
[57] C. R. Qi. L. Yi, H. Su, and L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. NIPS, 2017. 2, 3, S, 6,
[58] S. Shi. C. Guo, L. Jiang, Z. Wang. J. Shi, X. Wang, and H. Li. PV-RCNN: Point- Voxel Feature Set Abstraction for 3D Object Detection. CVPR, 2019, 8
[59] S. Shi, X. Wang, and H. Li. PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. CVPR, 2019. 8
[60] W. Shi, Ragunathan, and Rajkumar. Point-GNN: Graph Neural Network for 3D Object Detection in Point Cloud. CVPR, 2020.
[61] D. Sun, E. B. Sudderth, and M. J. Black. Layered image motion with explicit occlusions, temporal consistency, and depth ordering. NIPS, 2010. 3
[62] D. Sun, F. B. Sudderth, and M. J. Black. Layered segmentation and optical flow estimation over time. CVPR, 2012.
[63] J. Sun, Y. Dai, X. Zhang. J. Xu, R. Ai, W. Gu, and X. Chen. Efficient Spatial-Temporal Information Fusion for LiDAR Based 3D Moving Object Segmentation. IROS, 2022.
[64] H. Thomas, B. Agro, M. Gridseth, J. Zhang, and T. D. Barfoot Self-Supervised Learning of Lidar Segmentation for Autonomous Indoor Navigation. ICRA, 2021. 3
[65] H. Thomas. C. R. Qi. J. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas. KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV, 2019. 3
[66] 1. Tishehenko, S. Lombardi, M. R. Oswald, and M. Pollefeys. Self self-supervised learning of non-rigid residual flow and ego-motion. 3DV, 2020.
[67] A. Vaswani, N. Shazcer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. NIPS, 2017. 3, 16
[68] T. Vu, K. Kim, T. M. Luu, X. T. Nguyen, and C. D. Yoo. SoftGroup for 3D Instance Segmentation on Point Clouds. CVPR, 2022. 3
[69] J. Wagemans, H. Elder, M. Kubovy, S. E. Palmer, M. A. Peterson, M. Singh, and R. von der Heydt. A
century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bullerin, 138(6):1172 1217, 2012. 1
[70] W. Wang, R. Yu, Q. Huang, and U. Neumann. SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation. CVPR, 2018. 1,3
[71] Z. Wang. C. Shen, R. Li, C. Zhang. and G. Lin. RigidFlow: Self Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior. CVPR, 2022.
13

---

[72] Y. Wei, Z. Wang. Y. Rao, J. Lu, and J. Zhou. PV-RAFT: Point-Voxxe Voxel Correlation Fields for Scene Flow Estimation of Point Clouds. CVPR, 2021. 3
[73] M. Wertheimer. Untersachungen zur Lehre you der Gestalt. Psychologische Forschung, (4):301-350, 1923.
[74] W. Wu, Z. Y. Wang, Z. Li, W. Liu, and L. Fuxin. PoinIPWC-Net: Cost Volume on Point Clouds for (Self-JSupervised Scene Flow Estimation. ECCV, 2020. 3,9
[75] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. 7hu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang. L. Yi, A. X. Chang,
L. J. Guibas, and H. Su. SAPIEN: A SimulAted Part-based Interactive ENvironment. CVPR, 2020. 5, 6.
[76] B. Yang. J. Wang. R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni. Learning Object Bounding Boxes for 3D Instance Segmentation Point Clouds. NewrIPS, 2019. 1,3
[77] C. Yang, H. Lamdouar, B. Lu, A. Zisserman, and W. Xic. Self-supervised Video Object Segmentation by Motion Grouping. ICCV, 2021. 2,
[78] Y. Yang, A. Loquercio, D. Scaramuzza, and Soatto. Unsupervised moving object detection via contextual
information separation. CVPR, 2019. 2
[79] Y. Yang and B. Yang. Promising Elusive? Unsupervised Object Segmentation from Real-world Single Images. arXiv:2210002344 2022. 3
[80] L. Yi, H. Huang, D. Liu, F. Kalogerakis, H. Su, and L.. Guibas. Deep part induction from articulated object pairs. SIGGRAPH Asia, 37(6), 2018. 3, 6
[81] L. Yi, V. G. Kim, D. Ceylan, Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, and L. J. Guibas. scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph., 2016. 6
[82] Y. Zeng, Y. Qian, Z. Zhu, J. Hou, H. Yuan, and Y. He. CorrNet3D: Unsupervised End-to-end Learning of Dense Correspondence for 3D Point Clouds. CVPR, 2021.
[83] M. Zhai, X. Xiang, N. Lv, and X. Kong. Optical Flow and Scene Flow Estimation: Survey. Pattern Recognition, 2021.
[84] Y. Zhou and O. Tuzel. VoxeINet: End-to-Bin fedd Learning for Point Cloud Based 3D Object Detection. CVPR, 2018.
[85] D. Zoran, R. Kabra, and D. J. Rezende. PARTS: Unsupervised segmentation with slots attention and independence maximization. ICCV, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix A.6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.4
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because it would be too computationally expensive
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "UPO-QTUtmeT", "reviewer": "Reviewer_Qxwd", "summary": "This paper introduced a self-supervised approach to do instance segmentation on point clouds. \nThe key contribution is a set of auxiliary losses to ensure segmentation consistency under per-object rigid transformation and view-variance.\nThe paper showed large improvement over other unsupervised approaches, which are based on clustering and motion segmentation, and in some cases, the proposed method performs competitively compared to supervised baselines.\n\n\n---\n\n*I thank the authors for the answer and in particular I appreciate the added experiments. Final recommendation: While I think the proposed method has large potential for improvement, it's relevant to the community as it is the first paper for unsupervised detection. **I maintain my original score.** However the response of the authors provided renewed insights, shedding light to limitations the actual usability due to high training time and the limited generalisability of the learned features. I think it's only fair if the authors mention these in the main paper and link the experiment results to the appendix.*", "questions": "- Are $T_k$ and $O_k^t$ jointly optimized? \n- What is the training time required to perform the iterative optimization? How does it compare to the supervised version? \n- It seems like neither MaskFormer and SceneFlow nets are pretrained. Why? For each dataset, do you need to train these nets from scratch? Do the trained nets to transfer well to another dataset?\n- Is the dynamic loss able to recover small objects misclassified as part of another larger object or background? I'm concerned that the estimation $T_k$ will be prone to fit the larger object and ignore the smaller one. This may likely make the segmentation mask fit tighter to the larger object. But I don't see any incentive to segment the smaller object as a new object, say in $o_{n,k'}$", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- the first unsupervised method to perform instance segmentation on point clouds (although I'm not a domain expert and couldn't really verify this claim) \n- thorough experiments and ablation study on various datasets\n- a novel iterative optimization procedure to improve the segmentation mask via improving scene flow\n\n**Weakness**\n- The method seems only to be able to detect rigid moving objects, although L155-156 suggests that the geometry invariance loss can also help detect static objects. In general, the object segmentation network doesn't seem to have any fundamental issues in predicting masks for static objects. The dynamic loss also does not penalize static objects (e.g. $M^t = 0, T_k = I$). It would be good if the authors can further clarify the limitation of the proposed method in segmenting static objects.\n- I'm not sure the geometry smooth term is well designed. For one, it could lead to blurry prediction boundaries? To me, it makes sense to weigh the smoothness term based on the predicted $T_k$. \n- The bold font in a mixed table of supervised and unsupervised comparison is a bit confusing, especially in Table 4, all other methods seem to be better than OGC but OGC is marked bold. \n- Also regarding table formatting, it would be good to indicate whether the metric is larger the better or smaller the better, e.g. with $\\uparrow$ and $\\downarrow$\n- One interesting benefit of unsupervised training is that it can be used as a pertaining procedure to boost performance when the data annotation is sparse or for transferring to another domain. The others have shown domain transfer to KITTI-Det, but the baselines are all supervised methods. What's missing is to demonstrate the advantage of the OGC(unsup) as a pertaining step for semi-supervised learning with partial training data, preferably doing so while comparing other unsupervised methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OZce4svjOwx", "reviewer": "Reviewer_dUuM", "summary": "This paper proposed a 3D object segmentation method that can be trained in an unsupervised fashion. The core underlying assumption is the rigid motion dynamics and shape invariance between point cloud sequences. By carefully designing geometry consistency losses, supervision signals can be effectively extracted from a pair of point clouds. The paper has demonstrated promising results on multiple different datasets and scenarios.", "questions": "1) Motion rigidity is an excellent clue for learning object segmentation. However, in certain datasets, or for certain sensor modality, the actual shape invariance and rigid dynamic might not hold in the sensory data. For example, rolling shutter effect is a very well known effect that distorts high speed moving objects captured by a rolling shutter camera. For point clouds captured by mechanical lidar, motion distortion caused by scene flow, especially high speed moving objects, such as vehicles coming from the opposite direction, is usually very obvious. This paper has made great progress in leveraging object rigidity, but it would be interesting to see, how different degrees of motions can affect the proposed method. Especially, for cases like mechanical lidar point cloud, if motion distortion can affect the proposed method.", "limitations": "1) The proposed method relies on the object rigidity for object segmentation. But there are objects, such as articulated buses, semi-truck with trailers, that changes shapes during their movement. The proposed method might not be able to handle such cases well.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) This paper carefully designed multiple loss functions, which powers the unsupervised learning of object segmentation. Removing the need for large scale human annotation is a critical step towards making such method applicable in different new fields and settings.\n2) This paper demonstrated great results on multiple datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Jr1XjfC81x4", "reviewer": "Reviewer_eGGN", "summary": "This paper studies the problem of unsupervised 3D object segmentation from a point cloud. In particular, the proposed approach learns the scene flow in a self-supervised manner which is used as supervision signal and enforces the object geometric consistency for segmenting the static objects. The proposed unsupervised approach is evaluated on various synthetic and real dataset to demonstrate the performance.", "questions": "1)\tHow is the MaskFormer [8] trained? Is a pre-trained model applied to obtain the coarse object mask?\n2)\tPlease analyse Influence of the smoothness parameter H for over segmentation.\n3)\tCould the method handle incomplete point cloud? \n4)\tIt would be great to show the proposed method can handle static objects with qualitative results.\n\n5)\tBased on the reviewer’s understanding, the proposed approach mainly handle foreground objects without road. A large number of points belongs to floor or road which can be easily removed by plane fitting. The baseline clustering results should become much better. \n", "limitations": "The paper has discussed limitations of the proposed method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1)\tUnsupervised 3D object segmentation from point cloud is overlooked in the computer vision field. The paper proposed a solution to solve the problem.\n2)\tPoint cloud motion is applied as supervision signal for 3D object point segmentation. The motion consistency indeed provides strong information for 3D object point segmentation.\n3)\tThe geometry invariance loss is applied to encourage the segmentation of static objects in the scene.\n4)  The method is evaluated on both synthetic and real datasets.\n\n# Weaknesses\n1)\tThe training loss quality will be influenced by the self-supervised scene flow estimation approach.\n2)\tThe proposed method may fail if two objects are close with similar motion. \n3)\tThere are still over-segmentation issues in the results.\n4)   The proposed method can only handle objects from a finite dataset which hinders its generalisation ability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Hso6iEZsXVm", "reviewer": "Reviewer_8Drm", "summary": "The paper proposes a pipeline of 3D object segmentation from point clouds. The key idea is to add several regularization losses to enforce geometry consistency on object motion. The proposed method is evaluated on datasets for object part instance segmentation, general object segmentation, and scene flow estimation, where it achieves competitive performance.", "questions": "The reviewer was concerned that the authors made a wild claim “we introduce the first unsupervised multi-object segmentation pipeline on 3D point clouds, without needing any human annotations in training” as this has been previously explored in the scene flow community.", "limitations": "The evaluation of the SAPIEN and OGC-DR for this paper seems to be limited as these toy-like datasets have zero background and well-separated objects. The reviewer would expect more exploration of more complicated LIDAR datasets while the paper is currently limited on a small KITTI-SF dataset.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: The proposed pipeline is a novel combination of existing techniques. Only limited work has been proposed to address the problem of unsupervised 3D object segmentation. This paper leverages self-supervised scene flow estimation models to obtain initial motion estimation from a pair of point clouds and a neural network to obtain initial object masks, which are then used to supervise the learning of the object segmentation backbone, via introducing regularization techniques to enforce rigid transformation on objects, local smoothness, and invariance on semantic predictions. Each subcomponent by itself is not new, but the reviewer gives the credit to the combination.\n\n**Quality**: The submission is not technically sound. Segmenting 3D objects from the dynamics of point clouds is not a new concept, which has been explored in the scene flow estimation community as the motion segmentation task, e.g., experiments in FlowNet3D [1]. While in this submission, the authors explicitly add an object mask model and combine its prediction with the scene flow prediction to obtain auxiliary supervision signals.  The authors identify the creation of auxiliary supervision signals using motion dynamics as the key challenge and push it as the key contribution. However, the resulting three losses, i.e., the multi-object dynamic rigid consistency loss,  the object shape smoothness prior loss, and the object shape invariance loss are well-known techniques in scene flow estimation and augmentation-based self-supervised learning. \n- The multi-object dynamic rigid consistency loss.  The concept has been explored in Rigid3DSceneFlow[2], where they add rigid-motion constraints for both ego-motion and object motion.\n- The object shape smoothness prior loss. It shares the same spirit of local smoothness losses that are widely used in the scene flow community, where the key idea is to have a similar motion prediction in nearby points. Here the prior loss is a minor modification by changing the motion prediction to the object mask prediction.\n- The object shape invariance loss is highly relevant to the unsupervised semantic segmentation pipeline. The authors resolve the binding problem, e.g., point-to-label, using the Hungarian algorithm between predictions from augmented versions of point clouds and then check the consistency between these predictions. The high-level concept comes from augmentation-based self-supervised learning, still, the proposed loss is somewhat novel.\n\nOverall, the reviewer considers the paper has limited novelty due to the above discussions.\n\n\n[1] FlowNet3D: Learning Scene Flow in 3D Point Clouds. CVPR 2019\n\n[2] Weakly Supervised Learning of Rigid 3D Scene Flow. CVPR 2021\n \n \n**Clarity:** The submission has some issues.\n\nIt is not clear to the reviewer on the object segmentation network. Is it a supervised model? How do the authors obtain the initial multiple object masks without any human annotations? The authors mentioned the transformer for mask prediction. But it remains unclear to the reviewer how the transformer is trained. \n \nEquation (3) is a wrong notation to ignore the assignment. It didn’t make sense to directly associate two sets of object masks without a `permutation’.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept