Representation Learning for Object Detection from Unlabeled Point Cloud Sequences
Authors: Xiangru Huang
Yue Wang
Vitor Campagnolo Guizilini
Rares Andrei Ambrus
Adrien Gaidon
Justin Solomon
Venue: CoRL 2022 Poster (2022)
Keywords: Representation learning
object detection
point cloud sequences

Abstract: Although unlabeled 3D data is easy to collect, state-of-the-art machine learning techniques for 3D object detection still rely on difficult-to-obtain manual annotations. To reduce dependence on the expensive and error-prone process of manual labeling, we propose a technique for representation learning from unlabeled LiDAR point cloud sequences. Our key insight is that moving objects can be reliably detected from point cloud sequences without the need for human-labeled 3D bounding boxes. In a single LiDAR frame extracted from a sequence, the set of moving objects provides sufficient supervision for single-frame object detection. By designing appropriate pretext tasks, we learn point cloud features that generalize to both moving and static unseen objects. We apply these features to object detection, achieving strong performance on self-supervised representation learning and unsupervised object detection tasks. 

Representation Learning for Object Detection from Unlabeled Point Cloud Sequences
Xiangru Huang MIT CSAIL xrhuang @ essai.mmi.eed
Yue Wang" NVIDIA Research & MIT CSAIL
Vitor Guizilini
Toyota Research Institute (TRI)
yuewang@csail.mit.ed mit.du
vitor, sioorguizziliiie ienelliini@ttigggle grobblllbal
Rares Ambrus
Adrien Gaidon
Justin Solomon
Toyota Research Institute (TRI) rares anbeus @ triigllobal
Toyota Research Institute (TRI) adrien gaidon @tri.global
MIT CSAIL
jsolomon @ mit.edu
Abstract: Although unlabeled 3D data is easy to collect, state-of-the-art machine learning techniques for 3D object detection still rely on difficulttto-obbain manual annotations. To reduce dependence on the expensive and error-prone process manual labeling, we propose a technique for representation learning from unlabeled LiDAR point cloud sequences. Our key insight is that moving objects can be reliably detected from point cloud sequences without the need for human- labeled 3D bounding boxes. In single LiDAR frame extracted from a sequence, the set of moving objects provides sufficient supervision for single-frame object detection. By designing appropriate pretext tasks, learn point cloud features that generalize to both moving and static unseen objects. We apply these features object detection, achieving strong performance self-supervised representation learning and unsupervised object detection tasks. Code is available at https: //github. com/ xi
Keywords: Representation learning, object detection, point cloud sequences
Among the modalities used for object detection in autonomous driving, LiDAR point clouds capture accurate 3D scene structure, yielding state-of-the- performance [1, 2]. Yet, sparsity and irregularity make it difficult for models to generalize to complicated real-world environments. Moreover, object detection requires several tasks to be solved jointly, including foreground backurouud segmentation, instance segmentation, object localization, and classification. This results in a high demand for human labels of object locations, velocities, orientations, and other properties.
Unlike expensive human- generated labels, unlabeled point cloud sequences can easily be collected by autonomous vehicles with LiDAR sensors whenever they are on the road. These temporally- ordered sequences contain more information than individual frames. For example, we can estimate correspondences between adjacent point clouds. Recent works distill information learned from point cloud sequences to train single-frane models for motion estimation, object detection, prediction, and motion tracking [3, 4, 5, 7, 8, 9, 10, 11]. These works typically focus on sub-module relevant to object detection (e.g., scene flow), or require many labeled sequences to promote generalization.
We propose representation learning approach to learning features for object detection from unlabeled LiDAR point cloud sequences. Unlike previous works, ours is capable of learning features without 3D bounding box annotations, with competitive performance on object detection benchmarks among unsupervised and self-supervised approaches. To deliver generalization from limited labeled data, we use geometry processing techniques derive a pseudo-label generator. This generator ingests unlabeled point cloud sequences and produces annotations for pretext tasks like motion segmentation and moving object detection. Then, we use the generated annotations to pretrain a single-frame feature extractor that can be used for downstream tasks like object detection.
Our pseudo-label generator identifies moving objects as sequences of point clusters across frames (Figure 2). Unlike existing single-frame LiDAR segmentation methods that focus on point connec- tivity [12, 13, 14), estimate motion and planar structures from each sequence. This information
Work done while at MIT.
6th Conference on Robot I Learning (CoRI. 2022), Auckland, New Zealand.

---

object
Trezing
Parudo-task
Input plin sequence
Ticee classfication
Single-frame motion segmentation
inference
Trating daa
PacudottaskIi II:
Single object Figure I: Overview of our approach. Detected moving objected from sequences are fed into later self-supervised tasks.
enables us to separate out moving objects reliably, yielding highly-accurale moving object detection. By combining all detected point clusters in the sequence, we further estimate the location, orientation, and category of each moving object. To promote generalization and mitigate the bias of working exclusively with moving objects, we propagate estimated pseudo-labels to static objects following variant of [11]. The key difference of our approach is that we check the consistency of predicted object labels across frames to prune incorrect predictions, improving robustness of label propagation.
To summarize, our contributions are as follows:
An algorithm for the detection of moving objects from unlabeled point cloud sequences, which enables self-supervised representation learning for point cloud data.
Two representation learning pseudo-tasks leveraging our detected moving objects for the down- stream task of object detection, without requiring manually-annotated labels at any stage of training. Competitive self-supervised object detection accuracy and mate-of-the-art unsupervised object discovery accuracy.
Related Work
Point cloud self-supervised learning. Self-supervision is popular for representation learning from unlabeled data. For point clouds, several self-supervised pretext tasks have been proposed. Sauder and Sievers [15] split point clouds into voxels and reassemble shuffled voxels. Zhang al. [16] enforce consistency between point cloud and voxel representations. Xie et al. [17] synthesize transformed point clouds and enforce consistency. P4Contrast uses point-pixel pairs to learn complementary features from point clouds and images [18]. Liu et al. [19] include negative pairs in P4Contrast. Tian al. [20] learn to discover unseen objects from image and LiDAR clues. Our work is closely related to representation learning and unsupervised object discovery method, using relationships between adjacent frames to augment available information.
3D object detection. 3D object detection works can be categorized by data type, e.g. voxels [21], points [22, 23), range images [24, 25], and hybrid representations [26, 27, 28). Many works incorpo- rate remporally-adjacen tensor, using 3D convolutions for efficiency. Hu et [30] incorporate temporal voxel occupancy estimated from raycasting. Several models deal with spatio-temporal information, such as Trans- formers [31] and LSTMs [32]. Ku et al. [33], Chen al. [34], Xu et al. [35] combine modalities like point clouds and images into single framework. Frustum-Pointhie leverages 2D object detectors to form a frustum crop of points and then uses PointNet to aggregate features [36]. Beyond visual input, Yang et al. [37] use high-definition maps to boost performance of 3D object detectors. Liang et al. [38] argue that multi-tasking improves representations over single-tasking, Our method also closely related to [39], which learns perceptual model for unknown classes.
frames to improve detection. Luo et al. [29] stack frames into a sparse 4D
Learning from point cloud sequences. Point cloud sequence datasets have grown in scale and diversity, covering tasks like 3D reconstruction [40, 41]. semantic segmentation [42], and object detection [1, 2, 43]. Although sequences often yield better performance than single frames, they require new representations. Early works treated them 4D grids [44]. MeteorNet includes spatio- temporal neighborhoods for dynamic point cloud sequences [45]. CaSPR normalizes the temporal dimension [9], yielding time-continuous representation. Mersch et al. [5] convert point sequences

---

Car
Pedestrian
Cyclist
Box Count
4352210
2037627
49518
Moving Box Count
1161019
1427211
42411
Moving Trace Count
12496
13052
438
Velocity in (0.05, 2] m/s
850640
1044324
37051
Turning Angle < 3"
1153865
1379915
41546
Table 1: Waymo Open training dataset statistics [1] (798 sequences), including the number of (moving or all)
bounding boxes, the number unique moving objects the velocity and turning angle distribution all moving bounding boxes. We focus on the fraction of moving boxes that follow a smooth trajectory.
into stacked 2D range images, processed using 3D convolution. Qi al. [11] use motion and complementary views to help detect moving and static objects. Compared to previous work, our approach performs object detection without requiring annotated 3D bounding boxes.
2 Approach Overview
Our framework takes two main steps, summarized Figure 1: (1) extract a set of moving objects (defined below object traces) from temporally-ordered sequence of unlabeled point clouds; and (2) train a feature extraction module using these object traces. We do not require human-labeled 3D bounding boxes at any stage. Here, we introduce the concept of object traces and discuss why they are necessary for our approach. We then elaborate the key observations and challenges in each step.
Object traces. Point cloud segmentation generally involves finding instances of objects, represented by sets of point clusters }C+} where C, E RM, x3 In point cloud sequences, object instances appear interested in object traces, which are sequences
consistently in consecutive frames. Therefore, of point clusters across consecutive frames corresponding to the same object.
Formally, given a point cloud sequence P PN)) (PI...,
where P; the 1-th frame from LiDAR camera, object trace is a demporally-ordert of clusters = {Cr, C, where C. S P. for any 1 1 < N. Figure shows example.
sequence
Our key observation that many object classes relevant for autonomous driving are dynamic, including pedestrians, vehicles, and cyclists. Statistics from the Waymo dataset show that many ground truth object traces follow smooth trajectories (Table 1). Intuitively, the length and motion consistency of object traces reduce uncertainty in their detection, enabling our robust non-learning trace detection algorithms from unlabeled sequences. Detected traces can be used in subsequent pseudo-tasks.
Figure 2: An object trace is a sequence of point clusters moving along smooth trajectory. Here, we highlight an object trace of a moving vehicle.
Object trace detection. Detecting object traces is inevitably disturbed by interactions between objects and the environment and irregular motion and sampling density across frames. Even super- vised methods suffer from detection uncertainty. Here, we focus on objects moving along smooth trajectories. Our detector optimizes for detection precision instead of coverage. This choice only adds a mild bias to the detected objects since velocities are independent from geometric appearance.
Our object trace detection follows standard proposal-and-rreection framework. For each sequence, we propose candidate clusters corresponding movable objects. Then, Kalman filtering acquires object traces according to their motion. We collect subset of object traces with smooth trajectories.
Self-supervision. Given our high-quality detected object traces from unlabeled point cloud sequences, we design two self-supervised tasks that enable representation learning for point clouds.
The rest of our paper is organized as follows. $3 details our object trace detection algorithm. $4 introduces self- sule-vueeeeseed tasks using detected object traces. $5 shows results object trace detection and representation learning for object detection. $6 concludes and suggests future directions.
3 Object Trace Detection
Our object trace detection algorithm involves three steps: (1) preprocessing, (2) object cluster proposal, and (3) object trace tracking. We elaborate each step below.

---

(a) Input Sequence
Ground removal
(c) Claster proposal
(d) Trace detection
(c) Groun truth
Figure Object trace detection. In (a), we visualize the input point cloud sequence in world coordinates. Figures (b) to (d) show steps of trace detection. In (c), we highlight subset of proposed object clusters in red. In (d), we color each detected trace differently. (e), we visualize the ground truth, where vehicles are red and pedestrians Sequence preprocessing. We apply two preprocessing steps each point cloud sequence before passing it to our detection algorithm. First, we bring all frames into the world coordinate system using the provided ego-motion (Figure 1, left). Second, remove points the ground, which typically account for 50% to 90% of the points; this improves detection efficiency and robustness. Appendix A provides preprocessing details.
green.
Object cluster proposal. Since object sizes vary across classes, we use point cloud segmentation to extract point clusters corresponding to object instances. Unlike existing LiDAR point cloud segmentation methods, e.g. [13], we only extract segments corresponding to moving object instances.
Given a point cloud sequence, we use motion as cue for segmentation. We estimate the velocity 0; R of each point p; and compute pairwise proximity score 8ij via
IP:
12
|0; - u,
8ij -exp
where on 0.5 m and o, = 10 m/s. a, is chosen to be large to reduce variance in velocity estimation. We then run normalized cuts [46] with edge weights {sij) } and select clusters with average velocity > 0.1 m/s and belong to the same frame.
To estimate motion, adapt two state-of the-art art methods to point cloud sequences (details in $5). Our first variant "NeuralSF" follows et al. [47]. To accommodate point cloud sequences, we optimize for a spatiotemporal velocity field fo R IR3 mapping any point location E R and time to a velocity E R-3. We normalize (x. t) to j0, 1]4 and optimize network parameters 0 using chamfer distance between adjacent frames. Our second alternative "SSL-SceneFlow" follows Mittal et al. [48], who estimate scene flow between adjacent frames. They train using self-supervision on KITTI and nuScenes. We apply their pretrained model each pair of adjacent frames.
Object trace tracking. Assuming each point cluster rigidly transforms across frames, we adopt a multi-objeet tracking algorithm using Kalman filtering track cluster motion [49]. Specifically, use a Kalman filter to track the velocity of each object's ' center from a sequence of centers. For each cluster, if tracking succeeds, the output of tracking is a trace ($2). Appendix B provides details.
4 Self-Supervised Tasks
Given a set of object traces from moving object instances in LiDAR sequences, we design self- supervised tasks that yield point cloud features for downstream tasks like single- frame object detection.

---

Polosiia
Viddd
(a) Input object traces
(b) Trace Classification
(c) Box inference
Figure 4: Box inference. For visualization, (c) highlights the estimated bounding boxes.
Here we introduce two self-supervised tasks in $4.1 and $4.2, used to train a feature extractor with single-frame point cloud input. $4.3 provides details of our feature extraction module.
4.1 Task I: Motion Segmentation
We consider motion segmentation as the first self-supervised task. Since we extract traces with high precision, we use them segmentation masks to train a single-frame model differentiating moving objects from the background. We train pointwise classifier that classifies each feature vector as either moving non-moving, implemented as multi-layer perceptron (MLP) combining ReLU [50] and BatchNorm [51] layers beyond the architecture $4.3. Our loss negative log-likelihood.
To generalize from moving to static objects, we augment each scene with clusters corresponding moving objects from other scenes. particular, for each scene, we randomly choose 50 point clusters and place them at random locations.
point
4.2 Task I:: Object Detection
Task I is straightforward but fails all available information. For example, object orientations can be inferred from motion but are ignored. Here, we estimate 3D bounding boxes and class labels from each object trace and use this information to train a 3D object detector; same box regression component and regression/clasifccttion loss as Yin et al. [52]. Estimating 3D bounding boxes to train the object detector from traces involves several subproblems:
Figure We use the
Registration: We estimate the velocity of each object cluster in each trace while enforcing smoothness. This enables us to approximately reconstruct the object by incorporating geometry collected from multiple frames.
Trace classification: We categorize each object trace into one of the movable object classes. This is the only sub-pru brobleoblem that requires labeled data. We emphasize that class label much cheaper to acquire compared to 3D bounding box annotations.
3D bounding box estimation: Given estimated object class labels, we estimate the bounding boxes for each object class. We propagate the box size estimate from densely reconstructed objects to sparse objects each single frame, since the former provide high-confidence estimates.
The end result of the procedure above is a set of 3D bounding boxes that can be used to train tongle-framee object detector. Details of these steps provided below.
Registration. As observed in Table 1, most objects in the Waymo dataset follow smooth trajectories. For robustness against irregular sampling of LiDAR point clouds, we use this property to help estimate a velocity u; E 3 for each object cluster C. of frame We optimize
min Σ R;
Al|l:
Σ PEC,
1p U; q112/ QEC,
min
C
(A.)):
|-j==
where À = 1. The estimated [D,} brings all object points to the same coordinate system, forming a denser point cloud of the object instance, enabling trace classification and 3D box estimation.
Trace classification. Since object class labels are semantic rather than geometric, some minimal supervision is needed to distinguish classes (car, pedestrian, cyclist). We extract ground truth object class labels for 1,500 object instances (600 cars, 600 pedestrians and 300 cyclists) out of around millions object instances (or 26,000 object traces) in our training dataset to train an object classifier
5

---

Vehicle
Pedestrian
Cyclist
IoU=0.6
loU=0.7
IoU=0,4
Io5=0.5
IoU=0.4
1oU=0.5
AP 1.2
AP
AP
1.2 AP
1.1 AP
1.2 AP
L1 AP
1.2 AP
1.1
1.2
AP
L1 AP
1.2 AP
LPP
17.5
17.0
6.1
5,9
28,4
26.2
21.2
19.6
59.9
57.8
48.5
46.8
Ours-LP2
43.7
39.6
15.1
13.8
45.6
40.1
31.6
27.2
63.7
61.2
49.7
47.3
Ours-no-seg
52.8
47.8
21.4
19.3
51.4
45.4
32.9
28.0
63.3
60.7
48.7
46.3
Ours
53.6
48.8
22.3
20.6
51.6
45.9
33.7
28.9
64.6
62.0
50.1
47.6
Table Single-frame object detection results for unsupervised learning approaches. Here LP1/LP2 stands for the trained model before the first/second iteration of label propagation.
that takes object trace and outputs an object class label. An additional object class "Other" is included to deal with outliers. We label 3,000 instances of outliers as "Other". Our object trace classifier is adapted from Point Transformer [53] and Voxel-basee architectures, with 5 transition- down layers of dimension 32, 64, 128, 256, and 256, respectively. Each trace is represented as a 4D matrix with each row representing the location and time of point.
Bounding box estimation. In LiDAR object detection, each 3D bounding box is represented by a 7D vector with location, size, and orientation. Since object sizes vary among classes, we group object traces based on the estimated object class labels. For each object class, we observe that there is a fraction of objects that are densely captured by LiDAR while the rest only represent object parts.
Therefore, we learn model that regresses the bounding box size for each object trace, assuming does not change along each trace. For each class, we select the top 20% of object traces ranked by number of points. We then use the velocity computed in (2) to bring point clusters into the same coordinate system and compute 3D bounding box that covers all points with minimal volume. The resulting bounding box size vectors are used to train a model that regresses bounding box size. We use the model to predict bounding box sizes from all other object traces. The model architecture and input data representation of the regression model are also adopted similarly as trace classification.
Given the box size, we solve an optimization problem to estimate box location and orientation:
min b.
Σ (1-))
(b; by) + d2 (b;, C.).
Here, b; represents box attributes, d1 penalizes differences in orientation and enforces smoothness of box locations, and d2 encourages the i-th box b, cover the i-th point cluster C.. See appendix C.
Moving-to-statin label propagation. The resulting 3D bounding boxes only represent moving objects in each sequence. To promote generalization, we notice that static clusters of movable objects can be extracted but were rejected in the object cluster tracking step (see section 3). We train a classifier on registered object traces and ask it to find out static clusters that are geometrically similar. In addition, we train a single-frame object detector using estimated 3D bounding boxes and apply to predict 3D bounding boxes on the training data. Due the geometric similarity between moving and static objects, the object detector generates 3D bounding boxes for static objects. To improve robustness, we then apply [49] verify consistency of cemporall--ajjaee
bounding boxes and reject false positives. The verified 3D bounding boxes for static objects are added to the training set train another single-frame object detector from scratch. We repeat this procedure twice, yielding single-frame object detector that reasonably generalizes to moving and static objects. Appendix D provides details and visualization.
4.3 Model Architecture
We modify the CenterPoint architecture [54, 52] to construct our single-frame feature extractor, which outputs 256-dimensional feature for each 2D grid cell of size 0.1m 0.1m in the x- plane. The architecture is identical to the "VoxeINet" variant of CenterPoint that uses the SpConv Library for feature extraction [55]. The bounding box regression layer is modified according to self-supervised tasks discussed later. The training parameters are set to the same as the Waymo dataset defaults.
5 Experiments
We perform experiments related each step of our proposed approach. Unlike most self-supervised learning works that require fine-tuning on labeled data, our self-supervised tasks are closer to the

---

(a) Detected Object Traces
Ground Truth Object Points
Figure 5: Qualitative results of detected object traces. Left: Results of our object trace detection algorithm, each object trace is in different color. Right: Ground truth points corresponding to moving static objects with different color (red: vehicle, blue: cyclist, green: pedestrian).
downstream tasks, enabling us to train a single-frame object detector with only small set of class labels are needed but not any 3D box labels. We therefore perform two sets of evaluation on object detection algorithms, corresponding to the unsupervised setting and the self-supervised setting. In addition, we evaluate the quality of our detected object traces to demonstrate the reliability of our object trace detection algorithm.
$5.1 compares variants of our object trace detection algorithms in terms of the quality of the object traces. $5.2 compares our unsupervised single-frame object detector trained via self- supervueee tasks described in $4 without any annotated 3D bounding boxes. $5.3 compares our single-frame object detector trained under different settings.
Dataset. We conduct our experiments on the Waymo Open Dataset for high-quality, dense point cloud sequences [1]. The dataset contains 798 training sequences and 202 validation sequences, with each sequence containing ~ 200 frames with sampling frequency of 10Hz.
5.1 Object Trace Detection Results
Here, we evaluate the quality of detected object traces using the methods in $3. We compare two algorithmics, NeuralSF and SSL-SceneFlow. Both adapt state-of-the-art scene flow estimation algorithms to point cloud sequences. We use the Waymo Open Dataset training split for both.
Evaluation protocol. We report the mean Intersection over Union (mloU) between the detected object traces and the ground truth moving objects. For each detected point cluster, we find the 3D bounding box closest to the geometric center of the clus- We compute the IoU between points the box and the ones in the cluster. We also use the class labels of the box to categorize each cluster. If loU is zero, we categorize the detected cluster as "Other." We use the class labels for evaluation purpose only and report the evaluation results in Table 3.
Box count
Car
Pedestrian
Cyclist
Other
Neural-SB SSL-SF GT
1027229
729275
29178
348452
731480
631745
28369
543206
1161019
1427211
42411
Mean loU
Car
Pedestrian
Cyclist
Other
Ncural-SF
70.3
95.5
96.5
SSL-SF
67.4
90.0
91.6
Table Object trace detection results.
Analysis. Table 3 provides object trace detection results; Figure 5 qualitatively compares our detected object traces using Neural-SF the ground truth object points. Both algorithms achieve reasonable performance finding object traces from point cloud sequences. Neural-SF outperforms SSL-SF, possibly due to the joint estimation of motion among all frames, which provides robustness against false positive and false negative motions and promotes smoothness of motion.
5.2 Unsupervised Object Detection Results
Here, we evaluate the performance of our single-frame object detector trained with minimal class labels. We use object traces from Neural-SF, due to their superior quality. Since multiple iterations of label propagation are involved ($4), report the performance for both our final model and the
7

---

intermediate models after each label propagation iteration; we also try to remove motion segmentation (task and repeat the procedure for object detection (task II) only.
Evaluation protocol. We report 1.1 and 1.2 Average Precision (AP) on the validation split of the Waymo dataset for vehicles, pedestrians, and eyclists. We use two sets of loU thresholds compute AP for each object class: {0.6, 0.7) for vehicles, {0.4, 0.5} for pedestrians, and {0.4, 0.5) for cyclists. One of the IoU thresholds for each class is identical to the standard choice. See Table 4.
Analysis. Our unsupervised single-frame object detector works well for all classes (see Table 2). Among all three object classes, we achieved the highest precision cyclist class, which reflects the quality and coverage of our detected object traces for each object class reported $5.1. Empiricailly, we found that adding more iterations of label propagation do not improve overall performance. A possible reason is that the label propagation algorithm is sensitive to hyper-parameter,
5.3 Self-supervised Object Detection Results
In this section, we compare with two state-of toefthe-rrt self-supervised learning approaches: PointCon- trast [17] and DepthContrast [16]. These two methods work single-fire legae-frame point clouds but do not test on the Waymo Open dataset. We implement these two methods with a CenterPoint [52] pipeline. For ablation study, we train our model with three options: 1) (task-I) only task I's loss is used, 2) (task-Il) only task II's loss is used, 3) (all) the sum both tasks' loss is used.
Following the self-supervised learning routine, we split our training set into two parts and use all training point cloud sequences to pretrain a feature extraction module without using any labels. Then, we fine-tune the pretrained feature extraction module on 10% of the training point cloud sequences. The evaluation metries follow $5.2; we use standard IoU thresholds for this experiment, i.c., 0.7, 0.5 and 0.5 for vehicle, pedestrian and cyclist, respectively.
Analysis. Table 4 summarizes our results. PointContrast and DepthContrast improve overall AP by 0.6% and 0.2%, resp. Our self- supervised tasks achieved 0.8%. 1.9% and 2.6% improvement for task (motion segmen- tation), task II (object detection), and all tasks, resp. Among all three classes, we achieved the highest improvement on cyclists due to quality of the detected object traces, agrecing with the results from $5.2 and $5.1.
Vehicle
Pedestrian
Cyclist
(10% train)
60.6
60.7
65.4
PointContrast
61.3
61.3
66.0
DepthContrast
60.9
60.7
65.7
Ours-task-I
61.5
61.4
66.3
Ours-task-11
62.2
62.6
67.6
Ours-all
63.3
63.0
68.4
(100% train)
66.5
67.7
69.3
Table Single-frame object detection results for self-supervised learning.
6 Discussion and Conclusion
Our object trace detection algorithm reliably extracts objects that consistently and smoothly move in a point cloud sequence. These casily-extracted moving object traces provide critical information for downstream object detection without supervision, uncovered using self-supervised tasks. Our design choices differ from other auto-labeling approaches, emphasizing quality over quantity of the pseudo-labels. Even though some class labels are needed to achieve state-of-the class-aware object detection performance, our work reveals the power and ease of obtaining high- precision/low-cove coverrae pseudo-labels to train complex 3D shape analysis tasks like object detection.
Limitations and Future Work. Our method requires minimal amount of categorical labels to distinguish between different objects; we view this requirement as sensible, in that the type of moving object (vehicle, pedestrian, cyclist) requires semantic knowledge. Moreover, the painstaking process of annotating object bounding boxes in typical 3D supervision is not needed in setting: we just need a class label for our pre-detected object boxes (one of three discrete options, rather than a position of box in 3D). If such supervision is removed, a candidate replacement of the learned object classifiers might be to cluster the moving objects based on their geometry alone; our attempts at this unsupervised approach decreased performance. Incorporating self-supervised unsupervised object classifier models might be a promising direction to better automate this pipeline.
Our method focuses on movable objects like pedestrians, cyclists, and vehicles, since other classes will not captured by object trace detection. Different pretext tasks are needed for detecting static object classes. One could consider apatio-temporal structures and complementary views.
8

---

Acknowledgments
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants W91 INF201001 and W91 INF21 10293, of Air Force Office of Scientific Research award FA9550-19-1-033
of National Science Foundation grants IIS-1838071 and CHS-1955697, from the CSAIL Systems that Learn program, from the MIT-IBM Watson Al Laboratory, from the Toyota-CSAIL Joint Research Center, and from gift from Adobe Systems. The Toyota Research Institute provided funds to support this work.
References
[1] Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[2] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[3] Y. Wang, A. Fathi, J. Wu, T. A. Funkhouser, and J. M. Solomon. Multi-frame to single-frame: Knowledge distillation for 3d object detection. ECCV Workshop on Perception for Autonomous Driving, 2020.
[4] C. Luo, X. Yang, and A. Yuille. Self-supervised pillar motion learning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[5] B. Mersch, X. Chen, J. Behley, and C. Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In 5th Annual Conference on Robot Learning, 2021.
[6] Y. Shi, X. Cao, and B. Zhou. Self-supervised learning of part mobility from point cloud sequence. In Computer Graphics Forum, 2021.
[7] H. Wang, L. Yang, X. Rong. J. Feng, and Y. Tian. Self-supervised 4d spatio-temporal feature learning via order prediction of sequential point cloud clips. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[8] Yuan, X. Li, and Y. Fang. Deeptracking-net: 3d tracking with unsupervised learning of continuous flow, arXiv preprini arXiv:2006. 13848, 2020.
[9] D. Rempe, T. Birdal, Y. Zhao, Z. Gojeic, S. Sridhar, and J. Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[10] S. A. Baur, D. J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger. SLIM: Self-supervised LiDAR scene flow and motion segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[11] C. C. Qi, Y. Zhou, M. Najibi, P. Sun, Vo, Deng, and D. Anguelov. Offboard 3d object detection from point cloud sequences. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[12] I. Bogoslavskyi and C. Stachniss. Fast Range Image- Segmentation of Sparse 3D Laser Scans for Online Operation. In International Conference on Intelligent Robots and Systems (IROS), 2016. URL htttp://ww/ ipb. uni-bonn de/ppf/. s/dooss/oolslooooo lrrgossaaskii16irrr pdf.
[13] I. Bogoslavskyi and C. Stachniss. Efficient online segmentation for sparse 3d laser scans. PFG-Journal of Photogrammetry, Remote Sensing and Geoinformation Science, 85(1):41-52 2017.
[14] P. M. Chu, S. Cho, Y. W. Park, and K. Cho. Fast point cloud segmentation based on flood-fill algorithm. IEFE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pages 656,659,40 2017.

---

[15] J. Sauder and B. Sievers. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[16] Z. Zhang, R. Girdhar, A. Joulin, and I. Misra. Self-supervised pretraining of 3d features on any point-cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[17] S. Xie, J. Gu, D. Guo, C. R. Qi, L. Guibas, and O. Litany. Pointcontrast: training for 3d point cloud understanding. The European Conference on Computer Vision, 2020.
Unsupervised pre-
[18] Y. Liu, L. Yi, S. Zhang, Q. Fan, T. Funkhouser, and H. Dong. P4Contrast: Contrastive learning with pairs of point-pixel pairs for RGB-D scene understanding, ArXiv, abs/2012.13089, 2020.
[19] Y. Liu, Q. Fan. S. Zhang. H. Dong, T. A. Funkhouser, and L. Yi. Contrastive Multimodal Fusion with TupleInfoNCE. ArXiv, abs/2107 0020050 2021.
[20] H. Tian, Y. Chen, Dai, Z. Zhang, and Zhu. Unsupervised object detection with lidar clues. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2021.
[21] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[22] Y. Wang and J. M. Solomon. Object dgenn: 3d object detection using dynamic graphs. In Advances in Neural Information Processing Systems, 2021.
[23] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[24] G. P. Meyer, A. Laddha, F. Kee, C. An efficient probabilistic 3d object detector for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
and C. K. Wellington. Lasernet:
[25] L. Fan, X. Xiong, F. Wang, N. Wang, and Z. Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
[26] Y. Zhou, P. Sun, Y. Zhang, D. Anguelov, J. Gao. T. Ouyang, J. Guo, J. Ngiam, and V. Vasudevan. End-to-End Multi View Fusion for 3D Object Detection in LiDAR Point Clouds. In The Conference on Robot Learning (CoRL), 2019.
[27] Y. Wang, A. Fathi, A. Kundu, D. Ross, C. Pantofaru, T. Funkhouser, and J. Solomon. Pillar- based object detection for autonomous driving. In The European Conference on Computer Vision, 2020.
[28] Z. Zhang, B. Sun, H. Yang, and Q.-X. Huang. H3DNet: 3D Object Detection Using Hybrid Geometric Primitives. In The European Conference Computer Vision, 2020.
[29] W. Luo. B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3569-3577, 2018.
[30] P. Hu, J. Ziglar, D. Held, and D. Ramanan. What you see what you get: Exploiting visibility for 3d object detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2020.
[31] J. Yin, J. Shen, C. Guan, D. Zhou, and R. Yang. Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
10

---

[32] R. Huang, W. Zhang, A. Kundu, C. Pantofaru. D. A. Ross, T. Funkhouser, and A. Fathi. An Istm approach to temporal 3d object detection in lidar point clouds. In European Conference on Computer Vision 2020.
[33] Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander. Joint 3d proposal generation and object detection from view aggregation. In The International Conference on Intelligent Robots and Systems (IROS), 2018.
[34] X. Chen, H. Ma, Wan, B. Li, and Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2016.
[35] D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2018.
[36] C. Qi, W. Liu, C. Wu, H. Su, and L. Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[37] B. Yang, M. Liang, and R. Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In The Conference Robot Learning (CORL), 2018.
[38] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun. Multi-task multi- sensor fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[39] K. Wong, S. Wang, M. Ren, M. Liang, and R. Urtasun. Identifying unknown instances for autonomous driving. The Conference on Robot Learning (CORL), 2019.
[40] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. NieBner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1828-5839, 2017.
[41] S. Choi, Q.-Y. Zhou, S. Miller, and V. Koltun. A Large Dataset of Object Scans. 1602 2016.
[42] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- mantickitti: dataset for semantic scene understanding lidar sequences. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
[43] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11), 2013.
[44] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[45] X. Liu, M. Yan, and J. Bohg. Meteornet: Deep learning on dynamic 3d point cloud sequences. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
[46] Shi and J. Malik. Normalized euts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888 18088888881888888111111111111188 2000.
[47] X. Li, J. Kaesemodel Pontes, and Lucey. Neural scene flow prior. In Advances in Neural Information Processing Systems, 2021.
[48] H. Mittal, B. Okorn, and D. Held. Just go with the flow: Self-supervised scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[49] X. Weng and K. Kitani, A baseline for 3d multi-object tracking. a7Xiv:1907.03911 1(2):6, 2019.
arXiv preprint

---

[50] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In International conference machine learning. 2010.
[51] Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. PMLR, 2015.
[52] T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[53] H. Zhao, L. Jiang. J. Jia, H. Torr, and V. Koltun. Point transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[54] T. Yin et al. Center-based 3D Object Detection and Tracking. https://github.com com/
tianveiy/CeeteePPinn, 2021.
[55] Y. Yan et SpConv: Spatially Sparse Convolution Library. https://github.com com/ travellert9/spconty 2021.
12


Reviews:
[{"review_id": "s42tLV8y_gM", "reviewer": "Reviewer_NRFL", "summary": "This paper proposes an unsupervised learning pipeline to detect dynamic objects from a sequence of point cloud measurements. The proposed pipeline consists of object traces detection, motion segmentation, and single-frame object detection. The single-frame object detection method has been tested on Waymo Open Dataset and provides improved performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed pipeline that utilizes point cloud sequence for dynamic object detection is novel. This design enables dynamic object detection with unlabeled data.\n- The overall proposed pipeline is reasonable. \n- The experiments are performed on real-world data.\n\n**Weakness:**\n- The proposed method consists of multiple modules that can perform multiple tasks—resulting in a heavy paper that includes all aspects of object trace detection, motion segmentation, and object detection. \n- This reviewer finds the methodology part of the paper unclear. In particular, I am not confident in replicating the methods as not all steps are communicated. I recommend adding one or more algorithmic implementations of the proposed automatic labeling (main contribution) with precise explanations of inputs, algorithms, and outputs. \n-  Focusing more on one aspect, e.g., object detection and the advantage of self-supervised learning, might be better for a conference paper. For example, the comparison between two object trace detection methods (NeuralSF and SSL-SceneFlow) can be removed or moved to the appendix since these are two published methods. \n- A sequence of measurements is required to perform \"single-frame\" object detection. Thus it increases the required amount of data. In addition, the performance of object detection is limited by the performance of object trace detection.\n- Self-supervision is great. But it is also better to report some supervised methods to see the existing gap. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a framework for dynamic object detection by detecting object traces using a sequence of point clouds. The paper contains a good idea on automatic labeling of LIDAR data. But it requires a major revision and improved clarity. See my comments in the Weaknesses section. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Since the object trace detection accuracy limits the object detection performance, how is the performance affected by the object's motion? For example, the moving velocity, sudden movements, and turning angle.\n- Kalman filter is mentioned on page 3, but it is unclear how it is included as it is never discussed later. \n- In keywords: \"detection\" in misspelled as \"detection\".\n- What are \"pretext tasks\"?"}}, {"review_id": "j-CznRzl56", "reviewer": "Reviewer_442W", "summary": "The paper proposed a pretraining method for 3D object detection in point clouds. Leveraging the fact that a large portion of objects are moving smoothly in a sequence of frames in the Waymo dataset, and that moving objects leaving a trace in a sequence of point clouds are relatively easy to extract without manual labeling, pseudo 3D labels of objects can be automatically extracted, and the network can be pretrained on the detection task and related tasks. Experiments are conducted on the Waymo dataset and achieved better performance than existing pretraining methods. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Proposed to leverage the object traces as a clue for automatic labeling for the pretraining of the object detection task, which is an insightful observation and inspiring. \n2. Designed a series of steps to extract object traces and estimate the velocity and 3D bounding boxes of the objects from them. The steps are overall explained well and have practical value. \n3. Experimental results are reported on each step of the proposed method and support the claimed improvement of the paper. \n\nWeaknesses:\n1. The description of the network training is somewhat vague. For example, It is not clear how the task 1 and 2 are combined in the method. Are they two heads in the network and trained together? Or are they trained sequentially? Accordingly it is not clear how the networks in table 2 are trained (whether each task is included and whether they are iterated). \n2. Table 2 is not referenced in the paper. It is hard to find the corresponding paragraph in the main text. \n3. From table 1, the moving cars only account for 1/4 of all cars in the dataset. How is this problem addressed? Is it through the moving-to-static label propagation by adding newly detected objects to the pseudo labels in two iterations of training? Does adding a third iteration help? More discussion in section 5.2 could help, which also better supports the claim of benefit using high-precision/low-coverage pseudo labels. The latter is an interesting claim but it is not discussed much in the main text. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposed a practical pretraining method for point cloud object detection task. The approach of leveraging object traces is inspiring. The experimental results are generally convincing. Some minor revision on the organization and additional explanation and discussion are desired before it qualifies for publication. Therefore I suggest weak acceptance. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Explanation of how tasks 1 and 2 work together. \n2. Reference to table 2 in the main text. \n3. Explanation of how the problem of low proportion of moving cars is addressed. \n4. Discussion on the claim that using high-precision / low coverage pseudo labels is beneficial. "}}, {"review_id": "Zcwnl-8CaUn", "reviewer": "Reviewer_Thig", "summary": "This paper aims at reducing the effort of point cloud labeling by learning feature representations from point cloud sequences. The authors propose a method to extract moving object traces from a sequence of pose-aligned points clouds. The traces are computed by removing the ground plane and tracking point clusters. The points are clustered based on spatial proximity and scene flow between all point pairs in the sequence. The object traces are used to pre-train a single-scan feature extractor on tasks like motion segmentation and object detection without the need for 3D bounding box labels. Finally, the experimental evaluation shows the effectiveness of the object trace detection as well as the performance improvement by the proposed pre-training strategy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of focusing on easy-to-extract high-quality labels is interesting and the experimental evaluation shows that these labels can be used to improve object detection.\n- Two scene flow methods in the experiments are trained on KITTI and nuScenes data and still deliver suitable results on the Waymo data. This experiment validates their use for extracting object traces based on position and velocity similarity.\n- The paper is in general well written and easy to follow.\n\nWeaknesses:\n- The method relies on a \"provided ego-motion\". It is unclear how the performance is affected when odometry has to be estimated online possibly resulting in more drift.\n- The experimental analysis in Section 5.2 is incomplete since it only covers the performance for different classes. I recommend commenting on the effect of the different pre-training stages.\n- The first downstream task is a single-frame motion segmentation. However, I doubt that this task is meaningful since motion can not be reliably estimated from a single scan.\n- The object cluster proposal computes pairwise proximity scores between all points in the sequence. Since sequences of 200 frames result in a lot of point correspondences, I wonder how this affects the runtime of the object trace detection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides an interesting approach to using temporal LiDAR data to extract high-quality object traces used for representation learning of single-scan downstream tasks. The experiments show the effectiveness of the trained object detector and that the detection performance can be increased with fine-tuning compared to existing representation learning approaches. However, I still see some weaknesses and issues mainly affecting the contribution of self-supervision and advice resolving the unclarities.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "My main concern is regarding the use of the term \"self-supervised\". The authors claim to \"propose a technique for representation learning from unlabeled LiDAR point cloud sequences\". They mention that a \"minimal amount of categorical labels\" is still needed. In my view, the claim of performing \"self-supervised tasks\" does not hold since in fact labels are used, even though they are easier to get than bounding box annotations. Also, it is not clear if the ground truth poses are used for pre-processing which one could see as another form of labeling if manual inspection and validation of poses are carried out to obtain a good ground truth odometry. In general, I recommend updating the terminology and not using the term \"self-supervised tasks\" which is too strong in my view.\n\nMinor issues:\n- It is very hard to see details in Fig. 1 on the right or in Figure 3 at the top. Is it possible to zoom in to better focus on details than the complete scene?\n- What is the color coding in Figure 5 on the right? Using the same color makes it hard to distinguish individual traces here.\n- What extractly are L1 and L2 AP? Is this a metric coming from the Waymo Open Dataset? Please specify.\n- The section linking is broken (always jumps to the first page).\n- Keyword \"object detectioni\" is misspelled.\n- When mentioning the d1 and d2 losses in Eqn. (3), please also cite [3] as done in the supplementary material."}}, {"review_id": "3z1UpipeeVA", "reviewer": "Reviewer_FBsS", "summary": "This paper proposes an approach for detecting moving objects from unlabelled point cloud sequences, without any annotations. The detected object traces are then used to produce point cloud features for downstream tasks like single-frame object detection using self-supervised tasks. The experiments are run on the Waymo Open Dataset. \n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Manual annotation is a major bottleneck for scaling up the amount of data that can be used for learning. This paper uses the idea of using motion to separate out moving objects. This can be used on any unlabelled point cloud sequence, which can be easily collected without the need for expensive bounding box annotation. Further, the approach includes self-supervised pretext tasks to use these traces to perform useful downstream tasks.  \n\n\n\n2. The paper includes thorough experimental evaluation that evaluate the quality of object traces, as well as the point cloud features for downstream tasks, showing that the proposed approach can produce reasonable traces and object detection results as compared to prior approaches. The authors also include qualitative visualizations that show the structure of the detected traces. \n\n\nWeaknesses \n\n1. The first step of the approach is extracting object traces from multi-timestep data. The paper assumes that objects move along a smooth trajectory in order to extract this. This might be a reasonable assumption for the waymo dataset included in the experiments, however it limits the general applicability of the approach to other settings, such as a manipulation robot interacting with objects to learn skills, which often have non-smooth behavior. The smoothness assumption is also used in the self-supervised object-detection task which is used to produce 3D bounding boxes from the object traces. \n\n2. Central to the approach is extracting objects based on motion. There might be some objects of interest however that do not move, that we might still want to model. Though the paper does include moving to static label propagation, where 3D bounding boxes generalize due to visual similarity, this might be a special case of the dataset considered, and not hold in general for other domains. \n\n3. While the paper shows evidence for the efficacy of the approach in obtaining useful bounding boxes for key object categories, the paper doesn’t include any experiments showing that these are sufficient for control. Evidence that any loss in the quality of produced bounding boxes does not lead to major loss in downstream control performance would increase the community’s confidence in adopting this approach. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend acceptance, since using unlabelled point cloud sequences has the potential to scale up the amount of data used for learning, leading to much better performance for systems. The proposed idea relies on identifying objects based on motion, and while this has some issues as discussed, it can be a powerful source of self-supervision. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Issues \n1. Can the approach be adapted to handle non-smooth trajectories? Or is this a core, critical assumption?\n2. What pretext tasks could be constructed to extract representations for stationary objects?\n3. Are the produced bounding boxes (after the object-detection pretext task) sufficient for reliable control? How much is performance affected, if at all ? (As compared to using labelled data).\n\n"}}]

Author rebuttal:

Decision: Accept (poster)