4D Unsupervised Object Discovery
Authors: Yuqi Wang
Yuntao Chen
Zhaoxiang Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised Object Detection

Abstract: Object discovery is a core task in computer vision. While fast progresses have been made in supervised object detection, its unsupervised counterpart remains largely unexplored. With the growth of data volume, the expensive cost of annotations is the major limitation hindering further study.  Therefore, discovering objects without annotations has great significance. However, this task seems impractical on still-image or point cloud alone due to the lack of discriminative information. Previous studies underlook the crucial temporal information and constraints naturally behind multi-modal inputs. In this paper, we propose 4D unsupervised object discovery, jointly discovering objects from 4D data -- 3D point clouds and 2D RGB images with temporal information. We present the first practical approach for this task by proposing a ClusterNet on 3D point clouds, which is jointly iteratively optimized with a 2D localization network. Extensive experiments on the large-scale Waymo Open Dataset suggest that the localization network and ClusterNet achieve competitive performance on both class-agnostic 2D object detection and 3D instance segmentation, bridging the gap between unsupervised methods and full supervised ones. Codes and models will be made available at https://github.com/Robertwyq/LSMOL.

4D Unsupervised Object Discovery
Yuqi Wang 1.2 Yuntao Chen
Zhaoxiang Zhang 1,2,3
Center for Research Intelligent Perception and Computing (CRIPAC), National Laboratory of Pattern Recognition (NLPR). Institute of Automation, Chinese Academy of Sciences (CASIA)
School of Artificial Intelligence, University of Chinese Academy of Sciences Centre for Artificial Intelligence and Robotics, HKISI_CAS
{ {wangyuqi 20200zzaaxianng zaaaaooiiiinnzzannaaaaa atw.s/iiaa........ cn chenyuntao080gaail. com
Abstract
Object discovery is core task in computer vision. While fast progresses have been made in supervised object detection, its unsupervised counterpart remains largely unexplored. With the growth of data volume, the expensive cost of annotations is the major limitation hindering further study. Therefore, discovering objects without annotations has great significance. However, this task seems impractical on still-image point cloud alone due to the lack of discriminative information. Previous studies underlook the crucial temporal information and constraints natu- rally behind multi-modal inputs. In this paper, propose 4D unsupervised object discovery, jointly discovering objects from 4D data 3D point clouds and 2D RGB images with temporal information. We present the first practical approach for this task by proposing a ClusterNer on 3D point clouds, which is jointly it- eratively optimized with 2D localization network. Extensive experiments the large-scale Waymo Open Dataset suggest that the localization network and ClusterNet achieve competitive performance both class-agnostic 2D object detection and 3D) instance segmentation, bridging the gap between unsupervised methods and full supervised ones. Codes and models will be made available htops://githune com hwtp//obbetttyyg/LS00......
Introduction
Computer vision researchers have been trying to locate objects in complex scenes without human annotations for a long time. Current supervised methods achieve remarkable performance on 2D) detection [31, 15, 30, 38, 6] and 3D detection [49, 27, 34, 33, 47], benefiting from high-capacin models and massive annotated data, but tend fail for scenarios that lack training data. Therefore, unsupervised object discovery is critical for relieving the demand for training labels in deep networks, where raw data are infinite and cheap, but annotations are limited and expensive.
However, unsupervised object discovery in complex scenes used to believe impractical. Only a few studies pay attention this field and achieve limited performance in simple scenarios, far inferior to the supervised model. Recent methods [35, 42] discover objects on 2D still-image utilizing the self-supervised learning [7, 43] to distinguish primary objects from the background, then fine-tune localization network using the pseudo label. Although these methods outperform the previous generation of object proposal methods [39, 2, 50], their detection results are still far behind supervised models. Furthermore, contrastive learning guided methods have difficulty in distinguishing different instances within the same category, Alternatively, the 3D point cloud can be
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

decomposed into different class- class-agnostic instances based on proximity cues [5, 41, but due to lack of semantic information, is difficult identify the foreground instances. These problems can be mitigated by the complementary characteristics of 2D RGB images and 3D point clouds. The point cloud data provides accurate location information, while the RGB data contains rich texture and color information. Therefore, [37] proposed to aid unsupervised object detection with LiDAR clues, but it still depends on self-supervised models [14] to identify foreground objects. In summary, all previous methods rely heavily on the self-supervised learning models and overlook the important information from the time dimension.
To these ends, we propose new task named 4D insupervised object discovery, discovering objects uttliziig 4D data 3D point clouds and 2D RGB images with temporal information [25]. The task needs to joint discover objects on RGB images as in 2D Object Detection and objects on 3D point clouds as in 3D Instance Segmentation. Thanks to the popularization of LiDAR sensors in autonomous driving and consumer electronics (e.g., iPad Pro), such 4D data has become much more readily available, indicating the great potential of this task for general application.
In this paper, we present the first practical solution for 4D unsupervised object discovery. We proposed a joint iterative optimization for ClusterNer on 3D point cloud and localization network in RGB images, utilizing the spatio-temporal consistency from multi-modality. Specifically, the ClusterNet was trained with supervision from motion cues initially, which can be obtained from temporally consecutive point clouds. The 3D instance segmentation output by ClusterNet can be further projected to the 2D image as supervision for the localization network. Conversely, 2D detection can also help to refine the 3D instance segmentation by utilizing appearance information. In this way, the 21) localization network and 3D ClusterNet can benefit from each other through joint optimization. Temporal information could serve as a constraint in the optimization.
Our main contributions are as follows: (I) we proposed new task termed 4D Unsupervised Object Discovery, aiming at jointly discovering the objects in the 2D image and 3D Point Cloud without manual annotations.
(2) we proposed ClusterNet on 3D point clouds for 3D instance segmentation, which is jointly iterative optimized with 2D localization network. (3) Experiments on the Waymo Open Dataset [36] suggest the feasibility of the task and the effectiveness of our approach. We outperform the statc-of-the-arr unsupervised object discovery by a significant margin. superior to supervised methods with limited annotations, and even comparable to supervised methods with full annotations.
2 Related work
2.1 Supervised object detection
Object detection from Image. 2D object detection has made great progress in recent years. Two-
stage methods represented by the RCNN family [13, 31, 15] extract region proposals first and refine them with deep neural networks. One-stage methods like YOLO [30], SSD [22] and RetinaNet [21]
predict the class-wise bounding box in one-shot based on the anchors. FCOS (38] and CenterNet [48] further detect objects without predefined anchors.
Object detection from Point Cloud. LiDAR -ased 3D object detection develops rapidly along with autonomous driving. Point-based methods [28, 27, 46] directly estimated 3D bounding boxes from point clouds. The computing efficiency is affected by the number of points, these methods usually suitable for indoor scenes. Voxel-based methods [49, 45, 33] operate on the 3D voxelized point cloud are capable for large outdoor scenes. However, voxel resolution can greatly affect performance but is limited by computational constraints. Second [45] and PVRCNN [33] further apply sparse 3D) convolutions to reduce compute. CenterPoint [47] extends the idea of anchor-free from 2D detection and proposes a center-based representation of bird-eye view (BEV).
2.2 Unsupervised object discovery
Bottom-up clustering. Clustering methods combine similar elements based on proximity cues, appli- cable to point clouds and image data. Selective Search [39], MCG [2] and Edge Box [50] can propose a large number of candidate objects with the help of appearance cues, but it is difficult to identify objects from the background. Similarly, point cloud data can decompose into distinet segments according to density-based methods [10, 5, 32] but is unable to determine which is foreground.

---

L
bi.... bj
N 0
S
2D View
3D View
Movable
Static
View
3D View
2D-3D Cues
Motion Cues
Temporal Cues
The pipeline of 4D unsupervised object discovery. The input is the corresponding 2D frames and 3D point clouds. The task needs to discover objects on both images and point clouds without manual annotations. The overall process can be divided into two steps: (1) 3D instance initialization and (2) joint iterative optimization. (1) 3D instance initialization: motion cues serve as the initial cues for training the ClusterNet. (2) Joint iterative optimization: the localization network and ClusterNet are optimized jointly by 2D-3D cues and temporal cues.
Figure 1:
Top-down learning. Recently, self-supervised learning [8, 1,, 734 are capable to learn discriminate features without labels. Therefore, many methods attempt to introduce such properties to discriminate foreground objects without manual annotations. LOST [35] utilized a pre-trained DINO [7] to extract primary object attention from the background as the pseudo label and then finetuned an object detector. FreeSOLO [42] further proposed a unified framework for generating pseudo masks and iterative training. However, the performance relies heavily on the pre-trained self-supervised model, which determines the upper limits of such methods. Furthermore, such attention-based methods learned by contrastive learning also have the problem of distinguishing different instances within the same category. Our approach adopts top-down learning as well. Instead of aiding by an external supervieed model, we look for geometric information to discover objects in the scene naturally.
Algorithm
3.1 Task definition and algorithm overview
The task of 4D Unsupervised Object Discovery is defined as follows. As shown in Figure 1, the input of video clips recorded in both 2D video frames I' and 3D point clouds p' at frame during training. Since the point cloud and image data provide complementary information about location and appearance, they can serve as the natural cues guiding the training process mutually. During inference, the trained localization network Lø, is applied to still- till-image for 2D object detection, and the trained ClusterNet No, applied to the point cloud for 3D instance segmentation.
(b...... bb) = (I'),
= No.
(1)
(b/.... ... b(} are the 2D bounding box predictions by localization network Lø, at frame 1. (E)..... 5 are the 3D instance segments output by ClusterNet No.- k and n denotes the instance index.
07 8 arg min fLo, (1'), (P'))) 0.00
3

---

Our solution exploits the spatio-temporal consistency on 2D video frames and 3D point clouds. The algorithm can be formulated into a joint optimization function f in Eq.2. 01 and 02 are the parameters of the network need to optimize. Temporal information serve as the natural constraint in function f. The localization network 1-o, utilized Faster R-CNN as default. We propose a ClusterNet No, for 3D> instance segmentation. Detail implementation will discuss in section 3.2. The major challenge is the optimization for function f without annotations. To overcome the challenge, we seek for motion cues, 2D-3D ches and temporal cues to serve as the supervision. All these cues are extracted naturally in the informative 4D data. (1) motion cues, represented as 3D scene flow, can distinguish movable segments from the background. It uses to train the ClusterNet No: 0; initially. (2) 2D-3D cues, reflecting the mapping between LiDAR points and RGB pixels, can be used as a bridge to optimize the L and No, iteratively. It indicates the output of either network can be further used to optimize another network. (3) temporal cues, encouraging the temporal-consistent discovery in 21) and 3D view, can serve the constraint to optimize the function together. More details will introduce in section 3.3.
3.2 ClusterNet
ClusterNet generates 3D instance segmentation from raw point clouds. As shown in Figure 2, given a point cloud P e {(x, y, 2);, = 1, ...., N), the network is able give each point a class type Y; E (1,0) (indicating foreground or background) and instance ID d; E {1.....)) n). Thus we can obtain candidate segments & {(x, y. ر)ا = I.d; i} on the point cloud. n represents the number of instance segments in one frame of point cloud, and it is different in each frame.
Network design. The model first voxelized 3D points (x, y, 2), and extract voxelized features by transformer-bast feature extractor [11]. We further project these voxelized features back to each point. The feature dimensions of points become + C (3 means XYZ and C denotes the embedding dim). Inspired by the VoteNet [27]. we leverage a voting module to predict the class type and center offset for each point. Specifically, the voting module is realized with multi-layer perception (MLP) network. The voting module takes point feature fi E R3*C and outputs the Euclidean space offset Ax; E R² and class type prediction y,. The final loss is the weighted sum of the class prediction and center regression:
L = Lcenter + L
The class prediction loss Lets choose the focal loss [21] to balance the points of foreground and background. The predicted 3D offset Ax; is supervised by a regression loss:
Lcenter
E1 1r =]]
M
where 1[y]- = 1] indicates whether a point belongs the foreground according the ground truth y/. .M is the total number of foreground points. Ar, is the ground truth offset from the point position X; the instance center it belongs to. According to spatial proximity, we could further group the points into candidate instance segments with the predicted class type and center offset.
3D instance initialization. It is more challenging to obtain the supervision signal without manual annotation than the network design. The model was trained initially by motion cues. Specifically, motion provides strong cues for identifying foreground points and grouping parts into objects since moving points generally belong to objects and have the same motion pattern if they belong to the same instance. We could estimate the 3D scene flow S from the sequence of point clouds p' using the unsupervised method [19] at frame f. 3D scene flow describes the motion of all the 3D points in the scene, represented as S {Vxx, Vy) V2)
N).
Combining the scene flow (vx, Vy, Vz), and point location in 2D (u, v), and 3D (x, y, Z)j, we can obtain (u, v.x, y. 2, Vx. Vy. Vz); for each point p; in Pt. Then, we cluster the points with HDBSCAN [5] to divide the scan into mt segments, which will be the instance candidates Em. However, these instance candidates contain both foreground and background segments. We further assign each point P; of segment , a binary label y; to distinguish foreground points using the motion cues (3D scene flow), as shown in Eq. 5.
y; =
1[1
Σ PrEE;
I[IIS' > o]} > ml
(5)
[E
4

---

Backbone
Vate
Class
Chustering
Vote
Center
Figure 2: Overview of the ClusterNet architecture. A backbone extracts voxelized features for point clouds, given an input point cloud N points with XYZ coordinates. Each point predicts a class type and center through voting module. Then the points are clustered into instance segmentation.
in which [[] is the indicator function. 16y1 represents the total number of points belonging to segment 5 Pr E E a point in segment Ej. ||S" (p))l2 represents the velocity the point pt. and or denotes the threshold for velocity. l determines the ratio of being a foreground object. o = 0.05 and 1/ 0.8 moving regard it as foreground object, and all the
by default. y = means the point belongs to foreground segments. When the proportion points in the segment is greater than the threshold points it contains are labelled as foreground. These foreground segments selected by motion serve as the pseudo ground truth to train the ClusterNet initially.
3.3 Joint iterative optimization
ClusterNet trained by the motion cues serves as the initial weights for 02, which is the initialization (iter 0) of joint iterative optimization. Although movable objects can separate from the background with the motion cues, there are many static objects (e.g., parked cars or pedestrians waiting for traffic lights) in the scenes. Discovering both movable and static objects relies further joint optimization by 2D-3D cues and temporal cues. In section 3.3.1, we introduce the specific process of joint iterative optimization. Specifically, The 3D segments output by No, can project the 2D image to train the L.,, and the 2D proposals output by L-o, can lift back to 3D view to train the No:- Temporal consistency ensures the objects appear continuously in 2D and 3D views, which critical constraint in optimization. The joint optimization can be iterated several times since the 2D localization network and 3D ClusterNet benefit from each other. In section 3.3.2, will introduce the technical design for static object discovery.
3.3.1 Model training
In Eq. 6, our goal is to optimize the 01 and f without annotations. /' and p denote the RGB image and point cloud at frame 1. It challenging to optimize both parameters simultaneously, so we divide the optimization process into two iterative steps: 2D step and 3D step.
0 0 arg min f(Lo, 01,02
2D Step:
0r arg min (I'), (P)).
(6)
3D Step:
03 = arg min f(L, (P.
6h
2D step. In this step, the 02 fixed and optimized 0 Since the ClusterNet Ne, are able generate 3D instance segments En in 3D space, we can further project the 3D instance segments to 2D image plane by the transformation Ter (from the LiDAR sensor to the camera) and projection matrix P pe (from camera to pixels) defined by the camera intrinsic.
(1) P Tel 1
in which u denotes the pixel location in the 2D image plane. and x represents the 3D position of LiDAR points. Hence we can obtain the object point sets (w),..,,,, in the 2D image plane by projecting the LiDAR points of 3D instance segments (E1....,x,) E.). The 2D bounding boxes (bj....b;, derived from projected object point sets (w.),.,,,,, can use to optimize the weights of localization network Lø.
5

---

3D step. In this step, the 01 is fixed and optimized 02. The localization network Le, can output 2D bounding box predictions 1b9....bbi bg) based on the image uppearance information. It enables us to discover more objects in the scene (e.g.. parked cars regarded as background by motion cues). We can get the updated 2D object set b" by Eq. 8 (box loU set to 0.3 in Mon-Maximum Suppression).
b" NMSS( .... bn)) U (b9.....bf)) b(1))
Since many 3D instance segments may have been labelled as background by motion cues before, later refined the label with the help of the 2D object set b" Although the projection from LiDAR to the image is con-invertible without dense depth maps, could still utilize the mapping between the LiDAR point and image pixels. It suggests using the LiDAR points within the 2D bounding box to relabel the 3D instance segments. However, the bounding box may contain many LiDAR points corresponding to different 3D instance segments. Practically, we only consider the primary segment ș (with most points) inside the bounding box and relabel the primary segment the foreground object. With the refined label, we obtain the updated 3D object set g" of 3D instance segments (5) En). which further utilize to optimize the weights for ClusterNet No..
Temporal cues. Temporal information can be integrated into the 2D step and 3D step as extra constraints. As shown in Eq. 9, b'. represent the predicted 2D bounding box set and predicted 3D segments set for frame by La, and Ne.. b', E. denote the pseudo annotation for 2D bounding boxes (ib,,..., bū,)) and 3D instance segments (ffi..... (, from previous 2D step and 3D step. L'D is the loss for the localization network, and L3D is the loss for ClusterNet, which is introduced in Eq. 3. Lsmoosh encourages that the same object has consistent object labels across frames. The constraint can be added to both 2D views and 3D views. Therefore, it can help find new potential objects and filter wrong annotations across time. More details are illustrated in Appendix C.
f(L., (P'), LbD ('', Ssmoorh (b') + LED(E'
(006EEE.) .g.) Ssmooth (g.)
2D step
step
3.3.2 Static object discovery
Static object discovery is crucial in joint iterative training since the initialization by motion could handle movable objects well. During the joint iterative training, two technical designs are important for static object discovery. One is from the aspect of visual appearance, the other is from the aspect of temporal information.
Discover static objects by visual appearance. 2D localization network learns the object representa- tion by visual appearance. It indicates the good generalization ability for static objects since movable objects and static objects usually have similar visual appearances. However, a critical design is the selection of positive and negative samples in model training. Initially, the 2D pseudo annotations generated by motion cues mainly come from moving objects. It is crucial to avoid static objects becoming negative samples so that the model can have better generalization ability static objects. Table 4 compares different sampling strategies for the training.
Discover static objects by temporal information. Temporal information is also beneficial for static object discovery. Due to the occlusion in the 2D view, is more applicable to discover potential new objects by tracking in the 3D view. Practically, we used Kalman filtering for 3D tracking, and rediscover new objects in the static tracklets (center offset between the start and end frames less than 3 meters). Since we only focus on static objects, the mean center of the tracklet would be good prediction for lost objects.
Experiments
4.1 Dataset and implementation details
We evaluate our method on the challenging Waymo Open Dataset (WOD) [36], which provides 3D point clouds and 2D RGB image data that is suitable for our task setting. It of great significance to verify our unsupervised method under such real and large-scale complex scene.
Dataset. Waymo Open Dataset [36] is a recently released large-scale dataset for autonomous driving. We utilize point clouds from the 'top' LiDAR (64 channels, maximum distance of 75 meters),

---

and video frames (at a resolution of 1280x 1920 pixels) from the 'front' camera. The training and validation sets contain around 158k and 40k frames, respectively, All training frames and validation frames are manually annotated with 2D bounding boxes and 3D bounding boxes, which are capable of evaluating the performance of 2D object detection and 3D instance segmentation. Furthermore, WOD also provides the scene flow annotation in the latest version [17], which can illustrate the upper potential of our method.
Evaluation protocol. Evaluation is conducted on the annotated validation set of WOD. We evaluate the performance of 2D object detection and 3D instance segmentation. The dataset contains four annotated object categories ('vehicles', 'pedestrians', 'cyclists', and 'sign'), ) We test the class agnossic average precision (AP) score for vehicles, pedestrians, and cyclists. For 2D object detection, the AP score is reported at the box ditersection-ovvr-uiiee
(IoU) threshold of 0.5. For better analysis, results are also evaluated on small (area < <32 pixels), medium (32 pixels < area < 962 pixels) and large objects (area > 962 pixels). We also calculated the average recall (AR) to measure the ability of object discovery. For 3D instance segmentation, no previous metrics have been proposed on WOD. Referring to the 2D AP metrics, we propose to compute 3D AP score based on the IoU between predicted instance point sets and the ground truth. The ground truth for the instance segmentation can be obtained by labelling the point within 3D bounding boxes. The 3D AP score is reported at the point sets IoU threshold of 0.7 and 0.9, denoted as AP" and AP*) respectively. We also calculated the recall and foreground IoU for better analysis, which measure the ability of object discovery from more perspectives. Note here the AP denotes 2D object detection AP50 score. AP& denotes the 3D instance segmentation AP" score.
Implementation details. Our implementation is based on the open-sourced code of mmdetec- tion3d [9] for 3D detection and detectron2 [44] for 2D detection. For 2D localization network, we utilize Faster R-CNN [3]] with FPN [20] by default, where ResNet-50 [16] is used as the backbone. The network is trained on 8 GPUs (A100) with 2 images per GPU for 12k iterations. The learning rate is initialized to 0.02 and is divided by 10 at the 6k and the 9k iterations. The weight decay and the momentum parameters are set as 10 and 0.9, respectively. For 3D ClusterNet, the input raw point clouds removed ground points first by [4] and remained the points that can only be seen on the front camera. The cluster range is [Om, 74.88m] for the X-axis, [-37 37.44m,, 37.44m] for the Y-axis and [-2m, 4m] for the Z-axis. The voxel size is (0.32m, 0.32m, 6m). The feature extractor for voxelized points is [1]], and the embedding dim for C is 128. In the focal loss for class prediction, we set = 2.0,0, a = 0.8. The balance weight A for Eq. is set to 5. During inference, we set the minimum number of points to 5 for clustering. The ClusterNet is trained on 8 GPUs (A100) with 2 point clouds per GPU for 12 epochs. The learning rate is initialized to 10-S and adopts the cyclic cosine strategy (target learning rate 10-3). For hyper-parameters in HDBSCAN [5], we set the min cluster size to 15, and the others follow the default. For more implementation details, please refer to Appendix A.
4.2
Main results
2D object detection. Table compares the results between annotation from manual and annotaion from our method (termed as ClusterNet) for class-agnostic 2D object detection. All the experiments in Table utilized the same model (Faster R-CNN [31]) for fair evaluation. For the 2D bounding box annotations, the distant boxes are LiDAR invisible. The manual-annotated supervised baseline is trained with LiDAR visible 2D box annotations for a fair comparison. Even though our method still has the gap 43.2 vs. 54.4 to supervised baseline using fully manual annotation (1 (1137k bounding boxes), we can outperform the case when the annotation is limited. Limited annotation is frequent in real-world applications. Compared with 10% manual annotation (127k bounding boxes), our method could achieve 43.2 AP without any manual annotations and beat the 33.8 AP by a large margin. Since our method relies on the motion cues estimated unsupervised, we proved that the performance could increase to an incredible 51.8 AP with ground truth scene flow, which is very close to the performance of fully manual annotation but without bounding box annotation. Since the previous unsupervised methods only focused on still 2D images and could not extract the objects from the background accurately, it is no surprise they could only achieve poor results in such challenging scenes. LOST [35] can only extract one primary object from the background, which does not apply to the driving scenes. Freesolo [42] often generates a large mask for row of cars, which can not distinguish specific instances. Felzenszwalb Segmentation [12] generate potential proposals by graph-based segmentation but lacks the ability to identify the foreground objects. Our method has great performance ad advantages over these still-image methods.
7

---

Table 1: Class-agnostic 2D object detection
annotation setting
Himages
#bboxes
network weights
initialized from
APS0
AP
AP30 M
APL
AR-50
S
AR M
ARZ
supervised
fully manual annotation
158k
1137k
ImageNet
20.5
72.4
35.5
fully munual unmotation
158k
1137k
scraich
52.5
23.5
67.6
86.3
62.3
34,9
80.0
93.3
10% manual annotation
15k
127k
ImageNet
33.8
5.5
45.3
74.9
36.1
9.7
48.6
76.7
10% manual annotation
15k
127k
scratcc
31.6
5.7
42.5
72.2
35.9
8.6
47.7
75.3
unspeevised
Felzenszwall [12]
158k
ImageNet
0.0
0.5
11.1
0.6
14.5
30.7
LOST [35]
158k
ImageNet
1.9
0.0
1.0
7.6
5.0
0.0
0.4
27.9
FreeSelo [-2]
158k
ImageNNt
1.0
0.2
1.0
1.9
2.2
0.0
0.1
12.7
ClusterNet (w/ scenetlow)
158k
scratch
51.8
21.3
70.2
89.5
60.8
30.2
81.2
94.8
ClosterNet
scratch
3D instance segmentation. Table 2 illustrates the effectiveness of ClusterNet 3D instance segmentation. Our ClusterNet achieved 26.2 AP3 and 19.2 AP without any annotation, superior to 10% supervised baseline 23.6 AP30 and 15.5 Ap with 397k 3D bounding boxes annotation. We proved that our method with accurate motion cues (ground truth scene flow) could achieve 42.0 AP and 33.2 AP even comparable to that supervised baseline with fully manual annotation (4268k 3D bounding boxes). setting. Figure 3 illustrates the object prediction of approach the WOD validation set.
No previous method can achieve such high performance under an unsupervised
Table Class-agnostic 3D instance segmentation
annotation setting
#point clouds #3D bboxes
AP
AP
Recall
Recall
loU
supervised
fully manual annotation
158k
4268k
45.7
37.3
75.1
65.1
92.2
10% manual annotation
15k
397k
23.6
15.5
61.8
48.7
81.6
unsupervised
ClusterNet (w/ gt sceneflow)
158k
0
42.0
33.2
61.7
52.3
88.1
ClusterNet
158k
0
26.2
19.2
40.0
32.8
64.9
Figure 3: Visualization for 2D object detection and 3D instance segmentation the WOD validation set. Our approach could achieve such incredible results without any annotations.
2D instance segmentation. We can also conduct instance segmentation by projecting the LiDAR points of 3D instance segments to the 2D image plane. The key difference is that instance segmentation masks other than object bounding boxes deriving from 3D instance segments as pseudo annotations. We utilized alpha shape [1] to generate the mask of object points (LiDAR points projected to the image). The localization network can change conveniently to Mask R-CNN [15] for instance segmentation without manual annotations. Some predictions Figure 4 and Appendix D. Because the Waymo Open Dataset did not provide the annotation for instance mask, the performance of instance segmentation cannot be evaluated quantitatively.
validation set are illustrated in

---

Figure 4: Instance segmentation by our approach when using Mask R-CNN [15] as our localization network, without manual annotations. Our method can generate high-quality instance masks.
4.3 Ablation studies
Analysis of multi cues for training. Table analyze the contributions of multi cues in our approach to the WOD validation set. The final results are obtained after three iterations of joint optimization. AP2D denotes the AP score for 2D object detection, and AP3 denotes the AP score for 3D instance segmentation. The ClusterNet was trained with the pseudo-annotations obtained by HDBSCAN Clustering [5] for the first time. So simple baseline is directly using the HDBSCAN for 3D instance segmentation and project on 2D for localization network training. comparison, we demonstrate the effectiveness of using ClusterNet: the performance increased by 4.9 AP2! (from 25.1 1 to 30.0) and 2.2 AP3 (from 4.6 6.8). Furthermore, the performance improved significantly by joint optimizing for the ClusterNet and localization network, with 2D-3D cues and temporal cues.
Table 3: Analysis of multi cues for training.
Method
point cloud motion cues
2D-3D cues
temporal cues AP2D AP3D 1
14.9
2.1
HDBSCAN
25.1
4.6
30.0
6.8
ClusterNet
40.4
25.7
43.2
26.2
Table 4: Ablation on sampling strategy.
Table Ablation on early stopping.
sampling strategy
Ap50
Ap50
A
AP50
iterations Ap50
AR50
(a) IoU_>0.7, IoU_<0.3
27.8
3.2
37.2
70.0
3000
30.0
43.3
(b) 1oU,>0.6, 10U-<0.4
28.2
3.3
36.9
71.7
6000
27.9
40.4
(c) IoU,>0.6, 0.1<1oU.<0..
30.0
4.3
39.4
73.2
12000
25.3
35.0
Training strategy for localization network. Training the localization network with the pseudo annotations generated by ClusterNet is quite different from training with manual annotations. Specifi- cally, the pseudo annotations will be noisy and incomplete before joint iterative training. Therefore, two points we found crucial for the initial training: (1) sampling strategy in RPN (Region Proposal Network) and (2) early stopping for training. The following ablation experiments are conducted for the first-time training of the localization network. Table 4 compares three different strategies: (a) sample anchors with box ToU 0.7 as positive example and box loU 0.3 as negative example, as in the standard Faster R-CNN; (b) sample anchors with box loU > 0.6 positive example and box loU < 0.4 as negative example; (c) sample anchors with box loU 0.6 as positive example, and box 0.1 < IoU < 0.4 as negative example. In this way, the strategy considerably reduces the chance

---

of sampling static objects as negative examples. Table 5 compares different training iterations and shows that early stopping improves the generalization performance. Since the pseudo annotations have noise, training for a long time may overfit the noise in the training set, leading to the degradation of generalization performance.
Joint iterative optimization. Table 6 presents the efffectiveness of our joint iterative optimization for the 2D localization network and 3D ClusterNet. Iteration represents the initial performance of ClusterNet trained by motion cues (estimated scene flow). Next, each iteration means a 2D step and a 3D step. Even though the model did not perform well at the beginning, with joint iterative optimization, both AP and AP improved rapidly. Applying more than one iteration improves the results, indicating that the 2D localization network and 3D ClusterNet can benefit from each other. We set the iteration number as 3 by default.
Minimum points for ClusterNet. Minimum points determine the minimum number of LiDAR points for 3D instance segments. Table 7 analyze the model performance under different parameters during the inference. We set the min points to 5 by default.
Table 6: Joint iterative optimization.
Table Minimum points for ClusterNet.
iterations AP
APM
min points
Ap3D
6.8
2
25.3
30.0
20.2
5
26.2
37.4
25.4
10
26.0
43.2
26.2
20
25.5
42.8
25.9
Discussion and conclusions
Discussion. Unsupervised object discovery used to believe infeasible due to the ambiguity of objects and the complexity scenarios. However, 4D data with the sequence of image frames and point clouds provide enough cues to discover the movable objects, even without manual annotation. The complementary information behind the 3D LiDAR points and 2D image and constraints from temporal are the critical factors for the success of unsupervised object discovery. With 4D sensor data readily available onboard, our approach shows extraordinary potential for scenarios with limited annotation. The only limitation is that method is suitable for movable objects (vehicles, pedestrians); static things (never move) like heds or chairs can not be discovered.
Conclusions. In this work, propose a task named 4D Unsupervised Object Discovery. The task needs to discover objects both on the image and point clouds without manual annotations. We present the first practical approach for this task by proposing ClusterNet for 3D instance segmentation and joint iterative optimization. Extensive experiments on the large-scale Waymo Open Dataset demonstrate the effectiveness of our approach. So far as we know, we are the first work to achieve such high performance for unsupervised 2D object detection and 3D instance segmentation, bridging the gap between unsupervised methods and supervised methods. Our work sheds light new perspective on the future study of unsupervised object discovery.
Societal Impacts. The development of unsupervised object discovery requires large datasets, intro- ducing privacy issues. The technology of unsupervised detection dramatically reduces the labelling cost; it may affect the people engaged in the labelling industry in the future. The elimination of human intervention may also cause some data annotators to lose their current jobs. Our approach only tests in driving scenes for effectiveness, which may lead to some wrong detection in other scenes.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their constructive comments. This work was supported in part by the Major Project for New Generation of AI (No.201 8AAA0104400), the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231), and the InnoHK program. The authors would like to thank Xizhou Zhu and Jifeng Dai for conceiving an early idea of this work. Also, our sincere and hearty appreciations go to Jiawei He, Lue Fan and Yuxi Wang, who polishes our paper and offers many valuable suggestions.
10

---

References
[1] Nataraj Akkiraju, Herbert Edelsbrunner, Michael Facello, Ping Fu, EP Mucke, and Carlos Varela. Alpha shapes: definition and software. In Proceedings International Computational Geometry Software Workshop. 1995.
[2] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In CVPR, 2014. 1, 15
[3] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In ICIP, 2016. 16
[4] Igor Bogoslavskyi and Cyrill Stachniss. Efficient online segmentation for sparse 3d laser scans. PFG-Journal of Photogrammetry Remote Sensing and Geoinformation Science, 2017. 2, 7, 14
[5] Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference knowledge discovery and data mining, 2013. 2, 4, 7, 9, 14, 15
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
3,
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML,, 2020.
[9] MMDetection3D Contributors. MMDettcction3D: Open pext-generation platform for general 3D object detection. attps://githii com httppnnnmmmla/mmmmttetttiiaio.it
2020.
[10] Bertrand Douillard, James Underwood, Noah Kuntz, Vsevolod Vlaskine, Alastair Quadros, Peter Morton, and Alon Frenkel. On the segmentation of 3d lidar point clouds. In ICRA, 2011.
2
[11] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang. and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. arXiv preprint arXiv:21 2021. 4, 7,
[12] Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. IJCV, 2004. 7, 8, 14, 15
[13] Ross Girshick. Fast r-cnn. In JCCV, 2015. 2015. 2
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020. 2,
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 1,
2, 8,
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Philipp Jund, Chris Sweeney, Nichola Abdo, Zhifeng Chen, and Jonathon Shlens. Scalable scene flow from point clouds in the real world. RAL, 2021. 7
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1rXiv:1412.6980, 2014. 14
[19] Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. Neural scene flow prior. NeurIPS, 2021. 4, 14
[20] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR. 2017. 7

---

[21] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017. 2,
[22] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016. 2
[23] Xiankai Lu, Wenguan Wang. Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In CVPR, 2019. 15
[24] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkine- Sorkine-Hornung, A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016. 15
[25] AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova. 4d-net for learned multi-modal alignment. In ICCV, 2021.
[26] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Corkine-Hornunt and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprini arXiv: 2017. 15
[27] Charles R Qi. Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. ICCV, 2019. 1, 4
[28] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017. 2
[29] Hazem Rashed, Mohamed Ramzy, Victor Vaquero, Ahmad El Sallab, Ganesh Sistu, and Senthil Yogamani. Fusemodnet: Real-time camera and lidar based moving object detection for robust low-light autonomous driving. In ICCV Workshops, 2019. 15
[30] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017. 1,2
[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015. 2, 14
[32] Radu Bogdan Rusu and Steve Cousins. 3d is here: Point cloud library (pel). In ICRA, 2011.
[33] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020. 1,2
[34] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[35] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and labels. arXiv preprini arXiv:2109. 14279, 2021. 1, 7, 8, 14, 15
[36] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 2,
[37] Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. In CVPR, 2021. 2
[38] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In ICCV, 2019. 1,2
[39] Jasper RR Uijlings, Koen EAA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 2013. 1,2
[40] Carles Ventura, Miriam Bellver, Andreu Girbau, Amaia Salvador, Ferran Marques, and Xavier Giro-i Nieto. Rvos: End-to-end recurrent network for video object segmentation. CVPR, 2019. 15
12

---

[41] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang. Immortal tracker: Tracklet never dies, arXiv preprint arX1v2211..1362,, 2021. 16
[42] Xinlong Wang, Zhiding Yu, Shalini Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning segment objects without annotations. arXiv preprini arXiv:2202.12188 12181, 2022. 1, 7.8
[43] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021. 1,
[44] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: thub. hww.aacooooeeeeeeceeeeeeeeeeeeee
2019. 7, 14
[45] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[46] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020. 2
[47] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021. 1,2
[48] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint 1rXiv:1904007850, 2019.
[49] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018. 1,2
[50] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014. 1,2
13


Reviews:
[{"review_id": "tU66dVVuOp", "reviewer": "Reviewer_WTqt", "summary": "The paper solves the problem of unsupervised object discovery by utilizing the 4D information (cloud point, and time), they designed an iterative refinement frameworks which do the 2D information refinement and 4D information refinement one-by-one (usually 4~5 iters). \n\nThe results achieves the best result under the new setting. A few ablations about the importance of cues, training strategies, etc are provided to help us to better understand the task and the methods. ", "questions": "Listed as above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n- Interesting and sound setting to explore the 4d unsupervised object discovery.\n- The writing is easy to follow from intuition, to method, and finally experimental parts.\n\n\nCon:\n- Figure 1 is not clear about the dataflow. I am totally missed.\n- One detail I am curious is that whether your network is initialized from scratch, or from another pretrained network. That counts about whether you can name it as 'unsupervised'.\n- The contribution claim of clusternet, can the author highlight the difference of it with VoteNet?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OriT8efpZTb", "reviewer": "Reviewer_nTyt", "summary": "The paper tackles the problem of object discovery. They propose the 4d unsupervised object discovery from 4D data - 3D point clouds and 2D RGB images with temporal information together with a new model, ClusterNet that is jointly iterative and optimized with a 2D localization network. Finally, results are reported on the Waymo Open Dataset.", "questions": "1. How is \"n\" chosen for the number of instances?\n\n2. How hard is to collect data with cloud points?\n\n3. Can you give more details about the fully supervised baseline? Does it use a previously published method?\n\n4. In the implementation details (lines 218-232) it is stated that for example for 2D localization network, Faster R-CNN is utilized. Does this mean that you also initialize some weights with a pre-trained Faster R-CNN model or that you rely on the architecture? This aspect needs to be clearly stated.", "limitations": "The authors discuss some of the limitations and some potential societal impact problems.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem. It proposes the task of object discovery from 3D point clouds together with 2D images along with the temporal information which can be of interest for the community. I find the joint optimization to be interesting.\n\nMy main concern related to this paper is the comparison with state of the art which I find to be quite limited. I would have wished to see more comparisons with other methods for 2D unsupervised object detection. Since the comparison is quite limited, it is hard to estimate the gain brought by the proposed approach. So, the only estimate that remains is the comparison with the supervised case. However, for this setup I feel like there are little details about the used architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JWAh5rlcnV-", "reviewer": "Reviewer_Nejz", "summary": "In this paper, the authors have proposed a novel task named *4D Unsupervised Object Discovery*, aiming at discovery the objects from 4D sequential data (3D point clouds + 2D RGB images), without any human annotation.\n\nFor this purpose, they proposed a method called *ClusterNet* to automatically discovery the objects in 2D images (bounding boxes) and the objects in 3D point clouds (point-wise segments), which can be jointly optimized on the 4D sequential data by taking the temporal information into account. \n\nThe generated labels are evaluated in 2D object detection and 3D instance segmentation, showing on par results with supervised methods and superior performances compared to other object discovery methods. Furthermore, the 2D object boxes can be converted to 2D object segments by projecting the 3D instance segments to 2D images, which allows for joint segmentation on 4D data.", "questions": "1. Since one important objective of object discovery is to reduce human effort by automatically generating object labels, retraining existing models with the generated object labels should be added to show the true efficiency of the proposed object discovery method. The performance gap between models trained with generated labels and human labels can serve as a key indicator for the final application in real world. \n\n2. As this method works on 4D data, the authors propose to initial object labels on 3D, then project to 2D and iteratively optimize them, I am wondering is it possible to initial object labels on both 2D and 3D by clustering moving objects with optical flows and scene flows. This is not necessary, but we might get better initial labels by looking at both modes.\n\n3. The current version of Sec 3.3 is a bit too abstractive, with many details hiding in several parts of the paper.\nFor example, it is not trivial to train the 2D localization network to discover static objects with initial proposals projected from 3D point clouds, a summarization of the training strategy (L276-290) could be added in Sec 3.3 to help the readers understand quickly how this was done.", "limitations": "Yes, the limitations are discussed in the last section of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* This paper proposes a novel task and a novel method with clear presentations, and thorough ablation studies supporting the validation of each component of the proposed method.\n\n* While existing object discovery works mainly focus at extracting discriminative information from pure 2D or 3D data, this work chooses to combine the best of both modes by leveraging the 4D sequential data, and to consider the temporal information as well as the multi-modality constraints. This removes the dependence of a strong self-supervised learning models, and compensate the weaknesses of each side.\n\n* The superior experimental results shows that the advantages of object discovery on 4D data compared to pure 2D or 3D.\n\nWeaknesses:\n\n* No experimental result was given to demonstrate the quality of discovered objects by training existing models with the generated labels instead of the human annotations.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept