UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes
Authors: Ted Lentsch
Holger Caesar
Dariu Gavrila
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Object Detection
Multi-Modal Data
Self-Supervision
Primary area: machine_vision
TL;DR: Camera and LiDAR data are jointly exploited to learn to detect 3D objects in an unsupervised manner.

Abstract: Unsupervised 3D object detection methods have emerged to leverage vast amounts of data without requiring manual labels for training. Recent approaches rely on dynamic objects for learning to detect mobile objects but penalize the detections of static instances during training. Multiple rounds of (self) training are used to add detected static instances to the set of training targets; this procedure to improve performance is computationally expensive. To address this, we propose the method UNION. We use spatial clustering and self-supervised scene flow to obtain a set of static and dynamic object proposals from LiDAR. Subsequently, object proposals' visual appearances are encoded to distinguish static objects in the foreground and background by selecting static instances that are visually similar to dynamic objects. As a result, static and dynamic mobile objects are obtained together, and existing detectors can be trained with a single training. In addition, we extend 3D object discovery to detection by using object appearance-based cluster labels as pseudo-class labels for training object classification. We conduct extensive experiments on the nuScenes dataset and increase the state-of-the-art performance for unsupervised 3D object discovery, i.e. UNION more than doubles the average precision to 38.4. The code is available at github.com/TedLentsch/UNION.

UNION: Unsupervised 3D Object Detection using Object Appearance-basedd Pseudo-Classes
Ted Lentsch Holger Caesar Dariu M. Gavrila
Department of Cognitive Robotics
Delft University of Technology
Abstract
Unsupervised 3D object detection methods have emerged to leverage vast amounts of data without requiring manual labels for training. Recent approaches rely on dynamic objects for learning to detect mobile objects but penalize the detections of static instances during training. Multiple rounds (self) training are used to add detected static instances to the set of training targets; this procedure to improve performance is computationally expensive. To address this, we propose the method UNION. We use spatial clustering and self-supervised scene flow to obtain set of static and dynamic object proposals from LiDAR. Subsequently, object proposals' visual appearances
encoded to distinguish static objects in the foreground and background by selecting static instances that are visually similar to dynamic objects. As result, static and dynamic mobile objects obtained together, and existing detectors can be trained with a single training. In addition, we extend 3D object discovery to detection by using object appearance-based cluster labels as pseudo-class labels for training object classification. We conduct extensive experiments on the nuScenes dataset and increase the state-of-the-art performance for unsupervised 3D object discovery, i.e. UNION more than doubles the average precision to 38.4. The code is available github. com/ TedLentsch/UNIOOO
LiDAR: Object Proposals On
Fusion: Appearance Clusters with Motion Statistics
Pedestrians
X Stacic Dynamic
X
30 Eboo
[se]
ty (ookoonan)
sy [se!
(trea)
X
X
X
Temporal: Motion Estimates
X
Opclists
X
Trmes
Only Static instanios
Appearance Embeddiin Feeture Space
Opnamin
Stati
Camera:
Visual Appearances an
Cluster Classification
Train Object Detector
Object
Appoarure Cluater Nobil Clastrif ii
tertiie reart Pyronii roee
f Pacuoo Mesia Aspesrazie Clastars
Embedding
Tran Standard
DINOV2
Module
Object
Model
Mobille
Mudil
Non-Mobik
Figure 1: UNION discovers mobile objects (e.g. cars, pedestrians, cyclists) an unsupervised manner by exploiting LiDAR, camera, and temporal information jointly. The key observation is that mobile objects can be distinguished from background objects (e.g. buildings, trees, poles) by grouping object proposals with similar visual appearance, i.e. clustering their appearance embeddings. and selecting appearance clusters that contain at least X % dynamic instances.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Object detection is one of the core tasks of computer vision, and it is integrated into the pipeline of many applications such as autonomous driving [11], person re-identification [30], and robotic manipulation [35]. During the past decade, the computer vision community has made tremendous progress detecting objects, especially learning-based methods. These supervised methods rely on manual annotations, i.e. each object instance is indicated by bounding box and class label. However, a massive amount of labeled training data is usually required for training those models, while labeling is expensive and laborious. This raises the question of how object detection models can be trained without direct supervision from manual labels.
Unsupervised object detection is relatively unexplored research field compared to its supervised counterpart. For camera images, recent work [4, 21] shows that the emergent behavior of models trained with self-supervised representation learning can be used for object discovery, i.e. object localization without determining a class label. The behavior implies that the learned features of those models contain information about the semantic segmentation of image, and thus, they can be used to distinguish foreground from background. Consequently, the extracted coarse object masks are used to train 2D object detectors [22, 26]. Although these methods perform well for images depicting a few instances with a clear background, they fail to achieve high performance for images with many instances, such as autonomous driving scenes [27]. In these scenes, instances are close to each other and, as a result, are not directly separable using off-the-shelf features.
On the other hand, spatial clustering is the main foree that drives 3D object discovery [27, 32]. In contrast to images, separating objects spatially is relatively easy in 3D space, but differentiating between clusters based on shape is challenging because of the density of the data (c.g. sparse LiDAR point clouds). Hence, temporal information is often exploited to identify dynamic points that most likely belong to mobile objects such as cars and pedestrians. In this context, we define mobile objects as objects that have the potential to move. Consequently, objects such as buildings and trees are considered non-mobile classes. The discovery of static foreground instances (e.g. parked cars and standing pedestrians) is usually achieved by performing self-training. Self-training is based on the assumption that a detector trained on dynamic objects has difficulty discriminating between the static and dynamic versions of the same object type. As a result, when such a detector is used for inference, it will also detect many static instances. The predicted objects are then used for retraining the detector, i.c. self-training, which is repeated multiple times until performance converges. Figure 2a gives a schematic overview of LiDAR-only methods for unsupervised 3D object discovery.
A significant drawback of iterative self-training is that training takes significantly longer due to the many rounds, e.g. sequentially training 5-10 times [2, 32], and there may be confirmation bias, i.e. incorrect predictions early on can be reinforced. Moreover, we hypothesize that training with only
LiDAR-based Bounding Box Generation
LiDAR-based Bounding
Multi-Modal-basee Bounding Box and Class Label Generation
Box Generation
Pseudo-Bounding Bounding Boxes A50
Pseudo
Boxes 080
Pseudo-Boo Bounding Boxes and Class Labels dDA
Training LiDAR-based Class-Agnostic Object Detector
Training LiDAR-basee Class-Agnostic Object Detector
Training LiDAR-based Multi-Class Object Detector
LiDAR-based Self- delf-Training
Multi-Modal Self-Training
No Self- Needed
(a) LiDAR 3D object discovery with LiDAR-based self-training
(b) LiDAR 3D object discovery with multi-modal self-training
(c) UNION: multi-moda multi- class 3D object discovery (ours)
Figure 2: Comparison of the various designs for unsupervised 3D object discovery. (a) Most
object discovery methods exploit LiDAR to generate -buuddin boxes and use these to train a detector in class-agnosti setting followed by self-training.
(b) Wang et al. [27] generate bouudo-b boxes similar to (a) but alternate between training a LiDAR -base detector and a camera-base based detector for self-training (c) We use multi-modal data for generating aseudo-bounding boxes and pseudo-class labels, and train a multi-class detector without requiring self-training.
2

---

dynamic objects degrades the final detection performance because (1) there is inconsistency during training and (2) the data distribution of static and dynamic objects differs. The training inconsistency entails that the detection of static objects is penalized during training, i.e. they are considered false positives, while the objective of object discovery methods is to detect both static and dynamic objects. Besides, static and dynamic objects are sensed differently as they typically occur at different positions with respect to the sensors, and as a result, their data distribution differs (e.g. point distribution).
We argue that multi-modal data should be used jointly for unsupervised 3D object discovery as each modality has its own strengths, e.g. cameras capture rich semantic information and LiDAR provides accurate spatial information. Existing work [27] does use mudti-modal data for unsupervised object discovery but not jointly. As shown Figure 2b, the training procedure consists of two parts: (1) training with LiDAR- -based aseudo-bounding belonging to dynamic instances and (2) multi-modal self-training to learn to detect static and dynamic objects. However, Wang et al. [27] ignore the fact that both modalities can be used at the same time for creating pseudo-bounding boxes.
Therefore, we propose our method, UNION (unsupervised multi-modal 3D object detection), that exploits the strengths of camera and LiDAR jointly, i.e. as a union. We extract object proposals by spatially clustering the non-ground points from LiDAR and leverage camera encode the visual appearance of each object proposal into an appearance embedding. Subsequently, we exploit the appearance similarity between static and dynamic foreground objects for discriminating between static foreground and background instances (see Figure 1). Finally, the identified objects and their appearance embeddings are used generate pseudo-bounding boxes and pseudo-class labels. which can be used to train existing 3D object detectors in an unsupervised manner using their original training protocol. Figure 2c shows the high-level concept of our method with the novelties underlined.
Our contributions are twofold. 1. We propose UNION, the first method that exploits camera, LiDAR, and temporal information jointly for training existing 3D object detectors an unsupervised manner. We reduce training complexity and time by avoiding iterative training protocols. We evaluate our method under various settings and set new state-of-the-art (SOTA) for object discovery the nuScenes [3] dataset. 2. Rather than training detector to only distinguish between foreground and background, we extend 3D object discovery multi-class 3D object detection. We utilize the appearance embeddings from UNION to create pseudo-class labels and train a multi-class detector.
Related work
Here, we review the work most related unsupervised multi-modal 3D object detection.
Unsupervised 2D object discovery methods learn to detect objects in images without manual annotations and usually only camera information. In general, heuristics utilized distinguish foreground from background. Some methods [10, 12, 13, 23] perform co-segmentation, which is the problem of simultaneously dividing multiple images into segments corresponding to different object classes. These methods rely on strong assumptions about the frequency of the common objects and the required computation scales quadratically with the dataset size.
Table 1: Overview of existing methods for unsupervised 3D object discovery. In the modality column, L. and C are abbreviations for LiDAR and camera, respectively. Kickstart indicates what object types are used for the first round of training, i.e. the training before the self-training. and S and are abbreviations for static and dynamic objects, respectively. These methods rely on repeated traversals of the same location for extracting dynamic objects from the scene.
Method
Year
Modality
Kickstart
Key Novelty
Code
MODEST [32]
2022
Dt
LiDAR-based self-training
Najibi al. [20]
2022
L
D
Spatio-temporal clustering
X
OYSTER [33]
2023
L
S+D
Near-long range generalization
DRIFT [17]
2023
Dt
Heuristic- based reward function
CPD [29]
2024
L
S+D
Prototype-based box refinement Trajectory optimization
LISO [2]
2024
D
LSMOL [27]
2022
L+C
D
Multi-modt modil self-trainie training Appearance- based pseudo-classes
UNION (ours)
2024
L+C
S+D
3

---

In contrast, recent advances in representation learning [4, 21] show that the features of models trained with self-supervised learning contain implicit information about the semantic segmentation of a camera image. As result, several works [16, 22, 25, 26, 28] exploit these models to get pseudo- bounding boxes and segmentation masks for training unsupervised detection models. L.OST [22] leverages the features of the self-supervised method DINO [4] to extract foreground instances, and the clustered foreground instances are used to train a multi-class object detector. TokenCut [28] revisits LOST and proposes to build graph of connected image regions to segment the image. CutLER [25]
extends TokenCut for detecting multiple objects in a single image.
We also exploit model trained with self-supervised learning [21]. However, we use the features to embed the appearance of 3D spatial clusters instead of differentiating between background and foreground in 2D camera image. LOST extends unsupervised 2D object discovery to detection by creating pseudo-class labels using the CLSS token of DINO for each object. Analogously, we extend the 3D case from discovery to detection, but our semantic embedding strategy differs from LOST. We extract camera- based features for a spatial cluster using its LiDAR points and aggregate these features to obtain the appearance embedding for a cluster.
Unsupervised 3D object discovery methods often only use LiDAR data detect objects in 3D space, Table 1 provides an overview of the related work and is further explained below. MODEST [32] exploits repeated traversals of the same location for extracting dynamic objects from a LiDAR point cloud and uses self- self-training to also learn to detect static objects. DRIFT [17] also relies on repeated traversals for discovering dynamic objects but has different self-training protocol. It uses heuristic- based reward function to rank the predicted bounding boxes, and the highest-ranked bounding boxes are used to retrain the detector. The need for repeated traversals imposes a strong requirement on the data collection and limits the amount of data that can be used from existing datasets. Similarly to DRIFT, CPD [29] uses multi-class bounding box templates based on size information from Wikipedia to obtain set of bounding boxes used for bounding box refinement, i.e. re-size and re-location.
Najibi et al. [20] and LISO [2] use self superviee scene flow estimation in combination with tracking to detect dynamic objects, and train a detector using these objects. On the other hand, OYSTER [33] extracts object clusters close to the LiDAR sensor and uses the translation equivariance of a CNN in combination with data augmentation to learn near-range to long-range generalization for discovering objects. As a result, OYSTER can detect both static and dynamic objects. However, OYSTER cannot distinguish foreground objects from the background, and consequently, the performance suffers from the number of detected false positives. In contrast, LSMOL uses camera data to learn to detect static objects in its multi-modal self-training. It uses self-supervised scene flow estimation to identify dynamic objects. After that, it exploits the similarity of visual appearance between static and dynamic objects to learn to detect both during self-training.
Similarly to [2. 20, 27], we use self-supervised scene flow [15] for detecting dynamic objects instead of relying on repeated traversals of the same location such as MODEST and DRIFT. In contrast to existing methods, we use the camera- camera-based appearance of dynamic objects to distinguish between background and static foreground objects before training the detector. As a result, we can directly discover static and dynamic objects without needing self-training. [17, 29] use multi-class bounding box templates to score and refine pseudo-bounding boxes. However, there is fundamental difference between these methods and our method in how the pseudo-classes are used to supervise the detector. [17, 29] provide class-agnostic useudo-bounding boxes for training detectors. In contrast, we use the appearance embeddings from UNION to create pseudo-class labels and train object classification. By doing this, we are the first to extend 3D object discovery to 3D object detection. Lastly, we do not assume any class-specific geometric prior (c.g. bounding box template) during training and inference.
Self-supervised feature clusters have been utilized for tasks related to unsupervised object detection. Drive&Segment [24] uses LiDAR segments in combination with self-supervised camera features to compute pixel-wise pseudo-labels for 2D unsupervised semantic segmentation (USS). The method constructs LiDAR- based 3D segments and computes an appearance embedding for each segment using DINO [4]. After that, lseudo-labels (i.e. cluster IDs) are obtained by clustering all appearance embeddings. CAUSE [14] also uses self-supervised features to obtain class prototypes for 2D USS.
Similarly to Drive&Segment, we compute an object appearance embedding for segments. However, we exploit appearance to distinguish foreground from background instances instead of assigning each camera pixel to one of the pseudo-classe More specifically, we are the first to select and discard segments based on the fraction of dynamic instances belonging to their appearance clusters.

---

3 Method
This section introduces our method UNION for unsupervised 3D object detection of static and dynamic mobile class instances. Our work leverages recent advances in self- seff-super learning to overcome the difficulty of distinguishing between static foreground and background instances in sparse 3D point clouds. First, we explain the task of unsupervised 3D object detection. After that, we describe the workings of UNION for learning to detect mobile objects without using manual labels.
3.1 Unsupervised 3D Object Detection
The task of 3D object detection is to detect, i.e. localize and classify, objects in 3D space. We consider upright 3D bounding boxes. Hence, it is assumed that roll and pitch are equal to zero. Each bounding box b = (ar, y, z, w. h, 0) consists of the center position (z, y, z), the length, width, and height (4, w. h). and the heading 0. In general, C unique semantic classes oą of interest are defined for a detection setting, and each object instance is assigned to one of the classes.
In the case of supervised learning, all objects labeled, and thus, they can be used targets for training a learnable detector. However, for the unsupervised case, only raw data is available, which means that heuristics should be used to create pseudo-labele that replace manual labels. These pseudo-labels are an approximation of the real labels that would be obtained by carefully manually annotating the scenes. As result, the performance of supervised training is the upper bound for the performance that can be achieved by using the same amount of data for training.
We aim to develop framework for training existing 3D object detectors without relying manual labels. Assume we have a mobile robot equipped with calibrated and sene-yynchronized sensors, including LiDAR, camera, GPS/GNSS, and IMU, Also, assume that LiDAR and the camera have an overlap in their field of view (Fo i.e. parts of the environment are sensed by both sensors.
Input. The input of the framework consists of the multi-modal data collected during one or multiple traversals by the robot described above. Since this is raw data and no manual annotations are involved, such a dataset is easy to acquire. A traversal is sequence of T time steps, and for each time step t, a single LiDAR point cloud and Q multi-view camera images are available. Let P; E RLx3 denote the L-point 3D LiDAR point cloud, and let Ig.5 E R #x W x3 denote RGB image g captured by camera q- q E Q. H and W denote image height and width, respectively. The projection matrix is available for projecting 3D points on the 2D image plane for each camera. In addition, extrinsic transformations and ego-motion compensation can be used to transform data across sensor frames and time.
Output. The framework's 's output consists of set of bounding boxes and pseudo-class labels for each time step of all traversals. These pseudo-labels can be used train existing 3D object detectors using their original protocol, where the only difference is that the targets used during training are pseudo-labels instead of manual labels. Let B, and C, denote the sets of bseudo-bounding boxes and pscudo-class labels for time step t, respectively. Both sets consist of N pseudo-instaneee where each pseudo-instance is defined by a gseudo-boundieg pseudo-class label Cn,t. Assume that there are K pseudo-classes, i.c. Cn,t E {1, {l., K) There is only one pseudo-class for class-agnostic object detection, and thus C,,t for all object instances.
box b,,t = (I,, 3/ 2, L, w. h, 0) and
3.2 Overview of UNION
The pipeline of UNION consists of two stages: (1) object proposal generation and (2) mobile object discovery. In Figure 1, these stages are represented by steps 1-3 and 4-5, respectively. The objective of the first stage is to generate a set of 3D object-like clusters and gather information about each cluster, i.e. the visual appearance and motion. This information is then used in the second stage to identify groups of mobile class instances and generate pseudo-labels. These labels are utilized to train existing object detectors (step 6 in Figure 1). The trained detector is used identically to the fully supervised case during inference. Section 3.3, we explain the process of object proposal generation. After that, we describe the discovery of mobile objects in Section 3.4.
3.3 Object proposal generation
Here, we discuss the four components used for generating object proposals (steps 1-3 in Figure 1).
5

---

Ground Removal. The first step for generating object proposals is to extract the non ground (ng) points Png. from all LiDAR point clouds Pt as the non-ground points may belong to mobile objects. We assume that the ground is flat, and we fit a linear plane for each point cloud using RANSAC [9]. We use an inlier threshold of cm for fitting the plane and consider all points that are more than 30 cm above the fitted ground plane as the non-ground points.
Spatial Clustering. The non-ground points are spatially clustered get object proposals, i.e. 3D segments. To deal with the sparsity of the point clouds, we aggregate for each non- non-ground point cloud P Rgg. the past and next M non-ground point clouds. We set M equal to 7 to obtain aggregated point cloud that is based on 15 scans, After aggregating the points clouds, HDBSCAN [19] is used to extract N object proposals On. These object proposals can be part of the foreground as well as the background. We set the minimum cluster size and cluster selection epsilon to 16 points and 0.50 m, respectively. Step in Figure illustrates the generation of these class-agnostic 3D object proposals.
Motion estimation. We estimate the motion status of the object proposals to determine whether each proposal is static or dynamic. The object proposals contain temporal information the non-ground points from multiple time steps have been aggregated before the spatial clustering. In other words, the motion can be observed when the 3D points of an object proposal are split into different sets based on their time step, i.e. undoing the aggregation. This is shown by step 2 in Figure 1.
We estimate the motion of each proposal using modified version of the SOTA self-supervised scene flow estimation method ICP-Flow [15]. We assume that mobile objects only move relative to the ground plane, so we limit scene flow estimation to 21) translation plus yaw rotation. A motion estimate Un is obtained for each object proposal by calculating the velocity magnitude of the estimated motion. We consider all object proposals with at least 0.50 m/s to be dynamic objects. As a result, we obtain the sets of static and dynamic object proposals O and OP for time step t, respectively.
Visual appearance encoding, We observe that objects from the same semantic class look visually similar. Therefore, we aim to compute a camera-base based encoding that can be used to search for visually similar-looking static objects using a reference dynamic object. We leverage the off-the-shelf vision foundation model DINOv2 [21] for encoding the camera images. This vision foundation model is part of a family of self- -uferuupee learning algorithms that leverage the concept of knowledge distillation to train neural networks without requiring any labeled data.
We compute feature map Fq.. E IR H xCr for each camera image In..- Here, HF, Wr, and Cp indicate the feature map's height, width, and number of feature channels, respectively. Subsequently, we use our embedding module to compute a visual appearance embedding on E RCr for each object proposal On. In the module, the LiDAR points of an object proposal On,e are projected the image plane, and we assign to each point p RS a camera-based feature vector fp E RCF using the computed feature map Fq.t- After that, single object proposal's feature vectors are averaged to obtain the visual appearance embedding an This process is illustrated by step 3 in Figure 1.
3.4 Mobile object discovery
Current methods for 3D object discovery start the training of the detector with dynamic objects only. which means that static mobile objects such as parked cars serve as negatives. This causes an inconsistent supervision signal during neural network optimization as the shape of those static mobile objects can be very similar to dynamic mobile objects of the same class. As result, the detector may learn to exploit the small differences in data distribution between static and dynamic mobile objects to be able to reduce the amount of false positives, i.e. static objects that are detected. The common strategy to improve detection performance is to perform expensive self-training in which static objects are added to the training targets. However, this also adds background instances, which lowers the detector's 's precision.
We aim to create pseudo- labels for both static and dynamic foreground instances because (1) this gives more consistent supervision signal during training, (2) it enlarges the set of targets for training. i.e. more samples, and (3) this removes the need for computationally expensive self self-training. We are confident that the set with dynamic proposals consists of mobile objects but the set of static object proposals contains both background objects (e.g. houses, poles, and bridges) and static foreground objects (e.g- parked cars and standing pedestrians). On the other hand, static and dynamic mobile objects from the same class have similar visual appearance, which is different from background

---

objects. Therefore, we exploit the visual appearance embeddings to search for static mobile objects in the set of static object proposals.
We cluster the appearance embeddings using the K-Means algorithm [18] to group visually similar- looking object proposals (see step 4 in Figure 1). The number of clusters K1 is set to 20. The clustering is done for all proposals together to ensure that there are enough dynamic objects to be able to differentiate between mobile and non-mobile clusters. As the obtained clusters differ in size, we calculate the fraction of dynamic instances X for each cluster to classify the clusters. We consider appearance clusters with at least % dynamic object proposals as mobile clusters, while the other clusters are non-mobile clusters. This is illustrated by step 5 in Figure 1. Classifying the appearance clusters means that both the of static object proposals and the set of dynamic object proposals are split into mobile and non non-mo object instances. As result, we can discover static mobile objects without requiring self- self-training, and we are robust against objects falsely labeled as dynamic.
We discard all non-mobile objects to obtain the set of mobile objects Omobile and compute a pseudo- bounding box (Ir, y, 2, w. h, 0) for each mobile object o; using the 3D bounding box fitting algorithm of MODEST [32]. These aseudo-bounding boxes can be used to train existing detectors (see step in Figure 1). In contrast existing methods that only do class-agnostic detection, we extend 3D object discovery to 3D object detection by clustering our mobile objects into K2 appearance-basse ggeudo-classed using K-Means, As result, we obtain a pseudo-class label c, E {1. ..., K2 } for each object that can be used for training multi-class detector, i.e. predicting bounding boxes and classes.
Optional for experiment 2 (multi-class detection). During inference, the appearances are exploited to match the aseudo-classed to R real classes. First, appearance prototypes are determined for the classes. The K₂ appearance cluster centers from K-Means used appearance prototypes ag E RCr, k {1... K₂} } for the pseudo-classed and for each real class, we use a single example image compute real class appearance prototypes ar E RCF Subsequently, the cosine similarity calculated between the prototypes of pseudo-classes and real classes, and each pseudo-class is assigned to the real class with the highest similarity with its appearance prototype. Note that this matching step requires negligible supervision as we do one-shot association during inference.
Experiments
This section describes the experimental setup, the used baselines, and the conducted experiments.
4.1 Experimental setup
Dataset. We evaluate our method on the challenging nuScenes [3] dataset. This large-scale autonomous driving dataset for 3D perception captured in diverse weather and lighting conditions across Boston and Singapore. It consists of 700, 150, and 150 scenes for training. validation, and testing, respectively. A scene is sequence of 20 and is annotated with 2 Hz. Each frame contains one LiDAR point cloud and six multi-view camera images.
Mobile object classes. The nuScenes dataset has 10 detection classes. Eight of these relate to mobile objects, and we only use these for evaluation. Note that the labels of these mobile object classes
(a) HDBSCAN
(b) Scene Flow
(c) UNION
(d) Ground Truth
Figure 3: Qualitative results for the UNION pipeline compared to the ground truth annotations. (a) HDBSCAN (step in Figure 1): object proposals (spatial clusters) in black. (b) Scene flow (step 2 in Figure 1): static and dynamic object proposals black and red, respectively. (c) UNION: static and dynamic mobile objects in green and red, respectively. (d) Ground truth: mobile objects in blue.
7

---

Table Class-agnostic object detection on the nuScenes validation set. Results are obtained by training CenterPoint [31] with the generated gseudo-bounding boxes. L and C are abbreviations for LiDAR and camera, respectively. Best performance in bold, and second-best underlined. ST stands for self-training which increases the computational cost of training. Results taken from [2].
Method
Labels
ST
APT t
NDS t
ATE +
ASE 4
AOE 4
AVE +
Supervised %
Human Human
X
27.8
26.3
0.456
0.309
1.302
1.307
Supervised 10 %
X
61.2
56.7
0.255
0.221
0.462
0,455
Supervised 100 %
Human
X
76.5
68.7
0.209
0.198
0.241
0.305
HDBSCAN [16]
13.8
15.9
0.574
0,522
1.601
1.531
OYSTER [29]
L
9.1
11.5
0.784
0,521
1.514
LISO [1]
L
10.9
13.9
0.750
0.409
1.062
UNION (ours)
L+C
X
38.4
31.2
0.589
0.497
0.874
0.836
are not used during training as our method UNION is fully unsupervised. For class-agnostic object detection (Section 4.3), the eight classes are grouped into a single object class for evaluation. On the other hand, for multi-class object detection (Section 4,4), we create three different classes, namely, (1) vehicle, (2) pedestrian, and (3) cyclist. The vehicle class combines the bus, car, construction vehicle, trailer, and truck classes, and the cyclist class combines the bicycle and motorcycle classes.
Detector. We CenterPoint [31] for all our experiments. CenterPoint is SOTA LiDAR- R-based 3D object detector. In contrast to traditional detection methods that generate bounding boxes around objects, CenterPoint uses keypoint- based approach, detecting and tracking the centers of objects.
Metrics. The two main metries that we consider are average precision (AP) [8] and the nuScenes detection score (NDS) [3], which are computed using the standard nuScenes evaluation protocol. AP obtained by integrating the recall versus precision curve for recalls and precisions larger than 0.1, and averaging over match thresholds of 0.5 m, 1.0m, 2.0 and 4.0 m. NDS is weighted average of AP and five true positive errors, i.e. translation (ATE), scale (ASE), orientation (AOE), velocity (AVE), and attribute (AAE). We do all our experiments without attribute estimation similar to [2], and thus, we set the true positive errors for attribute to 1.0 by default.
Baselines. We compare against three unsupervised baselines for class-agnostic object detection, namely (1) HDBSCAN [19], (2) OYSTER [33], and (3) LISO [2]. In addition, we also compare to training CenterPoint using supervised learning with different subsets of the labels, i.e. 1 %, 10 %, and 100 %. We cannot compare MODEST [32], Najibi al. [20]. DRIFT [17], CPD [29], and LSMOL [27]. MODEST and DRIFT need multiple traversals of the same location, which does not hold for all sequences in the nuScenes dataset. Furthermore, Najibi et al., CPD, and I.MOL did not release their code (at the time of submission) and did not provide performance nuScenes.
Existing 3D object discovery methods cannot do multi-class object detection. Therefore, we HDBSCAN in combination with class-based information as a baseline. Analogous to the class- agnostic setting, we also train CenterPoint using supervised learning for multi-class object detection.
4.2 Implementation
We use the framework MMDetection33 [5] for all our experiments and use their implementation of CenterPoint. More specifically, we use CenterPoint with pillars of 0.2 m as voxel encoder, do not use test time augmentation, and train for 20 epochs with a batch size of 4. All class-agnostic experiments are done without class-balanced grouping and sampling (CBGS) [34], while we do use CBGS for multi-class experiments to improve the performance of tail classes. The camera images were encoded with large vision transformer (ViT-L/14) [7] trained using DINOv2 [21] with registers [6]. We used 8 NVIDIA V100 32 GiB GPUs for conducting the experiments. The hyperparameters of UNION were tuned visually on ten frames of the nuScenes training split. We made our code publicly available.
4.3 Class-agnostic object detection
We evaluate the performance of UNION for class-agnostic object detection on the nuScenes validation split. As shown in Table 2, UNION outperforms all unsupervised baselines in terms AP and NDS. The best-performing unsupervised baseline is HDBSCAN, and UNION achieves an AP of more than
8

---

twice the AP of HDBSCAN. Both OYSTER and LISO score significantly lower than UNION despite
using tracking in combination with self-training to improve detection performance.
When UNION is compared to supervised training, it be that outperform training with 1 % of the labels, but we are still behind the performance
Dynamic Object Proposal Fraction (%)
35
of using 10 % of the labels. This indicates that the dataset size and the quality of the targets used for
30
25
training significantly impact detection performance. This is especially true for fine- tine-grained labels such as the orientation, the orientation convention depends on the object type. For example, determining the ori- entation of car is relatively easy as the length of the
20
15
10
Threshold 5%
car is typically much larger than the width, while for pedestrians, both dimensions are roughly the same. As result, unsupervised approaches have difficulty determining the correct orientation for objects such as pedestrians.
15
20
Appearance Cluster ID
Figure 4: The dynamic object proposal frac- tions the visual appearance clusters. We use threshold of 5 % for selecting clusters.
Figure provides qualitative results the generated pseudo-boute bounding boxes for example scene that was used for training the detector. The ground truth boxes are also shown. The figure shows the second sample from scene- 100 (training dataset). It can be seen that the scene flow can identify multiple dynamic objects, and the appearance clustering can discover static mobile objects, including vehicles and pedestrians, using those dynamic instances.
Figure 4 shows the percentage of dynamic object instances in the various appearance clusters, sorted by increasing percentage. We see clear uptick at around %, which we use in the experiments as threshold for distinguishing clusters of non-mobile objects (blue) versus mobile objects (orange).
Ablation study We now investigate the contri- bution of the various UNION components. Two intermediate representations of UNION can used for generating lseudo-labels, namely (1) the output the spatial clustering (step in Figure 1) and (2) the output of motion estimation (step 2 in Figure 1). The output of the spatial clustering is identical to the HDBSCAN baseline in Table 2. The motion estimation differs from the spatial clustering output in that the points of dynamic object proposals are corrected for the estimated motion. The results shown in Table for dif- ferent component-basee that improves performance. This aligns with our expectation that appearance clustering can select mobile objects from the sets of static and dynamic proposals while discarding the background objects.
Table 3: Class-agnostic object detection the nuScenes validation set for different configu- rations of UNION. Results are obtained by train- ing CenterPoint [31] with the
boxes. Best performance in bold.
Method
AP
NDS
HDBSCAN
13.8
15.9
+ Motion Estimation
14.1
20.0
+ Appearance Clustering
38.4
31.2
pseudo-labels. As can be seen, appearance clustering is the main component
Ablation study II. Table 4 compares UNION's class-agnostic 3D object detection performance for different camera encoders, namely DINOv2 [21] and I-JEPA [1]. DINOY2 can process high- resolution images, such as the camera images from nuScenes. In contrast, I-JEPA can only process square-shaped images maximum 448 by 448 pixels. As a result, the obtained feature maps from I-JEPA have lower resolution than the with DINOv2 outperforms UNION with I-JEPA by 15.6 in AP and 8.4 in NDS. Please note that our main paper contributions do not depend specific canonical steps (e.g. image encoding, scene flow estimation). If better approaches become available, UNION can incorporate them.
from DINOv2. The table shows that UNION
Table Image encoder ablation study for UNION. Best performance in bold.
Method
AP
NDS
ATE
ASE
AOE
AVE
DINOv2 ViT-L/14 w/ registers [21]
38.4
31.2
0.589
0.497
0.874
0.836
I-JEPA ViT-H/16 [1]
22.8
22.8
0.561
0.486
0.953
0.865

---

4.4 Multi-class object detection
As second type of experiment, we perform multi-class object detection on nuScenes with 3 different semantic classes, namely vehicle, pedestrian, and eyclist, see Section 4.1. The mobile objects discovered by UNION are clustered into K₂ pseudo-classes. For evaluation, we have used three example instances from the nuScenes training dataset to associate each pseudo-class with one of the 3 real classes. This association procedure is described in Section 3.4. We also assign real classes to the class-agnosti predictions of HDBSCAN and UNION from Section 4.3. We do this by computing a prototype bounding box for each class, i.e. we select the bounding box with the median 2D area. Subsequently, assign each class-agnostic bounding box to the real class of which the prototype bounding box has the highest intersection over union (IoU). We indicate this associating by size prior.
As shown in Table 5, UNION trained with 5 pseudo-classes performs the best and outperforms both HDBSCAN and UNION with the size prior in terms of AP and NDS. Differences pedestrian detection performance mainly cause this. The vehicle detection performance of UNION-05pe is slightly worse. We observe that the cyclist performance equal to zero for all configurations. From the pseudo-classes of multi-class UNION, there were 1, 1, 3. and 4 pseudo-classes assigned to the cyclist class for 5, 10, 15, and 20 pseudo-classes, respectively. Thus, it is not the case that all pseudo-classes are assigned to either the vehicle pedestrian class. However, the nuScenes evaluation protocol only integrates the precision-recal curve for precision and recall larger than 0.1. Therefore, also evaluated without clipping the precision-recall curve shown in the last column of Table 5, The results show that UNION-20pe significantly outperforms the baselines.
Table 5: Multi-class object detection on the nuScenes validation set. Results are obtained by training CenterPoint [31] with the generated pseudo-bounding boxes. SP stands for size prior and indicates that agnsstte predictions from Table are assigned to real classes based on their size. UNION-Xp stands for UNION trained with X pscudo-classes. L and C are abbreviations for LiDAR and camera, respectively. Best performance bold, and second-best underlined. Without clipping the precision-rec recall curve, clipping is the default for nuScenes evaluation [3].
Mobile Objects NDS
Vehicle AP
Ped.
Cyclist AP f
Cyclist APt t
Method
Labels
MAP
AP f
Supervised %
Human
24.3
28.3
39.3
31.8
1.8
4.7
Supervised 10
Human
45.9
47.9
65.3
57.6
14.9
22.3
Supervised 100 %
Human
67.4
62.6
80.7
77.7
43.7
52.5
HDBSCAN [16] SP
L
5.0
13.0
14.6
0.4
0.0
1.3
UNION (ours) + SP
L+C
12.7
19.7
34.8
3.4
0.0
1.6
UNION-05pc (ours)
L+C
24.0
24.0
30.3
41.6
0.0
0.8
UNION- 10pe (ours)
L+C
19.9
21.7
27.3
32.5
0.0
0.5
UNION-15pc (ours)
L+C
18.5
21.2
25.7
29.9
0.0
0.4
UNION-20pc (ours)
L+C
17.9
21.7
23.7
29.9
0.0
4.2
Conclusion
We proposed UNION, the first framework that exploits LiDAR, camera, and temporal information jointly for generating pscudo-bounding boxes to train existing object detectors in an unsupervised manner. Rather than training object detector to distinguish foreground and background objects, perform multi-clas object detection by clustering the visual appearance of objects and using them pseudo-class labels. We reduce computational time by avoiding iterative training protocols and self-training. We evaluated our method under various settings and increase the SOTA performance for unsupervised 3D object discovery, i.e. UNION more than doubles the average precision to 38.4.
Limitations and future work. possible limitation our work that we make implicit assumptions about the occurrence frequency of objects by clustering the object proposals in the appearance embedding feature space. Mobile objects that are rare will likely grouped with other objects, and as result, these objects may be discarded when grouped with static background objects. Future work entails extending our method to better deal with these rare classes. addition, the motion estimation could be based on radar detections as radars offer instant radial velocity estimation.
10

---

Acknowledgments and Disclosure of Funding
This research has been conducted as part of the EVENTS project, which funded by the European Union, under grant agreement No 101069614, Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent. M. Rabbat, Y. LeCun, and N. Ballas. Self- supervised learning from images with a joint-embeuding predictive architecture. In Computer Vision and Pattern Recognition (CVPR), pages 15619-1 15629,, 2023.
[2] S. Baur, F. Moosmann, and A. Geiger. LISO: Lidar-only self- self-supervised 3D object detection. In European Conference on Computer Vision (ECCV), 2024. 2, 2,3, 8
[3] H. Caesar, V. Bankiti, A.H. Lang. S. Vora, V.E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and
O. Beijbom. nuScenes: A multimodal dataset for autonomous driving. In Computer Vision and Pattern Recognition (CVPR), pages 11621-11631, 2020. 3, 7, 8, 10
[4] M. Caron, H. Touvron, L. Misra, H. Jégou. J. Mairal, P. Bojanowski. and A. Joulin. Emerging properties in self-supervised vision transformers. International Conference on Computer Vision (ICCV), pages 0650-9660, 2021. 2,4
[5] MMDetection3I Contributors. MMDettectoon3D: Open aa next-generation platform for general 3D object detection, https://githee com/ open- mml matee/mmmabbbbbabbm wwtpaaaammdmeettiiioooo on3d, 2020.
[6] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. International Conference on Learning Representations (ICLR), 2024. 8
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, Gelly, J. Uszkoreit, and Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021. 8
[8] M. Everingham, L. Van Gool, C.K.I. Williams, Winn, and A. Zisserman. The pascal visual ohject classes (voc) challenge. International Journal of Computer Vision (IJCV), 88:303-338, 2010.
[9] M.A. Fischler and R.C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395 1981. 6
[10] K. Hsu, Y. Lin, and Y. Chuang. Co-attention CNNs for unsupervised object co-segmentation. In Interna- tional Joint Conference on Artificial Intelligence (IJCAI), 2018.
[11] Y. Hu, J. Yang, I. Chen, K. Li, C. Sima, X. 7hu, S. Chai, S. Du, T. Lin, W. Wang, al. Planning-oriented autonomous driving. In Computer Vision and Patiern Recognition (CVPR), pages 17853-17862, 2023.
[12] A. Joulin, F. Bach, and J. Ponce. Discriminative clustering for image co-segmentation. In Computer Vision and Pattern Recognition (CVPR), pages 1943- 1950, 2010.
[13] A. Joulin, F. Bach, and J. Ponce. Multi-class coscgmentation. In Computer Vision and Pattern Recognition (CVPR), pages 249, 2012.
[14] J. Kim, B. Lee, and Y.M. Ro. Causal unsupervised semantic segmentation. arXiv:2310.07377 2023. 4
arXiv preprint
[15] Y. Lin and H. Caesar. ICP-Flow: LiDAR scene flow estimation with ICP. In Computer Vision and Pattern Recognition (CVPR), 2024. 4,
[16] Z. Lin, Z. Yang, and Y. Wang. Foreground guidance and multi-layer feature fusion for unsupervised object discovery with transformers. In Winter Conference on Applications of Computer Vision (WACV), pages 4043-4053, 2023. 4
[17] K. Luo. Z. Liu, X. Chen, Y. You. S. Benaim, C.P. Phoo, M. Campbell, W. Sun, B. Hariharan, and K.Q. Weinberger. Reward finetuning for faster and more accurate unsupervised object discovery. Neurul Information Processing Systems (NeurIPS), 36, 2023. 3, 4,
[18] J. MacQueen. Some methods for classification and analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Probability (BSMSP), volume 1, pages 281-297, 1967. 7
11

---

[19] L. McInnes. J. Healy, and S. Astels. HDBBSCNN Hierarchical density based clustering, Journal of Open Source Software (JOSS), 2(11): 2017. 6,
[20] M. Najibi, J. Ji, Y. Zhou, C.R. Qi, X. Yan, S. Ettinger, and D. Anguelov. Motion inspired unsupervised perception and prediction in autonomous driving. In European Conference on Computer Vision (ECCV), pages 424-443, 2022. 3, 8
[21] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby, et DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), pages 2835 8856, 2024. 2, 4, 6, 8, 9
[22] O. Siméoni, G. Puy. H.V. Vo, Roburin, Gidaris, A. Bursuc, Pérez, Marlet, and J. Ponce. Localizing objects with self-supervised transformers and no labels. In British Machine Vision Conference (BMVC), 2021. 2,
[23] S. Vicente, C. Rother, and V. Kolmogorov. Object cosegmentation. In Computer Vision and Pattern Recognition (CVPR), pages 2217-2224, 2011. 3
[24] A. Vobecky, D. Hurych, O. Siméoni, S. Gidaris, A. Bursuc, P. Pérez, and J. Sivic. DrivenSegment: Unsupervised semantic segmentation of urban scenes via cross-modal distillation. In European Conference Computer Vision (ECCV), pages 478 495, 2022. 4
[25] X. Wang, R. Girdhar, S.X. Yu, and I. Misra. Cut and learn for unsupervised object detection and instance segmentation. In Computer Vision and Pattern Recognition (CVPR), pages 3124-3134, 2023. 4
[26] X. Wang, Z. Yu, S. De Mello, J. Kautz, A. Anandkumar, C. Shen, and J.M. Alvarez. FreeSOLO: Learning to segment objects without annotations, In Computer Vision and Pattern Recognition (CVPR). pages 14186, 2022. 2.4
[27] Y. Wang. Y. Chen, and Z. Zhang. 4D unsupervised object discovery. In Neural Information Processing Systems (NeurIPS), pages 13563-35575 2022. 3, 4,8
[28] Y. Wang, X. Shen, S.X. Hu, Y. Yuan, J.L. Crowley, and D. Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Computer Vision and Puttern Recognition (CVPR). pages 14543-14553, 2022. 4
[29] H. Wu, Zhao, X. Huang, C. Wen, X. Li, and C. Wang. Commonsense prototype for outdoor unsupervised 3D object detection. In Computer Vision and Pattern Recognition (CVPR), pages 14968 14977, 2024. 3, 4, 8
[30] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S.C.H. Hoi. Deep learning for person re-identification: survey and outlook. Transactions on Pattern Analysis and Machine Intelligence (PAMI), (4(6)22772-2893 2021. 2
[31] T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3D object detection and tracking. In Computer Vision and Partern Recognition (CVPR), pages 11784-111 11793, 2021. 8, 9, 10
[32] Y. You, K. Luo, C.P. Phoo, W. Chao, W. Sun, B. Hariharan, M. Campbell, and K.Q. Weinberger. Learning to detect mobile objects from LiDAR scans without labels. In Computer Vision and Puttern Recognition (CVPR), pages 11 1130-1 140, 2022. 2, 3, 4, 7,
[33] L. Zhang, A.J. Yang, Y. Xiong, S. Casas, B. Yang, M. Ren, and R. Urtasun. Towards unsupervised object detection from LiDAR point clouds. Computer Vision and Pattern Recognition (CVPR), pages 9317-9328, 2023. 3, 4,8 8
[34] B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced grouping and sampling for point cloud 3D object detection. arXiv preprint 29iv:1908.09992 2019. 8
[35] B. Zitkovich, Yu, Xu, P. Xu, T. Xiao, Xia, J. Wu, P. Wohlhart, Welker, A. Wahid, al. RT-
2: lision-languaag-acctoon models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRI.), pages 2165-2183, 2133, 2023. 2
12

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This has been verified by the reviewers.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the last paragraph of the Conclusion section.
13

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This is not theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details are provided in the paper. In addition, we release the code and data GitHub for full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
14

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We release the code and data on GitHub, so full open access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: SoooeeuumissiooPPoo
/ //nips... /tippscc/pput wttp:cccppppppccccccceecocoppcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We described the used hyperparameters and framework (MMDeteccion3))) In addition, we release the code and data GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Cross-validation could not be performed due to computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the subsection Implementation of the Experiments section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
16

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //wur//eee....ee cc Spppulic//thissuuidddeeopeeeeeee
Answer: [Yes]
Justification: Sic.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Work focuses the immediate method and performance benefits.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: There are no such risks seen.
Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All contributors mentioned authors. There is an open license for the data/software on the web crediting the authors.
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See earlier comment open access of data and software.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Does not apply.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Existing data was used, i.e. the nuScenes dataset.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
19


Reviews:
[{"review_id": "tI0fxvtmny", "reviewer": "Reviewer_LFQD", "summary": "This paper introduces UNION, a novel method for unsupervised 3D object detection that leverages object appearance-based pseudo-classes. This paper addresses the challenge of training object detection models without manual annotations by using spatial clustering and self-supervised scene flow to generate static and dynamic object proposals from LiDAR data. It then encodes the visual appearances of these proposals to distinguish static objects in the foreground and background. The main contribution is the method design that jointly uses camera, LiDAR, and temporal information to train existing 3D object detectors in an unsupervised manner.", "questions": "See weakness.", "limitations": "The authors have provided a discussion on the limitations of their work. They acknowledge that the method makes implicit assumptions about the occurrence frequency of objects.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper is well-structured, with a clear abstract, introduction, methodology, experiments, and conclusion sections that logically flow from one to the next.\n2.\tThe use of figures and diagrams, such as Figure 1, effectively illustrates the process and contributes to the clarity of the UNION method.\n3.\tThe use of pseudo-classes based on object appearance for training classifiers is innovative, offering a new way to tackle multi-class object detection without relying on manual annotations.\n4.\tThe UNION method introduces a new approach to unsupervised learning by combining spatial clustering, self-supervised scene flow, and visual appearance encoding in a synergistic manner. This represents a creative fusion of existing ideas applied to the problem of 3D object detection.", "weaknesses": "1.\tThe claim of being 'the first to do unsupervised multi-class 3D object detection' requires scrutiny. It's noted that other works, such as the one by Wu et al. presented at CVPR 2024, also delve into unsupervised multi-class 3D object detection. If the paper is accepted, it would be prudent to revise this statement to reflect the current state of research accurately and avoid overstating the novelty of the approach.\n\n2.\tThe decision to solely utilize the nuScenes dataset for experiments raises questions about the breadth of the evaluation. The Waymo dataset, with its denser point clouds, might offer a better testbed for supervised tasks. Furthermore, the official multi-class metric of the Waymo dataset, which assesses the detection of vehicles, pedestrians, and cyclists, provides a more straightforward comparison to fully supervised detectors. It would be beneficial to consider a comparative analysis using the Waymo dataset to strengthen the paper's findings. \nRebuttal may not have time for this work, but can be part of the future.\n\n3.\tThe observed zero AP for the cyclist class is concerning and warrants a deeper investigation. It is suggested that the discussion should be expanded to address potential solutions to this issue. While the less sample is a contributing factor, it's also essential to explore other possibilities, such as the high similarity in appearance between pedestrians and cyclists, which might lead to misclassification and the pedestrian labels actually being cyclist labels.\n\n[1] Wu, Hai, et al. \"Commonsense Prototype for Outdoor Unsupervised 3D Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback.\n\n**First method to do unsupervised multi-class 3D object detection**\n\nOur submission did not compare to [8] as it was not peer-reviewed back then.\nNow that it has been published at CVPR'23, we will add the paper to our related work section and Table 1 (the overview) to accurately reflect the current state of the research field.\nThis concurrent work does not use the nuScenes dataset, and their code has not yet been fully released, according to their GitHub page, so we cannot compare with them.\nWu et al. use multi-class bounding box templates based on Wikipedia to obtain a set of high-quality bounding boxes used for bounding box refinement, i.e. re-size and re-location.\nHowever, there is a fundamental difference between UNION and their methods of how the pseudo-classes are used to supervise the detector, i.e. class-agnostic versus multi-class detection.\nThey do not train object classification as their trained detector only outputs class-agnostic bounding boxes, i.e. they do class-agnostic object detection.\nIn contrast, we not only have a class-agnostic object detection experiment (experiment 1) but also have a multi-class experiment as clarified in the global rebuttal text (experiment 2).\nIn addition, we do not assume any class-specific geometry prior during training and inference.\nFor our multi-class object detection, we create $K_2$ appearance-based pseudo-classes as targets for training object classification and train CenterPoint in a multi-class fashion.\nWe can assign the learned pseudo-classes to real classes after training, i.e. during inference, as we match the appearance prototype of each pseudo-class with an example appearance of the real objects that are relevant during inference, such as vehicles, pedestrians, and cyclists.\nTherefore, we are actually the first method to train unsupervised multi-class 3D object detection.\nIn our paper, we will explain more extensively how our contribution differs from the existing methods, such as [8].\n\n**Results on Waymo**\n\nWe agree that results on the Waymo dataset [9] would improve the breadth of the evaluation.\nHowever, we do not have enough time during the rebuttal to provide these as Waymo is a very large dataset, and it would take more than one week in total to run the UNION pipeline plus training CenterPoint.\nTherefore, we consider Waymo results as future work.\n\n**Cyclist performance for multi-class detection**\n\nFrom the pseudo-classes of multi-class UNION, there are 1, 1, 3, and 4 pseudo-classes assigned to the cyclist class for 5, 10, 15, and 20 pseudo-classes, respectively.\nSo, it is not the case that all pseudo-classes are assigned to either the vehicle or pedestrian class.\nHowever, the nuScenes evaluation protocol discards the precision-recall curve where the precision or recall is lower than 10 percent.\nAs a result, the average precision for the cyclist class for multi-class UNION is equal to 0, while cyclists are detected.\nTherefore, we also evaluated without clipping the precision-recall curve **(see the last column in Table 2 in the rebuttal PDF)**.\nThe results show that UNION with 20 pseudo-classes significantly outperforms the baselines with the size prior.\n\n[8] Wu et al. (2024). Commonsense Prototype for Outdoor Unsupervised 3D Object Detection. In Conference on Computer Vision and Pattern Recognition (CVPR).\n\n[9] Sun et al. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In Conference on Computer Vision and Pattern Recognition (CVPR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sb3p3D9Vll", "reviewer": "Reviewer_ZwY5", "summary": "The paper introduces UNION, an unsupervised 3D object detection method designed to detect both static and dynamic objects without manual labels. UNION utilizes spatial clustering and self-supervised scene flow to generate object proposals and employs visual appearance encoding to distinguish between static and dynamic objects. This approach enables a single training process, enhancing performance while reducing computational costs. Experiments on the nuScenes dataset show that UNION significantly improves the state-of-the-art performance in unsupervised object discovery, doubling the average precision score to 33.9.", "questions": "See weaknesses.", "limitations": "The authors acknowledge the limitations of their work and plan to address these in future research.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The approach is quite novel, successfully eliminating the dependence on self-training.\n2. The presentation is clear and easy to understand.\n3. The paper effectively highlights the similarities and differences with related literature.", "weaknesses": "The experiments are relatively limited.\n1.  Besides DINOv2, how do other image encoders affect the model's performance?\n2. In Table 4, the cyclist AP is 0. Is there a more detailed investigation into the reasons for this? Given that, as a small object, the pedestrian class shows a significant improvement in AP with UNION-10pc compared to UNION+Size prior, why does the cyclist class not exhibit a similar improvement?\n3. In Fig. 3, 5% was chosen as the threshold. Why was this value selected? The authors should provide more discussion or experimental support for this choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback.\n\n**Different image encoders**\n\nSee global rebuttal text: 'Different image encoders'.\n\n**Cyclist performance for multi-class detection**\n\nFrom the pseudo-classes of multi-class UNION, there are 1, 1, 3, and 4 pseudo-classes assigned to the cyclist class for 5, 10, 15, and 20 pseudo-classes, respectively.\nSo, it is not the case that all pseudo-classes are assigned to either the vehicle or pedestrian class.\nHowever, the nuScenes evaluation protocol discards the precision-recall curve where the precision or recall is lower than 10 percent.\nAs a result, the average precision for the cyclist class for multi-class UNION is equal to 0, while cyclists are detected.\nTherefore, we also evaluated without clipping the precision-recall curve **(see the last column in Table 2 in the rebuttal PDF)**.\nThe results show that UNION with 20 pseudo-classes significantly outperforms the baselines with the size prior.\n\n**Threshold of 5 percent for the fraction of dynamic instances X**\n\nAs stated in the global response of the rebuttal, we have further improved the self-supervised scene flow component's recall by also estimating the velocity for spatial clusters with relatively few LiDAR points, and we now use only 20 appearance clusters for filtering the non-mobile instances.\nThis improved recall makes the distinction between non-mobile and mobile appearance clusters more obvious **(see Figure 1 in the rebuttal PDF)**.\nIn this figure, it can be visually observed that the first 13 clusters (indicated in blue) all have a relatively low dynamic instance fraction compared to the other 7 clusters (indicated in orange), i.e. the fraction of cluster 14 (considered a mobile cluster) is roughly three times as high as the fraction of cluster 13 (considered a non-mobile cluster).\nAs a result of this observed difference, we selected 5 percent for splitting the fractions into two groups: (1) clusters with very few dynamic instances (non-mobile clusters) and (2) clusters with many dynamic instances (mobile clusters).\nIn addition, we empirically observed that the detection performance is very robust when using the 5 percent threshold while varying, for example, the number of appearance clusters, or when the threshold itself is changed too to, for example, 7.5 percent.\nThe intuition is that there should be a clear distinction between the dynamic instance fraction of non-mobile clusters and mobile clusters because non-mobile instances do not have the potential to be dynamic.\nIn contrast, mobile instances move very often, as most are part of the traffic.\nThere are multiple reasons why the non-mobile clusters do have some dynamic instances, including (1) false positives from the scene flow component, i.e. static objects are estimated to be dynamic (e.g. dynamic tree), and (2) some instances may be part of an appearance cluster while having an appearance dissimilar to the other instances in the appearance cluster because the instance is close to the border of the appearance cluster in feature space.\nNote that using a threshold equal to 0 percent, i.e. just using all object proposals, is equal to the output of the scene flow component and achieves an average precision (AP) of less than half of the performance of UNION (see Table 3 in our submission).\nSo the appearance clustering component is essential for achieving high performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5qebW7ZEe6", "reviewer": "Reviewer_Gw7r", "summary": "This paper explores the challenge of unsupervised 3D object detection, introducing UNION. UNION leverages camera, LiDAR, and temporal information jointly to train 3D object detectors without relying on self-training. The approach demonstrates strong performance particularly on the nuScenes dataset. Additionally, the authors tackle unsupervised multi-class 3D object detection by clustering object appearance embeddings and employing these clusters as pseudo-class labels.", "questions": "See weakness.", "limitations": "The authors have discussed the limitations of the work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies a significant problem in unsupervised 3D object discovery.\n\n2. The paper is clearly written, and its motivation is straightforward.\n\n3. This method introduces visual embedding to enable certain static objects to be generated during object proposal generation.\n\n4. This method has achieved good performance on nuScenes dataset.", "weaknesses": "1. Compared to prior work, the method's novelty is weak. The core innovation lies in leveraging self-supervised visual features to aid in object proposal extraction, thereby eliminating the need for self-training to acquire static vehicle data. \n2. The core exploration of this work revolves around object appearance embedding. The current study only investigates the features of DINOv2. It remains to be explored how other self-supervised features perform in comparison, or whether combining different self-supervised methods could yield better results.\n3. As for training the multi-class detector, I believe the choice of K is crucial. The authors use K=10 as default. It should conduct an ablation study on the impact of K.\n4. As depicted in Table 2, the discussion centers around the challenges in orientation estimation. To enhance clarity, it would be beneficial to include detailed metrics such as ATE, AOE, etc.\n5. In spatial clustering, when utilizing point cloud aggregation, how to deal with the dynamic objects, such as multiple moving cars whose point clouds may overlap.\n6. As shown in Table 4, the AP for cyclists is always zero. Does this imply limitations of the method for detecting long-tail objects? What are some potential directions for improving detection of these less frequent objects in the future?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback.\n\n**The method's novelty**\n\nUNION extracts object proposals by spatially clustering the non-ground points from LiDAR.\nSubsequently, the velocity of each object proposal is estimated using self-supervised scene flow, and the cameras are leveraged to encode its visual appearance into an object appearance embedding.\nThese different types of information are then fused in a novel way to discover mobile objects in 3D space, such as vehicles, pedestrians, and cyclists, namely the visual appearances are clustered into $K_{1}$ appearance clusters and the fraction of dynamic object proposals per appearance cluster is calculated.\nThe appearance similarity between static and dynamic mobile objects is utilized for discriminating between static mobile objects (e.g. parked vehicles) and background instances (e.g. trees and buildings), i.e. appearance clusters are labeled as mobile when they have at least 5 percent dynamic objects.\nThe instances of these mobile appearance clusters form together the set of (class-agnostic) mobile objects and can be used to train any 3D object detector in an unsupervised manner.\nCompared to existing methods such as OYSTER [1] and LISO [2], we do not rely on computationally expensive self-training to detect static mobile objects because we can discover them entirely unsupervised using their visual appearance.\nThis significantly reduces the training time of the detector as only a single training is required instead of multiple training rounds (e.g. sequentially training 5-10 times) while at the same time obtaining a much better detection performance **(see Table 1 in our rebuttal PDF)**.\n\n**Different image encoders**\n\nSee global rebuttal text: 'Different image encoders'.\n\n**Ablation for number of pseudo-classes $K_2$ in multi-class detection**\n\nWe have added three hyperparameter configurations for the multi-class detection, namely 5, 15, and 20 pseudo-classes **(see Table 2 in the rebuttal PDF)**.\nThe table shows that UNION with 5 pseudo-classes achieves the highest average precision (AP) and nuScenes detection score (NDS).\nIn general, the multi-class versions of UNION get a slightly lower AP for the vehicle class than 'UNION+Size prior' but a significantly higher AP for the pedestrian class.\nAs a result, multi-class UNION gets a relatively high AP for both the vehicle and pedestrian class compared to the baselines using the size prior.\n\n**True positive metrics for nuScenes**\n\nWe have added the average translation error (ATE), average scale error (ASE), average orientation error (AOE), and average velocity error (AVE) to **Table 1 in the rebuttal PDF**.\nNote that we set the average attribute error (AAE) equal to 1 by default for the task of class-agnostic object detection because object classes have different attributes in nuScenes so attributes lose their meaning when all mobile classes are combined into a single class.\nTherefore, this error is not shown in the table.\nWe use the official nuScenes formula for calculating the nuScenes detection score (NDS).\n\n**Dealing with dynamic objects when aggregating point clouds**\n\nFor a frame, we aggregate the past $M$ and future $M$ non-ground point clouds, i.e. $2M+1$ LiDAR scans are aggregated into a single coordinate system, to obtain a denser point cloud.\nConsequently, the spatial clustering may create a single spatial cluster for dynamic objects that are close to each other during this time interval.\nWe use $M=7$ for our experiments, resulting in a time interval of 0.70 second, and notice that it rarely happens that different objects are merged into a single spatial cluster.\nA potential solution for this would be to estimate the motion vector for each point of the spatial cluster (scene flow), and determine whether the motion across the entire cluster is consistent.\nIf this is not the case, a cluster may be split into multiple spatial sub-clusters.\nHowever, this heavily relies on an accurate scene flow estimation.\n\n**Cyclist performance for multi-class detection**\n\nFrom the pseudo-classes of multi-class UNION, there are 1, 1, 3, and 4 pseudo-classes assigned to the cyclist class for 5, 10, 15, and 20 pseudo-classes, respectively.\nSo, it is not the case that all pseudo-classes are assigned to either the vehicle or pedestrian class.\nHowever, the nuScenes evaluation protocol discards the precision-recall curve where the precision or recall is lower than 10 percent.\nAs a result, the average precision for the cyclist class for multi-class UNION is equal to 0, while cyclists are detected.\nTherefore, we also evaluated without clipping the precision-recall curve **(see the last column in Table 2 in the rebuttal PDF)**.\nThe results show that UNION with 20 pseudo-classes significantly outperforms the baselines with the size prior.\n\n**Potential solutions for improving less frequent objects**\n\nUNION assumes that objects are mobile (i.e. have the potential to move) and that mobile objects with a similar appearance occur relatively often.\nThe appearance clusters are made using K-means, meaning that a mobile object type that rarely occurs will most likely be part of an appearance cluster belonging to objects with a dissimilar appearance.\nAs a result, the mobile objects may be considered non-mobile.\nSome directions to improve this are (1) exploring other clustering mechanisms than K-means for creating appearance clusters and (2) creating learnable appearance embeddings (self-supervised) that split mobile and non-mobile objects in the appearance feature space better.\n\n[1] Zhang et al. (2023). Towards unsupervised object detection from lidar point clouds. In CVPR.\n\n[2] Baur et al. (2024). LISO: Lidar-only Self-Supervised 3D Object Detection. In ECCV.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7D8WWzg49x", "reviewer": "Reviewer_JvDj", "summary": "This work presents an unsupervised 3D object detection method named UNION, which exploits LiDAR, camera, and temporal information jointly for generating pseudo bounding boxes to train existing object detectors. In addition, the authors introduce an appearance-based clustering method to generate pseudo class labels and train the object detector in a multi-class fashion.", "questions": "-- Could the authors explain why OYSTER and LISO are not included in the multi-class experiments?\n\n-- Will self-training help improve the detector trained with UNION pseudo labels?\n\n-- For visual appearance encoding, DINOv2 usually can only get a lower-resolution feature map of the input image. How did the authors obtain point-wise features?\n\n-- Which variant of DINOv2 is used for the experiments in this paper?\n\n-- Could the authors provide the runtime analysis for each step of UNION?", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "++ The method has significant improvement on the accuracy compared to existing unsupervised methods. \n\n++ This method does not require the time-consuming multi-round self-training procedure.", "weaknesses": "-- There are not any qualitative results in the paper. Visualizations on the pseudo box/class generation and the comparative results would help readers understand the method more comprehensively.\n\n-- Detailed analysis of failure cases would make this paper stronger. For example, in which scenarios UNION would generate bad pseudo labels, and how each step of UNION contributes to different failure cases.\n\n-- Details about the HDBSCAN hyper-parameters are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback.\n\n**Qualitative results of pseudo-bounding box generation**\n\nWe provide qualitative results of the intermediate outputs of the UNION pipeline and the final generated pseudo-bounding boxes **in Figure 2 in the rebuttal PDF**.\nThis figure shows sample 2 from scene-1100, which is part of the nuScenes training dataset.\nIt can be seen that the scene flow can identify multiple dynamic objects, and the appearance clustering can discover static mobile objects, including vehicles and pedestrians, using those dynamic instances.\n\n**UNION failure cases**\n\nThe UNION pipeline consists of multiple components that each have their limitations, and as a result, UNION can fail to generate good pseudo-labels for some cases.\nThe spatial clustering using HDBSCAN fails, for example, to generate correct clusters if multiple objects are close to each other, when objects are partially occluded, or when objects are far from the LiDAR (few points).\nThis may be (partially) solved by using temporal tracking in combination with bounding box refinement.\nIn addition, the self-supervised scene flow may fail to estimate a correct velocity when the spatial cluster contains few points.\nThis also may be solved by using temporal tracking or other sensors such as radar.\nFinally, the parallax effect sometimes causes problems when computing the appearance embedding, e.g., LiDAR points are projected on the wrong object in the camera image.\nAs a result, objects may be part of the wrong appearance cluster and, thus, may be falsely labeled as mobile or non-mobile.\nOne failure case is shown at the bottom left of the UNION results in Figure 2.\nAccording to the ground truth, many pedestrians are located there, but UNION only discovers 2 of them (the bus stop partially occludes some and they have few LiDAR points).\n\n**HDBSCAN hyperparameters**\n\nWe will add an appendix to the paper to list all the hyperparameters we used for UNION.\nFor HDBSCAN, we used the implementation from scikit-learn, a Python module for machine learning built on top of SciPy.\nThe minimum cluster size is set to 16 points, and the cluster selection epsilon is set equal to 0.50 meters, i.e., clusters are merged if they are within half a meter of each other.\n\n**OYSTER and LISO are not included in the multi-class experiment**\n\nThe methods OYSTER [1] and LISO [2] are for unsupervised class-agnostic 3D object detection, i.e. they generate class-agnostic pseudo-bounding boxes that can be used for training existing detectors.\nSo, their pseudo-bounding boxes cannot be used for training multi-class detectors as no (pseudo-)class labels are available.\nThe same holds for HDBSCAN; therefore, we assigned class labels to the class-agnostic bounding boxes based on the size of the bounding boxes **(see Table 4 in the submission and Table 2 in the rebuttal PDF)**.\nHowever, the source code of OYSTER and LISO has not been released (yet), so we cannot use the size prior for these two methods.\nConsequently, we cannot compare multi-class UNION to 'OYSTER+Size prior' and 'LISO+Size prior' for the multi-class 3D object detection experiment.\n\n**Self-training with UNION**\n\nWe have experimented with self-training for CenterPoint trained with the class-agnostic pseudo-labels from UNION.\nWe used the following strategy for each self-training round: (1) predict with the trained CenterPoint on the training dataset from nuScenes, (2) filter the predictions using a score threshold, i.e. only keep the prediction if the predicted score is at least equal to the score threshold, and (3) train CenterPoint from scratch using the filtered predictions.\nWe did this for three rounds for three different score thresholds, namely 0.10, 0.20, and 0.30.\nThe average precision (AP) did not improve for any score thresholds.\nIn addition, we noticed that the AP decreased after each round of self-training.\nEspecially for score threshold 0.10, the performance drop was significant, i.e. the AP dropped to less than 20, while UNION achieved 38.4 **(see Table 1 in the rebuttal PDF)**.\nSo, we did not experience any benefit from self-training after training with the UNION pseudo-labels.\nThis is in line with the objective of UNION, discovering the static and dynamic mobile instances before training such that the computationally expensive self-training is not needed and that consistency during training enhances the detection performance (e.g. the detector is not penalized for detecting a parked car during training).\n\n**Point-wise camera-based features**\n\nThe used large vision transformer has a stride of 14, i.e. the spatial resolution of the obtained feature map is in each dimension 14 times smaller.\nEach LiDAR point is first projected to the image plane.\nAfter that, the pixel coordinate is transformed to feature map coordinates (i.e. dividing by the stride), and then bilinear interpolation is used to get a camera-based feature vector for each LiDAR point so that the lower spatial resolution does not constrain us.\n\n**DINOv2 version**\n\nSee global rebuttal text: 'Different image encoders'.\n\n**Runtime analysis of UNION**\n\nWe have computed the runtime statistics for creating the class-agnostic pseudo-labels with UNION, which are used for training the detector (CenterPoint).\nWe tested on a system with 2 Intel Xeon E5-2690 v4 CPUs (56 logical CPUs) and 8 NVIDIA V100 32GB GPUs.\nThe training dataset of nuScenes consists of 700 sequences.\nIn practice, we process the sequences in parallel in 8 threads.\nAs a result, it takes 9, 19, 7, 9, and 2 hours for the ground removal, spatial clustering, motion estimation (scene flow), image encoding (DINOv2), and appearance embedding, respectively. After that, the appearance clustering is done for all sequences together and this takes 1 hour.\n\n[1] Zhang et al. (2023). Towards unsupervised object detection from lidar point clouds. In CVPR.\n\n[2] Baur et al. (2024). LISO: Lidar-only Self-Supervised 3D Object Detection. In ECCV.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers (R1:LFQD, R2:ZwY5, R3:Gw7r, R4:JvDj) for their valuable and detailed feedback.
We appreciate that our method UNION was overall well-received e.g. 'represents a creative fusion of existing ideas applied to the problem of 3D object detection' (R1), 'is quite novel' (R2), and 'has significant improvement on the accuracy compared to existing unsupervised methods' (R4).
In addition, we are pleased that the paper is considered 'well-structured' (R1), and that it 'effectively highlights the similarities and differences with related literature' (R2) and 'studies a significant problem in unsupervised 3D object discovery' (R3).

**Clarification on primary and secondary  contributions (class-agnostics discovery,  multi-class detection)**

OYSTER [1] and LISO [2] only provide class-agnostic pseudo-bounding boxes for training (i.e. no pseudo-class labels).
Our first step is similar in that it provides class-agnostic bounding boxes for mobile objects.
Object proposals are clustered based on appearance embedding into $K_1$ appearance clusters, and mobile objects are obtained by selecting the appearance clusters with at least $X$ percent dynamic instances.
Unlike OYSTER and LISO, our approach does not require computationally intensive self-training iterations.
This is our primary contribution, described in Section 3 (lines 185-252) and evaluated in Section 4.3.

In a second step, however, we extend class-agnostic 3D object discovery to multi-class 3D object detection.
We do so by splitting the obtained set of (class-agnostic) mobile objects into $K_2$ appearance-based pseudo-classes, using the same object appearance embedding technique used in the first step.
During inference, we assign each pseudo-class to a real class using the appearance prototype of the pseudo-class and the appearance of the real classes, requiring a single appearance embedding per real class **(see Figure 1 in rebuttal PDF)**.
Note that this step requires negligible supervision as we do one-shot association during inference.
This is our secondary contribution, described in Section 3 (lines 252-255) and Section 4 (lines 336-345), and evaluated in Section 4.4.
We cannot compare to OYSTER and LISO for this task as they do not have pseudo-classes, i.e. they only have class-agnostic pseudo-bounding boxes, and their source code is not released.
We will adapt the paper text to make the above points more clear.

**Improved results**

We meanwhile improved the results for class-agnostic 3D object detection **(see updated Table 1 in the rebuttal PDF)**.
Previously, we did not use the velocity direction of dynamic objects to correct the orientation of the fitted bounding boxes and did not use the estimated velocity during training (velocity prediction is part of the nuScenes detection score (NDS)).
In addition, we tuned the scene flow estimation, resulting in better performance, i.e. significantly higher recall with slightly lower precision for determining whether an object is dynamic.
We also did a parameter optimization on $K_1$ (grid search) and now use 20 appearance clusters instead of 50.
This all combined increased the average precision (AP) by 4.5 to 38.4, which is more than 3.5 times higher than OYSTER and LISO.
Also, in this rebuttal **(see PDF)**, we present updated results for the multi-class detection, show the bounding box generation pipeline, provide an analysis for the cyclist performance, present results for different image encoders, and provide a runtime analysis for the entire UNION pipeline.

**Different image encoders**

In our submission, all experiments were conducted with a large vision transformer (ViT-L/14) trained using DINOv2 [3] with registers [4].
Table 3 compares UNION's class-agnostic 3D object detection performance for DINOv2 and I-JEPA [5].
DINOv2 uses contrastive learning, which ensures that the representations of different views of the same image are similar while the representations of different images are distinct, while I-JEPA predicts the representations of part of an image from the representations of other parts of the same image.
DINOv2 can process high-resolution images, such as the camera images from nuScenes.
In contrast, I-JEPA can only process square-shaped images of a maximum of 448 by 448 pixels.
As a result, the obtained feature maps from I-JEPA have a lower resolution than the ones from DINOv2.
From Table 3, it can be seen that UNION with DINOv2 outperforms UNION with I-JEPA by 15.6 in AP and 8.4 in NDS.
Please note that our main paper contributions do not depend on specific canonical steps of the UNION pipeline (e.g., image encoding, scene flow estimation). If better approaches become available, UNION can incorporate them.

**Outperforming shelf-supervision**

Recently, a new method named CM3D [6] was released on arXiv; this method has not been peer-reviewed yet.
CM3D is a 'shelf-supervised' 3D object detection method, i.e. off-the-shelve foundation models that were trained with manual labels such as SAM [7] are used, and it achieves an AP and NDS of 27.9 and 25.0, respectively.
In contrast, UNION is *entirely* unsupervised and outperforms CM3D by more than 10 points in AP and more than 5 points in NDS **(see Table 1 in the rebuttal PDF)**, demonstrating the effectiveness of the UNION pipeline, while not relying on manually labeled data sources.

[1] Zhang et al. (2023). Towards unsupervised object detection from lidar point clouds. In CVPR.

[2] Baur et al. (2024). LISO: Lidar-only Self-Supervised 3D Object Detection. In ECCV.

[3] Oquab et al. (2024). DINOv2: Learning robust visual features without supervision. In TMLR.

[4] Darcet et al. (2024). Vision Transformers Need Registers. In ICLR.

[5] Assran et al. (2023). Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR.

[6] Khurana, et al. (2024). Shelf-Supervised Multi-Modal Pre-Training for 3D Object Detection. arXiv.

[7] Kirillov et al. (2023). Segment anything. In ICCV.

Meta-review: This paper proposes an unsupervised object detection (or object discovery) method using 3D data. The input consists of multimodal data such as images and LiDAR point clouds, while the output is a set of pseudo-bounding boxes and pseudo-class labels of objects. One of the significant contributions of the proposed method is the elimination of time-consuming iterative self-training, which is commonly used in previous methods. The method effectively utilizes multimodal data and temporal information to distinguish between static and dynamic objects. The paper's contributions and comparisons with related work are very clearly stated and are presented in an easily understandable manner using figures and tables. The clarity of the paper, the novelty of the method, and its high performance were recognized by most reviewers, all of whom gave ratings leaning towards acceptance. Although there were some questions regarding missing experimental results, these were addressed well with additional experimental results in the authors' rebuttal, satisfying all reviewers and leading them to maintain their initial evaluations. Based on these points, it can be concluded that this paper is suitable for acceptance at this conference.
Decision: Accept (poster)