Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
Authors: Katie Z Luo
Zhenzhen Liu
Xiangyu Chen
Yurong You
Sagie Benaim
Cheng Perng Phoo
Mark Campbell
Wen Sun
Bharath Hariharan
Kilian Q Weinberger
Venue: NeurIPS 2023 poster (2023)
Keywords: Self Driving
Self-Supervised Object Discovery
Reward Ranked Finetuning

Abstract: Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector’s own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT.

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
Katic Luo Zhenzhen Liu
Xiangyu Chen Yurong You'
Sagie Benaim" Cheng Perng Phoo* 2 Kilian Q. Weinberger
Mark Campbell' Wen Sun
Bharath Hariharan'
Comell University, Ithaca, NY
"The Hebrew University of Jerusalem
Abstract
Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) improve machine learning models and align them with human preferences. Although very successful for Large Language Mod- els (LLMs), these advancements have not had comparable impact in research for autonomous vehicles-whed alignment with human expectations can be impera- tive. In this paper, we propose adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.c., boxes containing objects are scored higher than those without. We start from the detector's own pre- dictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https: / thub. com matteeluo8//Ra..
Introduction
Self-driving need to accurately detect the moving objects around them in order to move safely, Most modern 3D object detectors rely on supervised training from 3D bounding box labels. However, these 3D bounding box labels are hard to acquire from human annotation. Furthermore, this supervised approach relies on pre-decided vocabulary of classes, which can cause problems when the car encounters novel objects that were never annotated.
Our prior work, MODEST [55], introduced the first method train 3D detectors without labeled data. In that work, we point out that instead of specifying millions of labels, one can succinctly describe heuristics for what a good detector output should look like. For example, one can specify that detector boxes should mostly enclose transient foreground points rather than background ones; they should roughly be an appropriate size; their sides should be aligned with the LiDAR points; their bottom should touch the ground, etc. Although such heuristics by a detector, training a detector on them is hard for two reasons: First, these heuristics are often non-linear, non- non-differentiable functions of the detector parameters (for example, slight shift of the box can cause all foreground points to fall off.) Second, existing object detection pipelines use carefully designed training objectives that heavily rely on labeled boxes, that are difficult modify (for example, PointRCNN [40] infers point labels from box labels and uses these for training). For these reasons, MODEST had to utilize admittedly slow self-training pipeline incrementally incorporate common-sense heuristics.
great for scoring set of boxes proposed
"Denotes equal contribution. i "Correspondences could he directed to kz16@cornel1.com edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Hiuurs
Days
Hours
Days
mA (BEV @@o 0.5)
mAP (BEV @loU 0.73
15
Self Amin,, MODEST
DRIFT
No-FT
Self Train. MODEST * DRIFT No-FT
20
200
400
600
800
200
400
600
800
Training Epochs
Training Epochs
(a) Performance over Epochs IoU 0.5
(b) Performance over Epochs @ IoU 0.7
Figure 1: Detection performance on Lyft test data function of training epochs. DRIFT demonstrates significantly stronger performance and faster learning. With only 9 hours of training, outperforms both baselines that have been trained for days.
In this paper, we propose a new reward ranking based framework that utilizes these common-sense heuristics directly as reward signal. Our method relies finetuning with reward ranking [31, 27, 13, 33J, where given an initialized object detector, we finetune it for a predefined set of desirable detection properties. This bypasses the need to encode heuristics as differentiable loss functions and avoids the need to hand-engineer training paradigms for each kind of object detector. Recent success with reinforcement learning from human feedback (RLHF) has proven effective in improving machine learning models -in particular, large language models (LLMs) and aligning them with human preferences [31 27]. However, these advancements have not been applicable detection-based vision models that are trained with per-instance regression and are difficult to view under probabilistic framework. To address this challenge, we utilize insights from reward ranked finetuning [13], a non-probabilistic paradigm designed for finetuning of LLMs, which inspired us to develop a similar framework for object discovery.
We refer to our method as Discovery from require labels, and instead uses the Persistency Prior (PP) score [55, 3] as heuristic identify dynamic foreground points based on historical traversals. These foreground points give rise to rough (and noisy) label estimates [55], which we use to pre-train our detector. The resulting detector performs poorly but suffices to propose many boxes of roughly the right sizes that we can use for exploration. To facilitate reward ranked finetuning, we first propose reward function to score boxes. Ideally, only boxes that tightly contain objects (e.g. a car) should yield high rewards. We achieve this by combining several simple heuristics (e.g. high ratio of foreground points) and assuming some rough knowledge about the object dimensions. During each iteration of training, DRIFT performs the following steps: 1. the object detector proposes many boxes in a given point cloud scene; 2. the boxes are "jittered" through random perturbations (as a means of exploration); 3. the boxes scored according to the reward function; 4. the top-k% non-overlapping boxes are kept as pseudo-labels for gradient updates.
Finetuning (DRIFT). DRIFT does
We evaluate DRIFT on two large, real-world datasets [12, 24] and show that we significantly out- perform prior self-training methods both in efficiency and generalizability. Experimental results demonstrate that using reward ranked finetuning for object discovery under our framework can quickly converge to a solution that is on par with out-of-domain supervised object detectors within a few training epochs, suggesting that DRIFT may point towards a more general unsupervised learning formulation for object detectors in an in-the-wild setting.
2 Related Works
3D Object Detection. 3D object detection models usually take LiDAR point clouds or multi-view images as input and aim to produce tight bounding boxes that describe nearby objects [9, 52, 26, 39, 40, 53, 54, 34]. Existing methods generally assume the supervised setting, in which the detector trained with
bounding boxes. However, annotated data are often expensive obtain

---

and limited in quantity. Furthermore, in tasks such as self-driving, environments can have highly varied conditions, and detectors with supervised training often require adaptation with additional labels from the new environment [47].
Unsupervised Object Discovery. The unsupervised object discovery task aims to identify and localize salient objects without learning from labels. Most existing works perform discovery from 2D images [8. 14, 45, 36, 2. 43, 48] or depth camera frames [21, 23, 17, 25, 20, 1J. Discovery from 3D LiDAR point clouds is underexplored. [49] performs joint unsupervised 2D detection and 3D instance segmentation from sequential point clouds and images based on temporal, motion and correspondence cues. MODEST [55] pioneers in performing label-free 3D object detection. It exploits high-level common sense properties from unlabeled data and bootstraps a dynamic object detector via repeated self-training. Despite promising performance, it requires excessive training time, which makes it difficult for practical use and development.
Reward Fine- Tuning for Model Alignment. Recently, foundation models [6. 29, 11, 44, 35, 37] have been shown achieve strong performance in diverse tasks [5, 50], but sometimes produce outputs that do not align with human values [18, 30, 10]. A line of research aims to improve model alignment under the paradigm of Reinforcement Learning with Human Feedback (RLHF). Some pioneering works [41, 31, 33] learn a reward model and train foundation models with Proximal Policy Optimization (PPO) [38], but PPO is often expensive and unstable to train, and more importantly. requires a probabilistic output on the action space. This makes it hard to use for the object detection setting, which primarily uses regression- based losses. Reward ranked finetuning [27, 13] a simplified alternative paradigm. It samples from foundation model itself, filters generations using the reward model, and conducts supervised finetuning with the filtered generations.
3 Discovery from Reward- Finetuning
Our framework, DRIFT, is inspired by the recent success of reward ranked finetuning methods for improving model alignment in the NLP community [13, 27J. We show that a similar approach can be adapted for 3D object discovery.
Problem Setup. We wish to obtain dynamic object detection model on LiDAR data, i.e., a model to detect mobile objects in the LiDAR point clouds, without human annotations. Let P E R x3 denote N-point 3D point cloud captured by LiDAR from which wish to discover objects. We assume inputs of unlabeled point clouds collected by a car equipped with synchronized sensors including LiDAR (for point clouds) and GPS/INS (for accurate position and orientation). Since no annotation is involved, such a dataset is easy to acquire from daily driving routines; we additionally assume it to cover some locations with multiple scans at different times for computation of PP-score.
Dynamic Point Proposals. DRIFT leverages prior works that use unsupervised point clouds to extract foreground- backoouuud-bukgroudd segmentation proposals. While many works [22, 3] have promising dynamic foreground segmentation results, in this work we rely on point Persistency Prior score (PP-score) [55] for its accuracy and leave the extension of other proposal methods to future work. For the purpose of this research, dynamic foreground points constitute LiDAR points reflecting off traffic participants (e.g. cars, bicyclists, pedestrians).
Using historical LiDAR sweeps collected at nearby locations of our point cloud P, the PP-score [3, 55] T(P) E [o, 1|"0 can provide an informative estimate on the per-point persistence, i.e., whether point belongs persistent background not. The PP-score is defined the normalized entropy over past point densities, based on the assumption that background space such as ground, trees, and buildings tend to exhibit consistent point densities across different LiDAR scans (high entropy), whereas points associated with mobile objects exhibit high density only if an object is present (low entropy).
3.1 Rewarding "Good" Dynamic Boxes
We first establish reward function that evaluates the quality of set of bounding boxes for dynamic objects in a scene. We denote a set of M dynamic objects bounding boxes as (b), bw }. where each bounding box b, is represented as upright box with parameters (æ4, Vi, 2;, w,, 4, h, 0.). defining the box's 's center, width, length, height, and z-rotation, respectively. The scoring function scores the validity of the bounding boxes, given the observed point cloud P. In practice, a reward
3

---

0.1
0.0
Dynamic
0.0
Shape Prior
Alignment
Dynamic Point Counts
Figure 2: Illustration of the reward components. The reward encourages boxes that have proper shape and alignment, and capture more dynamic points and few background points.
function that is positively correlated with IoU should suffice. We present our proposed reward function which aims capture only dynamic points, filter nonsensical boxes, enforce correct size, and encourage proper box aligoment to the captured dynamic points.
Shape Prior Reward. We enforce a box to not deviate significantly from a set of prototype sizes (WT L., J1,11 h1), (We, Ic, h (Fig. 2 left). We assume the shape prior distribution is mixture of C' isstropic Gaussians with mixture weights n,, diagonal variances E,, and corresponding means as (w; l,, he). These low-level statisties may acquired directly from the dataset, or from vehicle specs and sales data available online [47]. In practice, we scale the mixtures such that the probabilities the Gaussian means are equal for stability reasons. With this, the shape prior reward for box is computed as:
Rahape () Ps(b).
Alignment Reward. Due to the nature of LiDAR sensing, the points will mostly fall on the lateral surfaces of object. Therefore, a well-formed box should have dynamic points approximately close to the boundary of a box (Fig. middle). As [55] shows, PP-score allows for easy separation of dynamic and persistent background points. Let Payn denote the set of dynamic points, and let Pog be that of background points. In practice, since the PP-score is an approximation ground-trute persistence, we define Payn {p| (p) 0.6} and Pog {pl (p) 0.9).
Given a box b, we denote all points within and close to the box (b)). Only points within ((b) contribute to the reward of In practice, we let (b)) consist of all points within a x2 scaled up version of b (with identi- cal center and rotation). To score box b, we design reward function that identifies how "typical" the dy- namic points within (b)) are. For each dynamic point E O(b) Payn. compute the scaling factor $p.b required so that the rescaled box touches p with one of its sides; i.c., if point is inside the box, the box would have to be scaled down (8p.b < 1) to touch the point, if the point is outside must be scaled up (8p.b 1). We assume that Sp.6 roughly follows a Gaussian distribution centered near the box boundary, and visualize the actual distribution in Fig. 3.
Points Distribution
0.06
0.04
0.02
00
Centre
0.5
Box Edge
1.5
2.0
Scale
Figure 3: Distribution of dynamic points ground truth bounding boxes. We observe that dynamic points near hounding boxes fall in an approximate Gaussian distri- bution centered near box edges (sp.b 0.8).
We define our reward as the likelihood under this Gaussian distribution over scaling parameters. We approximate as a Gaussian with Syper-parametent mean Mscale and a variance dscale. Our reward is the product of the probability of each point in O(b):
Ralign (b)
II
N(sp. M(sp.b/Mscao Mscales Tscale).
PEo(b)riPP.
Common Sense Heuristics and Filtering. Lastly, proper bounding box must capture the dynamic points, and avoid capturing the background points (Fig. 2, right). This heuristic can be encoded by

---

simple weighted point count for each bounding box b:
Reount (b)
Adya
Payn O(b)] - Po n O(b)).
Intuitively, assigns a reward in proportion to the number of dynamic points captured by the box, and penalty in proportion the number of background points captured.
Furthermore, boxes violating common sense should be assigned a low reward. We filter boxes that are too high up or too low from the ground, including those with too few dynamic points, are too small or too large by directly assigning a reward of O. In practice, we filter boxes that contain fewer than 4 dynamic points, or that have more than 80% persistent points, similar [55].
In summary, the reward function is designed be
{o Dishape Rehape (b) + Aatign Ralign (b) + Reoun
if obeys common sense, otherwise.
R(b)
3.2 Exploration Strategy for Improved Discovery
We assume simple exploration strategy for identifying good box proposals. Given set of current object proposals given by the detector, we locally perturb the boxes in the output space:
Bexplore P (Bo, o):
where Bxxpoore is the set of explored boxes, perturbed from the model proposals By. For each box b E Bu, a set of explored boxes are sampled according to standard Gaussian noise along the position and size dimensions and uniform noise for orientation:
b
explore
center
o² I²)),
explorre
NNb o2 besplore ~ u(b)) o, by + +
(5)
Furthermore, encourage proposals of boxes foreground regions, we take in- spiration from [56, 40] and re-use PP-score as point-level semantic segmentation (fore- ground vs background) labels, with which the detector is encouraged to propose boxes at points that have low PP-scores (i.e.are likely to be foreground points). Following (56), for each point pæ P with prediction Dr. we assign its target classification label
Algorithm 1 Rewardd-Inentivized Finetuning
Input:
Base object detector fe. unlabeled LiDAR dataset D, exploration noise a,, sample size n, filter budget k, reward function Rt PP-scores T. repeat
P~D
p Sample point cloud. Run detector for proposals. b Sample boxes. Score boxes in set.
By + fo (P)
B Bexllrre ~ P (B).
r BEEB)
y: as:
B NMS (Bexpore Bo,
Bup + Filter(B, r,
b> Keep top k% boxes
[1 T(:)) <r. or 0i 1, (6)
Update 0 with Bugp for step
y (0 otherwise.
until converged
In effect, this encourages all non- non-persistent points (i.e., low T(P:)) to propose boxes near dynamic regions for better exploration.
3.3 Rewarr-Incentiivzeed Finetuning
The reward function allows us to quickly evaluate proposed bounding boxes B and the task 3D object discovery could be reduced to an optimization problem the total reward in box set space:
1
arg max
R(b), bEB
B
where the sum is taken over the boxes in the set B. Although a direct optimization for B is not plausible due to the non-pon polpoolll search space and discontinuity in R, can serve as effective guidance to facilitate model finetuning. The underlying intuition is similar to curriculum learning [4, 28, 46]: the object detection model takes small steps to improve from its current predictions towards 3 by following the direction provided by the maximum R direction in local space.
As illustrated in Alg. 1, in each training iteration, we first the object detector perform inference on point cloud P and propose set of dynamic objects Bo in the scene. To explore directions
5

---

of improvement with the non-differentiable reward function R. we sample n boxes from Bo (with replacement) and add an i.i.d Gaussian noise on their location and size, and an uniform noise on orientation following Eq. 4. These sampled boxes are then ranked by the reward function R. in which the top non-overlapping boxes are selected by Non-Maximum Suppression (NMS) as training targets to finetune the object detector. Note that since DRIFT treats the model training/inference procedures as black boxes, it can be applied to any 3D object detection model.
In practice, it is observed that neural networks can acquire task knowledge from imperfect demonstra- tions [16, 51, 32]. MODEST [55] pre-trained the 3D object detector on noisy seed labels produced by DBSCAN [15] clustering on spatial and PP-score. We follow [55] and initialize our 3D object detector model trained with discovered seed labels.
4 Experiments
Datasets, We experimented with two different datasets: Lyft Level 5 Perception dataset [24] and 1thaca-365 dataset [12]. To the best of our knowledge, these are the two publicly available datasets that contain multiple traversals of multiple locations with accurate 6-DoF localization and 3D) bounding box labels for traffic participants.
In the Lyft dataset, we experiment with the same split provided by [55], where the train set and test set are geographically separated. It consists of 11,873 train scenes and 4,901 test scenes. For the 1thaca365 dataset, we experimented with the full dataset which consists of 57,107 scenes for training and 1,644 for testing. For both datasets, we do not use any -annaatdd labels in training. To show the generalizability of our method, we conduct the development on the Lyft dataset, i.e. all the hyperparameters of our approach are finalized through experiments Lyft, and we use the exact same set of hyperparameters for all experiments in Ithaca365.
Evaluation. Following [55], we combine all traffic participants to a single mobile object class and evaluate the detector's performance on this class. Note that the labels are not used during training but solely for evaluation. For Lyft, we report the mean average precision (mAP) of the detector with the intersection over union (loU) thresholds at 0.5 and 0.7 in bird-eye- perspective. Note that mAP at 0.7 IoU threshold is stricter and harder metric and was not evaluated in [55], and we include it to emphasize the effectiveness of our method. For Ithaca365, we adopt metrics similar to those in [7]: evaluate mean average precision (mAP) for dynamic objects under {0.5, 1, 2, 4)m thresholds that determine the match between detection and ground truth; we also compute types of true positive metrics (TP metries), including ATE, ASE and AOE for measuring translation, scale and orientation errors. These TP metrics are computed under a match distance threshold of 2m; additionally, we also compute a distance- based breakdown (0-30m, 30-50m, 50-80m) for these metrics.
Implementation. We use PointRCNN [40] as our default architecture and we use the implementation provided by OpenPCDet [42]. We train DRIFT with 120 epochs in Lyft and 30 epochs in 1thaca365 as the default setting, and observe that the performance generally improves with more training epochs (Fig. 1). We use Xshape = 1, Aatign = 1, Adyn 0.001 and Ang = 0.001. We use Maeate 0.8 and o Sscate = 0.2 for the alignment reward. We define the shape priors based on four typical types of traffic participants: Car, Pedestrian, Cyclist, and Truck. Specifically, we use the mean and standard deviation of box sizes of each class in the Lyft dataset, but we show that they generalize well to other domains like 1thaca365 and are not sensitive (Tab. 2) The exact prototype sizes S and other implementation details can be found in the supplementary materials.
Baselines. To the best of our knowledge, MODEST (55] is the only prior work on this problem and we compare our method DRIFT against it with various variants of MODEST: (1) No Finetuning: the model trained with seed labels from PP-score without repeated self-training (MODEST (RO) in [55]; (2) Self-Training (i ep): the model initialized with (1) and self-trained with i epochs without PP-score filtering; (3) MODEST (i ep): the model initialized with (1) and self-trained with i epochs with PP-score filtering (full MODEST model). For self-training
in (2) and (3), we adopt 60 epochs for each self- tiffttraiiin round in the Lyft dataset (same as that in [55]) and 30 epochs for the 1thaca365 dataset. To ensure a fair comparison, DRIFT is also initialized from (1) and use the same detector configurations as the baselines. Following [55], we also compare with the supervised counterparts trained with human-annotated labels from the same dataset (Lyft or Ithaca365) and from another out-of-domain dataset (KITTI).

---

Table 1: Detection performance on Lyft. DRIFT outperforms both baselines with 10% training time, and approaches the performance of the aut-of-domant supervised detector trained on KITTI. Please refer the setup Sec. for the metries.
mAP IoU @ 0.5 (1)
mAP IoU @ 0.7 (1)
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
No Finetuning
44.1
21.1
1.2
23.9
24.4
6.0
0.1
10.5
Self-Train. (60 ep)
50.0
29.0
3.4
28.6
32.5
10.0
0.3
14.0
Self-Train. (600 ep)
56.7
41.1
9.1
37.2
35.1
20.7
1.6
19.9
MODEST (60 ep)
49.6
29.7
3.4
28.8
31.3
10.2
0.3
14.4
MODEST (600 ep)
56.4
45.4
11.3
39.6
33.6
18.6
1.4
18.8
DRIFT (30 ep)
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
DRIFT (60 ep)
60.3
43.8
14.6
41.8
42.0
29.2
5.8
26.7
DRIFT (120 ep)
61.4
45.1
21.7
45.3
42.7
31.7
29.6
Sup. on KITTI
71.9
49.8
22.2
49.9
47.0
26.2
6.4
27.9
Sup. Lyft
76.9
60.2
37.5
60.4
62.7
50.9
28.2
48.5
Table 2: Detection performance on Ithaca365, We observe DRIFT outperforms both baselines with significantly less training time. Please refer to the setup of Sec. 4 for the metrics.
mAP (1)
Errors 0-80m (L)
Method
0-30
30-50
50-80
0-80
ATE
ASE
AOE
No Finetuning
18.7
4.8
0.0
7.7
1.17
0.60
1.64
Self-Train. (30 ep)
25.9
9.2
1.2
12.4
1.08
0.62
1.57
Self-Train. (300 ep)
16.3
3.6
1.8
6.8
1.19
0.74
1.57
MODEST (30 ep)
14.6
0.7
0.0
3.7
0.83
0.52
1.53
MODEST (300 ep)
27.5
26.3
21.0
27.1
1.06
0.67
1.09
DRIFT (15 ep)
39.1
24.3
17.7
28.0
0.73
0.33
1.23
DRIFT (30 ep)
47.1
31.2
22.9
35.1
0.49
0.35
1.20
Sup. on KITTI
59.8
28.3
4.0
32.0
0.26
0.22
0.46
Sup. Ithaca365
75.7
48.3
22.6
51.5
0.18
0.13
0.33
Dynamic Object Detection Results. We report the perfor- mance of DRIFT and baseline detectors Lyft in Tab. 1, and show the performance over the training epochs in Fig. We re- port the performance Ithaca365 in Tab. 2. Notably, DRIFT demonstrates significantly faster learning and strong perfor- mance. It provides more than 10x speedup compared the baselines. On Lyft, DRIFT's performance at 60 epochs already surpasses the performance of both baselines at 600 epochs (10 self-training rounds) and approaches the performance of the out-of-domain supervised detector trained on KITTI [19]. On 1thaca365, its performance at 30 epochs significantly surpasses both baselines trained 300 epochs. It even outperforms the out-of-domain supervised detector trained on KITTI in mAP.
Table 3: Analysis on rewards of boxes produced by different detec- tors. We report mean and std of box reward on the Lyft dataset.
Models
Mean
StD.
Rand, Boxes
0.02
0.15
Self-Train.
0.44
0.57
MODEST
0.57
0.58
DRIFT
0.61
0.64
Ground Truth
1.00
0.42
Observe that the self- self-training performance starts collapsing with more rounds of self seff-training, and does not continue to improve.
Fig. 4 visualizes the detection
Ground truth boxes colored in green, predictions
from the detector without fine-tuning are in yellow, and predictions from DRIFT are in red. We observe that the detector without fine-tuning occasionally produces false positive predictions, produces boxes with incorrect sizes, or misses moving objects, while DRIFT produces accurate detection.
Rewards ablations. We report the average reward per box for ground truth boxes, random boxes, and predicted boxes from different detectors in Tab. The ground truth boxes have the highest rewards on average, while the random boxes have the lowest. This indicates that the reward reasonably reflects

---

No Fine-tune
DRIFT
No Fine-tune
DRIFT
Lyft Dataset
Ithaca366 Dataset
Figure 4: Visualization of detections. Qualitative results on two scenes from Lyft [24] and Ithaca365 [12] datasets. Ground truth boxes are labeled with green in the LiDAR figures and predictions without fine-tuning and DRIFT are in yellow and red, respectively. We observe DRIFT learns to produce accurate detection with correct shape and localization.
Table 4: Ablation the reward components. We report the mAP (0-80m) on Lyft. Re- moving components significantly degrades performance.
Table 5: Ablation the alignment reward's Macate- We report the mAP (0-80m) on the Lyft dataset. We show the detection perfor- mance not sensitive the choice of Pacale
Table 6: Ablation the alignment reward's aeate- We report the mAP (0-80m) on the Lyft dataset. The de- tection performance is not sensitive the variance.
mAP (0 80m)
Filter
Shape
Align.
loU 0.5
loU 0.7
mAP (0 80m)
0.6
0.0
mAP (0 80m)
Ascole
0.0
0.0
Pseale
IoU 0.5
IoU 0.7
IoU 0.5
IoU 0.7
0.0
0.0
0.1
34.2
20.9
22.4
3.2
0.8
38.3
23.1
0.2
38.3
23.1
38.3
23.1
0.9
38.1
23.4
0.3
38.1
20.0
the quality of the bounding box. And we observe the boxes predicted by DRIFT have higher rewards than those predicted by the baseline detectors.
Ablation study on the components of our reward is presented in Tab. 4, and visualization is shown in more of the components.
Fig. 5. Detection performance significantly drops when remove For example, when only common sense filtering is used, the detector just predicts boxes around foreground points. Without the shape prior reward, the detector predicts boxes with incorrect sizes.
Ablations Tab. and Tab. present the sensitivity analysis of choices of Mssale and Oscale- DRIFT achieves stable performance across reasonable choices of Mseate and Tsaale showing the robustness of our method.
Exploration. We study the necessity of the exploration component and the effect of incorporating other sources for box sampling. In Tab. 7, we compare exploration (1) sampling 200 boxes from box predictions, (2) sampling 100 from proposals near dynamic points and 100 from predictions, and (3) sampling 100 from seed labels and 100 from predictions. Observe that the exploration component is crucial for our method; by performing local exploration instead of simply updating from its own predictions, DRIFT avoids confirmation bias and ensures that labels improve over what it predicts. Furthermore, results show that sampling from the box predictions is sufficient for obtaining good performance; other sources do not provide obvious benefits.
We also explore the effects of modifying the exploration strategy. Tab. compares the detector performance of using sample size of 50, 100 and 200, and noise scale (i.e. variation) of 0.3 0.6. Each detector is trained for 30 epochs. At noise scale 0.3, increasing the sample size from 50 to 200 significantly improves the detection performance. Using noise scale 0.6 significantly reduces the detection performance, indicating that smaller noise may be preferable.
Filtering Budget of the Ranked Boxes. We study the effect of the choice of top 1% for filtering boxes by reward ranking. Tab. presents the detection performance with top 55% 65%, 75% and 85%. DRIFT is robust to the choice of k, with slightly decreased performance when & is too high.

---

Table 7: Detection performance with ad- ditional sampling sources exploration. Sampling near the predictions is suffi- cient; additional sources do not provide obvious benefits. Trained for 60 ep.
Table 8: Ablation exploration pa- rameters. Large sampling size and moderate noise scale are preferable.
Table 9: Ablation the choice top 4% for box filtering by reward ranking. DRIFT is robust to different values of l.
mAP (0 80m)
Noise Samples
InU 0.5
IoU 0.7
mAP (0- 80m)
mAP (0- 80m)
50
32.9
14.3
Top
ToU 0.5
ToU
0.3
100
36.4
18.9
ToU 0.5
IoU
No Exploration
0.0
0.0
200
38.3
23.1
55%
37.0
21.0
Sample near pred.
41.8
26.7
50
0.0
65%
38.9
23.0
+ near dynamic
39.3
22.6
0.6
100
17.2
1.3
75%
38.3
23.1
init. seed labels
39.5
22.8
200
0.6
0.0
85%
35.8
20.2
Full Reward
Filter Reward Only
Self
MODEST
40
mAP (BRV @lol 0.7)
30
(-) Size Roward
(-) Alignment Reward
Car
Pedestrian
Cyclist
Truck
Figure 5: Visualization of reward ablation. Removing components leads to predictions with incorrect shape or position.
Figure Per-class BEV mAP at IoU 0.7. We assign each predicted box to the class with the most similar prototype size. DRIFT outperforms the baselines for all classes.
Extension to Detection with Classes. Our prototype sizes are defined by different classes of traffic participants. Thus, given predicted box from class agnostic detector, we can compute the likelihood of its size under the Gaussian prior of each prototype size, and assign it to the class with the highest likelihood. Fig. 6 presents the per-class performance of DRIFT and baselines under such assignment. All detectors are trained for 60 epochs. DRIFT outperforms both baselines, with especially significant improvement for the Car and Cyclist classes. More details can found in the supplementary materials.
Discussion and Conclusion
In this work, we propose a framework, DRIFT, to tackle object discovery without labels. Instead of requiring expensive 3D bounding box labels, our method utilizes succinetly described heuristics as reward signal to initialize and subsequently fine-tune differentiable reward signal, we propose a simple but very effective reward finetuning framework, inspired by recent successes of reinforcement learning in the NLP community. Compared to prior self-training based methods [55], such a framework is an order of magnitude faster to train, while achieving higher accuracy. Traditional self-training iteratively generates pseudo-labels and retrains the model, requiring convergence before generating the next set of pseudo-labels. In general, training detector mimic pseudo-labels training (confirmation bias). DRIFT addresses this issue by leveraging reinforcement learning principles, where the exploration component is crucial. Our method avoids confirmation bias by performing local exploration and ensures that labels improve over what it predicts. Thus, DRIFT is able perform updates per-training iteration opposed to per self-training round, which allows it to converge significantly faster and achieve higher performance.
object detector. To optimize this non-
lead to undesirable artifacts, further amplified by repeated
Limitations and Future Works. One limitation is that the current framework is geared explicitly towards dynamic objects. Static objects would require different heuristics, not based on PP-scores. Similarly, currently restricted our framework entirely to LiDAR signals. However, the reward

---

based framework is extremely flexible, and could easily be extended to other data modalities. For example, one could use image features to help identify objects inside of box proposals. Although in supervised settings image features have typically not added much in to the higher resolution LiDAR point clouds, in our unsupervised setting it is certainly possible that pixel information can help disambiguate objects from background. Further, we plan to explore the use of reward fine-tuning for other vision applications beyond object discovery.
Acknowledgments and Disclosure of Funding
This research is supported in part by grants from the National Science Foundation (III- 2107161. IIS 2144117, and IIS- 1724282), Nvidia Research, and DARPA Geometries of Learning (HR001 12290078). We also thank Wei-Lun Chao, Yihong Sun, Travis Zhang, and Junan Chen for their valuable discussion.
References
[1] Aikaterini Adam, Torsten Sattler, Konstantinos Karantzalos, and Tomas Pajdla. Objects can move: 3d change detection by geometric transformation consistency. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings. Part XXXIII, pages 108- Springer, 2022.
[2] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 328-335, 2014.
[3] Dan Barnes, Will Maddern, Geoffrey Pascoe, and Ingmar Posner. Driven to distraction: Self-supervised distractor learning for robust monocular visual odometry in urban environments. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1894-1900. IEEE, 2018. 2,
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[5] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2/08.07258 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances in neural information processing systems, 33:1 1877-199, 2020. 3
[7] Holger Caesar, Varun Bankiti, Alex H Lang, Sourahh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 1621-11631, 2020.
[8] Joao Carreira and Cristian Sminchisescu. Constrained parametric min-cuts for automatic object segmenta- tion. In 2010 IEEE Computer Suciety Conference on Computer Vision and Pattern Recognition, pages 3241-3248. IEEE, 2010.
[9] Xiaozhi Chen, Huimin Ma, Wan, Bo Li, and Tian Xia. Multi-view object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907-199 1915, 2017. 2
[10] Jaemin Cho. Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053 2022.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 222.:2204.023111. 2022. 3
[12] Carlos Diaz- Ruiz, Youya Xia, Yurong You, Jose Nino, Junan Chen, Josephine Monica, Xiangyu Chen, Katie Luo, Yan Wang. Mare Emond, et al. Ithaca365: Dataset and driving perception under repeated and challenging weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21383 213922 2022. 2, 6,
10

---

[13] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum. and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint urXiv:2304.001 2023. 2,3
[14] Ian Endres and Derek Hoiem. Category independent object proposals. ECCV (5), pages 575-588. Citeseer, 2010. 3
[15] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowci Xu, ct al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-222 231, 1996. 6
[16] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. corr abs/1802.05315 (2018). arXiv preprint arXiv: 1802. 6
[17] Germán García, Ekaterina Potapova, Thomas Werner, Michael Zillich, Markus Vineze, and Simone Frintrop. Saliency-based object discovery on rgb-d data with a late-fusion approach. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1866-1873. IEEE, 2015. 3
[18] Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A Smith. Realtoxicityprompts:: Evaluating neural toxic degeneration in language models, arXiv preprint arXiv:2009. 1462, 2020. 3
[19] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets rohotics: The kitti dataset. The International Journal of Robotics Research, 32(11):221-11237, 2013.
[20] Maciej Halber, Yifci Shi, Kai Xu, and Thomas Funkhouser. Resean: Inductive instance segmentation for indoor rgbd scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2541-2550 2019. 3
[21] Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d object discovery via multi-scene analysis. In 201 JEFE/SS International Conference Intelligent Robots and Systems, pages 4850-4856. IEEF, 2011.
[22] Xiangru Huang, Yue Wang, Vitor Guizilini, Rares Ambrus, Adrien Gaidon, and Justin Solomon. Repre- sentation learning for object detection from unlabeled point cloud sequences. In Conference on Robot Learning, 2022. 3
[23] Andrej Karpathy, Stephen Miller, and Li Fei-Fei. Object discovery in 3d scenes via shape analysis. In 2013 IEEE international conference on robotics and automation, pages 2088 2095. IEEE, 2013. 3
[24] R. Kesten, M. Usman, Houston, T. Pandya, K. Nadhamuni, Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, Kazakova, C. Tao, Platinsky, Jiang, and V. Shet. 2019. 2, 6,
Lyft level av dataset 2019, http:p/s//leeel..l...ooooooo
[25] Deyvid Kochanov, Aljoša Ošep, Jörg Stückler, and Bastian Leibe. Scene flow propagation for semantic mapping and object discovery in dynamic street scenes. In 2016 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1785 1792. IEEE, 2016.
[26] Alex II Lang. Sourabh Vora, Holger Caesar. Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[27] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-2766 2022. 3
[28] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: framework and survey. The Journal of Machine Learning Research, 21(1):7382-7431, 2020. 5
[29]
OpenAI. Gpt-4 technical report, 2023. 3
[30] Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4262 6274, 2021. 3
[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-22 -77444, 2022. 2,
11

---

[32] Deepak Pathak, Ross Girshick, Piotr Dollár. Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference on compuder vision and pattern recognition. pages 2701-2710, 2017. 6
[33] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint arXiv:2302.08244 2023. 2,
[34] Charles R Qi, Or Litany, Kaiming He, and Lconidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286, 2019. 2
[35] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages
PMLR, 2021. 3
[36] Zhile Ren and Gregory Shakhnarovich. Image segmentation by cascaded region agglomeration. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 2011-2018, 2013.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 3
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017. 3
[39] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remn: Point-voxel feature abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10538, 2020. 2
[40] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770-779, 2019. 2, 6,6
[41] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedhack. Advances Neural Information Processing Systems, 33:3008-3021, 2020. 2020, 3
[42] OpenPCDet Development Team. Openpedet: An spen-source toolbox for 3d object detection clouds. https: //gi://gitttt com/ com/open- ht////ppnnmmaallmm wmw.mm mmllaa//peePDeet, 2020. 6
[43] Hao Tian, Yuntao Chen, Jifeng Dai. Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5962-5972, 2021. 3
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023. 3
[45] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104:154-171, 2013. 3
[46] Chen Wang, Junfeng Ding. Xiangyu Chen, Zelin Ye, Jialu Wang, Ziruo Cai, and Cewu Lu. Tendencyrl: Multi-stage discriminative hints for efficient goal-oriented reverse curriculum learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3474-3488 IEEE, 2019. 5
[47] Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11713- 11723, 2020. 3,
[48] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz. Self- supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14543 14553,, 2022. 3
[49] Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. 4d unsupervised object discovery. arXiv preprint arXiv:2210.04801, 2022. 3
12

---

[50] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0740 07682, 2022.
[51] Yueh-Hua Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration, In International Conference on Machine Learning, pages 6818- 6827. PMLR, 2019. 6
[52] Yan Yan, Yuxing Mao, and Li. 18(10):3337, 2018.
Second: Sparsely embedded convolutional detection. Sensors,
[53] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11040-1 1048, 2020. 2
[54] Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to pearse-to-demse to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF international conference computer vision, pages 1951-1960, 2019.
[55] Yurong You, Katie Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark Campbell, and Kilian Weinberger. Learning detect mobile objects from lidar scans without labels. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1130 1140, 2022. 1,
3, 6,,,9,69
[56] Yurong You, Cheng Perng Phoo, Katic Luo, Travis Zhang, Wei-Lun Chao, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Unsupervised adaptation from repeated traversals for autonomous driving. Advances in Newral Information Processing Systems, 35:27716-27722 2022.
13

---

Supplementary Material: Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
S1 Additional Implementation Details
We train DRIFT four NVIDIA RTX A6000 GPUs, with batch size 10 per GPU. The values we use for the shape priors are reported in Tab. S10. In practice, we scale the mixtures such that the probabilities at the Gaussian means equal for stability reasons. The values in Tab. $10 computed from the box shapes in the Lyft dataset, but the method generalize well to other domains like Ithaca365. Bear in mind, all models trained on Ithaca365 directly uses the hyperparameters found in Lyft, suggesting the generalizability of our method. DRIFT achieves stable performance when alternative values used (Tab. S16).
Table S10: Shape prior values used in our implementation.
Width w
Length
Height
Mixture Component
Mean
StD.
Mean
StD.
Mean
StD.
(Car)
1.911
0.162
4.745
0.559
1.711
0.248
2 (Pedestrian)
0.780
0.153
0.797
0.182
1.745
0.177
3 (Truck)
2,832
0.278
9.403
3.145
3.299
0.430
4 (Cyclist)
0.613
0.256
1.752
0.326
1.364
0.343
S2 Extended Quantitative Results
$2.1 Full Tables from Main Paper
We provide results on all ranges for all ablation tables shown in the the main paper. In Tab. S11, we report the results of the aligmment reward ablation on Pecate In Tab. S12, we report ablation results on reward components. In Tab. S13 and Tab. S14 report ablation sample counts, respectively. In Tab. S15, we report ablation results for varying the filter budget.
exploration method and
Table S11: Ablation on the alignment reward's Pseate: We report the mAP (0-80m) on the Lyft dataset. This corresponds with Table of the main paper.
mAP loU 0.5
mAP IoU 0.7
Alignment Reward
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
f 0.8
60.1
40.2
9.1
38.3
39.0
24.2
3,6
23.1
/ 0.9
58.3
39.2
10.1
38.1
37.1
26.0
4.4
23.4
S2.2
Ablation on Shape Prior Reward
We additionally include an ablation study on the shape-prior reward in Tab. S16. We show that it is not sensitive to the choice variance used for each class. We assume that the means of the shape priors are user defined.
$2.3 Lower ToU and Recall Evaluation
report mAP results at IoU 0.25 match in Tab. $17. In particular, ToU 0.25 metrie evaluates "localization" ie. if there is bounding box with very small overlap with the ground truth. However,
14

---

Table S12: Ablation on the reward components. We report the mAP (0-80m) on the Lyft dataset. This table corresponds to Table 4 in the main paper.
mAP IoU
0.5
mAP IoU 0.7
Filtering
Size Prior
Align.
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
1.3
0.8
0.0
0.6
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0,0
0.0
0,0
/
V
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
V
41.2
18.2
1.7
22.4
5,3
5.0
0.3
3.2
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
Table S13: Detection performance with additional sampling sources in exploration. Sampling near the predictions sufficient; additional sources do not provide obvious benefits.
mAP IoU 0.5
mAP IoU 0.7
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
Sample near pred.
60.3
43.8
14.6
41.8
42.0
29.2
5.8
26.7
+ near dynamic
59.8
40.9
10.6
39.3
39.1
22.8
3.2
22.6
+ init. seed labels
60.3
40.7
10.4
39.5
38.4
23.9
3.3
22.8
our method excels at predicting the size and proper orientation, which is better captured at higher IoUs metrics. To summarize, as compared the prior work, MODEST can localize boxes well, but it's not able to figure out the size; in contrast, our method does both well. As stated in the main paper, use 0.5 and 0.7 to (1) following the KITTI and Lyft standards of reporting, and (2) emphasize the strength of our method.
In addition, report recall metrics in Tab. S18. We that the recall improves additional training, which suggests that part of the improvement comes from being able to better detect bounding boxes by locating them in the scene.
Table S18: Recall by epoch. We report class recall bers at different training epochs. Observe that DRIFT training improves the recall.
S2.4 Ablation class mixtures
We include additional experiments on the class type mixtures that we in the reward function of DRIFT to evaluate the robustness of method. We report the results using (the ground truth number), 5, and 6 factors in the Gaussian mixture in Tab. S19.
Recall (0 80m)
Epoch
IoU 0.5
IoU 0.7
$3 Extended Qualitative Visualizations
30
0.47
0.30
60
0.51
0.34
We showcase additional qualitative results in Fig. S8. Observe that DRIFT improves significantly over the model without fine-tuning, and close to supervised performance, without any labels.
90
0.53
0.36
120
0.56
0.38
15

---

Table S14: Ablation on sampling hyperparameters for exploration. Large sampling size and moderate noise scale preferable.
mAP IoU 0.5
mAP IoU 0.7
Noise
Samples
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
57.3
31.3
4.8
32.9
26.0
14.4
1.4
14.3
0.3
100
59.8
36.2
7.0
36.4
33.0
18.9
1.8
18.9
200
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
50
17.8
1.2
0.0
5.6
0.1
0.0
0.0
0.0
0.6
100
39.5
10.4
0.2
17.2
4.0
0.3
0.0
1.3
200
0.1
0.9
1.1
0.6
0.0
0.0
0.0
0.0
Table S15: Ablation on the choice of top k% for box filtering by reward ranking. DRIFT is robust to different values of k.
mAP IoU 0.5
mAP IoU 0.7
Top 1%
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
55%
60.0
36.5
7.5
37.0
36.5
20.1
2.3
21.0
65%
60.1
39.9
9.7
38.9
39.1
23.4
3.5
23.0
75%
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
85%
58.1
36.6
6.6
35.8
35.8
20.6
2.3
20.2
SelFTrain.
MODEST
DRIFT
mAP (BEV @ll 0.5)
Car
Pedestrian
Cyelist
Truck
Figure S7: Per-class BEV mAP at IoU 0.5. We assign each predicted box to the class with the most similar prototype size. This corresponds to Figure main paper.
Table $16:
Ablation on variance of the class shape priors.
mAP IoU 0.5
mAP IoU 0.7
Class Shape Var.
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
0.5 StD.
57.6
34.5
5.1
34.5
36.8
22.5
1.7
21.3
σ; 0.2 StD.
55.3
40.2
11.1
37.1
39.8
27.9
5.2
25.5
True Variance
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
16

---

Table S17: Detection performance on the Lyft dataset. We report the evaluation results at IoU 0.25. This metric is evaluating "localization", i.e. if there a bounding box with a tiny overlap with the GT. Our method matches the baselines on this match threshold, and surpasses on the more strict IoU thresholds of 0.5 and 0.7.
mAP IoU 0.25 (1)
0-30
30-50
50-80
0-80
No Finetuning Self-Train, (60 ep)
63.5
34.9
6.0
37.5
67.7
43.2
8.7
43.1
Self-Train, (600 ep)
67.7
48.0
13.3
45.5
MODEST (60 ep)
68.5
46.2
10.5
45.1
MODEST (600 ep)
73.6
56.8
21.0
53.6
DRIFT (60ep)
72.3
51.5
19.2
50.7
DRIFT (120 ep)
72.5
51.7
25.8
52.9
Supervised on KITTI
78.6
53.9
26.1
55.3
Supervised on Lyft
81.8
63.6
40.0
64.2
Table $19: Ablation on the number of factors in the class mixture. We test out additional class sizes other than the ground truth number of classes the label set (car, pedestrian, eyclist, truck). Observe that the number of factors does not significantly affect the performance of DRIFT.
mAP IoU 0.5
mAP IoU 0.7
Num. Factors
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
4 (GT)
60.1
40.2
9.1
38.3
39.0
24.2
3.6
23.1
5
59.8
37.9
7.4
37.0
38.6
24.7
2.9
23.0
58.5
38.3
6.1
36.0
39.1
23.9
2.6
22.7
Fine-tune
DRIFT
No Fine-tune
DRIFT
"25
Lyft Datasct
Ithaca365 Datasct
Figure S8: Qualitative visualizations. Additional visualizations the DRIFT method, as compared
to no fine-tuning.
17


Reviews:
[{"review_id": "TvXPN6b16H", "reviewer": "Reviewer_iC9C", "summary": "This work contributes to unsupervised object discovery in LIDAR point clouds. It defines a few typical properties of customary bounding boxes in LIDAR point clouds and then develops Rewards for a Reinforcement Learning algorithm to learn this, lacking a gradient for direct optimization.\n\nIt is based on Persistency Prior scores defined by the MODEST approach but adds several steps to improve bounding box selection. Furthermore they ablate many factors, e.g. the impact of enforcing a certain shape on bounding boxes. While they can only compare on a limited amount of epochs due to time constraints, possibly due to the plenty ablation studies, they show they outperform the existing approach in that domain.\n", "questions": "2 Related Works: 3D Object Detection: I would disagree it is ideal to train detectors in an unsupervised way. Maybe ideal could be a combination to achieve best performance at reasonable costs while being able to assign actual labels (car, pedestrian, etc.). Maybe a statement like that could be just removed.\n\nHow robust is the algorithm to the amount of traversals? This is not touched at all unless I overlooked it.\n\nJittered boxes and then NMS has strong similarity to region proposals in approaches like Mask R-CNN. Maybe it could be good comparing this historic approach in the Related Work.\n\nWhat is the impact of the quality of the first pre-trained detector? If there is a false negative detection, how can this algorithm improve a non-existing bounding box?\n\nIn Reward Finetuning for Model Alignment, what are \"human values\"? Do the models output unexpected or wrong output or are there ethical considerations? The term seems confusing.\n\nHow have the lambda factors and mu_scale and sigma_scale been found? Similarly how was the scale of the scaled up box selected when designing the Alignment Reward? Since the approach is supposed to find objects unsupervised, hyperparameter tuning and then manually checking the results by a human who looks at the bounding box detections in the point cloud may improve the results but not in a fair way. It goes against the idea of unsupervised object detection. \n\nIn 3.2 what is meant by \"In effect, this encourages all non-persistent points ... to propose boxes\". How do points propose boxes? This is not like Yolo where each grid cell is associated with a fixed number of region proposals right?\n\nCouldn't this approach potentially be expanded to actually label Car, Pedestrian, Cyclist and Truck based on the shape priors?\n\nIt would be worthwhile investigating the impact of false negatives to understand the impact of the approach vs. the failure of the first step of bounding box proposal. The authors did compare different ways of box predictions but that is different from quantifying the performance. Knowing how many false negatives are produced by the first step would already give some insight into this. Or is that the No Finetuning case?\n", "limitations": "There is only one comparable work which is an unavoidable limit. The authors solved this by also comparing against other tasks, i.e. the supervised case.\n\nDue to the deadline DRIFT was not trained to convergence, however it seems to outperform it's competitor already. For the camera ready final numbers should be entered.\n\nIn their limitations the authors address most of the concerns in Weaknesses.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper improves performance by a large margin in a very novel research area. Even if the approach itself may have limited use, it is straightforward to see the value e.g. in combination with a pre-trained traffic participant classifier.\n\nThe easy and intuitive explanation of the reward shaping is very clear. Figure 2 helps to get a very quick insight into the idea. The whole paper does a good job at combining intuitive explanations with mathematical formulations.\n\nThe approach has many individual steps and the pseudo-algorithm helps understanding.\n\nThe used Lyft Level 5 Perception dataset is a good choice for comparing with the state of the art.\n\nThe ablation studies are suitable to show that all used steps together achieve the performance.\n\nAll figure captions guide the reader nicely towards the main message that figure should convey.", "weaknesses": "It seems this work takes many ideas from MODEST so the idea of using an object detector and defining common sense properties is not novel to this work. It needs to be read with that contribution offset in mind. \n\nThe title is maybe a bit boasting. Teaching cars to see sounds like solving most if not all perceptual problems while this approach, coming from unsupervised object detection, only separates moving objects from background.\n\nHaving still a bit to train is not optimal but understandable. The paper feels a bit rough around the edges, the DRIFT abbreviation is explained twice.\n\nThe approach is a bit complex with many parameters to tune. Even though the ablation studies show the value of all individual components, it is hard to judge how much hyperparameter optimization vs. unsupervised modeling performance is present.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We address the individual points: \n\n> Novelty: compared to MODEST\n\nWe highlight that although MODEST and DRIFT both use commonsense properties, MODEST uses them only when generating seed labels and filtering between self-training rounds. In contrast, DRIFT directly incorporates common sense rules throughout the training process using the reward function, which is a significantly more efficient way of leveraging the knowledge. In addition, DRIFT takes a fundamentally different approach towards object discovery by leveraging imitation learning tools from Reinforcement Learning that are designed to handle the non-continuous, non-differentiable nature of the commonsense properties. In doing so, DRIFT displays significantly faster convergence and stronger performance than MODEST (Fig. 1).\n\n> Title and writing\n\nWe will adjust our title to make it more appropriate and descriptive of our task. We will remove the statement that it is ideal to train detectors in an unsupervised way and the repetitive explanation of the abbreviation.\n\n> Robustness to the amount of traversals\n\nThe Lyft dataset has around 5 repeated traversals per location, and Ithaca365 has around 20 traversals per location. DRIFT performs competitively on both datasets, indicating that it is robust to the amount of traversals, and the number of traversals needed does not have to be very large. \n\n> Comparison with Mask R-CNN\n\nThank you for pointing this out; we will include corresponding discussion in our final version. \n\n> What is the impact of the quality of the first pre-trained detector?\n\nThe pretrained detector can have bad quality (e.g. Tab. 1 “no finetuning”), but it needs to have some functionality and cannot be completely randomly initialized. A pretrained detector with better quality may further improve DRIFT’s performance.\n\n> ​​In Reward Finetuning for Model Alignment, what are \"human values\"?\n\nThe meaning of “human values” is context dependent. It could refer to output quality such as accuracy or coherence (e.g. [41] in Related Works) or ethical considerations (e.g. [30] in Related Works), etc. In the context of our task it refers to the common sense rules that the boxes should follow. We will make this clearer in our final version. \n\n> How have the [hyperparameters] been found? \n\nThe lambda factors were found by hyperparameter tuning on the Lyft dataset. Intuitively, the selected values aim to balance each component of the reward and prevent one from dominating the others. The mu_scale, sigma_scale and the x2 scaled up range were adopted based on common sense reasoning: The points should in general fall close to the inner surface of the bounding box, leading the mu_scale to be slightly smaller than 1. And we intuitively consider x2 scale a reasonable range to encompass points associated with the object a box aims to detect. Our ablation in Supp. Table 2 further indicates that reasonable alternative choices of mu_scale would yield similar performance. We additionally ablate the standard deviation sigma_scale, and observe that it is not sensitive to the choice of value:\n\n| Scale St. Dev. | IoU 0.5 | IoU 0.7 |\n|----------------|:-------:|:-------:|\n|   0.1 St. Dev. |   34.2  |   20.9  |\n|   0.2 St. Dev. |   38.3  |   23.1  |\n|   0.3 St. Dev. |   38.1  |   20.0  |\n\nWe would also like to highlight that we directly applied the hyperparameters we identified on the Lyft dataset to Ithaca365. Despite the differences in environment and data distribution, DRIFT attains strong performance on Ithaca365 using the hyperparameters from Lyft. This indicates that our default hyperparameters are very generalizable, and hyperparameter tuning or other human involvement may not be necessary when training DRIFT for a new domain. \n\n> In 3.2 what is meant by \"In effect, this encourages all non-persistent points ... to propose boxes\"?\n\nThis is referring to the first stage of PointRCNN, which classifies each point as either foreground or background, and generates box proposals from foreground points. A focal loss is applied to the foreground/background classification. Here, we modify the classification labels used for the loss, such that a non-persistent (low PP-score) point predicted as background is still treated as foreground. A similar approach has been adopted in [1]. This modification encourages the detector to generate proposals around non-persistent points. \n\n> Labeling based on the shape priors\n\nYes, DRIFT has the potential to provide a rough class labeling based on its shape priors, e.g. like what we did in the “Extension to Detection with Classes” section. It is worth noting that some classes (e.g. pedestrians vs. cyclists) can have similar sizes, and thus cannot be completely separated based on shape. Additional heuristics may be needed to improve the accuracy of the labeling. \n \n> False negatives \n\nAs the detector gets trained for longer, it becomes better at identifying dynamic objects and is less likely to have false negatives. We report the recall at various epochs below. We also empirically find that additionally sampling boxes around non-persistent points (as a way to capture potential false negatives) does not improve the end performance (Table 6). \n\n| Recall @ Ep. | IoU 0.5 |\n|--|:--:|\n| 30 | 0.47 |\n| 60 | 0.51 | \n| 90 | 0.53 |\n| 120 | 0.56 |\n\n> Final numbers \n\nWe provide the results below, and include discussion in ZfQG's response.\n\n|  | IoU 0.5 | IoU 0.7 |\n|--|:--:|:--:|\n| No Finetuning   |   23.9  |   10.5  |\n| DRIFT (300 ep)  |   44.6  |   31.3  |\n\n[1] You, Yurong, et al. 2022. Unsupervised Adaptation from Repeated Traversals for Autonomous Driving."}, {"review_id": "aaYR4XMzWE", "reviewer": "Reviewer_ZfQG", "summary": "This paper proposes DRIFT, a novel reward fine-tuning method for unsupervised object discovery with point cloud input. Specifically, three reward methods are proposed to identify good bounding boxes. First, shape prior reward prefers bounding boxes with similar sizes to the prototypes.  Second, an alignment reward gives high scores to boxes that have most of the LiDAR points near the box edge. Third, a filter reward follows the spirit that a good box should contain more dynamic points than background points. By using these reward methods for refining the object detector, DRIFT achieves more accurate object discovery, and it also converges much faster than the previous method.", "questions": "(1) Authors should provide complete experiment results and refine their experimental analysis correspondingly, as some of the experiments are still running at the time of submission (L234 & L258).\n\n(2) In L224, the authors say they include evaluation results with IoU at 0.25 in the supplementary, but I don't see these results. Authors should add these results and explain why they don't provide these results in the main paper.", "limitations": "The authors provide limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed reward fine-tuning method is well-motivated and improves object discovery accuracy compared to the previous work.\n\n(2) In addition to improving the detection performance, the proposed method can also greatly improve the training speed.\n\n(3) The expression of the paper is clear, and the figures are intuitive.", "weaknesses": "(1) Although intuitive, I think the prior that most points fall on the lateral surfaces of an object is too absolute, e.g., a lot of LiDAR points will fall on the hood and front window of the oncoming vehicles beside lateral surfaces. Apart from this, using Gaussian distribution as an approximation is also not accurate, because few points fall out of the box edge, as shown in Figure 3.\n\n(2) I think a missing ablation is the contribution of the exploration strategy to the final detection performance, similar to the ablation on the reward methods in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> About using the Gaussian distribution as an approximation\n\nWe agree with your point that the Gaussian distribution approximation might not be the most accurate one, but from our observation the hood or front window of vehicles only contribute to a small fraction of the point cloud (Figure 2 in the main paper shows an example) since lidar reflections are sparser on glasses or surfaces parallel to the beam. We also contend that this approximation is one of the simplest, robust approximations while yielding satisfactory performance. We showed ablation study in Table 5 in the main paper, Table 2 and 8 in the supplementary material about robust results of varying the Gaussian parameters. We hope this could serve as a first but simple and strong step, and encourage further research.\n\n> About the contribution of exploration strategy:\n\nWe note that we have included Table 6, 7, 8 in the main paper, Table 4, 5, 6 in the supplementary material for contributions of different exploration strategies. In addition to Table 6 in the main paper, we note that if we do not use any exploration at all, the detection performance will drop to 0 (we will add an additional line in Table 6 for this in the final version) due to confirmation bias (which is a key problem with self-training), in which case the model gets indulged in is own proposals and the training loss cannot provide meaningful gradients.\n\n> Final number\n\nWe report our results for up to 300 epochs below:\n\n|                 | IoU 0.5 | IoU 0.7 |\n|-----------------|:-------:|:-------:|\n| No Finetuning   |   23.9  |   10.5  |\n| MODEST (600 ep) |   39.6  |   18.8  |\n| DRIFT (30ep)    |   38.3  |   23.1  |\n| DRIFT (60 ep)   |   41.8  |   26.7  |\n| DRIFT (120 ep)  |   45.3  |   29.6  |\n| DRIFT (180 ep)  |   43.2  |   30.2  |\n| DRIFT (300 ep)  |   44.6  |   31.3  |\n\nWe observed that the method converged soon after the final reported numbers at 120 epoch. We will include the final number into the report.\n\n> About results for IoU 0.25:\n\nThanks for pointing this out! We accidentally missed this in the supplementary material, and we provide them below. We will add this to the final version.\n\nTable Results for IoU 0.25\n\n|                      | 0-30 | 30-50 | 50-80 | 0-80 |\n|----------------------|:----:|:-----:|:-----:|:----:|\n| No Finetuning        | 63.5 |  34.9 |  6.0  | 37.5 |\n| Self Train. (60 ep)  | 67.7 |  43.2 |  8.7  | 43.1 |\n| Self Train. (600 ep) | 67.7 |  48.0 |  13.3 | 45.5 |\n| MODEST (60 ep)       | 68.5 |  46.2 |  10.5 | 45.1 |\n| MODEST (600 ep)      | 73.6 |  56.8 |  21.0 | 53.6 |\n| DRIFT (60ep)         | 72.3 |  51.5 |  19.2 | 50.7 |\n| DRIFT (120 ep)       | 72.5 |  51.7 |  25.8 | 52.9 |\n| Supervised on KITTI  | 78.6 |  53.9 |  26.1 | 55.3 |\n| Supervised on Lyft   | 81.8 |  63.6 |  40.0 | 64.2 |\n\nOne thing to bear in mind is that IoU 0.25 metric is more evaluating “localization”, i.e. if there is a bounding box with a tiny overlap with the GT. However, our method excels at the size and proper orientation, which is better captured at higher IoUs metrics. Basically, MODEST can put a box, but it’s not able to figure out the size, but our method does both. As stated in the main paper, we use 0.5 and 0.7 for 1) following the KITTI and Lyft standards of reporting, 2) emphasizing the strength of our method."}, {"review_id": "FhgCPYqTtL", "reviewer": "Reviewer_dARk", "summary": "This paper proposes a new reinforcement-learning-based framework for unsupervised 3D object detection that uses these common-sense heuristics directly as a reward signal. Avoid handcrafting training examples for each object detector. Furthermore, under the premise of greatly accelerating the convergence speed of the model, this method improves the experimental effect.", "questions": "1. In line 40, can it be understood that the method proposed in this article is to perform finetune on the pre-trained MODEST through a reward-based method? If the answer is yes, is the time comparison in Figure 1 meaningful or fair? Moreover, LLM is mentioned more than once in the article, but this article does not seem to use a specific LLM model. I think this may cause some misunderstandings for readers at the beginning.\n2. The method proposed in this paper is more like a specific RAFT method in unsupervised object detection, and the novelty may be limited.\n3. The training steps mentioned in Line 56. As far as I know, many self-training methods can achieve the same goal, please tell us your advantages.\n4. Lines 126 to 132 in the article seem to give the boxes from the detector a priori information or heuristic constraints from the real world. You are modeling here by a mixture of Gaussian distributions, how to determine mixture weights? In line 115 section. What happens if you use a mixture factor greater than 4? It seems that the model only focuses on objects of these sizes. In line 131, what happens if you don't use the scale operation on it? please give an example.\n5. Is Figure 3 drawn from a real dataset? If so, please describe which data were used in the supplementary material. Can this be understood as prior information derived from daily data?\n6. The paragraph starting on line 140 means that by scaling, more points are included in the box, this seems to serve the same purpose as jittering mentioned in Sec 3.2 or Intro. What is the difference between them?\n7. The sensitivity analysis of mu_{scale} in line 156 should be Tab5.\n8. The ablation experiment of Tab4 should complement the other 3 groups and did not analyze the reason for this phenomenon, especially when the effect gap is huge. Because I want to know how the three components of the proposed Reward function affect the detection performance respectively\n9. Please confirm the experimental code for reproducing the MODEST in Tables 1 and 2. Whether there is a P2 filtering result seems to be far from the original MODEST paper.\n", "limitations": "One limitation is that this paper uses MODEST as the baseline. If the number of timestamps is not large, continuing to use the p2 score may aggravate the impact of noise on the model. This is a direction that needs to be addressed. Another limitation is that since it is unsupervised 3D object detection, not only static mobile objects should be detected, but all static objects should be detected, which may be beneficial to downstream tasks.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed reinforcement learning-based framework without encoding heuristics into differentiable loss functions and avoids the need to hand-engineer training paradigm.\n2. The method proposed by the author has a fast convergence speed and a great algorithm implementation value.\n3. The experiments in this article are relatively sufficient, and the charts are easy for readers to understand.", "weaknesses": "1. The content arrangement of this article does not seem to be very convenient to read and there are some careless quoting errors that refer to Questions.\n2. The author should provide further explanation or derivation in the formula part of the article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments, and will incorporate all of the edits/analysis that they suggest. Regarding the mention of LLMs, our method draws inspiration across many different fields (Reinforcement Learning, Object discovery, RLHF) and we had hoped to showcase an example of where such a bridge in these different fields has proven successful. We thank the reviewer for pointing out this potential point of confusion, and will clarify it in the final revision. \n\n> Finetune on the pre-trained MODEST clarification\n\nOur method is not fine-tuning a MODEST detector, and the time comparison in Fig. 1 is correct. The x-axis refers to the number of training epochs *from scratch* needed to obtain the values. The model we are fine-tuning from is pre-trained on noisy seed labels produced by DBSCAN clustering on spatial and PP-score (L203-206), plotted as the yellow dot in Fig. 1. \n\n> Novelty may be limited: similarity to RAFT\n\nThis work follows along a broader branch of works commonly known to the Reinforcement Learning (RL) and Controls community as Filtered/Top-K behavior cloning [1]. However, while this field is *well studied for controls applications*, we consider our work a pioneering effort in bringing its advantages into object discovery. The work RAFT [2] also cites RL as inspiration for improving generative modeling. However, the techniques of RL have never been used for object discovery; we demonstrate that by leveraging its tools to optimize non-continuous and non-differentiable rewards, we can obtain incredible performance (tables 1, 2). In summary: 1) we explore the task of object discovery as compared to generative modeling, 2) we formalize novel reward and exploration functions for the setting, which has never been studied before. We additionally hope to point out that, at the time of the submission, the work RAFT had not been published at an official conference venue, and should be considered a concurrent work to ours.\n\n> Question regarding size prior mixture of Gaussian distribution\n\nWe comment about shape templates and their generalizability in the general response, and additionally address it here. The mixture weights that we use are computed such that the probabilities at the Gaussian means are equal for stability reasons (L131). We show that the method is robust to mixture weights (Tab. 8) as well as number of mixture factors (general response).\n\n> Reward component analysis\n\nWe analyze the components of the reward function in Tab. 4, Fig. 5, and provide additional discussion into the findings. Removing the filtering step causes the object detector to be unable to distinguish between foreground objects and static background, thus the drop in performance (Tab. 4, L3). To better understand the reward ablation, we visualize the resulting predictions in Fig. 5 of the main paper. Removing the size reward causes the model to place bounding boxes of incorrect shape (Fig. 5, lower left). Removing the alignment reward results in boxes that have points in the center of the box, optimizing for large boxes that capture all the dynamic objects (Fig. 5, lower right). Because these failure cases are size and shape specific, the metric of mAP at IoU 0.7 and IoU 0.5 cannot capture these kinds of errors, thus the large drop in perceived performance (Tab. 4, L2&4). \n\n> Clarifications about Figure 3\n\nFigure 3 is plotted from real data, specifically the Lyft L5 object detection dataset. We will include this in the final version, and thank the reviewer for pointing out this error.\n\n> Clarification about scale verses jittering\n\nTo clarify, jittering is exploration of the box proposals to search for potentially better candidates, while scale is referring to the ratio near a particular box used to calculate the reward associated to the box. All boxes that were produced from jittering considers their own o(b) set of points when computing the reward (and thus, the scale), and the best boxes from all jittered boxes are used to update the model (L139-140, 170-172). \n\n> Clarification about baseline reproduction\n\nWe report the results on mAP at IoU 0.5 and 0.7, which is different from the results reported in MODEST (IoU 0.25, 0.5). We made this choice to follow the precedent set in prior object detection works, KITTI dataset and Lyft dataset. We used the official implementation for baseline reproduction. We will provide results for IoU 0.25 in the final submission, but note that at such a low IoU the metric captures only the localization accuracy of the model. Our work is able to localize objects as well as predict the correct shape completion, as shown in the higher performance at the more strict IoU matches (Tab. 1).\n\n> Clarification regarding number of timestamps and impact of noise\n\nWe hope to clarify that the computation uses repeated traversals as opposed to consecutive timestamps to compute P2-score, and is actually more robust to noise per-sequence in the current traversal as demonstrated by [3, 4]. \n\n> Limitation: discovery of static objects\n\nWe agree with the reviewer that this is a limitation of the current method (L304-305); however, we are limited by the data and labels that are present in an academic setting. In this work, we focus on object discovery for self-driving following prior self-driving object detection tasks. We note that there are only labels for dynamic objects, which is necessary for evaluation. In theory, however, one can define a reward function for any objects they desire to discover, both static or dynamic, and leave this to future works (L307). \n\n[1] Kumar et al. 2022. When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?\n\n[2] Dong et al. 2023. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (in submission)\n\n[3] Barnes et al., 2017. Driven to Distraction: Self-Sup. Distractor Learning for Robust Monocular Visual Odometry\n\n[4] You et al., 2022. Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception"}, {"review_id": "Nm8yttM7fN", "reviewer": "Reviewer_vhou", "summary": "The proposed DRIFT framework is an approach to realize object discovery without labels. DRIFT first extracts foreground proposals based on the PP-score method, and then leverages common-sense heuristics including shape prior, box alignment, and background point filtering, to reward proposed boxes. Reinforcement-learning-based optimization is adopted, maximizing the reward function in a local space. The experimental results demonstrate the superiority of DRIFT over prior self-training methods in terms of efficiency and generalizability.", "questions": "Please see weaknesses above.", "limitations": "The authors have discussed limitations at the end of the paper. The discussion is valuable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The framework for 3D object discovery that simplifies common-sense heuristics as Gaussian-based reward signals to fine-tune the detector is interesting, straightforward, and relatively novel. Since the object discovery paradigm does not incorporate pre-defined class labels, it is promising in out-of-domain object perception tasks.\n- Experiments are basically comprehensive, with ablations on key reward components and plenty of hyperparameters.\n- The paper is well-written, and the ideas are presented clearly, making it easy to follow the authors' arguments and understand their contributions.", "weaknesses": "- The authors mainly compare DRIFT with proposal-based (PP-score) baselines. \n    - A missing related work for comparison: [a]. [b] is a concurrent work, but is valuable to discuss it though a direct comparison is not feasible.\n    - The PP-score method requires multiple traversals to get a superior performance which could limit the generalizability in real-world applications. This helps DRIFT to achieve a closer out-of-domain supervised detector trained on KITTI in the Ithaca365 dataset. But in normal datasets without many traversals, DRIFT actually has a large gap compared to the out-of-domain supervised detector.\n    - Current heuristics behave poorly in pedestrians, as shown in the per-class BEV mAP of the pedestrian class in Fig. 6 & Supp-Fig. 1. Heuristics need careful designs and more sophisticated ones are probably necessary to achieve satisfactory results on all categories of objects.\n- The reviewer is curious about the training efficiency. Any more insights about why the proposed DRIFT converges much faster than self-training and MODEST?\n- Any more insights about the large StD. in Table 1?\n- Misc: Fig. 1 is not referenced in the main body.\n\n> [a] Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. ECCV 2022.\n>\n> [b] Towards Unsupervised Object Detection from LiDAR Point Clouds. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive reviews and thoughtful feedback. We address the individual points below:\n\n>Missing related works\n\nThank you for pointing to these papers. Unfortunately we did not find code for them, and thus could not compare with them during our rebuttal. We will instead include discussion on them in our final version.\n\n> Generalizability in real-world applications\n\nWe would like to highlight that in the real world, it is common for people to follow the same routine, or for different people to share the same route that can be pooled together to produce the multiple traversals. Therefore, while not all publicly available datasets contain multiple traversals, it is highly likely that they become accessible in large quantities when developing self-driving cars in practical contexts. \n\n> Performance on Pedestrians\n\nThe limited performance on pedestrians can be mainly attributed to two reasons: First of all, DRIFT itself does not predict classes for its detections, and we could only assign class labels during post-processing based on the most similar shape priors. Sometimes pedestrians are mislabeled as cyclists and vice versa, as they often have similar box sizes. Secondly, the classes are highly imbalanced in both datasets. For example, the Lyft dataset contains 92.47% cars and 5.91% pedestrians. This limits a detector’s ability to learn to detect the smaller classes. We leave this study to a future work.\n\n> Training efficiency\n\nOne potential reason is that DRIFT guides boxes to adhere to common sense heuristics using the reward function throughout the entire training process, while self-training and MODEST only leverage them for seed label generation and (for MODEST) filtering between self-training episodes. Therefore, DRIFT more effectively harnesses the heuristics, and leads the detector to learn to predict appropriate boxes much faster.  For additional details, see response to reviewer dARK, Similarity and advantages over self-training methods.\n\n> Large Standard Deviation in Supp. Table 1\n\nThe large StD.s can be caused by several reasons: First of all, there are inherent size variations among objects within the same class. For example, a compact car and a SUV both belong to the Car class but have distinct dimensions. Additionally, values in Supp. Table 1 are computed from the box shapes in the Lyft dataset, and there are variations in human annotations. \n\n> Missing reference for Supp. Figure 1\n\nThanks for pointing this out! We will add a reference for it in our final version."}, {"review_id": "9Vtyg4bK5l", "reviewer": "Reviewer_Baoq", "summary": "The paper addresses the research question of unsupervised 3D object location detection from LIDAR data in autonomous driving scenes.\n\nThe proposed method, DRIFT, improves upon the MODEST baseline by incorporating heuristics for judging objectness likelihood and using them as rewards within a reinforcement learning framework.\n\nDRIFT is evaluated on the LYFT and Ithaca benchmarks, demonstrating enhanced performance and training efficiency compared to the MODEST baseline.\n\n**Before rebuttal**:\n\nThe paper presents a promising contribution and could be a valuable addition to NeurIPS this year. However, since my familiarity lies more in the domain of object detection from 2D scenes, I may not be able to provide detailed insights into the previous work in the LIDAR domain.", "questions": "See the weaknesses. More like a suggestion: \n\nPlease state from the very start that your goal is to find generic (foreground vs. background) object regions, and NOT categorization. This maybe confusing to some readers like me, that when you use the term \"Object Discovery\", I look for a semanic categorization component as well. \n", "limitations": "Limitations are presented. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** The paper is well-written, well-executed, and effectively addresses the under-explored problem of unsupervised 3D object location detection in autonomous driving scenes. The proposed method, DRIFT, demonstrates the potential for real-life applications.\n\n**S2.** The proposed idea is well-grounded and logical. The incorporation of studied priors based on shape, size, and location to judge objectness likelihood aligns with human intuition. The utilization of a feedback loop through reward-based fine-tuning to improve model performance over time is a sensible approach.\n\n**S3.** The reported improvement achieved by the authors is significant. Despite training for only 30 epochs (compared to MODEST's 600 epochs), DRIFT outperforms the strong baseline on both datasets. Although there is still a noticeable gap when compared to supervised methods, the results demonstrate promising progress.", "weaknesses": "**W1.** It would have strengthened the paper to provide a context within the well-studied objectness literature [1-2]. The concept of objectness aims to identify low-level, generic cues that distinguish foreground regions from the background, such as edge distributions, boundary textures, and likely object size, shape, and location. The authors' goal aligns with this objective, albeit from a different modality (LIDAR). Drawing inspiration and techniques from the objectness literature could have been insightful.\n\n**W2.** An explanation of how the authors generated the shape templates (priors, prototypes) would have been beneficial. It is unclear how these templates were derived or selected.\n\n**W3.** Given the reliance on multiple heuristics in the method (which is reasonable and sound), it would have been valuable to investigate cross-dataset generalization, specifically from LYFT to Ithaca. Understanding how the measured statistics change across different domains and driving scenes would provide insights into the method's robustness and applicability.\n\n[1] BING: Binarized normed gradients for objectness estimation at 300fps, https://mmcheng.net/mftp/Papers/ObjectnessBING.pdf\n\n[2] Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments, https://www.mdpi.com/1424-8220/21/15/5116\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and appreciation of our contributions! We address individual questions below:\n\n> Include context within the well-studied objectness literature\n\nWe thank the reviewer for pointing out the comprehensive works [1, 2] and the explanation of the concept of objectness. Indeed, objectness is aligned with our goals in this work, especially its goals to identify how likely something is an object in *any category*. In particular, we were surprised to learn about the metrics that are used in this field of work that’s explicitly designed to handle class-free proposal generation, although from a different modality. In the future, we will use inspiration from these works to evaluate the works we do in this direction. We thank the reviewer for bringing this subject to our attention and will include extensive discussion into objectness estimation as prior for our work. \n\n> Explanation of how shape templates were generated\n\nWe show category-wise object size statistics (unit: meter) in the table below in the format of $\\mu$ ($\\sigma$):\n| Dataset | Category | Length | Height | Width |\n|--|:--:|:--:|:--:|:--:|\n| Lyft | Truck | 9.40 (3.15) | 3.30 (0.43) | 2.83 (0.27) |\n| Lyft | Cyclist | 1.75 (0.33) | 1.36 (0.34) | 0.61 (0.26) |\n| Lyft | Car | 4.74 (0.56) | 1.71 (0.25) | 1.91 (0.16) |\n| Lyft | Pedestrian | 0.80 (0.18) | 1.74 (0.18) | 0.78 (0.15) |\n| Ithaca365 | Truck | 6.09 (2.57) | 2.33 (0.77) | 2.22 (0.48) |\n| Ithaca365 | Cyclist | 1.75 (0.64) | 1.52 (0.51) | 0.71 (0.22) |\n| Ithaca365 | Car | 4.41 (0.22) | 1.55 (0.15) | 1.75 (0.12) |\n| Ithaca365 | Pedestrian | 0.60 (0.19) | 1.70 (0.14) | 0.61 (0.12) |\n\nWe selected our templates of the shape priors as well as the number of prototypes based on the Lyft L5 dataset statistics, following prior self-driving works using the classes: cars, pedestrians, cyclists, and trucks (Supplementary, Tab. 1). In this work, we assume domain knowledge over object sizes for which we wish to discover. However, we extensively ablate how much the priors and prototypes affect the final results, and find that our method is not sensitive to the values (Supplementary, Tab. 8), and generalizes to the Ithaca365 dataset *without changing the values* (Table 2). In addition, we found the method is not sensitive to the number of prototypes. We include this ablation in the response to reviewer dARK and again below: \n\n| Num. Factors | IoU 0.5 | IoU 0.7 |\n|--------------|:-------:|:-------:|\n|       4 (GT) |   38.3  |   23.1  |\n|            5 |   37.0  |   23.0  |\n|            6 |   36.0  |   22.7  |\n\nIncluding additional prototypes at 10, 15-meters doesn’t significantly change the results, following the intuition that the method will search for the size prototype that best describes/fit the dynamic points.\n\n> Cross-dataset generalization of heuristics (Lyft to Ithaca365)\n\nWe calculate the true $\\mu_{scale}^*$ and $\\sigma_{scale}^*$ from dataset statistics:\n|Dataset|$\\mu_{scale}^*$|$\\sigma_{scale}^*$|\n|--|:--:|:--:|\n|Lyft|0.9428|0.2772|\n\nTo confirm, we selected all of our heuristics and hyperparameters on the Lyft dataset, and directly transferred the method into the Ithaca365 dataset *without changing the values*, suggesting at least some generalizability in the method. \n\n> Goal ambiguity from the introduction text\n\nWe thank the reviewer for pointing out this possible cause of confusion, and will ensure that our final wording is clearer in our objectives.\n\n[1] BING: Binarized normed gradients for objectness estimation at 300fps\n\n[2] Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments\n"}]

Author rebuttal:
We express gratitude to the reviewers for their constructive feedback on our work and appreciate their acknowledgment that the writing is "well written" and "easy and intuitive" to follow [iC9C, ZfQG, vhou, Baoq]. To reiterate, our work introduces a novel adaptation of Reinforcement Learning (RL)-based methods for unsupervised object discovery from LiDAR points, which surpasses prior works in both accuracy and training efficiency. We appreciate the reviewers' recognition that "[our method] improves performance by a large margin in a very novel research area" [iC9C] and that our method is noted to "significantly improve over [prior works]" by all reviewers. Reviewer dARk concisely characterizes our work as a reinforcement learning-based framework that mitigates the need to encode heuristics into differentiable loss functions. In summary, we are thankful that reviewers have found our work to be "straightforward" [iC9C, vhou] and "well-motivated" [ZfQG, Baoq].

Additionally, we present a general comment regarding the advantages over self-training methods:

Our method, DRIFT, significantly outperforms vanilla self-training for object discovery and converges at a much faster rate (Table 1, 2, Figure 1: Self-Train baseline). Traditional self-training iteratively generates pseudo-labels and retrains the model, requiring convergence before generating the next set of pseudo-labels. In unsupervised scenarios, training a detector to mimic pseudo-labels from a model lacking ground truth supervision can lead to undesirable artifacts, further amplified by repeated training (confirmation bias). Our method mitigates the problem by drawing on the field of RL: the exploration component is crucial for our method (shown below), and is not present in traditional self-training methods. By performing local exploration instead of simply updating from its own predictions, DRIFT avoids confirmation bias and ensures that labels improve over what it predicts. Thus, DRIFT is able to perform updates per-training iteration as opposed to per self-training round (which is generally 60 epochs, times number of iterations) which allows it to converge significantly faster and achieve higher performance. 

| | IoU 0.5 | IoU 0.7 |
|--------------|:-------:|:-------:|
| No Exploration       |   0.0   |   0.0   |
| W/ Exploration       |   41.8  |   26.7  | 


We also provide a discussion on the shape templates and their generalizability, brought up by reviewers dARk and Baoq: 

The setting we’re studying assumes domain knowledge over object sizes for which we wish to discover, and we follow prior self-driving works using the classes: cars, pedestrians, cyclists, and trucks (Supplementary, Tab. 1). Our main results use the ground truth mean and variance class sizes computed on the Lyft dataset, and we show that it can generalize to the Ithaca365 dataset *without changing the values* (Table 2). We also ablate the mixture weights in the Supplementary and show that the results are not sensitive to the values chosen (Table 8):

| Shape St. Dev.   | IoU 0.5 | IoU 0.7 |
|------------------|---------|---------|
| 0.5 * I St. Dev. |    34.5 |   21.3  |
| 0.2 * I St. Dev. |    37.1 |   25.5  |
|    True St. Dev. |    38.3 |   23.1  |

Per reviewer dARk’s suggestion, we further ablate the number of mixture factors (class size priors) greater than 4. We compare to the ground truth number of mixture factors (4) to increasing the number of mixture factors at constant standard deviation and mixture weights:

| Num. Factors | IoU 0.5 | IoU 0.7 |
|--------------|:-------:|:-------:|
|       4 (GT) |   38.3  |   23.1  |
|            5 |   37.0  |   23.0  |
|            6 |   36.0  |   22.7  |

Here, we add in “pseudo-class” size priors that are larger than the largest class at 10 and 15-meter lengths, which do not exist in the data. However, even in such extreme cases we note that our method is robust to the number of mixture factors.

Meta-review: This paper investigates adapting a reward fine-tuning based reinforcement learning method for learning to detect objects from LiDAR points without utilizing training labels. The proposed approach aims to mimic human feedback by combining multiple heuristics into a reward function, instead of using the labels. The manuscript received the following ratings: weak accept,weak accept, borderline reject, weak accept and weak accept. While the reviewers appreciated the motivation, idea and performance of the method, they also raised some concerns. Authors responded to reviewer's queries by submitting a rebuttal. The rebuttal largely addressed the initial concerns raised by the reviewers. Given that four reviewers are generally positive about the paper as well as author's rebuttal, the recommendation is accept. Authors are strongly encouraged to take into account the suggestions of the reviewers as well as changes acknowledged in the rebuttal when preparing the final manuscript.
Decision: Accept (poster)