Title: Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self Driving

Self-Supervised Object Discovery

Reward Ranked Finetuning
Abstract: Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector’s own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT.

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Abstract

Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles-where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector's own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT .

1 Introduction

Self-driving cars need to accurately detect the moving objects around them in order to move safely. Most modern 3D object detectors rely on supervised training from 3D bounding box labels. However, these 3D bounding box labels are hard to acquire from human annotation. Furthermore, this supervised approach relies on a pre-decided vocabulary of classes, which can cause problems when the car encounters novel objects that were never annotated.

Our prior work, MODEST [55], introduced the first method to train 3D detectors without labeled data. In that work, we point out that instead of specifying millions of labels, one can succinctly describe heuristics for what a good detector output should look like. For example, one can specify that detector boxes should mostly enclose transient foreground points rather than background ones; they should roughly be of an appropriate size; their sides should be aligned with the LiDAR points; their bottom should touch the ground, etc. Although such heuristics are great for scoring a set of boxes proposed by a detector, training a detector on them is hard for two reasons: First, these heuristics are often non-linear, non-differentiable functions of the detector parameters (for example, a slight shift of the box can cause all foreground points to fall off.) Second, existing object detection pipelines use carefully designed training objectives that heavily rely on labeled boxes, that are difficult to modify (for example, PointRCNN [40] infers point labels from box labels and uses these for training). For these reasons, MODEST had to utilize an admittedly slow self-training pipeline to incrementally incorporate common-sense heuristics.

mAP (BEV @IoU 0.5)

20

30

40

50

0

200

400

600

800

Self Train.

MODEST

DRIFT

No-FT

5 Days

9 Hours

Training Epochs

mAP (BEV @IoU 0.7)

5

15

25

35

0

200

400

600

800

Self Train.

MODEST

DRIFT

No-FT

In this paper, we propose a new reward ranking based framework that utilizes these common-sense heuristics directly as a reward signal. Our method relies on finetuning with reward ranking [31, 27, 13, 33], where given an initialized object detector, we finetune it for a predefined set of desirable detection properties. This bypasses the need to encode heuristics as differentiable loss functions and avoids the need to hand-engineer training paradigms for each kind of object detector. Recent success with reinforcement learning from human feedback (RLHF) has proven effective in improving machine learning models -in particular, large language models (LLMs)- and aligning them with human preferences [31, 27]. However, these advancements have not been applicable to detection-based vision models that are trained with per-instance regression and are difficult to view under a probabilistic framework. To address this challenge, we utilize insights from reward ranked finetuning [13], a non-probabilistic paradigm designed for finetuning of LLMs, which inspired us to develop a similar framework for object discovery.

We refer to our method as Discovery from Reward-Incentivized Finetuning (DRIFT) . DRIFT does not require labels, and instead uses the Persistency Prior (PP) score [55, 3] as a heuristic to identify dynamic foreground points based on historical traversals. These foreground points give rise to rough (and noisy) label estimates [55], which we use to pre-train our detector. The resulting detector performs poorly but suffices to propose many boxes of roughly the right sizes that we can use for exploration. To facilitate reward ranked finetuning, we first propose a reward function to score boxes. Ideally, only boxes that tightly contain objects (e.g. a car) should yield high rewards. We achieve this by combining several simple heuristics (e.g. high ratio of foreground points) and assuming some rough knowledge about the object dimensions. During each iteration of training, DRIFT performs the following steps: 1. the object detector proposes many boxes in a given point cloud scene; 2. the boxes are 'jittered' through random perturbations (as a means of exploration); 3. the boxes are scored according to the reward function; 4. the topk % non-overlapping boxes are kept as pseudo-labels for gradient updates.

We evaluate DRIFT on two large, real-world datasets [12, 24] and show that we significantly outperform prior self-training methods both in efficiency and generalizability. Experimental results demonstrate that using reward ranked finetuning for object discovery under our framework can quickly converge to a solution that is on par with out-of-domain supervised object detectors within a few training epochs, suggesting that DRIFT may point towards a more general unsupervised learning formulation for object detectors in an in-the-wild setting.

2 Related Works

3D Object Detection. 3D object detection models usually take in LiDAR point clouds or multi-view images as input and aim to produce tight bounding boxes that describe nearby objects [9, 52, 26, 39, 40, 53, 54, 34]. Existing methods generally assume the supervised setting, in which the detector is trained with human-annotated bounding boxes. However, annotated data are often expensive to obtain

and limited in quantity. Furthermore, in tasks such as self-driving, environments can have highly varied conditions, and detectors with supervised training often require adaptation with additional labels from the new environment [47].

Unsupervised Object Discovery . The unsupervised object discovery task aims to identify and localize salient objects without learning from labels. Most existing works perform discovery from 2D images [8, 14, 45, 36, 2, 43, 48] or depth camera frames [21, 23, 17, 25, 20, 1]. Discovery from 3D LiDAR point clouds is underexplored. [49] performs joint unsupervised 2D detection and 3D instance segmentation from sequential point clouds and images based on temporal, motion and correspondence cues. MODEST [55] pioneers in performing label-free 3D object detection. It exploits high-level common sense properties from unlabeled data and bootstraps a dynamic object detector via repeated self-training. Despite promising performance, it requires excessive training time, which makes it difficult for practical use and development.

Reward Fine-Tuning for Model Alignment . Recently, foundation models [6, 29, 11, 44, 35, 37] have been shown to achieve strong performance in diverse tasks [5, 50], but sometimes produce outputs that do not align with human values [18, 30, 10]. A line of research aims to improve model alignment under the paradigm of Reinforcement Learning with Human Feedback (RLHF). Some pioneering works [41, 31, 33] learn a reward model and train foundation models with Proximal Policy Optimization (PPO) [38], but PPO is often expensive and unstable to train, and more importantly, requires a probabilistic output on the action space. This makes it hard to use for the object detection setting, which primarily uses regression-based losses. Reward ranked finetuning [27, 13] is a simplified alternative paradigm. It samples from a foundation model itself, filters generations using the reward model, and conducts supervised finetuning with the filtered generations.

3 Discovery from Reward-Incentivized Finetuning

Our framework, DRIFT, is inspired by the recent success of reward ranked finetuning methods for improving model alignment in the NLP community [13, 27]. We show that a similar approach can be adapted for 3D object discovery.

Problem Setup. We wish to obtain a dynamic object detection model on LiDAR data, i.e., a model to detect mobile objects in the LiDAR point clouds, without human annotations . Let P ∈ R N × 3 denote a N -point 3D point cloud captured by LiDAR from which we wish to discover objects. We assume inputs of unlabeled point clouds collected by a car equipped with synchronized sensors including LiDAR (for point clouds) and GPS/INS (for accurate position and orientation). Since no annotation is involved, such a dataset is easy to acquire from daily driving routines; we additionally assume it to cover some locations with multiple scans at different times for computation of PP-score.

Dynamic Point Proposals. DRIFT leverages prior works that use unsupervised point clouds to extract foreground-background segmentation proposals. While many works [22, 3] have promising dynamic foreground segmentation results, in this work we rely on point Persistency Prior score (PP-score) [55] for its accuracy and leave the extension of other proposal methods to future work. For the purpose of this research, dynamic foreground points constitute LiDAR points reflecting off traffic participants (e.g. cars, bicyclists, pedestrians).

Using historical LiDAR sweeps collected at nearby locations of our point cloud P , the PP-score [3, 55] τ ( P ) ∈ [0 , 1] N can provide an informative estimate on the per-point persistence, i.e., whether a point belongs to persistent background or not. The PP-score is defined as the normalized entropy over past point densities, based on the assumption that background space such as ground, trees, and buildings tend to exhibit consistent point densities across different LiDAR scans (high entropy), whereas points associated with mobile objects exhibit high density only if an object is present (low entropy).

3.1 Rewarding 'Good' Dynamic Boxes

We first establish a reward function that evaluates the quality of a set of bounding boxes for dynamic objects in a scene. We denote a set of M dynamic objects bounding boxes as B = { b 1 , . . . , b M } , where each bounding box b i is represented as an upright box with parameters ( x i , y i , z i , w i , l i , h i , θ i ) , defining the box's center, width, length, height, and z-rotation, respectively. The scoring function R scores the validity of the bounding boxes, given the observed point cloud P . In practice, a reward

Alignment

Shape Prior

Dynamic Point Counts

Proposal

Prototypes

7.0

0.1

0.0

0.0

Dynamic

Persistent

1.0

0.8

0.6

0.4

0.2

0.0

function that is positively correlated with IoU should suffice. We present our proposed reward function which aims to capture only dynamic points, filter nonsensical boxes, enforce correct size, and encourage proper box alignment to the captured dynamic points.

Shape Prior Reward. We enforce a box to not deviate significantly from a set of prototype sizes S = { ( w 1 , l 1 , h 1 ) , . . . , ( w C , l C , h C ) } (Fig. 2 left). We assume the shape prior distribution is a mixture of C isotropic Gaussians with mixture weights π i , diagonal variances Σ i , and corresponding means as ( w i , l i , h i ) . These low-level statistics may be acquired directly from the dataset, or from vehicle specs and sales data available online [47]. In practice, we scale the mixtures such that the probabilities at the Gaussian means are equal for stability reasons. With this, the shape prior reward for box b is computed as:

Alignment Reward. Due to the nature of LiDAR sensing, the points will mostly fall on the lateral surfaces of an object. Therefore, a well-formed box should have dynamic points approximately close to the boundary of a box (Fig. 2 middle). As [55] shows, PP-score allows for easy separation of dynamic and persistent background points. Let P dyn denote the set of dynamic points, and let P bg be that of background points. In practice, since the PP-score is an approximation of ground-truth persistence, we define P dyn = { p | τ ( p ) < 0 . 6 } and P bg = { p | τ ( p ) ≥ 0 . 9 } .

Given a box b , we denote all points within and close to the box as O ( b ) . Only points within O ( b ) contribute to the reward of b . In practice, we let O ( b ) consist of all points within a × 2 scaled up version of b (with identical center and rotation). To score a box b , we design a reward function that identifies how 'typical' the dynamic points within O ( b ) are. For each dynamic point p ∈ O ( b ) ∩ P dyn, we compute the scaling factor s p , b required so that the rescaled box touches p with one of its sides; i.e., if a point is inside the box, the box would have to be scaled down ( s p , b < 1 ) to touch the point, if the point is outside it must be scaled up ( s p , b > 1 ). We assume that s p , b roughly follows a Gaussian distribution centered near the box boundary, and visualize the actual distribution in Fig. 3.

Center

0.5

1.5

2.0

0.00

0.02

0.04

0.06

0.08

0.10

Points Distribution

Scale

We define our reward as the likelihood under this Gaussian distribution over scaling parameters. We approximate it as a Gaussian with hyper-parameters mean µ scale and a variance σ scale. Our reward is the product of the probability of each point in O ( b ) :

Common Sense Heuristics and Filtering. Lastly, a proper bounding box must capture the dynamic points, and avoid capturing the background points (Fig. 2, right). This heuristic can be encoded by a

simple weighted point count for each bounding box b :

Intuitively, it assigns a reward in proportion to the number of dynamic points captured by the box, and a penalty in proportion to the number of background points captured.

Furthermore, boxes violating common sense should be assigned a low reward. We filter boxes that are too high up or too low from the ground, including those with too few dynamic points, or are too small or too large by directly assigning a reward of 0. In practice, we filter boxes that contain fewer than 4 dynamic points, or that have more than 80% persistent points, similar to [55].

In summary, the reward function is designed to be

3.2 Exploration Strategy for Improved Discovery

We assume a simple exploration strategy for identifying good box proposals. Given a set of current object proposals given by the detector, we locally perturb the boxes in the output space:

where B explore is the set of explored boxes, perturbed from the model proposals B 0 . For each box b ∈ B 0 , a set of explored boxes are sampled according to a standard Gaussian noise along the position and size dimensions and uniform noise for orientation:

Furthermore, to encourage proposals of boxes in foreground regions, we take inspiration from [56, 40] and re-use PP-score as point-level semantic segmentation (foreground vs background) labels, with which the detector is encouraged to propose boxes at points that have low PP-scores (i.e.are likely to be foreground points). Following [56], for each point p i ∈ P with prediction ˆ y i , we assign its target classification label y i as:

In effect, this encourages all non-persistent points (i.e., low τ ( p )

i ) to propose boxes near dynamic regions for better exploration.

3.3 Reward-Incentivized Finetuning

The reward function R allows us to quickly evaluate proposed bounding boxes B and the task of 3D object discovery could be reduced to an optimization problem on the total reward in box set space:

where the sum is taken over the boxes in the set B . Although a direct optimization for B ∗ is not plausible due to the non-polynomial search space and discontinuity in R , R can serve as effective guidance to facilitate model finetuning. The underlying intuition is similar to curriculum learning [4, 28, 46]: the object detection model takes small steps to improve from its current predictions towards B ∗ by following the direction provided by the maximum R direction in a local space.

As illustrated in Alg. 1, in each training iteration, we first let the object detector perform inference on a point cloud P and propose a set of dynamic objects B 0 in the scene. To explore directions

Algorithm 1 Reward-Incentivized Finetuning

Input: Base object detector f θ , unlabeled LiDAR dataset D , exploration noise σ , sample size n , filter budget k , reward function R , PP-scores τ .

repeat

of improvement with the non-differentiable reward function R , we sample n boxes from B 0 (with replacement) and add an i.i.d. Gaussian noise on their location and size, and an uniform noise on orientation following Eq. 4. These sampled boxes are then ranked by the reward function R , in which the top k non-overlapping boxes are selected by Non-Maximum Suppression (NMS) as training targets to finetune the object detector. Note that since DRIFT treats the model training/inference procedures as black boxes, it can be applied to any 3D object detection model.

In practice, it is observed that neural networks can acquire task knowledge from imperfect demonstrations [16, 51, 32]. MODEST [55] pre-trained the 3D object detector on noisy seed labels produced by DBSCAN [15] clustering on spatial and PP-score. We follow [55] and initialize our 3D object detector a model trained with discovered seed labels.

4 Experiments

Datasets. We experimented with two different datasets: Lyft Level 5 Perception dataset [24] and Ithaca-365 dataset [12]. To the best of our knowledge, these are the two publicly available datasets that contain multiple traversals of multiple locations with accurate 6-DoF localization and 3D bounding box labels for traffic participants.

In the Lyft dataset, we experiment with the same split provided by [55], where the train set and test set are geographically separated. It consists of 11,873 train scenes and 4,901 test scenes. For the Ithaca365 dataset, we experimented with the full dataset which consists of 57,107 scenes for training and 1,644 for testing. For both datasets, we do not use any human-annotated labels in training. To show the generalizability of our method, we conduct the development on the Lyft dataset, i.e., all the hyperparameters of our approach are finalized through experiments on Lyft, and we use the exact same set of hyperparameters for all experiments in Ithaca365.

Evaluation. Following [55], we combine all traffic participants to a single mobile object class and evaluate the detector's performance on this class. Note that the labels are not used during training but solely for evaluation. For Lyft, we report the mean average precision (mAP) of the detector with the intersection over union (IoU) thresholds at 0.5 and 0.7 in bird-eye-view perspective. Note that mAP at 0.7 IoU threshold is a stricter and harder metric and was not evaluated in [55], and we include it to emphasize the effectiveness of our method. For Ithaca365, we adopt metrics similar to those in [7]: we evaluate mean average precision (mAP) for dynamic objects under { 0 . 5 , 1 , 2 , 4 } mthresholds that determine the match between detection and ground truth; we also compute 3 types of true positive metrics (TP metrics), including ATE, ASE and AOE for measuring translation, scale and orientation errors. These TP metrics are computed under a match distance threshold of 2 m; additionally, we also compute a distance-based breakdown (0-30m, 30-50m, 50-80m) for these metrics.

Implementation. Weuse PointRCNN [40] as our default architecture and we use the implementation provided by OpenPCDet [42]. We train DRIFT with 120 epochs in Lyft and 30 epochs in Ithaca365 as the default setting, and observe that the performance generally improves with more training epochs (Fig. 1). We use λ shape = 1 , λ align = 1 , λ dyn = 0 . 001 and λ bg = 0 . 001 . We use µ scale = 0 . 8 and σ scale = 0 . 2 for the alignment reward. We define the shape priors based on four typical types of traffic participants: Car, Pedestrian, Cyclist, and Truck. Specifically, we use the mean and standard deviation of box sizes of each class in the Lyft dataset, but we show that they generalize well to other domains like Ithaca365 and are not sensitive (Tab. 2) The exact prototype sizes S and other implementation details can be found in the supplementary materials.

Baselines. To the best of our knowledge, MODEST [55] is the only prior work on this problem and we compare our method DRIFT against it with various variants of MODEST: (1) No Finetuning: the model trained with seed labels from PP-score without repeated self-training (MODEST (R0)) in [55]; (2) Self-Training ( i ep): the model initialized with (1) and self-trained with i epochs without PP-score filtering; (3) MODEST ( i ep): the model initialized with (1) and self-trained with i epochs with PP-score filtering (full MODEST model). For self-training in (2) and (3), we adopt 60 epochs for each self-training round in the Lyft dataset (same as that in [55]) and 30 epochs for the Ithaca365 dataset. To ensure a fair comparison, DRIFT is also initialized from (1) and use the same detector configurations as the baselines. Following [55], we also compare with the supervised counterparts trained with human-annotated labels from the same dataset (Lyft or Ithaca365) and from another out-of-domain dataset (KITTI).

Dynamic Object Detection Results. We report the performance of DRIFT and baseline detectors on Lyft in Tab. 1, and show the performance over the training epochs in Fig. 1. We report the performance on Ithaca365 in Tab. 2. Notably, DRIFT demonstrates significantly faster learning and strong performance. It provides more than 10 × speedup as compared to the baselines. On Lyft, DRIFT's performance at 60 epochs already surpasses the performance of both baselines at 600 epochs (10 self-training rounds) and approaches the performance of the out-of-domain supervised detector trained on KITTI [19]. On Ithaca365, its performance at 30 epochs significantly surpasses both baselines trained at 300 epochs. It even outperforms the out-of-domain supervised detector trained on KITTI in mAP.

Observe that the self-training performance starts collapsing with more rounds of self-training, and does not continue to improve.

Fig. 4 visualizes the detection on two scenes. Ground truth boxes are colored in green, predictions from the detector without fine-tuning are in yellow, and predictions from DRIFT are in red. We observe that the detector without fine-tuning occasionally produces false positive predictions, produces boxes with incorrect sizes, or misses moving objects, while DRIFT produces accurate detection.

Rewards ablations. We report the average reward per box for ground truth boxes, random boxes, and predicted boxes from different detectors in Tab. 3. The ground truth boxes have the highest rewards on average, while the random boxes have the lowest. This indicates that the reward reasonably reflects

NoFine-tune

DRIFT

NoFine-tune

DRIFT

Lyft Dataset

Ithaca365Dataset

the quality of the bounding box. And we observe the boxes predicted by DRIFT have higher rewards than those predicted by the baseline detectors.

Ablation study on the components of our reward is presented in Tab. 4, and visualization is shown in Fig. 5. Detection performance significantly drops when we remove one or more of the components. For example, when only common sense filtering is used, the detector just predicts boxes around foreground points. Without the shape prior reward, the detector predicts boxes with incorrect sizes.

Ablations Tab. 5 and Tab. 6 present the sensitivity analysis of the choices of µ scale and σ scale. DRIFT achieves stable performance across reasonable choices of µ scale and σ scale, showing the robustness of our method.

Exploration. We study the necessity of the exploration component and the effect of incorporating other sources for box sampling. In Tab. 7, we compare no exploration to: (1) sampling 200 boxes from box predictions, (2) sampling 100 from proposals near dynamic points and 100 from predictions, and (3) sampling 100 from seed labels and 100 from predictions. Observe that the exploration component is crucial for our method; by performing local exploration instead of simply updating from its own predictions, DRIFT avoids confirmation bias and ensures that labels improve over what it predicts. Furthermore, results show that sampling from the box predictions is sufficient for obtaining good performance; other sources do not provide obvious benefits.

We also explore the effects of modifying the exploration strategy. Tab. 8 compares the detector performance of using sample size of 50, 100 and 200, and noise scale σ (i.e.variation) of 0.3 vs. 0.6. Each detector is trained for 30 epochs. At noise scale 0.3, increasing the sample size from 50 to 200 significantly improves the detection performance. Using noise scale 0.6 significantly reduces the detection performance, indicating that smaller noise may be preferable.

Filtering Budget of the Ranked Boxes. We study the effect of the choice of top k % for filtering boxes by reward ranking. Tab. 9 presents the detection performance with top 55%, 65%, 75% and 85%. DRIFT is robust to the choice of k , with slightly decreased performance when k is too high.

FullReward

FilterReward Only

1.0

80

0.6

0.4

0.2

0.0

(-) Size Reward

(-) Alignment Reward

mAP (BEV @IoU 0.7)

0

10

20

30

40

Car

Pedestrian

Cyclist

Truck

Self Train.

MODEST

DRIFT

Extension to Detection with Classes. Our prototype sizes are defined by different classes of traffic participants. Thus, given a predicted box from a class agnostic detector, we can compute the likelihood of its size under the Gaussian prior of each prototype size, and assign it to the class with the highest likelihood. Fig. 6 presents the per-class performance of DRIFT and baselines under such assignment. All detectors are trained for 60 epochs. DRIFT outperforms both baselines, with especially significant improvement for the Car and Cyclist classes. More details can be found in the supplementary materials.

5 Discussion and Conclusion

In this work, we propose a framework, DRIFT, to tackle object discovery without labels. Instead of requiring expensive 3D bounding box labels, our method utilizes succinctly described heuristics as a reward signal to initialize and subsequently fine-tune an object detector. To optimize this nondifferentiable reward signal, we propose a simple but very effective reward finetuning framework, inspired by recent successes of reinforcement learning in the NLP community. Compared to prior self-training based methods [55], such a framework is an order of magnitude faster to train, while achieving higher accuracy. Traditional self-training iteratively generates pseudo-labels and retrains the model, requiring convergence before generating the next set of pseudo-labels. In general, training a detector to mimic pseudo-labels can lead to undesirable artifacts, further amplified by repeated training (confirmation bias). DRIFT addresses this issue by leveraging reinforcement learning principles, where the exploration component is crucial. Our method avoids confirmation bias by performing local exploration and ensures that labels improve over what it predicts. Thus, DRIFT is able to perform updates per-training iteration as opposed to per self-training round, which allows it to converge significantly faster and achieve higher performance.

Limitations and Future Works. One limitation is that the current framework is geared explicitly towards dynamic objects. Static objects would require different heuristics, not based on PP-scores. Similarly, currently we restricted our framework entirely to LiDAR signals. However, the reward

based framework is extremely flexible, and could easily be extended to other data modalities. For example, one could use image features to help identify objects inside of box proposals. Although in supervised settings image features have typically not added much in to the higher resolution LiDAR point clouds, in our unsupervised setting it is certainly possible that pixel information can help disambiguate objects from background. Further, we plan to explore the use of reward fine-tuning for other vision applications beyond object discovery.

Acknowledgments and Disclosure of Funding

This research is supported in part by grants from the National Science Foundation (III2107161, IIS 2144117, and IIS-1724282), Nvidia Research, and DARPA Geometries of Learning (HR00112290078). We also thank Wei-Lun Chao, Yihong Sun, Travis Zhang, and Junan Chen for their valuable discussion.

References

Aikaterini Adam, Torsten Sattler, Konstantinos Karantzalos, and Tomas Pajdla. Objects can move: 3d change detection by geometric transformation consistency. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pages 108-124. Springer, 2022. 3

Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 328-335, 2014. 3

Dan Barnes, Will Maddern, Geoffrey Pascoe, and Ingmar Posner. Driven to distraction: Self-supervised distractor learning for robust monocular visual odometry in urban environments. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1894-1900. IEEE, 2018. 2, 3

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009. 5

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021. 3

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 3

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , pages 11621-11631, 2020. 6

Joao Carreira and Cristian Sminchisescu. Constrained parametric min-cuts for automatic object segmentation. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3241-3248. IEEE, 2010. 3

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 1907-1915, 2017. 2

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053 , 2022. 3

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. 3

Carlos A Diaz-Ruiz, Youya Xia, Yurong You, Jose Nino, Junan Chen, Josephine Monica, Xiangyu Chen, Katie Luo, Yan Wang, Marc Emond, et al. Ithaca365: Dataset and driving perception under repeated and challenging weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21383-21392, 2022. 2, 6, 8

Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 , 2023. 2, 3

Ian Endres and Derek Hoiem. Category independent object proposals. In ECCV (5) , pages 575-588. Citeseer, 2010. 3

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd , volume 96, pages 226-231, 1996. 6

Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. corr abs/1802.05313 (2018). arXiv preprint arXiv:1802.05313 , 1802. 6

Germán M García, Ekaterina Potapova, Thomas Werner, Michael Zillich, Markus Vincze, and Simone Frintrop. Saliency-based object discovery on rgb-d data with a late-fusion approach. In 2015 IEEE International Conference on Robotics and Automation (ICRA) , pages 1866-1873. IEEE, 2015. 3

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 , 2020. 3

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013. 7

Maciej Halber, Yifei Shi, Kai Xu, and Thomas Funkhouser. Rescan: Inductive instance segmentation for indoor rgbd scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2541-2550, 2019. 3

Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d object discovery via multi-scene analysis. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4850-4856. IEEE, 2011. 3

Xiangru Huang, Yue Wang, Vitor Guizilini, Rares Ambrus, Adrien Gaidon, and Justin Solomon. Representation learning for object detection from unlabeled point cloud sequences. In Conference on Robot Learning , 2022. 3

Andrej Karpathy, Stephen Miller, and Li Fei-Fei. Object discovery in 3d scenes via shape analysis. In 2013 IEEE international conference on robotics and automation , pages 2088-2095. IEEE, 2013. 3

R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet. Lyft level 5 av dataset 2019. urlhttps://level5.lyft.com/dataset/, 2019. 2, 6, 8

Deyvid Kochanov, Aljoša Ošep, Jörg Stückler, and Bastian Leibe. Scene flow propagation for semantic mapping and object discovery in dynamic street scenes. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1785-1792. IEEE, 2016. 3

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12697-12705, 2019. 2

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems , 35:27591-27609, 2022. 2, 3

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research , 21(1):7382-7431, 2020. 5

OpenAI. Gpt-4 technical report, 2023. 3

Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4262-4274, 2021. 3

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022. 2, 3

Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2701-2710, 2017. 6

André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint arXiv:2302.08242 , 2023. 2, 3

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9277-9286, 2019. 2

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021. 3

Zhile Ren and Gregory Shakhnarovich. Image segmentation by cascaded region agglomeration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2011-2018, 2013. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 3

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. 3

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10529-10538, 2020. 2

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 770-779, 2019. 1, 2, 5, 6

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020. 3

OpenPCDet Development Team. Openpcdet: An open-source toolbox for 3d object detection clouds. https://github.com/open-mmlab/OpenPCDet , 2020. 6

Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5962-5972, 2021. 3

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 3

Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision , 104:154-171, 2013. 3

Chen Wang, Junfeng Ding, Xiangyu Chen, Zelin Ye, Jialu Wang, Ziruo Cai, and Cewu Lu. Tendencyrl: Multi-stage discriminative hints for efficient goal-oriented reverse curriculum learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3474-3480. IEEE, 2019. 5

Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1171311723, 2020. 3, 4

Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz. Selfsupervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14543-14553, 2022. 3

Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. 4d unsupervised object discovery. arXiv preprint arXiv:2210.04801 , 2022. 3

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. 3

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning , pages 68186827. PMLR, 2019. 6

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 18(10):3337, 2018. 2

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11040-11048, 2020. 2

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1951-1960, 2019. 2

Yurong You, Katie Luo, Cheng Perng Phoo, Wei-Lun Chao, Wen Sun, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Learning to detect mobile objects from lidar scans without labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1130-1140, 2022. 1, 2, 3, 4, 5, 6, 9

Yurong You, Cheng Perng Phoo, Katie Luo, Travis Zhang, Wei-Lun Chao, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Unsupervised adaptation from repeated traversals for autonomous driving. Advances in Neural Information Processing Systems , 35:27716-27729, 2022. 5

Supplementary Material: Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

S1 Additional Implementation Details

We train DRIFT on four NVIDIA RTX A6000 GPUs, with batch size 10 per GPU. The values we use for the shape priors are reported in Tab. S10. In practice, we scale the mixtures such that the probabilities at the Gaussian means are equal for stability reasons. The values in Tab. S10 are computed from the box shapes in the Lyft dataset, but the method generalize well to other domains like Ithaca365. Bear in mind, all models trained on Ithaca365 directly uses the hyperparameters found in Lyft, suggesting the generalizability of our method. DRIFT achieves stable performance when alternative values are used (Tab. S16).

S2 Extended Quantitative Results

S2.1 Full Tables from Main Paper

We provide results on all ranges for all ablation tables shown in the the main paper. In Tab. S11, we report the results of the alignment reward ablation on µ scale . In Tab. S12, we report ablation results on reward components. In Tab. S13 and Tab. S14 we report ablation on exploration method and sample counts, respectively. In Tab. S15, we report ablation results for varying the filter budget.

S2.2 Ablation on Shape Prior Reward

We additionally include an ablation study on the shape-prior reward in Tab. S16. We show that it is not sensitive to the choice in variance used for each class. We assume that the means of the shape priors are user defined.

S2.3 Lower IoU and Recall Evaluation

We report mAP results at IoU 0.25 match in Tab. S17. In particular, IoU 0.25 metric evaluates 'localization", i.e. if there is a bounding box with a very small overlap with the ground truth. However,

our method excels at predicting the size and proper orientation, which is better captured at higher IoUs metrics. To summarize, as compared to the prior work, MODEST can localize boxes well, but it's not able to figure out the size; in contrast, our method does both well. As stated in the main paper, we use 0.5 and 0.7 to (1) following the KITTI and Lyft standards of reporting, and (2) emphasize the strength of our method.

In addition, we report recall metrics in Tab. S18. We see that the recall improves over additional training, which suggests that part of the improvement comes from being able to better detect bounding boxes by locating them in the scene.

S2.4 Ablation on class mixtures

We include additional experiments on the class type mixtures that we use in the reward function of DRIFT to evaluate the robustness of our method. We report the results of using 4 (the ground truth number), 5, and 6 factors in the Gaussian mixture in Tab. S19.

S3 Extended Qualitative Visualizations

We showcase additional qualitative results in Fig. S8. Observe that DRIFT improves significantly over the model without fine-tuning, and close to supervised performance, without any labels.

mAP (BEV @IoU 0.5)

0

20

40

60

Car

Pedestrian

Cyclist

Truck

Self Train.

MODEST

DRIFT

No Fine-tune

DRIFT

No Fine-tune

DRIFT

41

Lyft Dataset

Ithaca365Dataset

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the research question of unsupervised 3D object location detection from LIDAR data in autonomous driving scenes.\n\nThe proposed method, DRIFT, improves upon the MODEST baseline by incorporating heuristics for judging objectness likelihood and using them as rewards within a reinforcement learning framework.\n\nDRIFT is evaluated on the LYFT and Ithaca benchmarks, demonstrating enhanced performance and training efficiency compared to the MODEST baseline.\n\n**Before rebuttal**:\n\nThe paper presents a promising contribution and could be a valuable addition to NeurIPS this year. However, since my familiarity lies more in the domain of object detection from 2D scenes, I may not be able to provide detailed insights into the previous work in the LIDAR domain.\n\nSTRENGTHS:\n**S1.** The paper is well-written, well-executed, and effectively addresses the under-explored problem of unsupervised 3D object location detection in autonomous driving scenes. The proposed method, DRIFT, demonstrates the potential for real-life applications.\n\n**S2.** The proposed idea is well-grounded and logical. The incorporation of studied priors based on shape, size, and location to judge objectness likelihood aligns with human intuition. The utilization of a feedback loop through reward-based fine-tuning to improve model performance over time is a sensible approach.\n\n**S3.** The reported improvement achieved by the authors is significant. Despite training for only 30 epochs (compared to MODEST's 600 epochs), DRIFT outperforms the strong baseline on both datasets. Although there is still a noticeable gap when compared to supervised methods, the results demonstrate promising progress.\n\nWEAKNESSES:\n**W1.** It would have strengthened the paper to provide a context within the well-studied objectness literature [1-2]. The concept of objectness aims to identify low-level, generic cues that distinguish foreground regions from the background, such as edge distributions, boundary textures, and likely object size, shape, and location. The authors' goal aligns with this objective, albeit from a different modality (LIDAR). Drawing inspiration and techniques from the objectness literature could have been insightful.\n\n**W2.** An explanation of how the authors generated the shape templates (priors, prototypes) would have been beneficial. It is unclear how these templates were derived or selected.\n\n**W3.** Given the reliance on multiple heuristics in the method (which is reasonable and sound), it would have been valuable to investigate cross-dataset generalization, specifically from LYFT to Ithaca. Understanding how the measured statistics change across different domains and driving scenes would provide insights into the method's robustness and applicability.\n\n[1] BING: Binarized normed gradients for objectness estimation at 300fps, https://mmcheng.net/mftp/Papers/ObjectnessBING.pdf\n\n[2] Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments, https://www.mdpi.com/1424-8220/21/15/5116\n\nQUESTIONS:\nSee the weaknesses. More like a suggestion: \n\nPlease state from the very start that your goal is to find generic (foreground vs. background) object regions, and NOT categorization. This maybe confusing to some readers like me, that when you use the term \"Object Discovery\", I look for a semanic categorization component as well.\n\nLIMITATIONS:\nLimitations are presented.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed DRIFT framework is an approach to realize object discovery without labels. DRIFT first extracts foreground proposals based on the PP-score method, and then leverages common-sense heuristics including shape prior, box alignment, and background point filtering, to reward proposed boxes. Reinforcement-learning-based optimization is adopted, maximizing the reward function in a local space. The experimental results demonstrate the superiority of DRIFT over prior self-training methods in terms of efficiency and generalizability.\n\nSTRENGTHS:\n- The framework for 3D object discovery that simplifies common-sense heuristics as Gaussian-based reward signals to fine-tune the detector is interesting, straightforward, and relatively novel. Since the object discovery paradigm does not incorporate pre-defined class labels, it is promising in out-of-domain object perception tasks.\n- Experiments are basically comprehensive, with ablations on key reward components and plenty of hyperparameters.\n- The paper is well-written, and the ideas are presented clearly, making it easy to follow the authors' arguments and understand their contributions.\n\nWEAKNESSES:\n- The authors mainly compare DRIFT with proposal-based (PP-score) baselines. \n    - A missing related work for comparison: [a]. [b] is a concurrent work, but is valuable to discuss it though a direct comparison is not feasible.\n    - The PP-score method requires multiple traversals to get a superior performance which could limit the generalizability in real-world applications. This helps DRIFT to achieve a closer out-of-domain supervised detector trained on KITTI in the Ithaca365 dataset. But in normal datasets without many traversals, DRIFT actually has a large gap compared to the out-of-domain supervised detector.\n    - Current heuristics behave poorly in pedestrians, as shown in the per-class BEV mAP of the pedestrian class in Fig. 6 & Supp-Fig. 1. Heuristics need careful designs and more sophisticated ones are probably necessary to achieve satisfactory results on all categories of objects.\n- The reviewer is curious about the training efficiency. Any more insights about why the proposed DRIFT converges much faster than self-training and MODEST?\n- Any more insights about the large StD. in Table 1?\n- Misc: Fig. 1 is not referenced in the main body.\n\n> [a] Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving. ECCV 2022.\n>\n> [b] Towards Unsupervised Object Detection from LiDAR Point Clouds. CVPR 2023.\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nThe authors have discussed limitations at the end of the paper. The discussion is valuable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new reinforcement-learning-based framework for unsupervised 3D object detection that uses these common-sense heuristics directly as a reward signal. Avoid handcrafting training examples for each object detector. Furthermore, under the premise of greatly accelerating the convergence speed of the model, this method improves the experimental effect.\n\nSTRENGTHS:\n1. The proposed reinforcement learning-based framework without encoding heuristics into differentiable loss functions and avoids the need to hand-engineer training paradigm.\n2. The method proposed by the author has a fast convergence speed and a great algorithm implementation value.\n3. The experiments in this article are relatively sufficient, and the charts are easy for readers to understand.\n\nWEAKNESSES:\n1. The content arrangement of this article does not seem to be very convenient to read and there are some careless quoting errors that refer to Questions.\n2. The author should provide further explanation or derivation in the formula part of the article.\n\nQUESTIONS:\n1. In line 40, can it be understood that the method proposed in this article is to perform finetune on the pre-trained MODEST through a reward-based method? If the answer is yes, is the time comparison in Figure 1 meaningful or fair? Moreover, LLM is mentioned more than once in the article, but this article does not seem to use a specific LLM model. I think this may cause some misunderstandings for readers at the beginning.\n2. The method proposed in this paper is more like a specific RAFT method in unsupervised object detection, and the novelty may be limited.\n3. The training steps mentioned in Line 56. As far as I know, many self-training methods can achieve the same goal, please tell us your advantages.\n4. Lines 126 to 132 in the article seem to give the boxes from the detector a priori information or heuristic constraints from the real world. You are modeling here by a mixture of Gaussian distributions, how to determine mixture weights? In line 115 section. What happens if you use a mixture factor greater than 4? It seems that the model only focuses on objects of these sizes. In line 131, what happens if you don't use the scale operation on it? please give an example.\n5. Is Figure 3 drawn from a real dataset? If so, please describe which data were used in the supplementary material. Can this be understood as prior information derived from daily data?\n6. The paragraph starting on line 140 means that by scaling, more points are included in the box, this seems to serve the same purpose as jittering mentioned in Sec 3.2 or Intro. What is the difference between them?\n7. The sensitivity analysis of mu_{scale} in line 156 should be Tab5.\n8. The ablation experiment of Tab4 should complement the other 3 groups and did not analyze the reason for this phenomenon, especially when the effect gap is huge. Because I want to know how the three components of the proposed Reward function affect the detection performance respectively\n9. Please confirm the experimental code for reproducing the MODEST in Tables 1 and 2. Whether there is a P2 filtering result seems to be far from the original MODEST paper.\n\nLIMITATIONS:\nOne limitation is that this paper uses MODEST as the baseline. If the number of timestamps is not large, continuing to use the p2 score may aggravate the impact of noise on the model. This is a direction that needs to be addressed. Another limitation is that since it is unsupervised 3D object detection, not only static mobile objects should be detected, but all static objects should be detected, which may be beneficial to downstream tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DRIFT, a novel reward fine-tuning method for unsupervised object discovery with point cloud input. Specifically, three reward methods are proposed to identify good bounding boxes. First, shape prior reward prefers bounding boxes with similar sizes to the prototypes.  Second, an alignment reward gives high scores to boxes that have most of the LiDAR points near the box edge. Third, a filter reward follows the spirit that a good box should contain more dynamic points than background points. By using these reward methods for refining the object detector, DRIFT achieves more accurate object discovery, and it also converges much faster than the previous method.\n\nSTRENGTHS:\n(1) The proposed reward fine-tuning method is well-motivated and improves object discovery accuracy compared to the previous work.\n\n(2) In addition to improving the detection performance, the proposed method can also greatly improve the training speed.\n\n(3) The expression of the paper is clear, and the figures are intuitive.\n\nWEAKNESSES:\n(1) Although intuitive, I think the prior that most points fall on the lateral surfaces of an object is too absolute, e.g., a lot of LiDAR points will fall on the hood and front window of the oncoming vehicles beside lateral surfaces. Apart from this, using Gaussian distribution as an approximation is also not accurate, because few points fall out of the box edge, as shown in Figure 3.\n\n(2) I think a missing ablation is the contribution of the exploration strategy to the final detection performance, similar to the ablation on the reward methods in Table 4.\n\nQUESTIONS:\n(1) Authors should provide complete experiment results and refine their experimental analysis correspondingly, as some of the experiments are still running at the time of submission (L234 & L258).\n\n(2) In L224, the authors say they include evaluation results with IoU at 0.25 in the supplementary, but I don't see these results. Authors should add these results and explain why they don't provide these results in the main paper.\n\nLIMITATIONS:\nThe authors provide limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work contributes to unsupervised object discovery in LIDAR point clouds. It defines a few typical properties of customary bounding boxes in LIDAR point clouds and then develops Rewards for a Reinforcement Learning algorithm to learn this, lacking a gradient for direct optimization.\n\nIt is based on Persistency Prior scores defined by the MODEST approach but adds several steps to improve bounding box selection. Furthermore they ablate many factors, e.g. the impact of enforcing a certain shape on bounding boxes. While they can only compare on a limited amount of epochs due to time constraints, possibly due to the plenty ablation studies, they show they outperform the existing approach in that domain.\n\nSTRENGTHS:\nThe paper improves performance by a large margin in a very novel research area. Even if the approach itself may have limited use, it is straightforward to see the value e.g. in combination with a pre-trained traffic participant classifier.\n\nThe easy and intuitive explanation of the reward shaping is very clear. Figure 2 helps to get a very quick insight into the idea. The whole paper does a good job at combining intuitive explanations with mathematical formulations.\n\nThe approach has many individual steps and the pseudo-algorithm helps understanding.\n\nThe used Lyft Level 5 Perception dataset is a good choice for comparing with the state of the art.\n\nThe ablation studies are suitable to show that all used steps together achieve the performance.\n\nAll figure captions guide the reader nicely towards the main message that figure should convey.\n\nWEAKNESSES:\nIt seems this work takes many ideas from MODEST so the idea of using an object detector and defining common sense properties is not novel to this work. It needs to be read with that contribution offset in mind. \n\nThe title is maybe a bit boasting. Teaching cars to see sounds like solving most if not all perceptual problems while this approach, coming from unsupervised object detection, only separates moving objects from background.\n\nHaving still a bit to train is not optimal but understandable. The paper feels a bit rough around the edges, the DRIFT abbreviation is explained twice.\n\nThe approach is a bit complex with many parameters to tune. Even though the ablation studies show the value of all individual components, it is hard to judge how much hyperparameter optimization vs. unsupervised modeling performance is present.\n\nQUESTIONS:\n2 Related Works: 3D Object Detection: I would disagree it is ideal to train detectors in an unsupervised way. Maybe ideal could be a combination to achieve best performance at reasonable costs while being able to assign actual labels (car, pedestrian, etc.). Maybe a statement like that could be just removed.\n\nHow robust is the algorithm to the amount of traversals? This is not touched at all unless I overlooked it.\n\nJittered boxes and then NMS has strong similarity to region proposals in approaches like Mask R-CNN. Maybe it could be good comparing this historic approach in the Related Work.\n\nWhat is the impact of the quality of the first pre-trained detector? If there is a false negative detection, how can this algorithm improve a non-existing bounding box?\n\nIn Reward Finetuning for Model Alignment, what are \"human values\"? Do the models output unexpected or wrong output or are there ethical considerations? The term seems confusing.\n\nHow have the lambda factors and mu_scale and sigma_scale been found? Similarly how was the scale of the scaled up box selected when designing the Alignment Reward? Since the approach is supposed to find objects unsupervised, hyperparameter tuning and then manually checking the results by a human who looks at the bounding box detections in the point cloud may improve the results but not in a fair way. It goes against the idea of unsupervised object detection. \n\nIn 3.2 what is meant by \"In effect, this encourages all non-persistent points ... to propose boxes\". How do points propose boxes? This is not like Yolo where each grid cell is associated with a fixed number of region proposals right?\n\nCouldn't this approach potentially be expanded to actually label Car, Pedestrian, Cyclist and Truck based on the shape priors?\n\nIt would be worthwhile investigating the impact of false negatives to understand the impact of the approach vs. the failure of the first step of bounding box proposal. The authors did compare different ways of box predictions but that is different from quantifying the performance. Knowing how many false negatives are produced by the first step would already give some insight into this. Or is that the No Finetuning case?\n\nLIMITATIONS:\nThere is only one comparable work which is an unavoidable limit. The authors solved this by also comparing against other tasks, i.e. the supervised case.\n\nDue to the deadline DRIFT was not trained to convergence, however it seems to outperform it's competitor already. For the camera ready final numbers should be entered.\n\nIn their limitations the authors address most of the concerns in Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}