SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Katie Z Luo\*1,† Zhenzhen Liu\*1 Xiangyu Chen\*1 Yurong You\*1 Sagie Benaim2 Cheng Perng Phoo
Mark Campbell<sup>1</sup> Wen Sun<sup>1</sup> Bharath Hariharan<sup>1</sup> Kilian Q. Weinberger<sup>1</sup>
<sup>1</sup>Cornell University, Ithaca, NY <sup>2</sup>The Hebrew University of Jerusalem
Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector's own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT.
Self-driving cars need to accurately detect the moving objects around them in order to move safely. Most modern 3D object detectors rely on supervised training from 3D bounding box labels. However, these 3D bounding box labels are hard to acquire from human annotation. Furthermore, this supervised approach relies on a pre-decided vocabulary of classes, which can cause problems when the car encounters novel objects that were never annotated.
Our prior work, MODEST [55], introduced the first method to train 3D detectors without labeled data. In that work, we point out that instead of specifying millions of labels, one can succinctly describe heuristics for what a good detector output should look like. For example, one can specify that detector boxes should mostly enclose transient foreground points rather than background ones; they should roughly be of an appropriate size; their sides should be aligned with the LiDAR points; their bottom should touch the ground, etc. Although such heuristics are great for scoring a set of boxes proposed by a detector, training a detector on them is hard for two reasons: First, these heuristics are often non-linear, non-differentiable functions of the detector parameters (for example, a slight shift of the box can cause all foreground points to fall off.) Second, existing object detection pipelines use carefully designed training objectives that heavily rely on labeled boxes, that are difficult to modify (for example, PointRCNN [40] infers point labels from box labels and uses these for training). For these reasons, MODEST had to utilize an admittedly slow self-training pipeline to incrementally incorporate common-sense heuristics.
*Denotes equal contribution.
†Correspondences could be directed to kz16@cornell. edu
(a) Performance over Epochs @ IoU 0.5
(b) Performance over Epochs @ IoU 0.7
Figure 1: Detection performance on Lyft test data as a function of training epochs. DRIFT demonstrates significantly stronger performance and faster learning. With only 9 hours of training, it outperforms both baselines that have been trained for days.
In this paper, we propose a new reward ranking based framework that utilizes these common-sense heuristics directly as a reward signal. Our method relies on finetuning with reward ranking [31, 27, 13, 33], where given an initialized object detector, we finetune it for a predefined set of desirable detection properties. This bypasses the need to encode heuristics as differentiable loss functions and avoids the need to hand-engine training paradigms for each kind of object detector. Recent success with reinforcement learning from human feedback (RLHF) has proven effective in improving machine learning models—in particular, large language models (LLMs)—and aligning them with human preferences [31, 27]. However, these advancements have not been applicable to detection-based vision models that are trained with per-instance regression and are difficult to view under a probabilistic framework. To address this challenge, we utilize insights from reward ranked finetuning [13], a non-probabilistic paradigm designed for finetuning of LLMs, which inspired us to develop a similar framework for object discovery.
We refer to our method as Discovery from Reward-Incentivized Finetuning (DRIFT). DRIFT does not require labels, and instead uses the Persistence Prior (PP) score [55, 3] as a heuristic to identify dynamic foreground points based on historical traversals. These foreground points give rise to rough (and noisy) label estimates [55], which we use to pre-train our detector. The resulting detector performs poorly but suffices to propose many boxes of roughly the right sizes that we can use for exploration. To facilitate reward ranked finetuning, we first propose a reward function to score boxes. Ideally, only boxes that tightly contain objects (e.g. a car) should yield high rewards. We achieve this by combining several simple heuristics (e.g. high ratio of foreground points) and assuming some rough knowledge about the object dimensions. During each iteration of training, DRIFT performs the following steps: 1. the object detector proposes many boxes in a given point cloud scene; 2. the boxes are "jittered" through random perturbations (as a means of exploration); 3. the boxes are scored according to the reward function; 4. the top- $k\%$  non-overlapping boxes are kept as pseudo-labels for gradient updates.
We evaluate DRIFT on two large, real-world datasets [12, 24] and show that we significantly outperform prior self-training methods both in efficiency and generalizability. Experimental results demonstrate that using reward ranked finetuning for object discovery under our framework can quickly converge to a solution that is on par with out-of-domain supervised object detectors within a few training epochs, suggesting that DRIFT may point towards a more general unsupervised learning formulation for object detectors in an in-the-wild setting.
3D Object Detection. 3D object detection models usually take in LiDAR point clouds or multi-view images as input and aim to produce tight bounding boxes that describe nearby objects [9, 52, 26, 39, 40, 53, 54, 34]. Existing methods generally assume the supervised setting, in which the detector is trained with human-annotated bounding boxes. However, annotated data are often expensive to obtain
and limited in quantity. Furthermore, in tasks such as self-driving, environments can have highly varied conditions, and detectors with supervised training often require adaptation with additional labels from the new environment [47].
Unsupervised Object Discovery. The unsupervised object discovery task aims to identify and localize salient objects without learning from labels. Most existing works perform discovery from 2D images [8, 14, 45, 36, 2, 43, 48] or depth camera frames [21, 23, 17, 25, 20, 1]. Discovery from 3D LiDAR point clouds is underexplored. [49] performs joint unsupervised 2D detection and 3D instance segmentation from sequential point clouds and images based on temporal, motion and correspondence cues. MODEST [55] pioneers in performing label-free 3D object detection. It exploits high-level common sense properties from unlabeled data and bootstraps a dynamic object detector via repeated self-training. Despite promising performance, it requires excessive training time, which makes it difficult for practical use and development.
Reward Fine-Tuning for Model Alignment. Recently, foundation models [6, 29, 11, 44, 35, 37] have been shown to achieve strong performance in diverse tasks [5, 50], but sometimes produce outputs that do not align with human values [18, 30, 10]. A line of research aims to improve model alignment under the paradigm of Reinforcement Learning with Human Feedback (RLHF). Some pioneering works [41, 31, 33] learn a reward model and train foundation models with Proximal Policy Optimization (PPO) [38], but PPO is often expensive and unstable to train, and more importantly, requires a probabilistic output on the action space. This makes it hard to use for the object detection setting, which primarily uses regression-based losses. Reward ranked finetuning [27, 13] is a simplified alternative paradigm. It samples from a foundation model itself, filters generations using the reward model, and conducts supervised finetuning with the filtered generations.
3 Discovery from Reward-Incentivized Finetuning
Our framework, DRIFT, is inspired by the recent success of reward ranked finetuning methods for improving model alignment in the NLP community [13, 27]. We show that a similar approach can be adapted for 3D object discovery.
Problem Setup. We wish to obtain a dynamic object detection model on LiDAR data, i.e., a model to detect mobile objects in the LiDAR point clouds, without human annotations. Let  $P \in \mathcal{R}^{N \times 3}$  denote a  $N$ -point 3D point cloud captured by LiDAR from which we wish to discover objects. We assume inputs of unlabeled point clouds collected by a car equipped with synchronized sensors including LiDAR (for point clouds) and GPS/INS (for accurate position and orientation). Since no annotation is involved, such a dataset is easy to acquire from daily driving routines; we additionally assume it to cover some locations with multiple scans at different times for computation of PP-score.
Dynamic Point Proposals. DRIFT leverages prior works that use unsupervised point clouds to extract foreground-background segmentation proposals. While many works [22, 3] have promising dynamic foreground segmentation results, in this work we rely on point Persistence Prior score (PP-score) [55] for its accuracy and leave the extension of other proposal methods to future work. For the purpose of this research, dynamic foreground points constitute LiDAR points reflecting off traffic participants (e.g. cars, bicyclists, pedestrians).
Using historical LiDAR sweeps collected at nearby locations of our point cloud  $P$ , the PP-score [3, 55]  $\tau(P) \in [0,1]^N$  can provide an informative estimate on the per-point persistence, i.e., whether a point belongs to persistent background or not. The PP-score is defined as the normalized entropy over past point densities, based on the assumption that background space such as ground, trees, and buildings tend to exhibit consistent point densities across different LiDAR scans (high entropy), whereas points associated with mobile objects exhibit high density only if an object is present (low entropy).
3.1 Rewarding "Good" Dynamic Boxes
We first establish a reward function that evaluates the quality of a set of bounding boxes for dynamic objects in a scene. We denote a set of  $M$  dynamic objects bounding boxes as  $\mathcal{B} = \{b_1, \ldots, b_M\}$ , where each bounding box  $b_i$  is represented as an upright box with parameters  $(x_i, y_i, z_i, w_i, l_i, h_i, \theta_i)$ , defining the box's center, width, length, height, and z-rotation, respectively. The scoring function  $R$  scores the validity of the bounding boxes, given the observed point cloud  $P$ . In practice, a reward
Shape Prior
Alignment
Figure 2: Illustration of the reward components. The reward encourages boxes that have proper shape and alignment, and capture more dynamic points and few background points.
Dynamic Point Counts
function that is positively correlated with IoU should suffice. We present our proposed reward function which aims to capture only dynamic points, filter nonsensical boxes, enforce correct size, and encourage proper box alignment to the captured dynamic points.
Shape Prior Reward. We enforce a box to not deviate significantly from a set of prototype sizes  $S = \{(\overline{w}_1, \bar{l}_1, \bar{h}_1), \dots, (\overline{w}_C, \bar{l}_C, \bar{h}_C)\}$  (Fig. 2 left). We assume the shape prior distribution is a mixture of  $C$  isotropic Gaussians with mixture weights  $\pi_i$ , diagonal variances  $\Sigma_i$ , and corresponding means as  $(\overline{w}_i, \bar{l}_i, \bar{h}_i)$ . These low-level statistics may be acquired directly from the dataset, or from vehicle specs and sales data available online [47]. In practice, we scale the mixtures such that the probabilities at the Gaussian means are equal for stability reasons. With this, the shape prior reward for box  $b$  is computed as:
$$
R _ {\text {s h a p e}} (\boldsymbol {b}) = P _ {S} (\boldsymbol {b}). \tag {1}
$$
Alignment Reward. Due to the nature of LiDAR sensing, the points will mostly fall on the lateral surfaces of an object. Therefore, a well-formed box should have dynamic points approximately close to the boundary of a box (Fig. 2 middle). As [55] shows, PP-score allows for easy separation of dynamic and persistent background points. Let  $P_{\mathrm{dyn}}$  denote the set of dynamic points, and let  $P_{\mathrm{bg}}$  be that of background points. In practice, since the PP-score is an approximation of ground-truth persistence, we define  $P_{\mathrm{dyn}} = \{p|\tau(p) < 0.6\}$  and  $P_{\mathrm{bg}} = \{p|\tau(p) \geq 0.9\}$ .
Given a box  $\pmb{b}$ , we denote all points within and close to the box as  $\mathcal{O}(b)$ . Only points within  $\mathcal{O}(b)$  contribute to the reward of  $\pmb{b}$ . In practice, we let  $\mathcal{O}(b)$  consist of all points within a  $\times 2$  scaled up version of  $\pmb{b}$  (with identical center and rotation). To score a box  $\pmb{b}$ , we design a reward function that identifies how "typical" the dynamic points within  $\mathcal{O}(b)$  are. For each dynamic point  $p \in \mathcal{O}(b) \cap P_{\mathrm{dyn}}$ , we compute the scaling factor  $s_{p,b}$  required so that the rescaled box touches  $p$  with one of its sides; i.e., if a point is inside the box, the box would have to be scaled down ( $s_{p,b} < 1$ ) to touch the point, if the point is outside it must be scaled up ( $s_{p,b} > 1$ ). We assume that  $s_{p,b}$  roughly follows a Gaussian distribution centered near the box boundary, and visualize the actual distribution in Fig. 3.
Figure 3: Distribution of dynamic points near ground truth bounding boxes. We observe that dynamic points near bounding boxes fall in an approximate Gaussian distribution centered near box edges ( $s_{p,b} \approx 0.8$ ).
We define our reward as the likelihood under this Gaussian
distribution over scaling parameters. We approximate it as a Gaussian with hyper-parameters mean  $\mu_{\mathrm{scale}}$  and a variance  $\sigma_{\mathrm{scale}}$ . Our reward is the product of the probability of each point in  $\mathcal{O}(\pmb {b})$ :
$$
R _ {\text {a l i g n}} (\boldsymbol {b}) = \prod_ {\boldsymbol {p} \in o (\boldsymbol {b}) \cap \boldsymbol {P} _ {\mathrm {d y n}}} \mathcal {N} \left(s _ {\boldsymbol {p}, \boldsymbol {b}} \mid \mu_ {\text {s c a l e}}, \sigma_ {\text {s c a l e}}\right). \tag {2}
$$
Common Sense Heuristics and Filtering. Lastly, a proper bounding box must capture the dynamic points, and avoid capturing the background points (Fig. 2, right). This heuristic can be encoded by a
simple weighted point count for each bounding box  $\pmb{b}$ :
$$
R _ {\text {c o u n t}} (\boldsymbol {b}) = \lambda_ {\text {d y n}} \cdot | P _ {\text {d y n}} \cap \mathcal {O} (\boldsymbol {b}) | - \lambda_ {\text {b g}} \cdot | P _ {\text {b g}} \cap \mathcal {O} (\boldsymbol {b}) |.
$$
Intuitively, it assigns a reward in proportion to the number of dynamic points captured by the box, and a penalty in proportion to the number of background points captured.
Furthermore, boxes violating common sense should be assigned a low reward. We filter boxes that are too high up or too low from the ground, including those with too few dynamic points, or are too small or too large by directly assigning a reward of 0. In practice, we filter boxes that contain fewer than 4 dynamic points, or that have more than  $80\%$  persistent points, similar to [55].
In summary, the reward function is designed to be
$$
R (\boldsymbol {b}) = \left\{ \begin{array}{l l} \lambda_ {\text {s h a p e}} \cdot R _ {\text {s h a p e}} (\boldsymbol {b}) + \lambda_ {\text {a l i g n}} \cdot R _ {\text {a l i g n}} (\boldsymbol {b}) + R _ {\text {c o u n t}} (\boldsymbol {b}) & \text {i f o b e y s c o m m o n s e n s e}, \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {3}
$$
3.2 Exploration Strategy for Improved Discovery
We assume a simple exploration strategy for identifying good box proposals. Given a set of current object proposals given by the detector, we locally perturb the boxes in the output space:
$$
\mathcal {B} _ {\text {e x p l o r e}} \sim \mathcal {P} \left(\mathcal {B} _ {0}, \sigma\right), \tag {4}
$$
where  $\mathcal{B}_{\mathrm{explore}}$  is the set of explored boxes, perturbed from the model proposals  $\mathcal{B}_0$ . For each box  $b \in \mathcal{B}_0$ , a set of explored boxes are sampled according to a standard Gaussian noise along the position and size dimensions and uniform noise for orientation:
$$
\boldsymbol {b} _ {\text {e x p l o r e}} ^ {\text {c e n t e r}} \sim \mathcal {N} \left(\boldsymbol {b} _ {0} ^ {\text {c e n t e r}}, \sigma^ {2} \boldsymbol {I}\right), \boldsymbol {b} _ {\text {e x p l o r e}} ^ {\text {s i z e}} \sim \mathcal {N} \left(\boldsymbol {b} _ {0} ^ {\text {s i z e}}, \sigma^ {2} \boldsymbol {I}\right), \boldsymbol {b} _ {\text {e x p l o r e}} ^ {\theta} \sim \mathcal {U} \left(\boldsymbol {b} _ {0} ^ {\theta} - \sigma , \boldsymbol {b} _ {0} ^ {\theta} + \sigma\right). \tag {5}
$$
Furthermore, to encourage proposals of boxes in foreground regions, we take inspiration from [56, 40] and re-use PP-score as point-level semantic segmentation (foreground vs background) labels, with which the detector is encouraged to propose boxes at points that have low PP-scores (i.e. are likely to be foreground points). Following [56], for each point  $\pmb{p_i} \in \pmb{P}$  with prediction  $\hat{\pmb{y}}_i$ , we assign its target classification label  $\pmb{y}_i$  as:
$$
\boldsymbol {y} _ {i} = \left\{ \begin{array}{l l} 1 & \text {i f} \tau (\boldsymbol {p} _ {i}) <   \tau_ {L} \text {o r} \hat {\boldsymbol {y}} _ {i} = \mathbf {1}, \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {6}
$$
Algorithm 1 Reward-Incentivized Finetuning
Input: Base object detector  $f_{\theta}$ , unlabeled LiDAR dataset  $\mathcal{D}$ , exploration noise  $\sigma$ , sample size  $n$ , filter budget  $k$ , reward function  $R$ , PP-scores  $\tau$ .
$P\sim \mathcal{D}$   Sample point cloud.
$\mathcal{B}_0 \gets f_{\theta}(\pmb{P})$  Run detector for proposals.
$\mathcal{B}_{\mathrm{explore}} \sim \mathcal{P}(\mathcal{B}_0, \sigma)$  ▷ Sample  $n$  boxes.
$\pmb{r}\gets \{R(\pmb {b})\}_{\pmb {b}\in \mathcal{B}_{\mathrm{explore}}\cup \mathcal{B}_0}$  ▷ Score boxes in set.
$\mathcal{B}\gets \mathrm{NMS}\left(\mathcal{B}_{\mathrm{explore}}\cup \mathcal{B}_0,\boldsymbol {r}\right)$
$\mathcal{B}_{\mathrm{top}} \gets \text{Filter}(\mathcal{B}, r, k) \quad \triangleright$  Keep top  $k\%$  boxes
Update  $\theta$  with  $\mathcal{B}_{\mathrm{top}}$  for 1 step
until converged
In effect, this encourages all non-persistent
points (i.e., low  $\tau(\pmb{p}_i)$ ) to propose boxes near dynamic regions for better exploration.
3.3 Reward-Incentivized Finetuning
The reward function  $R$  allows us to quickly evaluate proposed bounding boxes  $\mathcal{B}$  and the task of 3D object discovery could be reduced to an optimization problem on the total reward in box set space:
$$
\mathcal {B} ^ {*} = \underset {\mathcal {B}} {\arg \max } \sum_ {\boldsymbol {b} \in \mathcal {B}} R (\boldsymbol {b}), \tag {7}
$$
where the sum is taken over the boxes in the set  $\mathcal{B}$ . Although a direct optimization for  $\mathcal{B}^*$  is not plausible due to the non-polynomial search space and discontinuity in  $R$ ,  $R$  can serve as effective guidance to facilitate model finetuning. The underlying intuition is similar to curriculum learning [4, 28, 46]: the object detection model takes small steps to improve from its current predictions towards  $\mathcal{B}^*$  by following the direction provided by the maximum  $R$  direction in a local space.
As illustrated in Alg. 1, in each training iteration, we first let the object detector perform inference on a point cloud  $P$  and propose a set of dynamic objects  $\mathcal{B}_0$  in the scene. To explore directions
of improvement with the non-differentiable reward function  $R$ , we sample  $n$  boxes from  $\mathcal{B}_0$  (with replacement) and add an i.i.d. Gaussian noise on their location and size, and an uniform noise on orientation following Eq. 4. These sampled boxes are then ranked by the reward function  $R$ , in which the top  $k$  non-overlapping boxes are selected by Non-Maximum Suppression (NMS) as training targets to finetune the object detector. Note that since DRIFT treats the model training/inference procedures as black boxes, it can be applied to any 3D object detection model.
In practice, it is observed that neural networks can acquire task knowledge from imperfect demonstrations [16, 51, 32]. MODEST [55] pre-trained the 3D object detector on noisy seed labels produced by DBSCAN [15] clustering on spatial and PP-score. We follow [55] and initialize our 3D object detector a model trained with discovered seed labels.
Datasets. We experimented with two different datasets: Lyft Level 5 Perception dataset [24] and Ithaca-365 dataset [12]. To the best of our knowledge, these are the two publicly available datasets that contain multiple traversals of multiple locations with accurate 6-DoF localization and 3D bounding box labels for traffic participants.
In the Lyft dataset, we experiment with the same split provided by [55], where the train set and test set are geographically separated. It consists of 11,873 train scenes and 4,901 test scenes. For the Ithaca365 dataset, we experimented with the full dataset which consists of 57,107 scenes for training and 1,644 for testing. For both datasets, we do not use any human-annotated labels in training. To show the generalizability of our method, we conduct the development on the Lyft dataset, i.e., all the hyperparameters of our approach are finalized through experiments on Lyft, and we use the exact same set of hyperparameters for all experiments in Ithaca365.
Evaluation. Following [55], we combine all traffic participants to a single mobile object class and evaluate the detector's performance on this class. Note that the labels are not used during training but solely for evaluation. For Lyft, we report the mean average precision (mAP) of the detector with the intersection over union (IoU) thresholds at 0.5 and 0.7 in bird-eye-view perspective. Note that mAP at 0.7 IoU threshold is a stricter and harder metric and was not evaluated in [55], and we include it to emphasize the effectiveness of our method. For Ithaca365, we adopt metrics similar to those in [7]: we evaluate mean average precision (mAP) for dynamic objects under  $\{0.5,1,2,4\}\mathrm{m}$  thresholds that determine the match between detection and ground truth; we also compute 3 types of true positive metrics (TP metrics), including ATE, ASE and AOE for measuring translation, scale and orientation errors. These TP metrics are computed under a match distance threshold of  $2\mathrm{m}$ ; additionally, we also compute a distance-based breakdown ( $0 - 30\mathrm{m}$ ,  $30 - 50\mathrm{m}$ ,  $50 - 80\mathrm{m}$ ) for these metrics.