Just Label What You Need: Fine-Grained Active Selection for P&P through Partially Labeled Scenes
Authors: Sean Segal
Nishanth Kumar
Sergio Casas
Wenyuan Zeng
Mengye Ren
Jingkang Wang
Raquel Urtasun
Venue: CoRL2021 Poster (2021)
Keywords: Self-Driving
Active Learning

Abstract: Self-driving vehicles must perceive and predict the future positions of nearby actors to avoid collisions and drive safely. A deep learning module is often responsible for this task, requiring large-scale, high-quality training datasets. Due to high labeling costs, active learning approaches are an appealing solution to maximizing model performance for a given labeling budget. However, despite its appeal, there has been little scientific analysis of active learning approaches for the perception and prediction (P&P) problem. In this work, we study active learning techniques for P&P and find that the traditional active learning formulation is ill-suited. We thus introduce generalizations that ensure that our approach is both cost-aware and allows for fine-grained selection of examples through partially labeled scenes. Extensive experiments on a real-world dataset suggest significant improvements across perception, prediction, and downstream planning tasks. 

Just Label What You Need: Fine-Grained Active Selection for P&P through Partially Labeled Scenes
Sean Segal Nishanth Kumar Sergio Casas Wenyuan Zeng!?
Mengye Ren 12 Jingkang Wang Raquel Urtasun
Waabi', University of Toronto", MIT CSAIL
(ssegal, scasas, wzeng, mren, jwang, urtasun)8waabi ai, ajkecsail.mite edu
Self-driving vehicles must perceive and predict the future positions of nearby actors to avoid collisions and drive safely. A deep learning module is of- ten responsible for this task, requiring large-scale, high-quality training datasets. Due to high labeling costs, active learning approaches are an appealing solution to maximizing model performance for a given labeling budget. However, despite its appeal, there has been little scientific analysis of active learning approaches for the perception and prediction (P&P) problem. In this work, study active learn- ing techniques for P&P and find that the traditional active learning formulation is ill-suited. We thus introduce generalizations that ensure that our approach is both cost-aware and allows for fine- ine-grained selection of examples through partially la- beled scenes. Extensive experiments on real-world dataset suggest significant improvements across perception, prediction, and downstream planning tasks.
Abstract:
1 Introduction
For self-driving vehicles to safely plan a route, they must perceive nearby actors and predict their future locations. In self-driving stack, learned perception and prediction (P&P) model re- sponsible for this task, taking sensor data input and producing object detections and future predictions. These models require large-scale, high-quality training datasets due to the high dimen- sional sensor inputs and long tail of future outcomes. While self-driving companies collect massive amounts of data from real-world driving, annotating the data remains major bottleneck. Further- more, some of the data may be less interesting for model training - e.g., a prediction dataset with many parked vehicles is less informative than one with highly interactive, moving actors. Therefore, the choice of which examples label is crucial maximize performance for given budget.
Given particular model, we seek to determine the examples most likely to improve performance when labeled. This problem is well-studied in the field of active learning and recent work has shown impressive performance gains over random selection in many tasks, including image classification, semantic segmentation and 2D object detection [1, 2, 3]. Active learning presents a promising framework to employ in real-world self-driving development, where models can continually improve as new batches of examples are selected iteratively for labeling (see Figure 1).
Despite the appeal of active learning, few approaches have been developed for self-driving. Sci- entific analysis is limited to object detection [4, 5], with no approaches designed for P&P. When applying active selection to this task, we find the traditional formulation to be ill-suited. First, while approaches typically assume fixed labeling costs per example, annotation costs can vary drastically as they depend on the number of actors present. Furthermore, the spatial label structure can be ex- ploited support partial labeling, enabling fine-grained active selection. Specifically, with small modifications, P&P models can be trained from partial supervision. This allows the active learner to select specific actors in scene without requiring the remaining actors to be selected, as they may be
'Equal Contribution
I This work was done by all authors while Uber ATG
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

Algorithm Active Learning Selection
Params: K, N
Inputs: Xu. x-( .M (model)
Outputs: x .M
(N)
Train from Partial Supervision
1: for i 1. ......... do
N
2: S(0) -- score (M, Xu | ((())))
3: Q < select top k(St)), K)
4: x(++1) t x UQU
Label Best Regions
Score Unlabeled
5: M tr-tainnM,, M
Examples
6: end for
Figure 1: (Left) Overview of fine-grat fine-grained active selection. (Right) Selection Algorithm.
uninteresting for model improvement. With these differences in mind, we introduce fine fineggranee,, cost-aware selection along with specific scoring criteria for active selection in the P&P setting.
We leverage real-world, large-scale dataset to analyze the effects of partial labeling and active selection for P&P. First, we study models trained on partial supervision, without active selection, and observe improved performance from sparser labeling. Next, we analyze the added benefits of fine-grained active selection and observe significant gains compared traditional approaches. events, translating improvements in
Further analysis shows the gains most signifiant downstream planning performance. All together, our analysis suggests that the dominant paradigm of labeling entire self-driving scenes is not the most efficient use of fixed labeling budget and that more fine-grained active selection may be required to most effectively select examples for labeling.
Related Work
While perception and prediction have traditionally been handled sep- arately, [6, 7] introduce models jointly perform both tasks, improving performance and efficiency. Among the exciting progress made in both tasks over recent years, most relevant to our work are improvements in prediction representations, allowing models to better characterize uncertainty. Ex- amples of representations include trajectories [7, 8), probabilistic occupancy maps [9, 10), Gaussian mixtures [11], implicit latent variable models [12], and auto-regressive models [13].
Perception and Prediction:
Active Learning: We focus on pool- active learning, which new training examples are queried from large, unlabeled pool [14]. One class of approaches seeks to characterize model uncertainty, measured via model disagreement [15, 16], entropy [17, 18), learned loss prediction [1], discriminator score [19], and select examples with high uncertainty for labeling. While often effective, uncertainty based approaches are prone to selecting subset of similar examples when computational constraints require large batches of examples to be selected before retraining. This motivates diversity- bassed approaches [20, 21, 22, 23] which seek to find a representative subset of the unlabeled pool. [3, 24, 25] introduce approaches which balance both uncertainty and diversity in the selection process. Most related to our domain are [4, 5, 26, 27, 28] which study active learning approaches for object detection. While most approaches assume fixed labeling costs per example, [29, 30) have explored explicitly modeling individual labeling costs part of the selection process. Leveraging partially labeled data for fine-grained active selection has been explored in semantic segmentation [31, 32] and more generally, in the context of structured prediction problems [33].
Dataset Selection: Many self-driving datasets select examples manually [34], randomly, or via hardcoded rules. Argoverse describes rules-based criteria to mine interesting trajectories for predic- tion [35]. More recently, [36] proposed a set of complexity measures for dataset selection. Finally, [37] proposed tagging attributes of self-driving scenes, enabling retrieval for dataset curation.
Active Learning for P&P
Given the high labeling costs of P&P datasets, budgets be spent more efficiently by intelligently selecting examples for labeling. Active learning offers a promising solution, selecting examples

---

believed most likely to improve model performance. In this section, we review traditional pool- based active learning, where examples are iteratively selected from an unlabeled pool to build a high-quality labeled dataset. Then, we address shortcomings in the P&P setting by introducing a new paradigm which is both cost-aware and enables aine-graieed selection through partially labeled scenes, providing the flexibility to ensure budgets are spent effectivelyy. Finally, we provide concrete selection criteria used within framework optimize the model's s performance.
3.1 Traditional Active Learning
Self-driving companies can collect large-scale unlabeled real-world data when operating their vehi- cles. Our goal is select the best subset label improve model performance. We assume access to a large, unlabeled pool of examples, Xu. and an initial subset of labeled examples, x Each example x E Xy represents an input to our model f(x) and if selected, a labeling oracle returns the ground truth supervision, y = L(x). In the P&P setting, inputs x represent raw sensor observations and HD Maps, and labels y represent actor bounding boxes the current timestep and for the pre- diction horizon of T seconds. In each active learning iteration, select subset from the remaining unlabeled examples, Q(3) C Xu \ X ((i-1, query the labeling oracle, and add the examples to our labeled set. With each iteration, the model is retrained or fine-tuned with the latest dataset.
Traditionally, the active learner will select fixed number of examples at each iteration, [Q K. This implicitly assumes that each example x E Xy can be labeled for the same cost, an assumption clearly violated in the P&P setting, which we will relax in the next section. While a variety of approaches have been studied for active selection, we focus on methods which produce a scalar score for each example, Sxx E R. Scores represent some notion of informativeness where highly scored examples are believed to be most likely improve model performance. For example, measures of model uncertainty, such entropy, commonly used (see Section 3.3 for concrete scoring functions for P&P). As different models may benefit from different types of examples, most scoring approaches depend on the model's 's current state. After scores have been computed for the remaining unlabeled examples, the top K examples can be selected for labeling. This process repeats for N active learning iterations and summarized in Algorithm 1.
3.2 Fine-Grained Cost-Aware Active Learning
In this section, we generalize two aspects critical to the P&P setting, allowing for variable labeling costs and fine-grained selection through partial supervision.
Cost-Aware Active Learning: As a self-driving vehicle operates, the surrounding environment will change, leading to scenes with different labeling costs. Crowded scenes can contain hundreds of actors, which are each traditionally labeled with a precise bounding box. Sparser scenes, on the other hand, can be labeled with little manual effort. To account for these differences, we explicitly model the cost to label each example, C(x)) At each iteration, rather than select fixed K examples, the learner is instead given fixed budget B, which cannot be exceeded, Exeq C(x) B.
This formulation is generalization of the previous setting, which can be recovered by setting C(x) = 1 for all examples and B = K. In practice, labeling cost for P&P examples can be accurately modeled as linear function of the number of actors in the scene as most annotation time spent drawing detailed bounding boxes for each actor. This new formulation requires modifica- tions to our selection algorithm, since high scoring examples S(x) may also have high costs C(x). Therefore, rather than sorting by score, we can select examples with the highest value, V(x) = 8(x)
(()
Since the cost of labeling a region is unknown before labeling, we approximate it using the number of detections after non mon-maximum suppression (NMS) [38]. After labeling, the true cost is known, and active selection can continue iteratively until the budget reached. Since examples in the P&P setting represent large scenes, scores S(x) and costs C(x) can vary significantly as scenes can have few to many actors, each contributing to the total score and cost. As consequence, coarse-grained scoring will be suboptimal scenes may contain regions with high score and low cost (c.g., single
3

---

car performing a rare U-Turn) and other regions with low score and high cost (e.g., parking lot filled with many static vehicles). This motivates the need for more fine-grained scoring and selection. Next, we describe modifications to support partially labeled scenes, which will enable fine- ene-grained selection for better performance in the cost-aware active learning setting.
Partially Labeled Scenes: We generalize the labeling process to allow for partial labeling. Along with the added flexibility, this setting is also realistic in practice. Even as entire scenes are labeled to- day, annotation platforms often decompose work into subtasks, which can be more easily distributed and validated across a labeling team. As a simple extension, platforms could support querying labels for only particular regions. To support partial labels, we redefine an example XR as the scene aug- mented with a labeling region R, XR= (x, R). Given the of labels for the entire scene y L(x) and region R. each actor's bounding box label y; will either be fully contained in R, completely outside of R. or partially inside of R. For simplicity, we assume that if any part of the bounding box yi of an actor is inside R then it will be provided annotating all actors, even those that are only partially visible in the labeling region. More formally, the labeling oracle returns labels for an example xn. L(xn) = [y, : y: E R and y; E L(x))-
label. In practice, this translates to labelers
Training from Partial Supervision: We adapt training to support partial supervision by applying the loss only the labeled region, R. Importantly, we do not alter the network input x, since do not want to bias the network by changing input statistics. Therefore, the forward pass remains unchanged, y = fx). To compute the loss, we only consider the labels that we have received in R,
8) - (BB())++++ + Σ (p(y.,
(I)
YEER
Here, ér(-.-) represents traditional multi-task perception and prediction losses applied over the pos- itive examples in It and (R)) represents a "background" loss which encourages the network not to output detections for negative regions in R. For example, in experiments, Ep includes a prob- abilistic prediction loss, a bounding box regression loss and cross-entropy positive examples, whereas g (t) represents the hard negative mining loss, sampling only negative anchors from Rt
Fine-Grained Selection: Without restrictions on R. there are infinite regions consider for a given scene. Therefore, in order to efficiently score and select regions for labeling, we consider the set obtained by discretizing the entire scene into rectangular grid. Specifically, example into HW non-overlapping regions, X = (x, Rh.,)). By setting H = W = 1, we obtain a single region for each scene and recover the original formulation. As H and W increase, candidate regions become smaller, providing the learner more fine-grained precision for selection. Most steps of the selection process can remain unchanged. Scoring functions now operate over examples augmented with regions, (xx)) returning a score that only considers network predictions in R. Similarly, only the cost of labeling the queried region C'XXR) incurred when selecting x
divide each
As regions sizes shrink, we observe that the active learner is more likely to select large number of scenes, each labeled with very sparse supervision. While this dataset would contain many interest- ing actors, we also find this unconstrained selection results in significantly longer training times as examples are less densely labeled. Additionally, we observe training instabilities due to the imbal- ances between the amount of supervision available for each example. To alleviate these issues, we introduce sparsity regularizer, which requires that the active learner select a minimum number of positive examples M for any selected scene. Therefore, letting P(xx) represent the number of posi- tive examples in an example, our new formulation can be summarized by the following optimization problem solved by the learner at each iteration,
Σ S(xn)
Σ XREQO)
C(xn) < B and P(xR) 2 M V Xn E Q(i)
max Q
(2)
XREQO
We solve this optimization greedily by first selecting the highest scoring scene remaining, then selecting the highest regions in the scene until at least M actors are labeled. We continue selecting new scenes until the budget is reached.

---

2.0
Random Scenes
144
Randem Regions
5.0
maanADE (matseeS)
Care Set
Venkcie VenklenuPP@o
7.0-
LeAmt ces
Cio Grainod Fine Graned
1.7-
5.76
2.6
6.76
672
144
Number 120K
120 Number Labels
LOOK
120k
140K
Number of Labels
Figure 2: (Left) P&P performance when trained on partial labels varying densities, (Right) Performance of various selection approaches over N active learning iterations.
3.3 Selection Criteria
Our fine-grained, cost-aware active learning formulation introduced above generally supports any approach which provides an informativeness score S(xR) per example. In this work, we focus on uncertainty- based approaches, an extremely common active learning paradigm based on the as- sumption that training on uncertain examples are most likely improve future performance. As this approach depends model's characterization of uncertainty, we first describe our probabilistic P&P model, followed by possible uncertainty-measuue that can be used for scoring.
Following [6], we jointly train a model for both perception and prediction from LiDAR and HD map imputs. A model which naturally characterizes uncertainty over predictions is desirable, as these uncertainty estimates provide useful measure of informativeness for scoring. Therefore, leverage the output representation of [11], using mixture of K Gaussians represent the distri- bution of each actor's future positions. For simplicity, independence is assumed between timesteps of the prediction horizon. Thus the likelihood of particular actor trajectory, ys. can be written as,
Model:
K
ply.)
Σ Thh II (YRKA,E))
where A is the pdf 2D multivariate Gaussian with parameters p Ej and ã represent Gaus- sian mixture weights. These parameters, for each detected actor, are predicted by deep neural network trained with negative log likelihood (see Section 4 for more details). With knowledge of the model, we now introduce measures of uncertainty to use selection criteria. Due to the multi- task nature of the task, we present separate selection criteria for the detection and prediction task. In practice, a mix of both can be used to ensure performance improves across both tasks.
Detection Entropy: We focus on characterizing the uncertainty over the model's classification predictions for each anchor. For classification tasks, the uncertainty is typically estimated by cal- culating the entropy of the model's predicted probabilities. Given anchors A with associated probabilities Pa, the entropy of the predictions are given by,
(D(A) = Σ Pa lo + (1 Pa) log(1 Pa)
(4)
aEA
To score regions, we assume independence between anchors and sum the entropies of anchors in R.
Prediction Entropy: Computing prediction entropy naturally depends on the model's output rep- resentation. Our model outputs Gaussian mixture for each predicted actor. Unfortunately, there is no known closed form solution to computing this distribution's 's entropy [39]. Therefore, we are required to estimate the entropy via approximations. We explored various approximations, including a sample- monte-carlo estimate, but all performed similarly to or worse than an approximation via the entropy of the discrete categorical distribution induced by the mixture weights Tk.
Hp(y:) = Σ xt
Ty log Tk
(5)
Intuitively, this approximation is well-suited capture cases where the model is uncertain between multiple possible modes, which is likely representative of the true entropy of distribution. We use
5

---

Prediction (meanADE) 4
Downstream Planning
Selection
Straight
Left (m)
Right (m)
Stationary
Collision
L2
Lai. acc. (m/s²)
Jerk (m mm/s")
Progress f
(m)
(m)
(%)
(m)
(m)
Random Scenes
2.89
5.31
5.68
0.22
5.02
5.89
2.80
2.67
33.5
Random Regions
2.46
4.82
4.96
0.20
5.07
5.71
2.70
2.47
33.6
Core-Set
2.45
4.71
5.01
0.21
5.14
5.72
2.65
2.45
33.6
Learn
2.46
4.74
4.99
0.21
5.15
5.74
2.68
2.47
33.6
Coarse-Grained
2.44
4.79
5.03
0.22
5.17
5.71
2.67
2.44
33.8
Fine- Frne-Grained
2.29
4.52
4.91
0.21
4.63
5.56
2.62
2.38
33.7
Table 1: (Left) Prediction Performance By High Level Action (Right) Planning Performance.
this approximation due to its simplicity and computational efficiency while providing similar per- formance. Finally, we assume independence and sum the actor entropies within each region.
Experiments
In this section, we analyze the effects of partial labeling and fine seeeggrannd active selection. First, we explore partial labeling independent of active selection. Next, we explore the improvements provided by active selection for prediction. We find that a simple prediction entropy combined with fine-grained active selection outperforms various traditional scene-based approaches. More detailed analysis shows that fine-gre ime-ggrined selection enables the learner to better oversample labels exhibiting complex driving behaviors, resulting in better performance on these challenging behaviors in the test-set. In practice, we are most interested in the effects of these improvements on the downstream motion planning task, where we find significant improvements across most metrics. Finally, we observe similar improvements for perception when using detection entropy as the selection criterion.
Dataset: We leverage real-world large-scale dataset collected across multiple cities in North America. To simulate the active learning setting, we follow standard practice in active learning research and treat the large labeled dataset as an unlabeled pool. Active learning approaches select from this pool containing 100K scenes, with roughly 2 million labels. For each scene, we have access to LiDAR sweeps recorded 10Hz with a localized HD map given as input to the model. For evaluation, use the standard metrics of mAP@0.7 for detection and meanADE for prediction.
Implementation Details: For model implementation, we follow the exact details of the Gaussian mixture baseline from [12], an implementation of MTP [11] for the joint perception and prediction setting. To support partial labeling,
find necessary use sum instead of mean reduce losses in batch, ensuring actors in less densely labeled scenes
not up-weighted relative to those in more densely labeled scenes. All models are trained for 50 epochs, using budgeted training [40] for the learning rate schedule. We use H W 20 to discretize the entire scene into 400 rectangular regions for scoring and selection. Finally, for sparsity regularization, we set M 5.
4.1 P&P from Partial Supervision
To test the effects of partial labeling, we randomly select labels at varying levels of ground-truth density per scene. To ensure a fair comparison, the total labeling budget is fixed across densities. Specifically, let be the labeling density. When r 1, we recover the traditional fully labeled setting. When }. we first sample scenes randomly and then sample regions from half of each scene, providing labels only for selected regions. Notice that since we fix the labeling budget, selecting at lower densities will result in more scenes. Figure 2 shows the effect of partial labeling on P&P performance. Across all dataset sizes, we observe improved performance with lower density datasets. Benefits appear to saturate at r = 1.- as the sparser labeling density r = does not provide further improvements. Performance gains can be explained by the fact that more sparsely labeled datasets naturally include supervision from more scenes, improving the model's 's ability to generalize.
Our results suggest adopting sparse labeling to optimize P&P performance under a fixed labeling budget rather than label scenes entirely. However, note that in practice there is tradeoff between labeling at lower densities and model training times, datasets with less dense supervision must be trained for more iterations. We explore this further in our sparsity regularization experiments.

---

Figure 3: Qualitative Examples: Labels in regions selected by fine-grained selection shown in red. Selected regions tend to have moving vehicles performing interesting actions (e.g. U-Turns).
4.2 Fine-Grained Active Selection for Prediction
Experimental Sctup:
this experiment, we study active learning approaches for improved pre- diction. For each method, we sample an initial labeled set L
x=(0) of 40K vehicles from X To ensure that final datasets contain scenes with similar density of supervision, the initial data for fine-grained methods are partially labeled scenes, whereas coarse- gaatseegr approaches sample full scenes. For fair comparison, we the labeling budget at each active learning iteration. Specifically, for each of the N active learning iterations, the learner is given budget of 20K vehicles. After each iteration, the model is re-trained on its current set of labels and evaluated seperate held-out test set. The test-set is held constant across all approaches and contains traditional fully labeled scenes.
We compare fine tiee-graindd active selection to full scenes selected randomly (Random Scenes), partially labeled scenes at density r = selected randomly (Random ons), full scenes selected by prediction entropy CCoarse-Grainee baselines adapted to the P&P setting. The first a recent uncertainty-base bassd approach which learns to predict the loss of unlabeled examples, which we refer to as LearnLoss [1]. The second is the common diversity- -based approach of Core-Set selection [21]. See the supplementary for details.
Additional Baselines:
and two additional active learning
Prediction Performance: Results are shown in Figure All active selection techniques offer significant improvements over random selection. Interestingly, despite large differences in the se- lection criteria (e.g., uncertainty-basel vs. diversity- -bseed), scene- scene-based approaches achieve similar performance, indicating that gains may be saturated due to the inflexibility of selecting entire scenes. Surprisingly, simply labeling random regions appears to perform better than or similar to many of the coarse-grained active learning approaches. Finally, fine-grained selection offers the best per- formance. While the improvements may appear relatively small, we find consistent results across random seeds due the large dataset sizes. Additionally, aggregate prediction metries are aver- aged over more than 1M actors the test-set and may hide large differences between the specific behaviors of the prediction models, calling for more detailed analysis.
Performance By High Level Action: In Table 1, we break down the prediction performance by action: driving straight, turning left, turning right, and stationary. We notice that differences between selection algorithms become more apparent across actions associated with more difficult predictions (i.e., all non-stationary actions). These results are explained by the fact that fine-grained entropy selects more unpredictable moving actors compared to other selection approaches.
Following [12], we evaluate downstream performance on motion plan- ning, computing the collision rate, 1.2 error, lateral acceleration, jerk, and progress of a planner [41] given the predictions from each trained model. The results in Table show that fine-grained selection leads significant reduction in collisions and also outperforms all baselines across the remaining planning metrics, except progress, for which the differences are not significant (an obser-
Planning Performance:
7

---

Uncorstrameed
Sparilly Regulariced
mamile imetes)
1.6
Rendom Reglona
Coarse Graited
1.5
Fine Grained
1.4
0.62
100%
0.60
0.58
50%
25%
40K
63K
B3K
100K
120k
140K
Number of Labels
Figure 4: (Left)) The effect of sparsity regularization on performance (top) and training time (bot- tom). (Right) Multi-class detection performance for N iterations of Active Learning.
vation consistent with [12]). The results demonstrate that the challenging training examples selected by active selection leads to improved planning, which is ultimately most important for self-driving.
Qualitative Examples and Selection Statisties: Examples of regions selected by fine-grained selection, seen in Figure tend to include vehicle labels with moving actors, actors intersections, actors performing odd maneuvers (e.g.. U-Tur in the top-right example). Non-moving actors rarely selected. In the supplementary materials, we plot histograms of label statistics selected by each method after the final iteration of active learning. As expected, we notice that active-selection methods tend to sample more non-stationary vehicles and vehicles further from the SDV. This effect more apparent for fine-grained selection methods due to the additional flexibility provided by the partially labeled setup. Please see the supplementary for more detailed analysis.
Sparsity Regularization Ablation: In Figure 4, we ablate the effect of sparsity regularization on performance and number of scenes selected. As expected, at early iterations, we find unconstrained selection outperforms the sparsity regularized approach, as the unconstrained approach has the free- dom select a larger set of scenes, each with less supervision. However, later iterations, we observe the unconstrained selection performance degrades. This is likely caused by the imbalance between the amount of supervision available for each scene, which found empirically can lead to degraded performance. Beyond performance, there is an additional, perhaps more important, benefit of sparsity regularization. Since the active learner must select at least M actors per scene, the num- ber of scenes in the dataset grows linearly with each iteration. Alternatively, in the unconstrained approach, the dataset size explodes at early iterations until there a label for every scene in Xu-
4.3 Fine-Grained Active Selection for Perception
We additionally experiment with fine-grat fineigrained selection for improved perception. We follow simi- lar experimental setup, replacing prediction entropy with detection entropy. As all methods perform similarly on vehicle detection, we evaluate on the more challenging multi-class setting where ey- clists and pedestrians must be detected. Similar to the prediction setting, results Figure 4 show fine-gra une-ggrined selection is most effective. Interestingly, coarse- graiied selection is similar random scenes, likely explained by averaging effect from summing the entropies over the full scene.
Conclusion
We studied active learning techniques to intelligently select examples to label from unlabeled self- driving data logs for perception and prediction models. We found the traditional active learning setting ill-suited and introduced generalizations to account for variable labeling costs and enable fine-grained selection through partially labeled scenes. In our experiments, we found significant im- provements from partial labeling without any active selection, and further gains across perception, prediction and downstream planning by leveraging fine-grained active selection. Our results demon- strate that the dominant paradigm of labeling entire self-driving scenes may not be most efficient under fixed budget and that fine-grained selection is likely required for maximal efficiency.

---

References
[1] D. Yoo and I. S. Kweon. Learning loss for active learning. In CVPR, pages 93-102, 2019.
[2] S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In ICCV, pages 5972-5981, 2019.
[3] S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In ECCV, pages 137-153. Springer, 2020.
[4] E. Haussmann, M. Fenzi, K. Chitta, J. Ivanecky, H. Xu, D. Roy, A. Mittel, N. Koum- chatzky, C. Farabet, and J. M. Alvarez. Scalable active learning for object detection. ArXiv, abs/2004.04699, 2020.
[5] D. Feng. X. Wei, L. Rosenbaum, A. Maki, and K. Dietmayer. Deep active learning for efficient training of lidar 3d object detector. In /V, pages 667-674. IEEE, 2019.
[6] W. Luo, B. Yang, and R. Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In CVPR, pages 3569-3577, 2018.
[7] S. Casas, W. Luo, and R. Urtasun. Intentnet: Learning predict intention from sensor data. In CoRL, pages 947-956. PMLR, 2018.
[8] T. Phan-Minh, E. Grigore, F. Boulton, O. Beijbom, and E. M. Wolff. Coverne:: Multimodal behavior prediction using trajectory sets. CVPR, pages 14062-14071, 2020.
[9] A. Jain, S. Casas, R. Liao, Y. Xiong, S. Feng, S. Segal, and R. Urtasun. Discrete residual flow for probabilistic pedestrian behavior prediction. In CoRL, pages 407-419. PMLR, 2020.
[10] G. Oh and J.-S. Valois. Henaf: Hyper-conditioned neural autoregressive flow and its applica- tion for probabilistic occupancy map forecasting. In CVPR, pages 14550-14559, 2020.
[11] H. Cui, Radosavijevic, F--. Chou, T.-H. Lin, Nguyen, T.-K. Huang, J. Schneider, and N. Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In ICRA, pages 2090-2096. IEER, 2019.
[12] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for scene-consistent motion forecasting. ECCV, 2020.
[13] Y. C. Tang and R. Salakhutdinov. Multiple futures prediction. In NeurlPS, 2019.
[14] B. Settles. Active learning literature survey. 2009.
[15] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NIPS, 2017.
[16] W. H. Beluch, T. Genewein, A. Nürnberger, and J. M. Köhler. The power of ensembles for active learning in image classification. CVPR, pages 9368-9377, 2018.
[17] Y. Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. In ICML, pages 1183-1192, PMLR, 2017.
[18] A. Holub, P. Perona, and M. C. Burl. Entropy-basee active learning for object recognition. In CVPR Workshops, pages 1-8. IEEE, 2008.
[19] S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. ICCV, pages 5971-5980, 2019.
[20] H. T. Nguyen and A. Smeulders. Active learning using pre-clustering. In ICML, page 79, 2004.

---

[21] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set ap- proach. ICLR, 2018.
[22] Y. Guo. Active instance sampling via matrix partition. In NIPS, pages 802-810, 2010.
[23] D. Gudovskiy, A. Hodgkinson, T. Yamaguchi, and S. Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. In CVPR, pages 9041-9049, 2020.
[24] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. ICLR, 2020.
[25] M. Gao, Z. Zhang, G. Yu, Ó. Ank, L. S. Davis, and T. Pfister. Consistency- semi- supervised active learning: Towards minimizing labeling cost. In ECCV, pages 510-526. Springer, 2020.
[26] S. Sivaraman and M. M. Trivedi. Active learning for on-road vehicle detection: A comparative study. Machine vision and applications, 25(3)-999-611 2014.
[27] S. Roy, A. Unmesh, and V. Namboodiri. Deep active learning for object detection. In BMVC, page 91, 2018.
[28] A. Pardo, M. Xu, A. Thabet, Arbelaez, and B. Ghanem. Baod: budget-aware object detec- tion. In CVPR, pages 1247-1256, 2021.
[29] G. Wang, J.-N. Hwang, C. Rose, and F. Wallace. Uncertainty based active learning via sparse modeling for image classification. TIP, 28(1):316-329, 2018.
[30] K. Wang, D. Zhang, Y. Li, R. Zhang, and L. Lin. Cost-Effective Active Learning for Deep Image Classification. TCSVT, 27(12):551-2200, Dec. 2017. ISSN 1051-8215, 1558-2205. doi: 0001001100/TC3VV22666.2888899
arXiv: 1701.03551.
[31] T. Kasarla, G. Nagendar, G. M. Hegde, Balasubramanian, and C. Jawahar. Region-based active learning for efficient labeling semantic segmentation. In WACV, pages 1109-1117. IEEE, 2019.
[32] R. Mackowiak, P. Lenz, O. Ghori, F. Diego, O. Lange, and C. Rother. Cereals - cost-effective region-basse based active learning for semantic segmentation. BMVC, 2018.
[33] W. Luo, A. Schwing, and R. Urtasun. Latent structured active learning. NIPS, 26:728-736, 2013.
[34] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621-11611, 2020.
[35] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, pages 8748-877 8757, 2019.
[36] A. Sadat, S. Segal, S. Casas, J. Tu. B. Yang, R. Urtasun, and E. Yumer. Diverse complexity measures for dataset curation in self-driving. arXiv preprint arXiv:2101.066 2021.
[37] Segal, Kee, W. Luo, A. Sadat, E. Yumer, and Urtasun. Universal embeddings for spatio-temporal tagging of self-driving logs. CoRL, 2020.
[38] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained bast-based models. IEEE TPAMI, 3269)16277-1655, 2009.
[39] M. F. Huber, T. Bailey, H. Durrant- Whyte, and U. D. Hanebeck. On entropy approximation for gaussian mixture random vectors. In MFI, pages 181 181-188. IEEE, 2008.
10

---

[40] M. Li, E. Yumer, and D. Ramanan. Budgeted training: Rethinking deep neural network training under resource constraints. ICLR, 2020.
[41] A. Sadat, M. Ren, A. Pokrovsky, Y.-.. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages -956. IEEE, 2019.


Reviews:
[{"review_id": "W5yveUV8cw", "reviewer": "Reviewer_KVfT", "summary": "Self-driving vehicles require extensive data supervision in order to generalize in the real world. However, much of this labelling is wasted effort due to the \"long-tail problem\" - common situations (e.g. parked car) are easy to predict and make up the bulk of the dataset while rare (but crucial) situations make up a tiny fraction of the data-points. This paper proposes an active learning technique to make maximal use of fixed labelling budgets. The utility $S(r)$ and the cost $C(r)$ of each un-labelled region $r$ are explicitly modeled, and data points are chosen for labelling if they are expected to improve prediction performance most efficiently. Furthermore, the training loss is modified to model partially labelled scenes. These modifications enable improved performance v.s. baselines and strongly indicate that fully labelling scenes is a wasteful process.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the big ideas behind the work are effectively conveyed. The supplementary video is very helpful for understanding the paper and the qualitative examples help contextualize the work's contributions. The general idea of using active learning and partial labelling seems like a sensible way to attack the long-tail problem. Furthermore, the ablations do a good job of justifying hyper-parameter choices (e.g. scoring function parameterization and selection policy) in the method.\n\nHowever, the results section could use comparisons to real active learning baselines. The authors mention *LearnLoss* and *Core-Set* but show no results on those baselines, other than vague references of similarity to *Coarse-Grained* in the supplementary material. Real comparisons to prior active learning/labelling work will help us judge this paper's contributions fairly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is well executed and thorough investigation of partial labelling and active learning in P&P modeling context. However, it is missing comparisons against real baselines that are important to understand significance.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "* Please show comparisons v.s. baselines you mentioned as well as any other active learning baselines that have been previously tried in this space.\n* Please do a better job documenting the dataset. Did you collect it or did someone else? If you collected it, what sort of data distribution should we expect?"}}, {"review_id": "Fwt9gbhweh3", "reviewer": "Reviewer_PW9h", "summary": "This paper presents a way to introduce active learning in the Perception and Prediction (P&P) setting: it chooses to label the scenes partially instead of label the entire scene for a higher cost. It trains with partial supervision and only apply the loss on the labeled regions; essentially it will select a large number of scenes, each labelled very sparsely. The methods has been shown effective in P&P tasks and outperform prior baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It is a novel idea to explicitly model the cost and incorporate it in the active learning selection process. \n- Comprehensive literature review\n- Clear presentation of methods and algorithms\n\n\nOne concern I have on visualization is that in Figure 3: Qualitative Examples, the graphics do not seem to be very clear in presentation. The red boxes are quite small and I'm not sure are the white boxes meaningful? There could be more semantically meaningful explanations for the figures (although I like the idea of using qualitative examples).\n\nAnother question I have is in Figure 2 (Right): the experimental results of Fine-Grained is not too different from that using Random Regions, especially when using small amount of labels. Do the authors have an explanation on the similarity of results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend this paper since it presents an interesting active learning method in P&P setting that involves labelling cost, and introduces the idea of modularity in training samples. The main idea could potentially inspire active learning in other fields, for example image classification, robot learning etc.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please refer to the Strengths and Weaknesses section for more details.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "2QC39GyAj0y", "reviewer": "Reviewer_TXdJ", "summary": "The authors present a method for fine-grained active learning in the self-driving domain.  At each round of training, a learning agent scores unlabelled examples that seem most informative, and an oracle provides labels for them.  The agent then trains on all labeled examples seen so far and repeats.  The authors argue that performing this active selection on a per-instance basis rather than a per-scene basis improves performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is very well written and easy to follow.  Nothing is confusing, and the material is well presented.\n\nThe experiments are thorough and the authors have presented good baselines to disentangle the effects of active learning vs. fine-grained/instance-based labelling.\n\nThe results seem to provide useful insight for practitioners in this space.\n\nWeaknesses:\nThere is not a great deal of technical novelty here.  The key idea is to do active learning over instances rather than full scenes, which leads to a fairly straightforward implementation.\n\nThe authors heavily promote their approach based on active learning, but the most important result of the paper seems to be that if you have a limited labeling budget, it’s much better to partially label many scenes than densely label a few, regardless of whether you use active labeling or just random selection.  The difference between active fine-grained and random fine-grained (random regions) is quite small in the immediate perception/planning results (Figure 2.)  There is more of a gap for the downstream planning results (Table 1.) but here the gaps between all methods seem relatively small.  In general it would be very helpful if the authors provided some guidance as to what constitutes a significant improvement in this domain.  The authors comment that the difference of 0.1 “Progress” where their model is weaker than the random baseline is not significant, but do not provide information on what threshold would be considered significant here.  I understand that with self-driving cars every collision counts, but I also have a limited understanding of how well performance on these held-out test sets relates to real-world performance.  More explanation here would be helpful.\n\nAlong these lines, please provide units for the various error rates in Table 1 to give a better idea of what these numbers mean.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am very much on the fence with regards to this paper.  It is well presented, and well argued, but the minimal technical novelty and results which are marginally better, but not dominating other methods makes it hard to strongly argue in favor of acceptance.  Part of this stems from my own lack of familiarity with what constitutes a significant improvement in this domain.  I would improve my score if the authors can convincingly argue that these results are significant and useful and/or other reviewers can vouch for their significance.\n\nAlso, if there was an option for Impact between 3 and 4 (something like \"This work is likely to have some impact within a limited domain\") I would apply it here.  It is reasonable to see this paper having some impact within the self-driving community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "It would be great if you can provide more clarity on what constitutes significant improvement in this domain, and make a strong argument that the improvements here will have a strong impact on real-world performance.\n\nLabel/explain the units in Table 1 please."}}, {"review_id": "19tzidQK43h", "reviewer": "Reviewer_TDyJ", "summary": "The paper proposes a labeling method for active learning for self-driving applications. In particular, the paper focuses on the percpetion and prediction problem. Unlike traditional active learning frameworks that assume each example in the dataset can be labeled for the same cost, the proposed method explicitly model the cost to label each example. The motivation is, given a fixed labeling budget, more examples (lower cost examples) can be labeled. The authors proposed to include the cost to label in the scoring function, the function used in active learning frameworks to pick examles that are believed to be most likely to improve the model performance. \nFurthermore, the paper introduces partial labeling, i.e., labeling only particular regions of an example. The authors show that given a fixed labeling budget, the performance will improve by training with more examples but equal number of labels. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is well-structured and the results support the claims. \n* The idea is quite simple but seems to have a considerable effect on the results \n\nWeaknesses \n* The numbers/results provided lack measures like confidence intervals to ensure statistical significance. \n* One of the core contributions of the paper, i.e., predicting the cost of labeling for each example, is not clearly described. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is not really clear how partial labeling would work for the prediction problem. An actor can leave the region in subsequent frames, therefore, the labeling region can change across consecutive frames. \n\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* An important contribution of the paper is the \"cost-aware\" part of it. However, this component is not described well and it is not really clear how the cost of each example is predicted. \n\n* For all experimental results, a confidence interval measure is required to ensure statistical significance. \n\n* Line 117: NMS is not defined. \n* Line 166: The sentence is incomplete"}}]

Author rebuttal:

Meta-review: All reviewers agree that the idea proposed in the paper is interesting and that the paper is well presented. Three of four reviewers rate the paper's impact to be limited, although I can see this to generalize to other domains as well. There are two issues with the paper in its current form though: I agree with reviewer TDyJ who criticizes the lack of confidence intervals to understand the statistical significance of the results. Second, reviewer KVfT recommends a comparison with prior active learning work - something I would love to see as well. If these issues were to be addressed I would recommend acceptance. 
Decision: Accept (Poster)