Exploring Active 3D Object Detection from a Generalization Perspective
Authors: Yadan Luo
Zhuoxiao Chen
Zijian Wang
Xin Yu
Zi Huang
Mahsa Baktashmotlagh
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Active Learning
3D Object Detection
Lidar Point Clouds

Abstract: To alleviate the high annotation cost in LiDAR-based 3D object detection, active learning is a promising solution that learns to select only a small portion of unlabeled data to annotate, without compromising model performance. Our empirical study, however, suggests that mainstream uncertainty-based and diversity-based active learning policies are not effective when applied in the 3D detection task, as they fail to balance the trade-off between point cloud informativeness and box-level annotation costs. To overcome this limitation, we jointly investigate three novel criteria in our framework CRB for point cloud acquisition - label conciseness, feature representativeness and 
geometric balance, which hierarchically filters out the point clouds of redundant 3D bounding box labels, latent features and geometric characteristics (e.g., point cloud density) from the unlabeled sample pool and greedily selects informative ones with fewer objects to annotate. Our theoretical analysis demonstrates that the proposed criteria aligns the marginal distributions of the selected subset and the prior distributions of the unseen test set, and minimizes the upper bound of the generalization error. To validate the effectiveness and applicability of CRB, we conduct extensive experiments on the two benchmark 3D object detection datasets of KITTI and Waymo and examine both one-stage (i.e., Second) and two-stage 3D detector (i.e., PV-RCNN). Experiments evidence that the proposed approach outperforms existing active learning strategies and achieves fully supervised performance requiring $1\%$ and $8\%$ annotations of bounding boxes and point clouds, respectively. 

Published as a conference paper at ICLR 2023
EXPLORING ACTIVE 3D OBJECT DETECTION FROM A GENERALIZATION PERSPECTIVE
Yadan Luo", Zhuoxiao Chen". Zijian Wang, Xin Yu, Zi Huang, Mahsa Baktashmotlagh The University of Queensland, Australia
ABSTRACT
To alleviate the high annotation cost in LiDAR-based 3D object detection, ac- tive learning a promising solution that learns to select only a small portion of unlabeled data to annotate, without compromising model performance. Our em- pirical study, however, suggests that mainstream uncertainty-bane based active learning policies are not effective when applied in the 3D detection task, as they fail to balance the trade-off between point cloud informativeness and box-level annotation costs. To overcome this limitation, tigate three novel criteria in our framework CRB for point cloud acquisition label conciseness, feature representativeness and geometric balance, which hier- archically filters out the point clouds of redundant 3D bounding box labels, latent features and geometric characteristics (e.g., point cloud density) from the unla- beled sample pool and greedily selects informative ones with fewer objects to annotate. Our theoretical analysis demonstrates that the proposed criteria aligns the marginal distributions of the selected subset and the prior distributions of the unseen test set, and minimizes the upper bound of the generalization error. To validate the effectiveness and applicability of CRB, we conduct extensive experi- ments on the two benchmark 3D object detection datasets of KITTI and Waymo and examine both one-stage (i.e., SECOND) and two-stage 3D detectors (i.e., Pv- RCNN). Experiments evidence that the proposed approach outperforms existing active learning strategies and achieves fully supervised performance requiring 1% and 8% annotations of bounding boxes and point clouds, respectively. Source code: https://gttuu..com// http://giiibbooooooooy///////aaa Luydaaa/CRRBBBaBaaeeeeaaaa
and diversity-
jointly inves-
INTRODUCTION
IIDAR-bascd 3D object detection plays an indispensable role in 3D scene understanding with a wide range of applications such autonomous driving (Deng al. 2021 Wang et al.) (2020) and robotics (Ahmed et al., 2018 Montes et al. 2020 Wang et 2019). The emerging stream of 3D detection models enables accurate recognition the cost of large-scale labeled point clouds, where 7-degree of freedom (DOF) 3D bounding boxes consisting of position, size, and orientation information- for each object are annotated. In the benchmark datasets like Waymo (Sun al. 2020 there are over 12 million LiDAR boxes, for which, labeling a precise 3D box takes more than 100 seconds for an annotator (Song al., 2015). This prerequisite for the performance boost greatly hinders the feasibility of applying models to the wild, especially when the annotation budget is limited.
To alleviate this limitation, active learning (AI.) aims reduce labeling costs by querying labels for only small portion of unlabeled data. The criterion- based query selection process iteratively selects the most beneficial samples for the subsequent model training until the labeling budget is run out. The criterion is expected to quantify the sample informativeness using the heuristics derived from
sample uncertainty (Gal et al. 2017 Du et al., 2021 Caramalau lu et al. 2021 Yuan et al. 2021 Choi
et al.) 2021 Zhang et al. 2020 Shi & Li 2019) and sample diversity (Ma et al.] 2021 Gudovskiy
et al. 2020 Gao et al. 2020 Sinha et al. 2019 Pinsler et al. 2019). In particular, uncertainty-driven
approaches focus the samples that the model is the least confident of their labels, thus searching
for the candidates with: maximum entropy (MacKay 1992 Shannon 1948 Kim et al. 2021b Sid-
diqui et al. 2020; Shi & Yu 2019), disagreement among different experts (Freund al. 1992 Tran
et al. 2019 minimum posterior probability of a predicted class (Wang et 2017), or the samples
"Equal contribution. Correspondence Yadan Luo <y.lun@ng.edu.uu>>

---

Published as a conference paper at ICLR 2023
with reducible yet maximum estimated error (Roy& & McCallum, 2001 Yoo & Kweon 2019: Kim
eet al. (22IIa). On the other hand, diversity- beased methods try to find the most representative samples
to avoid sample redundancy. To this end, they form subsets that are sufficiently diverse describe
the entire data pool by making use of the greedy coreset algorithms (Sener & Savarese 2018), or
the clustering algorithms (Nguyen & Smeulders 20044) Recent works (Liu al.) [202]] Citovsky
it al. 2021| Kirsch et al.) 2019 Houussy et al. (011)) combine the aforementioned heuristics: they
measure uncertainty as the gradient magnitude of samples (Ash et al. 2020) or its second-order met-
rics (Liu et al., (02T) at the final layer of neural networks, and then select samples with gradients spanning a diverse set of directions. While effective, the hybrid approaches commonly cause heavy computational overhead, since gradient computation is required for each sample in the unlabeled pool. Another stream of works apply active learning to 2D/3D object detection tasks (Feng et al.) 2019 Schmidtt et al.] 22020; Wang et al., 2022] Wu et al. 2022 Tang et al.] (00T), by leveraging ensemble (Beluch et al. 2018) or Monte Carlo (MC) dropout (Gal & Ghahramani (016) algorithms to estimate the classification and localization uncertainty of bounding boxes for images/point clouds acquisition (more details in Appendix 1). Nevertheless, those AL methods generally favor the point clouds with more objects, which have higher chance of containing uncertain and diverse objects. With a fixed annotation budget, it is far from optimal to select such point clouds, since more clicks are required to form 3D box annotations.
To overcome the above limitations, we propose to learn AL criteria for cost-efficient sample acqui- sition the 3D box level by empirically studying its relationship with optimizing the generalization upper bound. Specifically, we propose three selection criteria for cost-effective point cloud acqui- sition, termed as CRB, i.e., label conciseness, feature representativeness and geometric balance. Specifically, we divide the sample selection process into three stages: (1) To alleviate the issues of label redundancy and class imbalance, and to ensure label conciseness, we firstly calculate the entropy of bounding box label predictions and only pick top Ki point clouds for Stage 2; (2) We then examine the feature representativeness of candidates by formulating the task as the K2-medoids problem on the gradient space. To jointly consider the impact of classification and regression ob- jectives on gradients, we enable the Monte Carlo dropout (Mc-DROPOUT) and construct the hy- pothetical labels by averaging predictions from multiple stochastic forward passes. (3) Finally, to maintain the geometric balance property, we minimize the KI. divergence between the marginal distributions of point cloud density of each predicted bounding box. This makes the trained detector predict more accurate localization and size of objects, and recognize both close (i.e., dense) and distant (i.e., sparse) objects the test time, using minimum number of annotations. We base our criterion design on our theoretical analysis of optimizing the upper bound of the generalization risk, which can be reformulated as distribution aligmment of the selected subset and the test set. Note that since the empirical distribution of the test set is not observable during training, WLOG, we make an appropriate assumption of its prior distribution.
Contributions. Our work is pioneering study in active learning for 3D object detection, aiming to boost the detection performance at the lowest cost of bounding box-level annotations. To this end, we propose a hierarchical active learning scheme for 3D object detection, which progressively filters candidates according to the derived selection criteria without triggering heavy computation. Extensive experiments conducted demonstrate that the proposed CRB strategy can consistently out- perform all the state-of-the-ar AL basclines on two large-scale 3D detection datasets irrespective of the detector architecture. To enhance the reproducibility of work and accelerate future work in this new research direction, we develop an act factive-3D-der toolbox, which accommodates various AL approaches and 3D detectors.
2
METHODOL.OGY
2.1 PROBLEM FORMULATION
In this section, we mathematically formulate the problem of active learning for 3D object detection and set up the notations. Given orderless LiDAR point cloud p {xr, y, 2, e) with 3D location (I. y. 2) and reflectance e, the goal of 3D object detection is to localize the objects of interest as set of 3D bounding boxes B = (bx ke ( Nal with NB indicating the number of detected bounding boxes, and predict the associated box labels Y {uu ] E E = (I,.... , C......,,,..... with C being the number of classes to prediet. Each bounding box b represents the relative center position (pz, py. Pa) to the object ground planes, the box size (I, w, h). and the heading angle 8. Mainstream 3D object detectors
use point clouds P to extract point-level features E IR WWWZ --F
(Shi et al. 2019
Yang et al. 2019
2

---

Published as a conference paper at ICLR 2023
Rox level Point Density
Dg sim
Labeled Subset Ds
Human Annotator
Conciseness
Representativeness
Balance
Re- Araitain
Label Query
Vecel Encader
Selected Point
Conyy Block
Camv- Cam--Block Block
Clasetir
Clouds
Drapout Conry Blodd
Extraction
Conv-Block
Dropout
Infer
Active Learning
Regressioo
Sparao
divergence Donsn
Policy
Heads
Box Head
Roi Head
Pre-train
Budget N,
Label Sampling
Prototypes Selection
Greedy Search
Labeled Training Set D
Unlabeled Pool Du
Durofsiee
MC Dropout
Da of Kip
Hypo thetical Regression Labels
................................ ......... ................................
Figure An illustrative flowchart of the proposed CRB framework for active selection of point
clouds. Motivated by optimizing the generalization risk, the derived strategy hierarchically selects point clouds that have non-redundant bounding box labels, latent gradients and geometric character- istics to mitigate the gap with the test set and minimize annotation costs.
2020) or by voxelization (Shi et al. 2020), with W, L, F representing width, length, and channels of the feature map. The feature map æ is passed to a classifier f(;;wy parameterized by wy and regression heads g(: wx, (e.g., box refinement and ROI regression) parameterized by Wy. The output of the model is the detected bounding boxes B (δ&) with the associated box labels ý = {ik) } from anchored areas. The loss functions gels and greg for classification (e.g., regularized cross entropy loss Oberman & Calder (018)) and regression (e.g., mean absolute error/ regularization Qi et al. (2020)) are assumed to be Lipschitz continuous. As shown in the left half of Figure [ in an active learning pipeline, a small set of labeled point clouds Dr = {(P, B, [emm and large pool of raw point clouds Dú ((P)))eEm) are provided at training time, with and being total number of point clouds and m < For each active learning round
[R]. and based on
the criterion defined by an active learning policy, we select a subset of raw data (Pz (PibSE(N) from Du and query the labels of 3D bounding boxes from oracle Ω P - B x to construct Ds = (PP, B, Y), JEIN.I The 3D detection model is pre-trained with Dr for active selection, and then retrained with Dş UD, until the selected samples reach the final budget B, iee, ER N, B.
2.2 THEORETICAL MOTIVATION
The core question of active 3D detection is how to design a proper criterion, based on which fixed number of unlabeled point clouds can be selected to achieve minimum empirical risk Ry [[( g: w)) on the test set Dr and minimum annotation time. Below, inspired by (Mansour et al., 2009 Ben- David et al. 2010), we derive the following generalization bound for active 3D detection so that the desired acquisition criteria can be obtained by optimizing the generalization risk.
(VC) dimension d with f and g being the classification and regression branches, respectively. The Ds and Dr represent the empirical distribution induced by samples drawn from the acquired subset Ds and the test set Dr, and & the loss function bounded by J. It is proven that V 8 E (0, 1), and V VS. E H. with probability at least - 8 the following inequality holds,
Theorem 2.1. Let H be hypothesis space of Kupnik-Chervonent
Ry RT[e(J, ()) < Rs(eWS
w)] + disc DT) X const,
where const = const=3J((
log 2N-
+ V log 2 + / /ddog(aNN//d) 2N:
2d (eN(e//)
N.
Ni
Notably, = "rr[[(( w")] Rg(E((" "";;""""""""") denotes the joint risk of the optimal hypoth- esis 8r and g" with w" being the model weights. Nr and N, indicate the number of samples in the Ds and Dr. The proof can be found in the supplementary material.
Remark. The first term indicates the training error on the selected subsets, which assumed to be trivial based on the zero training assumption (Sener & Savarese] 2018). To obtain tight up- per bound of the generalization risk, the optimal subset Dis can be determined via minimizing the discrepancy distance of empirical distribution of two sets, i.e.,
D's = arg min disc (s, Dr). DD
Below, define the discrepancy distance for the 3D object detection task.
3

---

Published as a conference paper at ICLR 2023
Definition 1. For any f. f.g. f' E H. the discrepancy between the distribution of the selected sets Da and unlabeled pool Dr can be formulated as,
disc(Ds. Dr) - sup
(D. - ED E.g (
sup
S.
9.9'
where the bounded expected loss for any classification and regression functions are symmetric and satisfy the triangle inequality.
Remark. As 3D object detection is naturally an integration of classification and regression tasks, mitigating the set discrepancy is basically aligning the inputs and outputs of each branch. Therefore, with the detector freezed during the active selection, finding an optimal D'g can be interpreted as en- hancing the acquired set's 's' (1) Label Conciseness: aligning marginal label distribution of bounding boxes, (2) Feature Representativennss: aligning marginal distribution of the latent representations of point clouds, and (3) Geometric Balance: aligning marginal distribution of geometric character- istics of point clouds and predicted bounding boxes, and can be written as: Representativenes
Conciseness
Balance
D's FJ arg min dA(Pys Pv,)+ dA Pxr) + dA(P (B) Po(Pr.Br)
(1)
DeCDu Here, Ps and Pr represent the point clouds in the selected set and the ones in the test set. o(.) indicates the geometric descriptor of point clouds and d distance (Kifer al. 2004) which can be estimated by a finite set of samples. For latent features Xg and XT, we only focus on the features greg
that differ from the training sets, since FD, gole 0 and E,, 0 based on the zero training error assumption. Considering that test samples and their associated labels are not observable during training, we make an assumption on the prior distributions of test data. WLOG, we assume that the prior distribution of bounding box labels and geometric features are uniform. Note that we can adopt the KL-divergence for the implementation of d assuming that latent representations follow the univariate Gaussian distribution.
Connections with existing AL approaches. The proposed criteria jointly optimize the discrep- ancy distance for both tasks with three objectives, which shows the connections with existing AL strategies. The uncertainty- based methods focus strongly on the first term, based on the assumption that learning more difficult samples will help to improve the suprema of the loss. This rigorous assumption can result a bias towards hard samples, which will be accumulated and amplified across iterations. Diversity- based methods put more effort into minimizing the second term, aiming to align the distributions in the latent subspace. However, the diversity- -bsed approaches are unable to discover the latent features specified for regression, which can be critical when dealing with detection problem. We introduce the third term for the 3D detection task, motivated by the fact that aligning the geometric characteristics of point clouds helps preserve the fine-grained details of objects. leading to more accurate regression. Our empirical study provided in Sec. 3.3 suggests jointly optimizing three terms can lead to the best performance.
2.3 OUR APPROACH
To optimize the three criteria outlined in Eq. I we derive an AL scheme consisting of three components. In particular, to reduce the computational overhead, we hierarchically filter the sam- ples that meet the selection criteria (illustrated in Fig. [ we first pick K₁, candidates by con- cise label sampling (Stage 1), from which we select K-2 representative prototypes (Stage 2), with K1 << n. Finally, we leverage greedy search (Stage 3) to find the N, prototypes that match with the prior marginal distribution of test data. The hierarchical sampling scheme can save ((nn - -T2 + (n K2)TTa cost, with Tz and T3 indicating the runtime of criterion evaluation. The algorithm is summarized in the supplemental material. In the following, we describe the details of the three stages.
Stage 1: Concise Label Sampling (CLs). By using label conciseness as a sampling criterion, we aim to alleviate label redundancy and align the source label distribution with the target prior label distribution. Particularly, we find a subset Dū, of size Kc, that minimizes Kullback-Leibler (KL) divergence between the probability distribution Py, and the uniform distribution Py,. To this end, we formulate the KL-divergence with Shannon entropy H-)) and define an optimization problem of maximizing the entropy of the label distributions:
DKL(Pr, Pr,) -Y,,s,) + log Fs.
(2)
D', arg min DKL (Prs, PY,) = arg max H (Ys,). CDu
(3)
Da. CDu
4

---

Published as a conference paper at ICLR 2023
where log Ys, = logKi indicates the number of values Ys, can take on, which is a constant. Note that Py, is a uniform distribution, and we removed the constant values from the formulations. We pass all point clouds {(P) hxeui from the unlabeled pool to the detector and extract the predictive The label entropy of
labels {ūr); for NB bounding boxes, with D: = arg maxxeecy I w)y. the j-th point cloud H HY,,s) can be calculated as,
elbi=e//Na
H(v = E Pi.e log Pi,e Pi.
(4)
Σe elii=c//Na
Based on the calculated entropy scores, we filter out the top-K. candidates and validate them through the Stage 2 representative prototype selection.
Stage 2: Representative Prototype Selection (RPs). In this stage, we aim to to identify whether the subsets cover the unique knowledge encoded only in Dy and not in Dr by measuring the feature representativeness with gradient vectors of point clouds. Motivated by this, find the represen- tative prototypes on the gradient space 9 to form the subset Dsy, where magnitude and orientation represent the uncertainty and diversity of the new knowledge. For classification problem, gradi- ents can be retrieved by feeding the hypothetical label ý arg maX P(y(ar to the networks. However, the gradient extraction for regression problem is not explored yet in the literature, due to the fact that the hypothetical labels for regression heads cannot be directly obtained. To mitigate this, we propose to enable Monte Carlo dropout (MC-DROPOUT) at the Stage 1, and get the averag- ing predictions B of M stochastic forward passes through the model as the hypothetical labels for regression loss:
B
ga Wtd wy), we Bernoulli p),
(5)
M
Gs, = (V ere (g(z). B: w,), Ds2h
(6)
with p indicating the dropout rate, wn the random variable of the dropout layer, and é the param- eters of the convolutional layer of the shared block. The gradient maps Gs, E G can be extracted from shared layers and calculated by the chain rule. Since the gradients for test samples are not observable, make assumption that its prior distribution follows Gaussian distribution, which allows us to rewrite the optimization function as,
D's = arg min DKL(PX || Pxr) ~ arg min DKL(PGs, PG7) Da, CDs,
Ds, CDs
a + (US² - PT)
min log OT
= Kg-medoids(Gss)
=
Dag CDs
SS
28
2
with J(Sg. 0gg (JT,, and σT) being the mean and a standard deviation of the univariate Gaussian distribution of the selected set (test set), respectively. Based on Eq. the task of finding rep- resentative set can be viewed as picking K22 prototypes (i.e., Kz-medoids) from the clustered data, so that the centroids (mean value) of the selected subset and the test set can be naturally matched. The variance s, and dy, basically, the distance of each point to its prototypes will be minimized simultaneously. We test different approaches for selecting prototypes Sec. 3.3
Stage 3: Greedy Point Density Balancing (GPDB). The third criterion adopted is geometric bal- ance, which targets at aligning the distribution of selected prototypes with the marginal distribution of testing point clouds. As point clouds typically consist of thousands (if not millions) of points, is computationally expensive to directly align the meta features (e.g., coordinates) of points. Fur- thermore, in representation learning for point clouds, the common practice of using voxel-based architecture typically relies on quantized representations of point clouds and loses the object details due to the limited perception range of voxels. Therefore, we utilize the point density d(..)) within each bounding box to preserve the geometric characteristics of an object in 3D point clouds. By aligning the geometric characteristic of the selected set and unlabeled pool, the fine-tuned detector is expected to predict more accurate localization and size of bounding boxes and recognize both close (i.e., dense) and distant (i.e., sparse) objects at the test time. The probability density function (PDF) of the point density is not given and has to be estimated from the bounding box predictions. To this end, we adopt Kernel Density Estimation (KDE) using a finite set of samples from each class which can be computed as:
5

---

Published as a conference paper at ICLR 2023
No
o(P.B) -((P.B). Kerl
p(o(P,B)
(8)
NBh
h
3=1
with > O being the pre-defined bandwidth that can determine the smoothing of the resulting density function. We use Gaussian kernel for the kernel function Ker(.). With the PDF defined, the optimization problem of selecting the final candidate sets Ds of size N, for the label query is:
D's = arg min (xL(O(Ps, || ((Pr, Br)),
DsCDs²
where :) measures the point density for each bounding box. We use greedy search to find the optimal combinations from the subset Ds, that can minimize the KL distance to the uniform distribution p(o(Pr,Br) ~ uniform(ax ax.)) The upper bound an and lower bound at of the uniform distribution are set to the 95% density interval, i.e., p( < ((P,BB B,) < <ons) = 95% for every predicted bounding box j. Notably, the density of each bounding box is recorded during the Stage 1, which will not cause any computation overhead. The analysis of time complexity against other active learning methods is presented Sec. 3.4
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets. KITTI (Gegger et al. 2012 is one of the most representative datasets for point cloud based object detection. The dataset consists of 3,712 training samples (i.e., point clouds) and 3,769 val samples. The dataset includes total of 80,256 labeled objects with three commonly used classes for autonomous driving: cars, pedestrians, and cyclists. The Waymo Open dataset (Sun et al. 2020) is a challenging testbed for autonomous driving, containing 158,361 training samples and 40,077 testing samples. The sampling intervals for KITTI and Waymo are set and 10, respectively.
Generic AL Baselines. We implemented the following five generic AL baselines of which the im- plementation details can be found in the supplementary material. (1) RAND: is a basic sampling method that selects N,, samples at random for each selection round; (2) ENTROPY (Wang & Shang 2014]: is an uncertainty-based active learning approach that targets the classification head of the de- tector, and selects the top N,- ranked samples based on the entropy of the sample's ss predicted label; (3) LLAL (Yoo & Kweon 2019): is an uncertainty-bast
method that adopts an auxiliary network to predict an indicative loss and enables select samples for which the model likely to produce wrong predictions; (4) CORESET (Sener & Savarese 2018): : is dliversitybbasee method perform- ing the core-set selection that uses the greedy furthest first search on both labeled and unlabeled embeddings at each round; and (5) BADGE (Ash et al.) (020): is a hybrid approach that samples instances that are disparate and of high magnitude when presented in a hallucinated gradient space.
Applied AL Baselines for 2D and 3D Detection. For fair comparison, we also compared three variants of the deep active learning method for 3D detection and adapted one 2D active detection method to our 3D detector. (6) MC-MI (Feng al., (219)) utilized Monte Carlo dropout associated with mutual information to determine the uncertainty of point clouds. (7) MC-REG: Additionally, to verify the importance of the uncertainty in regression, we design an uncertainty-basel baseline that determines the regression uncertainty via conducting M-round MC-DROPOUT stochastic passes at the test time. The variances of predictive results are then calculated, and the samples with the top-N, greatest variance will be selected for label acquisition. We further adapted two applied AL methods for 2D detection to a 3D detection setting, where (8) LT/C (Kao et al. 2018) measures the class- specific localization tightness, i.e., the changes from the intermediate proposal to the final bounding box and (9) CONSENSUS (Schmidt et al.) 2020) calculates the variation ratio of minimum IoU value for each Rol-match of 3D boxes.
3.2 COMPARISONS AGAINST ACTIVE LEARNING METHODS
Quantitative Analysis. We conducted comprehensive experiments on the KITTI and Waymo datasets to demonstrate the effectiveness of the proposed approach. The Kú and K2 are empiri- cally set to 300 and 200 for KITTI and 2, 000 and 1, 200 for Waymo. Under a fixed budget of point clouds, the performance of 3D and BEV detection achieved by different AL policies are reported in Figure 2 with standard deviation of three trials shown in shaded regions. We clearly observe that CRB consistently outperforms all of-te-of-the-ut AL methods by a noticeable margin, irrespec- tive of the number of annotated bounding boxes and difficulty settings. It is worth noting that, on the

---

Published as a conference paper at ICLR 2023
Table 1: Performance comparisons (3D AP scores) with generic AL and applied AL for detection on KITTI val set with % queried bounding boxes.
CAR
PEDESTRIAN
CYCLIST
AVERAGE
Method
HARD
MOD.
HARD
EASY
MOD.
HARD
EASY
MOD.
HARD
AL-Det Generie
CORESET
87.77
77.73
72.95
47.27
41.97
38.19
81.73
59.72
55.66
72.26
59.81
55.59
BADGE
89.96
75.78
70.54
51.94
46.24
40.98
84.11
62.29
58.12
75.34
61.44
56.55
LLAL
89.95
78.65
75.32
56.34
49.87
45.97
75.55
60.35
55.36
73.94
62.95
58.88
MC-REG
88.85
76.21
73.47
31.81
29.79
73.98
55.23
51.85
66.21
54.41
51.70
MC-MI
86,28
75.58
71.56
41.05
37.50
33.83
86,26
60.22
$6.04
71.19
57.77
53,81
CONSENSUS
90.14
78.01
74.28
56.43
49.30
44.80
78.46
55.77
53.73
75.01
61.09
$7.60
LTfe
73.87
83.72
63.21
75.88
63.23
58.89
CRB
90.98
79.02
74.04
64.17
54.80
50.82
86,96
67.45
63.56
80.70
67.81
62.81
(6PT5) Masterto
18
RANO
LIAL
MCRED
ENTROPY
ENTROPY MCRED
LIAL
MCR
Eh TROPY
CORESCT
CORESCT
BADKE
BADKE
BADGE
ORB
ORB
C13
100
1000
1500
2000
2530
1000 1600
500
1000
1500
2000
2500
2000 3000
10000
20000
30000
40000
63000
70
35
6BV Motecoo
35
JPHNN Levolz
BBY 1
RMC
LLAL
MCREO
CORESCT
LUAL
MGRE0
CONEECT
MCHEE
LLAL
CONESETT
BADGE
BADGE
CRB
CRB
500 1000 1500 2000 2520
3000 3500
500
3000 Numror Sourdi Boass
1000 1500
2000
2500
3800
10000
20000 30000
40020
50000
humburo Bounding Bosss
Nambal Bounding
Figure 2: 3D and BEV mAP (%) of CRB and AL baselines on the KITTI and Waymo val split.
KITTI dataset, the annotation time for the proposed CRB is 3 times faster than RAND, while achiev- ing a comparable performance. Moreover, AL baselines for regression and classification tasks (e.g., LlAL) for regression only tasks (e.g., MC-REG) generally obtain higher scores yet leading to higher labeling costs than the ceciaficaationoorinntt methods (e.g., ENTROPY).
Table rreports the major experimental results of the state-of-the-ee thrtt---ttttttttoo generic AL methods and applied AI approaches for 2D and 3D) detection the KITTI dataset. It is observed that LIALL and LT/C achieve competitive results, as the acquisition criteria adopted jointly consider the classification and regression task. Our proposed CRB improves the 3D mAP scores by 6.7% which validates the effectiveness of minimizing the generalization risk.
Qualitative Analysis. To intuitively understand the merits of our proposed active 3D detection strat- egy. Figure 33 demonstrates that the 3D detection results yielded by RAND (bottom left) and CRB selection (bottom right) from the corresponding image (upper row). Both 3D detectors are trained under the budget of 1K annotated bounding boxes. False positives and corrected predictions are in- dicated with red and green boxes. is observed that, under the same condition, CRB produces more accurate and more confident predictions than RAND. Besides, looking at the cyclist highlighted in the orange box in Figure 3. the detector trained with RAND produces a significantly lower confi- dence score compared to approach. This confirms that the samples selected by CRB are aligned better with the test cases. More visualizations can be found in the supplemental material.
3.3 ABLATION STUDY
Study of Active Selection Criteria. Sableerreporee the performance comparisons of six variants of the proposed CRB method and the basic random selection baseline (I-st row) on the KITTI dataset. We report the 3D and BEV mAP metrics at all difficulty levels with 1,000 bounding boxes annotated. We observe that only applying GPDB (4-th row) produces 12.5% lower scores and greater variance than the full model (the last row). However, with CLS (6-th row), the performance increases by approximately 10% with the minimum variance. This phenomenon evidences the importance of optimizing the discrepancy for both classification and regression tasks. It's further shown that

---

Published as a conference paper at ICLR 2023
Table 2: Ablative study of different active learning criteria on the KITTI val split. 3D and BEV AP
scores (%) are reported when 1,000 bounding boxes are annotated.
3D Detection mAP
BEV Detection mAP
CLS
RPS
GPDB
EASY
MODERATE
HARD
EASY
MODERATE
HARD
70.70
58.27
54.69-1.30
75.37
64.5441.08
61.36+1161
77. 77.7621.70
64.5611.39
59.54.1.13
81.07=1.67
69.76
65.0 65.01
V
14.13
61.65
57.70
78.85_2.31
67.07
63.47a1.21
V
69.11 13.22
56.12+12.74
52.85
73.57=10.44
62.49+10.62
59.45
V
76. 76.19
62.81
58.0321.18
80.73_0.92
68.67
64.422-0.2
76. 00.78
64..00a1.07
59.68
50.71=0.22
70.01
65.47
79.03 11.39
65.86
61.06
82.60
70.74
66.41 +1.22
A case study of active 3D detection performance RAND (bottom left) and CRB (bottom right) under the budge of 1,000 annotated bounding boxes. False positive (corrected predictions) are highlighted in red (green) boxes. The orange box denotes the detection with low confidence.
Figure 3:
moving any selection criteria from the proposed CRB triggers a drop on mAP scores, confirming the importance of each in sample-efficient AL strategy.
Sensitivity to Prototype Selection. We examine the sensitivity of performance to different proto- type selection methods used in the RPS module the KITTI dataset (moderate difficulty level). Particularly, In Figure T (right), we show the performance of our approach using different prototype selection methods of the Gaussian mixture model (GMM), K-MEANS, and K-MEANS++. To fairly reflect the trend in the performance curves, we run two trials for each prototype selection approach and plot the mean and the variance bars. K-MEANS is slightly more stable than the other two, with higher time complexity and better representation learning. It is observed that there is very little variation (~ 1.5%) in the performance of our approach when using different prototype selection methods. This confirms that the CRB'S superiority over existing baselines is not coming from the prototype selection method.
Figure 4: Results KITTI val set with varying KDE bandwidth h (left) and pro- totype selection approaches (right) with increasing queried bounding boxes.
30 NAPPN) Modinda
50 (1 Mastrate
80
GAIM
hw5
Koldwas
I-3
K-Meara**
Number Selected Sounding Booes
Number Selected Bourding Booax
Sensitivity to Bandwidth h. Figure depicts the results of CRB with the bandwidth h varying in {3. 5, 7,9). Choosing the optimal bandwidth value h. can avoid under-smoothing (h < h²) and over-smoothing (h h²) in KDE. Except h = 3 which yields large variation, CRB with the bandwidth of all other values reach similar detection results within the 2% absolute difference on 3D mAP. This evidences that the CRB is robust different values of bandwidth.
Sensitivity to Detector Architecture. We validate the sensitivity of performance to choices of one- stage and two-stage detectors. Table 4 reports the results with the SECOND detection backbone on the KITTI dataset. With only 3% queried 3D bounding boxes, it is observed that the proposed CRB approach consistently outperforms the SOTA generic active learning approaches across a range of detection difficulties, improving 4.7% and 2.8% 3D mAP and BEV mAP scores.
8

---

Published as a conference paper at ICLR 2023
Table 3: Performance comparisons on KITTI val set w.r. varying thresholds Kj and K2 after two rounds of active selection (8% point clouds. 1% bounding boxes). Results are reported with 3D AP with 40 recall positions. indicates the reported performance of the backbone trained with the full
labeled set (100%).
CAR
PEDESTRIAN
CYCLIST
AVERAGE
K1
X2
BASY
MOD.
HARD
BASY
MOD.
HARD
EASY
MOD.
HARD
EASY
MOD.
HARD
500
400
90.04
79.08
74.66
57.11
51.10
51.12
81.97
63.40
59.62
76.50
64.53
60.10
500
300
90.98
79.02
74.04
61.17
54.80
50.82
86.96
67.45
63.56
80.70
67.81
62.81
400
300
91.30
79.21
74.00
62.93
55.67
49.27
79.02
60.50
56.74
77.75
65.13
60.00
300
200
90.45
78.81
73.44
65.00
55.91
51.12
84.82
65.77
61.53
80.09
67.32
62.05
PV-RCNN" 92.56
84.36
82.48
64.26
56.67
51.91
88.88
71.95
66.78
81.75
70.99
67.06
Sensitivity Analysis of Thresholds K₂ and K2. We examine the sensitivity of our approach to different values of threshold parameters Ki and K2. We report the mean average precision (mAP) on the KITTI dataset, including both 3D and BEV views at all difficulty levels. We check four possible combinations of K,, and K22 and show the results in Table 3
We can observe that MODERATE and HARD levels. there is only 3.28% and 2.81% fluctuation on average mAP. In the last row, we further report the accuracy achieved by the backbone detector trained with all labeled training data and larger batch size. With only 8% point clouds and 1% annotated bounding boxes, CRB achieves a comparable performance to the full model.
3.4 COMPLEXITY ANALYSIS
Table shows the time complexity of training and active selection for different active learning approaches. n indicates the total number of unlabeled point clouds, N, is the quantity selected, and E' is the training epochs, with N, < n. We can clearly observe that, training stage, the complexity of all AL strategies is O(En), except LLAL that needs extra epochs Eσ to train the loss prediction module. At the active selection stage, RAND randomly generates N, indices to retrieve samples from the pool. CORESET computes pairwise distances between the embedding of selected samples and unlabeled samples that yields the time complexity of (N,n). BADGE iterates through the gradients of all unlabeled samples passing gradients into K-MEANS++ algorithm, with the complexity of (N,,)) bounded by K-MEANS++. Given K1, se N,,, the time complexity of our method is On log + 2N?).. with On log))) being the complexity of sorting the entropy scores in CLS, and O(N?) coming from Ky-medoids and greedy search in Rps and GPDB. Note that, in our case, O(n log 2N? < O(N,)). The complexity of simple ranking- based baselines is On ln)( due to sorting the sample acquisition scores. Comparing our method with recent oate-offthe-arts, LLAL has the highest training complexity, and BADGE and CORESET have the highest selection complexity. Unlike the existing baseline, training and selection complexities of the proposed CRB are upper bounded by the reasonable asymptotic growth rates.
Table 4: AL Results with one-stage 3D detector SECOND.
Table 5: Complexity Analysis.
3D Detection mAP
BEV Detection mAP
AL Stratogy
Training
Solection
EASY
MODERATE
HARD
EASY
MODERATE
HARD
RAND
O(En)
O(N.)
ENTROPY
O(En)
Ol Olglog))
RAND
75.23
60.83
56.55
80.20
67.56
63.30
MC-REG
O(En)
Oin login)
LLAL
72.02
58.96
$4.21
79.50
66.82
62.48
Oin log m))
CORESET
74.74
79.71
CORESET
O(Ex)
((N,
BADGE
75.38
61.65
56.72
80,81
68.83
64.17
BADGE
O(En)
O(N,o)
CRB
78.96
64.27
$9.60
83.28
70.49
66.09
CRB
O(Ex)
( log n + 2N2)
4
DISCUSSION
This paper studies three novel criteria for sample-efficient active 3D object detection, that can effec- tively achieve high performance with minimum costs of 3D box annotations and runtime complexity. We theoretically analyze the relationship between finding the optimal acquired subset and mitigating the sets discrepancy. The framework is versatile and can accommodate existing AL strategies to pro- vide in-depth insights into heuristic design. The limitation of this work lies in a set of assumptions made on the prior distribution of the test data, which could be violated in practice. For more dis- cussions, please refer to Sec. A. in Appendix. In contrast, it opens opportunity of adopting our framework for active domain adaptation, where the target distribution is accessible for alignment. Addressing these two avenues is left for future work.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by Australian Research Council (CE200100025),
REPRODUCIBILITY STATEMENT
The source code of the developed active 3D detection toolbox is available in the supplementary material, which accommodates various AL approaches and one-stage and two-stage 3D detectors. We specify the settings of hyper-parameters, the training scheme and the implementation details of our model and AL baselines in Sec. of the supplementary material. We show the proofs of Theorem 2.T 2.inn in Sec. C followed by the overview of the algorithm in Sec. D in the supplementary material. We repeat the experiments on the KITTI dataset 3 times with different initial labeled sets and show the standard deviation in plots and tables.
ETHICS STATEMENT
Our work may have a positive impact communities reduce the costs of annotation, computa- tion, and carbon footprint. The high-performing AL strategy greatly enhances the feasibility and practicability of 3D detection in critical yet data-scarce fields such medical imaging. We did use crowdsourcing and did not conduct research with human subjects in our experiments. We cited the creators when using existing assets (e.g., code, data, models).
REFERENCES
Syeda Mariam Ahmed, Yan 7hi Tan, Chee-Meng Chew, Abdullah Mamun, and Fook Seng Wong- Edge and corner detection for unorganized 3d point clouds with application to robotic welding. In Proc. International Conference on Intelligent Robots and Systems (IROS), pp. 7350-7355, 2018.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy. John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In Proc. International Confer- ence on Learning Representations (ICLR), 2020.
William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. The power of ensem- bles for active learning in image classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9368-9377, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Journal of Machine Learning, 79 (1-2):151-175, 2010.
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 9583-9592, 2021.
Jiwoong Choi, Ismail Elezi, Hyuk-Jae Lee, Clément Farabet, and Jose M. Alvarez. Active leamn ing for deep object detection via probabilistic modeling. In Proc. International Conference on Computer Vision (ICCV), pp. 10244-10253, 2021.
Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Ros- tamizadeh, and Sanjiv Kumar. Batch active learning scale. In Proc. Annual Conference on Neural Information Processing (NeurIPS), pp. 11933-11944, 2021.
Boyang Deng, Charles R. Qi, Mahyar Najibi, Thomas A. Funkhouser, Yin Zhou, and Dragomir Anguelov. Revisiting 3d object detection from an egocentric perspective. In Proc. Annual Con- ference on Neural Information Processing (NeurIPS), pp. 26066-26079, 2021.
Pan Du, Suyun Zhao, Hui Chen, Shuwen Chai, Hong Chen, and Cuiping Li. Contrastive coding for active learning under class distribution mismatch. In Proc. International Conference on Computer Vision (ICCV), pp. 8907-8916 2021.
10

---

Published as a conference paper at ICLR 2023
Di Feng, Xiao Wei, Lars Rosenbaum, Atsuto Maki, and Klaus Dietmayer. Deep active learning for efficient training of a lidar 3d object detector. Proc. Intelligent Vehicles Symposimm, (IV), pp- 667-674, 2019.
Yoav Freund, H. Sebastian Seung. Eli Shamir, and Naftali Tishby. Information, prediction, and query by committee. In Proc. Annual Conference Neural Information Processing (NeurIPS), pp. 483-490, 1992.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Proc. International Conference on Machine Learning (ICML), volume 48, pp. 1050-1059, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proc. International Conference on Machine Learning (ICML), volume 70, pp- 1183-1192, 2017.
Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ömer Arik, Larry S. Davis, and Tomas Pfister. Consistency- based semi superviie active learning: Towards minimizing labeling cost. In Proc. European Conference on Computer Vision (ECCV), volume 12355, pp. 510-526, 2020.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354-3361, 2012.
Denis A. Gudovskiy, Alec Hodgkinson, Takuya Yamaguchi, and Sotaro Tsukizawa. Deep active learning for biased datasets via fisher kernel self-supervision. Proc. IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp.
2020.
Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. CoRR, abs/1 1 12.5745, 2011.
Chieh-Chi Kao, Teng- Yok Lee, Pradeep Sen, and Ming- Yu Liu. Localization-awan active learning for object detection, In Proc. Asian Conference on Computer (ACCV), pp. 506-522, 2018.
Daniel Kifer, Shai Ben-David, and Johannes Gehrke Detecting change in data streams. In (e)Proceedings of International Conference on Very Large Data Bases ((VLDB), pp. 180-191. Morgan Kaufmann, 2004.
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational ad- versarial active learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8166-8175, 2021a.
Yoon- Yeon-Yeong Kim, Kyungwoo Song. JoonHo Jang, and Il-Chul Moon. LADA: look-ahead data acquisition via augmentation for deep active learning. In Proc. Annual Conference Neural Information Processing (NeurIPS), pp. 19919-22930, 2021b.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch ac- quisition for deep bayesian active learning. In Proc. Annual Conference on Neural Information Processing (NeurIPS), pp. 7024-7035 2019.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In Proc. International Conference on Computer Vision (ICCV), pp. 9254-9263, 2021.
Shuang Ma. Zhaoyang Zeng. Daniel McDuff, and Yale Song. Active contrastive learning of audio- visual video representations. In Proc. International Conference Learning Representations (ICLR), 2021.
David J. C. MacKay. Information-based objective functions for active data selection. Journal of Neural Computation, 4(4):590-604 604, 1992.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Proc. Conference Learning Theory (COLT), 2009.

---

Published as a conference paper at ICLR 2023
Hector A. Montes, Justin Le Louedec, Grzegorz Cielniak, and Tom Duckett. Real-time detection of broccoli crops in 3d point clouds for autonomous robotic harvesting. In Proc. International Conference on Intelligent Robots and Systems (IROS), pp. 10483-10488, 2020.
Hieu Tat Nguyen and Arnold W. M. Smeulders. Active learning using pre- preclustering. In Carla E.
Brodley (ed.), Proc. International Conference on Machine Learning (ICML), 2004.
Adam M. Oberman and Jeff Calder. Lipschitz regularized deep neural networks converge and gen- eralize. CoRR, abs/ 808,00940, 2018.
Robert Pinsler, Jonathan Gordon, Eric T. Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. In Proc. Annual Conference on Neural Information Processing (NeurIPS), pp. 6356-6367, 2019.
Jun Qi, Jun Du, Sabato Marco Siniscalchi, Xiaoli Ma, and Chin-Hui Lee. On mean absolute error for deep neural network based vector-to-vector regression. IEEE Signal Processing Letters, 27: 1485-1489, 2020.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proc. International Conference Machine Learning (ICML), pp. 441 448, 2001.
Sebastian Schmidt, Qing Rao, Julian Tatsch, and Alois C. Knoll. Advanced active learning strategies for object detection. In Proc. Intelligent Vehicles Symposium, (IV), pp. 871-876, 2020.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In Proc. International Conference on Learning Representations (ICLR), 2018.
Claude F. Shannon. mathematical theory of communication. The Bell System Technical Journal, 27(3):379-4233 1948.
Feng Shi and Yu-Feng Li. Rapid performance gain through active model reuse. In Proc. Interna- tional Joint Conference on Artificial Intelligence (ICAA)) pp. 3404-3410 2019.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proc. IEEE Conference on Computer Vision and Pattern Recogni- fion (CVPR), pp. 770-779. Computer Vision Foundation IEEE, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. PV-RCNN: poxet-voxel feature set abstraction for 3d object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10535, 2020.
Weishi Shi and Qi Yu. Integrating bayesian and discriminative sparse kernel machines for multi- class active learning. In Proc. Annual Conference on Neural Information Processing (NeurIPS),
pp. 2292-2291, 2019.
Yawar Siddiqui, Julien Valentin, and Matthias Nießner. Viewal: Active learning with viewpoint entropy for semantic segmentation. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9430-9444 2020.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. Proc. International Conference on Computer Vision (ICCV), pp. 5971-5980, 2019.
Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene under- standing benchmark suite. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 567-576, 2015.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proc. IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 2443-2451, 2020.
12

---

Published as a conference paper at ICLR 2023
Sing-Peng Tang, Xiu-Shen Wei, Borui Zhao, and Sheng-Jun Huang. Qbox: Partial transfer learning with active querying for object detection. Journal of IEEE Transactions on Neural Networks and Learning Systems, pp. 1-13, 2021. doi: 10.1 10.1009/
Toan Tran, Thanh-Toan Do, Ian D. Reid, and Gustavo Carneiro. Bayesian generative active deep learning. In Proc. International Conference on Machine Learning (ICML), volume 97, pp. 6295- 6304, 2019.
Dan Wang and Yi Shang. new active labeling method for deep learning. In Proc. International Joint Conference on Neural Networks (IJCNN, pp. 112-119, 2014.
Jun Wang, Shiyi Lan, Mingfei Gao, and Larry S. Davis. Infofocus: 3d object detection for tonomous driving with dynamic information modeling. In Proc. European Conference on Com-
puter Vision (ECCV), volume 12355, pp. 405-420, 2020.
Keze Wang. Dongyu Zhang, Ya Li, Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. Journal of IEEE Transactions on Circuits and Systems for Video Technology, 27(12):2591-2666 2017.
Li Wang, Ruifeng Li, Jingwen Sun, Xingxing Liu, Lijun Zhao, Hock Soon Seah, Chee Kwang Quah, and Budianto Tandianus. Multi-view fusion-based 3d object detection for robot indoor scene perception. Sensors, 19(19):4092, 2019.
Xiao Wang, Xiang Xiang, Baochang Zhang, Xuhui Liu, Jianying Zheng, and Qingl ei Hu, Weakly supervised object detection based on active learning. Journal of Neural Processing Letters, pp- 1-15, 2022.
Jiaxi Wu, Jiaxin Chen, and Di Huang. Entropy-based active learning for object detection with pro- gressive diversity constraint. In Proc. IEFE Conference on Computer Vision and Puttern Recog- nition (CVPR), pp. 9387-9396. IEEE, 2022.
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. STD: sparse-to-dense 3d object detector for point cloud. In Proc. IEEÉ Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1951-1960. IEFE, 2019.
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11037- 11045. Computer Vision Foundation / IEEE, 2020.
Donggeun Yoo and In So Kweon. Learning loss for active learning. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 93-102, 2019.
Tianning Yuan, Fang Wan, Mengying Fu, Jianzhuang Liu, Songcen Xu, Xiangyang Ji, and Qixiang Ye. Multiple instance active learning for object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5330-5339, 2021.
Beichen Zhang. Liang Li, Shijie Yang. Shuhui Wang. Zheng-Jun Zha, and Qingming Huang. State- relabeling adversarial active learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2020.
13


Reviews:
[{"review_id": "kHKzbvnQb2", "reviewer": "Reviewer_cyaq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the task of 3D object detection with only a small portion of labeled data. Conventional active learning-based approaches show a promising solution, but they fail to balance the trade-off between point cloud informativeness and box-level annotation costs. To overcome this limitation, this paper jointly investigates three novel criteria for point cloud acquisition, i.e., label conciseness, feature representativeness and geometric balance. Extensive experiments are conducted on KITTI and WOD benchmarks to demonstrated the superiority of the proposed method.", "summary_of_the_review": "This paper proposes a novel CRB framework for 3D object detection with active learning. The proposed method is novel, and achieves superior performance on both KITTI and WDO benchmarks. Although there are some questions mentioned above, I think the overall quality of this paper is good.", "strength_and_weaknesses": "**Strength**\n1. The motivation of this paper is solid, and the proposed framework CRB is novel.\n2. The paper is well written.\n3. Extensive experiments are conducted to demonstrate the superiority of the proposed method.\n\n**Weakness**\n1. In Sec. 2.1, I am very confused how to annotate a point cloud $\\mathcal{P}$ with a bounding box $\\mathcal{B}$ and its associated box label $\\mathcal{Y}$?\n2. From Fig. 2 in the main paper and Fig. 5 in the Suppl., the proposed method seems to get inferior results when a low budget is required. Considering the scale of Waymo is bigger than KITTI, what is the performance on WOD under the same setting as Tab.1?\n3. In B.2, there are several hyper-parameters about active learning protocols. So how to set values of them? From Tab.3, results are sensitive to the setting of $\\mathcal{K}1$ and $\\mathcal{K}2$, does both thresholds need to be carefully tuned when different number of annotated bounding boxes are available?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hA5VRZDY1PT", "reviewer": "Reviewer_8qfL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed three new criteria for 3D box-level active learning in LiDAR object detection. The first part(Eq. 4) encourages selecting more class-balance point clouds. The second part advocates selecting point clouds with more diverse gradients w.r.t. pseudo classification and regression labels. The las thet part favors point clouds with similar object point density as of the labelled set. Empirical studies showed the proposed three components are mostly effective and out-perform existing active learning methods on some datasets.", "summary_of_the_review": "It is easy to see that the authors put a lot of efforts in this work and tried to connect the proposed selection criteria with some theoretical supports. Although there are some errors and unclear writings in the method part, but the proposed methods are backed up by at least solid experiments on the KITTI datasets, so I am leaning to recommend for its acceptance.", "strength_and_weaknesses": "Pros:\n1. The authors gave a good summary of existing active learning methods for LiDAR object detection, which highlighted the significance and popularity of the problem under study.\n2. The authors developed an active-3D-det toolbox which could be quite helpful for setting fair and rigorous baselines for future researches.\n3. Experiments on KITTI showed promising results.\n\nCons:\n1. There are quite some factual errors in the method part of this work.\n    1. PointNet series did not encode free-point cloud into a 3D tensor. On the bottom of Page 2, “Mainstream 3D object detectors encode free-form point clouds P into a regular feature map x ∈ R W ·L·F directly (Qi et al., 2017) or by voxelization (Shi et al., 2020), with W, L, F representing width, length, and channels of the feature map.” \n    2. Wrong assumptions about loss function. The authors assumed that “The loss functions for classification and regression are assumed to be Lipschitz continuous”, but a simple binary cross entropy loss is not Lipschitz continuous for p in (0, beta]. Also I failed to connect the Lipschitz continuous of loss function with anything in Section 2.2.\n2. Section 2.2 is quite vague to me, I don’t how Theorm 2.1 is connected to Eq 1 and what are Ps in Eq 1\n3. A uniform prior of target labels as assumed in Eq. 2 is too far from the observed reality in LiDAR object detection \n4. Stage 2 in section 2.3 is quite hard to understand. Each step in Eq. 7 seems too big a leap for me and proper discussion are not made in the text part.\n5. Only the KITTI experiments showed promising results. For the Waymo experiments in Figure 2, CRB is out-performed by LLAL and ENTROPY in the low data regime while almost on par with RAND baseline in the high data regime. I find there are implementations for other datasets like nuScenes and Lyft in the supplementary codes, which makes me wonder if the method generalizes well across different datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bwCJf7TVfs", "reviewer": "Reviewer_BMpC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of active 3D object detection from point clouds. The main problem addressed in this paper is how to design a proper criterion for cost-efficient sample acquisition. In this work, the authors jointly investigate three novel criteria in the framework CRB for point cloud acquisition (label conciseness, feature representativeness and geometric balance) which hierarchically filters out the point clouds of redundant 3D bounding box labels. \n\nIn particular, (1)To alleviate the issues of label redundancy and class imbalance, the entropy of bounding box label predictions are used to pick top *K*1 point clouds. (2)Monte Carlo dropout and the hypothetical labels by averaging predictions from multiple stochastic forward passes are used for considering the impact of classification and regression objectives on gradients. (3) Finally, to maintain the *geometric balance* property, we minimize the KL divergence between the marginal distributions of point cloud density of each predicted bounding box. \n", "summary_of_the_review": "This paper is overall a goog paper, my main concern is the above weakness .\nThe author's response addressed my concerns well, so my final recommendation is accept", "strength_and_weaknesses": "Strengths:\n\nThe paper is well written, the experiments are convincing, and the proposed method is simple to implement. Figure 1 nicely illustrates the overall framework of the method.\n\nThe work presents an exhaustive experimental evaluation which demonstrates the effectiveness of the presented approach on KITTI and Waymo datasets. \n\nAblation study (Table 2) and complexity analysis  (Table 4) support the experimental well.\n\nWeaknesses:\n\nAlthough the complexity analysis is given in the article, no specific time cost is given.\n\nThese experiments are all carried out under the autonomous driving dataset. Does this structure work for the 3D detection task of the indoor dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "941P21eQ0ew", "reviewer": "Reviewer_S6aF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores how activate learning can help reduce annotation costs in the field of object detection of 3D point clouds. Specifically, to achieve efficient active learning with limited fixed annotation budgets, three selection criteria termed CRB are proposed to learn better sample acquisition of the 3D boxes annotation. Extensive experiments on KITTI and Waymo validate the effectiveness of the proposed method. The paper is well-written and easy to follow. The experimental results demonstrate the effectiveness of the proposed hierarchical active learning scheme in active learning for 3D object detection.\n", "summary_of_the_review": "I'm not quite familiar with this topic, but this paper looks very comprehensive and solid, so I'm inclined to borderline accept.", "strength_and_weaknesses": "### Srength: \n- Activate learning in point cloud object detection is interesting and novel. I believe this paper can provide insights and inspiration to the related works in the 3D object detection area.\n\n- The theoretical analysis is sufficient and helpful to understand the method, and also provide support for the final performance validation.\n\n- The experiments are extensive and can validate the effectiveness of the proposed framework.\n\n\n### Weaknesses: \n- I would suggest the authors clearly clarify the major difference of the paper with existing works on object detection and point clouds processing. It seems that this paper is analyzed in an general optimized view, but it will be better if the author can explicitly illusrate the innovation and the fundamental difference with exisiting techniques.\n\n- For related work, I would suggest that the author add literature on 2D Detection and some of the latest works in 3D point cloud detectors. Just name a few: \n\n\t1. QBox: Partial Transfer Learning With Active Querying for Object Detection [2021, TNNLS]\n\t2. Towards Dynamic and Scalable Active Learning with Neural Architecture Adaption for Object Detection [2021, BMVC]: Add NAS into the AL loops.\n\t3. Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint [2022, CVPR]\n\t4. Weakly Supervised Object Detection Based on Active Learning [2022, NPL]\n\t5. Active Learning Strategies for Weakly-Supervised Object Detection [2022]\n\t6. Label-Efficient Point Cloud Semantic Segmentation: An Active Learning Approach [2021, CVPR]\n\t7. ReDAL: Region-based and Diversity-aware Active Learning for Point Cloud Semantic Segmentation [2021, ICCV]\n\t8. Active Learning for Point Cloud Semantic Segmentation via Spatial-Structural Diversity Reasoning [2022]\n\n- As for the experimental results in Table 1, it is a little bit counterintuitive: the proposed CRB with 1% labels can perform better than the general full-label training model. In my view, the active learning methods can achieve the performance very close to the full-label training model. The authors are suggested to explicitly clarify why the performance is even better than fully supervised model.\n\n-More experiments on various detectors (e.g., SECOND, PV-RCNN) are required to illustrate the effectiveness of the proposed criteria for general active 3D object detection. Additionally, indoor scenarios such as ScanNet should also be taken into consideration.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%