STONE: A Submodular Optimization Framework for Active 3D Object Detection
Authors: RUIYU MAO
Sarthak Kumar Maharana
Rishabh K Iyer
Yunhui Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Active learning
3D object detection
Primary area: active_learning
TL;DR: Unified active 3D object detection framework based on submodular optimization.

Abstract: 3D object detection is fundamentally important for various emerging applications, including autonomous driving and robotics. A key requirement for training an accurate 3D object detector is the availability of a large amount of LiDAR-based point cloud data. Unfortunately, labeling point cloud data is extremely challenging, as accurate 3D bounding boxes and semantic labels are required for each potential object. This paper proposes a unified active 3D object detection framework, for greatly reducing the labeling cost of training 3D object detectors. Our framework is based on a novel formulation of submodular optimization, specifically tailored to the problem of active 3D object detection. In particular, we address two fundamental challenges associated with active 3D object detection: data imbalance and the need to cover the distribution of the data, including LiDAR-based point cloud data of varying difficulty levels. Extensive experiments demonstrate that our method achieves state-of-the-art performance with high computational efficiency compared to existing active learning methods. The code is available at [https://github.com/RuiyuM/STONE](https://github.com/RuiyuM/STONE)

STONE:
A Submodular Optimization Framework for Active 3D Object Detection
Ruiyu Mao Sarthak Kumar Maharana Rishabh K Iyer
Yunhui Guo
Department of Computer Science
The University of Texas Dallas, Richardson, TX, USA
frxm210041, skm200005, rishabh. iyer, yunhui.guo/@uttallle gow.euualllll.....ocoooo edu
Abstract
3D object detection is fundamentally important for various emerging applications, including autonomous driving and robotics. A key requirement for training an accurate 3D object detector is the availability of a large amount of LiDAR-basee point cloud data. Unfortunately, labeling point cloud data extremely challenging, accurate 3D bounding boxes and semantic labels are required for each potential object. This paper proposes unified active 3D object detection framework, for greatly reducing the labeling cost of training 3D object detectors. Our framework is based on a novel formulation of submodular optimization, specifically tailored to the problem of active 3D object detection. In particular, we address two fun- damental challenges associated with active 3D object detection: data imbalance and the need to cover the distribution of the data, including LiDAR- R-based point cloud data of varying difficulty levels. Extensive experiments demonstrate that our method achieves state-of-the-art performance with high computational effi- ciency compared to existing active learning methods. The code available https://jthbb..co//uyyu///////
Introduction
In many emerging applications such autonomous driving, is critical to localize objects in 3D scene for accurate scene understanding 8 [56]. This is usually achieved by using a 3D detection model based on LiDAR point cloud data with oriented bounding boxes and semantic labels. While highly accurate recognition and localization of objects can be achieved due to recent advancements in deep learning, this performance often comes the expense of requiring large volume of labeled point cloud data, which much more costly to collect compared to typical RGB images 51 52
Active learning (AL.) is the standard method for reducing labeling costs in machine learning AL often starts with a small labeled set and iteratively selects the most informative samples for label acquisition from large pool of unlabeled data, given a labeling budget. The informativeness, of a selected sample, can be measured in various ways. For example, by computing the sampling uncertainty measured by maximum Shannon entropy [48] 32 or estimated model changes or by using greedy
45
finding the most representative samples to avoid sample redundancy [36 17 40
coreset algorithms [4 18 clustering- based approaches [38 57
42
Active learning has proven highly effective reducing labeling costs for recognition tasks, but its application in LiDAR-basee object detection remains limited and under-explored [11 22 43 Compared to standard recognition tasks, there are two fundamental challenges: 1) Depending on the specific scene, each category involves different difficulty levels (EASY, MODERATE, HARD), which are determined by the size, occlusion level, and truncation of 3D objects. Ideally, the selected labeled point cloud data should include varying difficulty levels. 2) Each 3D scene contain multiple objects, leading to highly imbalanced label distributions in the point cloud data. For example, most
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

point clouds include cars, but not cyclists. Addressing data imbalance is thus crucial for selecting informative point clouds to train the 3D object detector.
In a recent study, CRB 335 proposed three stages to label unlabeled point clouds hierarchically,
ensuring they are concise, representative, and geometrically balanced. One of the critical components of CRB is to achieve label balance in individual point clouds. However, it cannot guarantee the label distribution is balanced across all the labeled point clouds. More recently, KECOR (34] was proposed for characterizing sample informativeness in both classification and regression tasks with a unified measurement. The main idea is to select subset of point clouds that maximizes the kernel coding rate. This approach enables the selection of representative samples from the unlabeled point cloud. However, does not address the issue of data imbalance.
In this paper, we propose a unified submodular optimization framework, called STONE, for active 3D object detection, to address limitations in existing methods. Our framework leverages submodular functions [12] a classical tool for measuring set quality, due to its well-known property of diminishing returns. Building on these fundamental results, we introduce a novel formulation of submodular optimization for active 3D object detection. This formulation not only ensures that the selected point cloud achieves maximal coverage of the unlabeled point cloud but also addresses the issue of data imbalance. Based on the formulation, we propose a two-Stage algorithm. Firstly, we select representative point clouds using submodular function based on gradients computed from hypothetical labels using Monte Carlo dropout [13]. Secondly, we employ a greedy search algorithm to select unlabeled point clouds, aiming to balance the data distribution as measured by another submodular function. Our work makes the following core contributions. 1) We introduce the first submodular optimization framework for active 3D object detection, addressing two fundamental challenges in this domain. 2) We develop simple and efficient two-stage algorithm within the framework for selecting representative point clouds and addressing the issue of data imbalance. 3) We extensively validate the proposed framework on real-world autonomous driving datasets, including KITTI 15] and Waymo Open dataset [52], achieving state-of-the-art performance in active 3D object detection. Furthermore, additional results in active 2D object detection demonstrate the high generalizability of our proposed method.
2 Related Works
which involves alleviating labeling annotation costs by selecting the most representative samples from pool of unlabeled data, all the while not comprising model performance. Broadly speaking, AL can be categorized into two strategies uncertainty sampling and representative/diversitt sampling. Algorithms under represeettaive//diversig sampling select subsets of data that act as stand-ins or surrogates for the entire dataset []] Prior works have explored coreset-based subset selection 44 [8). clustering algorithms [38, 55 3]. and generative adversarial lcarning (16). In uncertainty sampling [3]] samples are selected by maximizing the Shannon entropy [32] of the posterior probability, minimizing the model's subsequent training eгror through variance reduction [6]. based on the maximum gradient magnitude [46), and considering the disagreement among a committee of model hypotheses [47 [54]. Hybrid sampling strategies are proposed recently [25 I 26 20), combining
Active Learning (AL) has been a deeply studied topic in machine learning 7 45
representativv/ddierrsss and uncertainty sampling, like BADGE [.. where the authors propose to use the gradient magnitude with respect to the classifier's parameters as measure of uncertainty. This approach selects samples whose gradients cover a wide range of directions.
Active Learning for 3D Object Detection. While AL has been actively studied and applied to image classification and regression tasks, it has recently been garnrring interest in the 3D object detection community. Initial works like MC-MI (II] make use of Monte-Cario (MC)-dropout [14] and Deep Ensembles 300 compute the Shannon entropy in the predicted labels and mutual information between class predictions and model parameters. CONSENSUS 4 estimates uncertainty using ensembles for 2D/3D object detection. While these works relied on generic metries to measure prediction uncertainty, two other works on 3D object detection, CRB 351 and KECOR [34], were proposed recently. In CRB, the method greedily searches and samples unlabeled point clouds that exhibit concise labels, representative features, and geometric balance. KECOR shows that, for sample selection, maximizing the kernel coding rate is beneficial and improves over task specific AL methods for 3D detection, at fast-running times.
2

---

Submodular Optimization. Submodular functions (see Section and thereby optimization, have found wide applications in the selection of data subsets [58] 29 active learning [27 28]. speech
recognition [59] 60], continual learning [53], and hyper-parameterr tuning (24 The effectiveness
of submodular functions primarily stems from their ability to model diversity via clustering in
representation learning. This enhances their capacity to discriminate between different data subsets
or classes while ensuring the preservation of unique and relevant features.
3 Background
3D Object Detection. LiDAR-based 3D object detection aims to localize and recognize objects in point cloud data using oriented bounding boxes and semantic labels. Point clouds are typically generated by LiDAR sensors by emitting pulsed light waves into the surrounding environment and then analyzing the time difference of receiving the bounced-back pulses. The newly generated orderless point P; = {(T, y/ r)} is represented by zyz spatial coordinates and reflectance r. Based on the point cloud data, ground
bounding boxes can be labeled as B, = {b: Ji where b, E I=? which include the relative center zyz spatial coordinates the object ground planes, the box size, the heading angle, and the box label, with their associated bounding box semantic labels C, = { N; represents the number of bounding boxes in the i-th point cloud. In 3D object detection, a 3D objector Mg with parameters 9 first extracts logits fi from the raw points. These logits are then processed, resulting in modified logits f/: which are subsequently used by classification head for predicting the semantic label. Additionally, a regression head will be used to predict the bounding boxes. Finally, the output of the 3D detector the predicted bounding box [ which includes the semantic labels {e TE i== for each bounding box, where c; E (1.2..... C} with C' being the total number of semantic classes, e.g., cars, eyclists and pedestrians.
N:
Active Learning for 3D Object Detection. Active 3D object detection aims to reduce the labeling cost for training the 3D detector. In the beginning stage, small number of labeled point clouds Dr. are randomly selected from the unlabeled data pool Dy to train the backbone 3D object detection model. During active learning, for each query iteration method will select I' number of unlabeled point clouds from Dy. These selected point clouds are then given to human annotator for labeling the bounded boxes and semantic labels, and the labeled point clouds Dş = {P,. B₁ (N,) along with the previously selected point clouds are combined to create selected point clouds Dz Dz U Ds for the backbone model to re-train. When the query round Q is reached or the total queried bounding box number, or budget, NQ is reached, it will stop repeating the above process, where No = > Ng-
{1, .....
Q}. 21.,.... given active learning
Submodular Functions and Optimization. A set function f. in the discrete space, defined as follows: f:20 F. where 2D is a power set of D f coming from the discrete space in R²",, is considered submodular if it satisfies the following property of diminishing returns,
S(A) + S(B)= Z [)AUB) + BB
A. C D. A similar alternative property is
f(AU (z)) f(A) 2 ((BU f(B)
(2)
A, C D, A and x & (2). Additionally. f is strictly monotone if f(A) < (BB for ACB. An important example of a submodular function, in the context of active learning, is the Shannon entropy [8). We refer the reader to 22 for proof of the submodularity of the Shannon entropy (48).
In order to maximize the representativeness of samples or subsets, a natural solution is to maximize f in the form of max f (A) with S being a constrained set and SC2", The idea is select a subset set that would accept feasible solutions. Since ff is monotone submodular, such discrete maximization problems, being NP-complete [10, can be guaranteed to be approximated a factor of 1 = 0.63
377 if maximized under S using a greedy approximation algorithm 2
4 Algorithms
4.1
A Submodular Optimization Approach to Active 3D Object Detection
To tackle the core challenges of active 3D object detection, we propose unified framework grounded in submodular optimization. Our objective is to select set of unlabeled point clouds that are (I)
3

---

SDMCB
GBSSS
Labeled Point Clouds
Oracle (Human)
Step -Balancc cloud (Equation 6)
Unisbeled Paint-Cloud Pool
Dropout
Selected Point Clouds
Voxel Encoder
Clacsifter
Conv Blocks
Retrain
Box Head
Roi Head
Select point clouds with highsst entropy values
Hypotherical Labels
y
Classifier
Stop Belance the Label distributions donnons of labeled point clouds (Equation 7)
Balanced
Infer
Active Learning
Conv-Blocks
Policy
Box Head
Rol Head
Gradlents
Labeled Training
Unlabeled Point-Cloud Pool
Elcon
Fasture-bssed submodulaar
Set
Figure 1: STONE: An illustrative pipeline of proposed active learning method for 3D object detection leveraging submodular functions.
representative, including various levels of difficulty, and, (2) preservative of the label distribution of the selected samples.
To achieve the first goal, leverage submodular function fi to measure the representativeness of the selected unlabeled point clouds Ds with respect to the whole unlabeled set Dy. This can be achieved by minimizing the absolute difference between fi (Dv) and fi (Ds) since D Dy. This translates to maximizing the absolute difference fi(D)) fi (Dv). To achieve the second goal, we leverage different submodular function to that once the selected unlabeled point clouds Ds are added to the labeled set DL, the overall quality, as measured by label distribution, will not decrease. In sum, aim to select unlabeled point clouds Ds from the unlabeled pool Dy that optimize the following two objectives.
max (Di(Ds) + fe(DD UD5) Dg
Unlike existing active learning methods that rely on sample uncertainty, such as Shannon entropy 48, (55, which tends select difficult point clouds, our proposed formulation selects samples of varying difficulty levels. Additionally, unlike existing algorithms [44] [8] or clustering- based approaches 38 3. our formulation also considers the selected labeled point clouds prevent data imbalance when training 3D detector. The formulation is also general, allowing for different choices of submodular functions. In practice, we use feature-based submodular function 59 2
as fi and the entropy of the label distribution fz. Although the proposed formulation is motivated by the problem of active 3D object detection, it can also be useful for active learning problems similar domains, such 2D object detection, we will demonstrate in the experiments. Finding the set Ds that solves Equation is NP-complete [10]. Therefore, we propose a simple algorithm called STONE to efficiently solve the problem as will be detailed below.
4.2 STONE
In order to optimize the submodular optimization framework in Equation 3 we have designed an active learning pipeline with two stages, aligning with the dual objectives of the submodular criteria. To enhance computational efficiency and simplify optimization, we have implemented a hierarchical structure eliminate unselected samples each stage, with the remaining samples comprising our final selection. Initially, we select I samples using the proposed Gradient -aadie Submodular Subset Selection (GBSSS), which maximizes diversity and coverage while minimizing redundancy from Dy. Subsequently, we select I2 samples from I through Submodular Optimization for Class Balancing (SDMCB). detailed explanation of these stages provided the following paragraphs. We illustrate proposed pipeline in Figure

---

4.2.1 Gradient-Basee Submodular Subset Selection (GBSSS)
To address the challenge outlined in the introduction- aariducttoo--aiiinng from variations in object size, occlusion, and truncation, which introduce different levels of difficulty across categories- -w need to identify representative and diverse samples not only across categories but also within each category with varying difficulty levels. This challenge can be addressed by using a submodular function fi for maximizing J (1s)) Si (Dtr). In the proposed gradient-based baaed submodular subset selection, we select subset of unlabeled point clouds from Dy that maximizes gradient feature coverage while ensuring diversity.
Due to the absence of ground semantic labels in the pool of unlabeled point clouds, we employ Monte Carlo dropout (MC-dropout) [1] at the detector head for each point cloud P: to generate multiple regression predictions, as in 35 By averaging these predictions, we obtain regression hypothetical labels as well as the classification hypothetical labels. We then calculate the loss function C; of the 3D detector Mg. with parameters 6, using the classification and regression hypothetical labels. Following backpropagation, we extract gradients VoL, from the fully connected (FC) layer of the detector head.
However, when the dataset is highly imbalanced, as is usually the case in 3D object detection, the gradients generated by the above approach become inaccurate for classes with fewer samples [39]. To address this issue, we introduce two novel reweighing approaches for handling regression loss Trey and classification loss Lat in 3D object detection for better computation of the gradients for rare classes. In particular, after calculating the regression loss for each bounding box, we calculate the average loss Lieg for class based on the regression hypothetical labels. For given label c, the regression loss reweighing factors can be expressed as We n.. where n. is the number of the bounding boxes that belong to class c. We then normalize We as we max(e,]" The purpose of doing so is two-fold - 1) The class-specific weights in T. are uniformly scaled to prevent bias toward frequently occurring classes. 2) This approach helps reduce overfitting to bounding boxes associated with those classes. Then, we perform an element-wiss wise multiplication between ú. and Ei to reaseale the loss, for each class c, and compute the mean, across all the semantic classes C, to get the final reweighed regression loss Erreg. as Lreg- = Ee= We Lreg- The reweighed regression loss treg places more emphasis on classes with fewer samples, which critical for computing the gradient.
we
we additionally introduce a new classification reweigh- ing loss function designed to handle class imbalances by adapting the classification margins according to the label distribution during the active learning stage. In essence, for rare classes, the distance of the samples from the decision boundary i.e., the margin, would have to be penalized more. This way the generalization error of minor classes can be improved without worsening the performance on frequent classes. However, penalizing the rare classes more might affect the margins of frequent classes, leading to a complex trade-off [4. In order to balance this, for the i-th point cloud, we leverage margin vector m., where the margin for class is defined mi,e Anc (n, is the class frequency of class e). We then subtract the predicted logits fa by the margin vector mn to reweigh the predicted logits. Finally, the reweighed logits and hypothetical classification labels 9; are used in the classification loss function Leie to compute the class-balane balanced classification loss Lote as,
Inspired by the theoretical formulation
L = Lets Sr m))
We then use the class-balanced balanced detection loss i. = Lreg + Lots to compute the gradient for each point cloud. After obtaining the gradients, we use Seature-based submodular function (59 fi to select the top I diverse samples from Du,
Σ 9 (v(Vols) P.EDs
max DgC D2
(5)
where ((x) = log(1 + 2). The concavity of g(z) ensures that the marginal gain of adding more instances of similar features decreases as more such instances are added. This means that the function will favor adding elements that introduce new features rather than redundant ones. For the score function [59], (1(:), we use gradients distribution entropy HVVoL) as the measure of informativeness. More details are given in the Appendix. A greedy algorithm is then applied to iteratively add samples to a subset, providing the most significant increase in marginal gain until the target size, Tı, is reached.
5

---

4.2.2 Submodular Diversity Maximization for Class Balancing (SDMCB)
One of the main challenges for active 3D object detec- tion is that each 3D scene can contain multiple semantic classes, such as cars and cyclists. As the queried point clouds increase, it will inevitably introduce an imbal- anced label distribution in Dr, eventually resulting in performance drops. This situation is more prominent in autonomous driving datasets. A recent study, CRB, attempts to address this imbalance issue by introducing individual point cloud balancing. However, it cannot guarantee a balanced label distribution in Du across multiple active learning queries. In Figure 2 we illus- trate the imbalance in label distribution entropy caused by existing methods B5 34). particularly KECOR, where label imbalance begins increase as the number of query rounds rises. Aiming to alleviate this label imbalance, we utilize entropy calculated from the predicted label distribution the submodular func- tion f2. This approach supports the principle of diminishing returns as Dr becomes more balanced. Our method is divided into two steps. In Step I, we select point clouds which have balanced label distribution based on the hypothetical labels. In Step 2. from the selected ones, we further choose a subset that can be labeled ensure that the label distribution is balanced in the labeled set Dr.
Labal Distribution Entrogy
1.5
1.4
1.2
STONE
CRB
1.1
KECOR
Number Active Query Round
Figure 2: The proposed STONE method more effectively maintains the balance of label distribution in active 3D object detec- tion. The plots show the cumulative label distribution entropy values on KITTI (15) validation set split with PV-RCNN [49
Step 1: Balance the label distribution of individual point cloud. To balance each individual point cloud, select the top K. most balanced samples according to their individual point cloud label distribution entropy. This selection process applies a heuristic that maximizes the immediate gain in entropy. To calculate the probability P..c of specific label within point cloud P, first normalize the count of predicted label c, denoted as ne. by the total number of predicted bounding boxes N; in P,. This normalization is done using softmax function and adjusts the raw count of label reflect its proportion relative to the total number of labels. Next, the entropy of the individual point cloud, H (Pt). is calculated as follows,
th /
H(P.) = - - log Pi.
66)
e N;
We calculate (P.)) for each unlabeled point cloud P. E Dy and identify the top Kc point cloud samples with the highest entropy values.
Step 2: Balance the label distribution of labeled point clouds. We then aim to choose a subset of point clouds from the selected ones in Step for balancing the label distribution of the labeled point clouds. To achieve this, we follow a similar approach from the previous step to calculate label probability. Instead of normalizing ne by the number of bounding boxes N, in the point cloud P.. we first compute the sum of the number of labeled bounding boxes NL.c of a certain class c in the labeled set and ne, which represents the total number of objects belonging to class once the point cloud P; is labeled. Then we compute the sum of total labeled bounding boxes NL and N₁ to find Yi which represents the normalized number of predicted label Finally. calculate Pi... indicating the cumulative label probability of certain class once the point cloud P; is labeled. The cumulative entropy H( (P:) of the cumulative label probability then can be computed as,
NI.C + ne
= - Pr. =(P:)= c=1
log Dr.e. Psc =
The
Nr. + N;
(P::) can be used to measure label balancing after the point cloud P; is added into the labeled set. To construct the final selected point clouds in the current query round q. we use greedy search algorithm to iteratively select samples from Dy that maximizes the cumulative entropy, to finally select the top
T2 samples. Thus, using this two-stage progressive approach, we achieve both individual point cloud label balance and label balance across all the labeled point clouds which crucial for the real- world application of active 3D object detection.

---

Experiments and Results
5.1 Experimental Setup
Datasets. For our experiments, we use the KITTI dataset [1], one of the commonly used datasets in autonomous driving tasks. The dataset consists of 3,712 training samples and 3,769 validation samples, which include total of 80,256 labeled objects. These objects include cars, pedestrians, and cyclists, each annotated with class categories and bounding boxes. We also use the more challenging dataset for 3D object detection in autonomous driving the Waymo Open dataset [52] It includes 158,361 training samples and 40,077 testing samples. In the KITTI dataset, task difficulty levels defined as EASY for fully visible objects, MODERATE for partially occluded objects, and HARD for significantly occluded objects. For the Waymo test set, the framework categorizes difficulty into two levels: LEVEL 1 with more than five LiDAR points inside the ground-truth bounding box and LEVEL 2 with less or equal to five points. The sampling intervals are set to 1 and 10 for KITTI and Waymo Open, respectively.
Baselines. We compare our work against several generic active learning baselines, 1) RANDOM: Naive sampling strategy that randomly selects fixed samples at every round: 2) ENTROPY (55 41:
Selects samples with the highest degree uncertainty as measured by entropy of the sample's posterior probability; 3) LLAL [62]: Task-agnostic method with parametric module that chooses samples where the model is likely to make wrong predictions, based on an indicative loss; 4) CORESET [44]: Greedy furthest-first method using the core-set selection both labeled and unlabeled embeddings. 5) BADGE [ Batch-mode AL method to select diverse samples, with high gradient magnitude, that span wide range of directions in the gradient space.
We also draw contrasts between our method and AL methods (and variants) for 2D/3D detection, 6) MC-MI [II]: Uses MC-dropout [14] to estimate model uncertainty and mutual information to select the most uncertain point cloud samples; 7) MC-REG (35]: Uses several rounds MC-dropout 14 to approximate the regression uncertainty and picks samples with the greatest variance for labeling: 8) LT/C [22] Adapted from 2D detection, it selects samples based on both localization uncertainty and classification confidence to improve model performance; 9) CONSENSUS [43]: uncertainty estimation and continuous training to reduce labeling efforts. 10) CRB (35] and 11) KECOR (34). To compare the performance of our method the 2D object detection task, we compare with 12) AL-MDN [5]: Constructs mixture density networks to estimate probability distributions for the outputs of localization and classification heads.
Employs ensemble-base
Evaluation Metrics. To maintain faimness with the baselines on KITTI, we measure the performance using Average Precision (AP) for 3D and Bird Eye View (BEV) detection, with rotated Intersection over Union (loU) thresholds of 0.7 for cars and 0.5 for pedestrians and cyclists, following [49]. For the Waymo Open, performance is measured using Average Precision (AP) and Average Precision Weighted by Heading (APH), with IoU thresholds of 0.7 for vehicles and 0.5 for pedestrians and eyclists.
Implementation Details. We train our proposed method and all baselines on a GPU cluster with NVIDIA RTX A5000 GPUs. To make fair comparisons, we adopt the implementation settings as outlined in CRB. All the baselines use PV-RCNN 49 as the backbone detection model.
Training settings. For KITTI and Waymo Open, the training batch sizes are set to 6 and 4 respectively. However, the evaluation batch sizes are set to 16 for both datasets. We optimize the network parameters using Adam with fixed learning rate of 0.01. For all the methods, we perform stochastic forward passes of the MC-Dropout [4].
Active learning parameters. For KITTI, we set I, and I2 to 400 and 300 respectively. In the case of Waymo Open, I; and I22 are set to 2,000 and 1,200 respectively. To ensure fairness, Ng is set to 100 in all the methods.
5.2 Results
KITTI Dataset Results. evaluate the performance of STONE against other baseline methods in
Table I We clearly notice that STONE outperforms all the prior active learning methods, irrespective of the detection difficulty level and backbone model. In particular with PV-RCNN [49 as backbone,
on average, observe 3D AP improvements of 1.47%, 0.84%, and 1.24%, over CRB for the EASY,
7

---

Table 3D AP) scores on KITTI validation set with 1% queried bounding boxes, using PV-RCNN as the backbone detection model.
CAR
Pedescrion
Cyclist
Average
Method
EASY
MOD:
HARD
EASY
MOD.
HARD
EASY
MOD.
HARD
EASY
MOD.
HARD
CORESET
87.77
77.73
72.95
47.27
41.97
38.19
81.73
59.72
72.25
59.81
BADGE 1
89.96
75.78
70.54
51.94
46.24
40.95
84.11
62.29
58. 12
75.34
61.44
65.55
LLAL
24
89.95
73.65
75.32
56.34
69.57
45.97
75.55
60.35
55.36
73.94
62.95
58.88
MC REC
88.85
76.21
13,47
35.82
31.81
29.79
73.98
55.25
51.85
66.21
54.41
$1,70
MC-MI
86.28
58
37.50
$6.04
CONSENSUS
78.01
74.28
49.50
55.27
75.01
LTVC PH
88.73
72
73.87
55.17
48.37
43.65
83.
63.21
59 16
75.88
63.23
58,89
CRB RM
90.98
79.00
74.04
64.17
$4.80
50.82
86
67.45
63. 63.56
80,70
67.81
62.81
KECOR
STONE
02,09
80.27
75.44
66.1
58.84
52,70
N8.31
67.14
64.01
82.17
68.75
64,55
MODERATE, and HARD evaluation modes. We observe improvements of 0.54%, 0.08%, and 0.63%
over KECOR respectively.
From Figure 3 it is evident that regardless of the detection difficulty level. STONE consistently surpasses other baseline methods. Specifically, compared to the previous state-of-the-an methods KECOR and CRB, with 1% of labeled bounding boxes from the entire pool of unlabeled point clouds at the HARD difficulty level, STONE on average 0.9% higher than KECOR and 1.9% higher than CRB. Although STONE requires 37% more annotated bounding boxes than KECOR, it achieves higher accuracy in earlier query rounds, resulting in faster training times due to fewer epochs and active queries needed. Compared CRB, even with 36% more labeled bounding boxes at the final round of active queries, STONE is still on average 0.84% higher across all the difficulty levels.
70
30 noiei Easy
Ked
Entopy
Gred
Bedge
Badge
M0/9
LUK.
010
CR1
091
1000
4000
1000
4002
10000
2000
3000
1000
Number
Selected Brunding Dinces
Number Selected Bounding Scoee
Number Selected Douncing flooes
Figure 3D mAP (%) of AL baselines on the KITTI validation set with PV-RCNN.
0.45
0.75
0.375
Fanc
0.325
Ford
Erbway
0.350
MCRED
LUA
0.355
0.75
STORE
O11
10000
4000C
50000
0.250
10000
20000
40002 50000
Figure 4: 3D mAP (%) of AL baselines the Waymo Open validation set with PV-RCNN.
Waymo Open Dataset Results. To further test the generality and robustness of STONE, we also evaluate our approach on the Waymo Open dataset using the APH score as the performance metric across the two levels of difficulty LEVEL 1 and LEVEL 2. As shown in Figure 4 STONE surpasses other baseline methods as the number of bounding boxes increases. Compared to previous state- of-the-art methods KECOR and CRB at 25,000 bounding boxes, STONE achieves higher accuracy for both levels of detection difficulty. Our method uses fewer active learning loops than KECOR, meaning it reaches 2% higher APH score with fewer training epochs. Additionally, when comparing STONE with CRB in the final active learning round, STONE achieves a 1% higher APH score on average with 24% fewer bounding boxes.
2D Object Detection Results. We also demonstrate the generalizability of our method on 2D object detection task. We perform experiments the PASCAL VOC [9] dataset, that contains 20 object

---

classes. For the experiments, we use VOCO7 trainval and VOCO7+12 trainval to train a Single Shot MultiBox Detector (SSD) [3] with a VGG-16 backbone [50 and test on VOCO7 test. We follow the training guidelines and setup as outlined in AL--MDNN 5 to reproduce the results. For all methods, Ng is set be 1000 for total of three queries. In 2 we summarize the results in terms of mean Average Precision (mAP), noting that our method either outperforms or is comparable to all baseline methods. the first query, we observe improvements of 2.93% and 2.43% over AL-MDN, gmm and AL-MDNeff of respectively. Our method, while achieving balance in label distribution, also scales well to large datasets with more semantic labels.
Computational Complexity. STONE achieves significant GPU memory savings compared to KECOR, which computes gradients of the out-
Table VOCO7 mAP(%% of STONE against AL basclines.
put of the ROI head's fully connected shared layer, resulting in a gradient matrix of high di- mensions and memory. In contrast, STONE fo- cuses gradient computation on the outputs of the classification and regression loss layers within the ROI head, which are much lower in dimen- sionality. On the KITTI dataset, with a batch size of 6 (for a fair comparison), our method consumes 10 GB of GPU memory, whereas KECOR consumes 24 GB, which is 140% more GPU memory. Additionally, STONE maintains similar running time to KECOR.
mAP in S images)
Method
1st (2k)
2nd (2k)
3nd (4k)
RANDOM
62.43:100100
66.36:20113
68.47::0.099
ENTROPY
411
62.43:2.100 10
66.85:0.122 12
68. 70:200200 18
CORESET
62.43:10.10
66.57:10.20
0.226
LLAL
K2
62.47:1:0.16
67.02:20.11
68.90:20.15
MC- MC-DROPOUT M
62.43-+-0119
67.10=-0.07
00.39--0.099
ENSEMBLE 19
62.43-10.10
0..11:10.26
69.26 2:114
AL-MDNamm
62.43:1:0100
67.32:10112
69.43:10.11
ALADDN./1 5
62.91-100166
67.61 0.11
69.66-1-0.17
STONE
65:34-0..4
67.01-10.47
69.03-00.55
Table 4: Stage- wise performance comparisons.
Table 3: Ablation on the reweighing factor.
3D in
Bounding Botcs
3D AP in %
Stages
EASY
MGD.
HARD
Componenis
EASY
MOD.
HARD
GBSSS
79.60
66.50
62.78
2623
w/o reweighing facter wio seweighing factor Lreg
50.81
68.78
62,88
GBSSS SUMCH (Step 2)
CIBSSS SDMCB :Step 1)
80.37
79.30
65.85
67.80
64.63
63.17
2473
1571
80.32
69.64
63.59
SDMCE
N0.13
07.83
62.59
1484
wio reweighing factor Lefa
50:22
68.08
SDMCB (Skep 1)
$0.95
60.44
64.31
1567
none
83.03
70.33
65.31
SDMCB (Ske
80.65
63.36
64.5
2514
All
8300
76.33
6 Ablation Study
We perform ablation studies, on KITTI, to assess the efficacy of our approach and to better understand the key mechanisms and components involved. In the ablation study, all the experiments use 5 rounds of active learning selection, acquiring a total of 500 point clouds for annotation. We present and discuss additional ablation experiments in the Appendix
Contributions of Lveg and Lis.- Given that our method incorporates both regression loss Ling and classification loss Lote to handle potential class imbalance, we have carried out extensive experiments to analyze their impact model performance. We first investigated whether regression or classification loss has a greater impact by using gradients generated from each loss separately during the active learning stage. The results showed that using only Lreg resulted in an average 3D AP drop of 0.77% for HARD while using only Lets resulted in an average 3D AP drop of 3. 3.12% for HARD on the KITTI validation dataset. This indicates that T.dts has a larger impact on our model's performance, and both losses are essential for optimal results.
Effect of the reweighing f'actor. To further evaluate the reweighing factor's importance on our method's performance, we perform three crucial experiments. In Table 3 (row 1), we study the effect of not having the reweighing factor on both the regression and classification losses. We observe performance drops of 3.78% 1.55%, and 2.45% for the EASY, MODERATE, and HARD levels of difficulty, respectively. We also observe similar performance drops in Table 2) and Table 3) with no reweighing factor for the regression or classification loss, respectively. This fundamental issue arises because, in contrast to classification tasks where single loss function is typically considered, detection tasks require the simultaneous optimization of multiple loss components. Specifically, reweighing only the classification regression loss in detection tasks can disrupt the

---

Table 5: 3D mAP(%) scores on KITTI validation set with 1 1% queried bounding boxes, using SECOND [6 as the backbone detection model.
SD Detection overgee MAP
BEV Desection aeerage mAP
Mechol
EASY
MOD.
HARD
EASY
MOD.
[ARD
Randoen
66.33
55.48
51.53
75.66
63.77
59.71
CORESET
66.86
48.97
73.08
56.95
LLAL
69.19
55.38
50.85
76.52
59.07
RADIG
51.23
76.07
$9.47
69.45
51.25
76.04
63.49
$3.40
CRB 35
72.53
53.00
78.84
65.82
61.25
КВСOК 34
74,05
60.38
55.34
50.00
68.20
63.20
STONE
76.86
64.04
58.75
$2.14
70,82
65.68
balance within the model, as it does not adequately model the interdependencies between these loss functions, leading to the suboptimal performance of the detector.
Relevance of each stage. In Table 4 we examine the importance and the relevance of each stage of our method. We assess the performance using 3D AP (%) and the number of bounding boxes annotated with 500 point cloud selections. From Table we observe that the removal of any single component leads to a drop 3D AP performance. particular, using the GBMSS stage only leads to drop of 3D AP by 3,43%, 3.83%, and 2.55% for the EASY, MCDERATE and HARD difficulty levels respectively. In such a scenario, the bounding boxes annotated increase by 71.45 71.43, which shoots up the labeling cost. It is interesting to note that while using only the SDMCB stage results in annotating the lowest bounding boxes, it also leads to a drop in performance. In conclusion, both stages of our method necessary for the optimal selection of samples, with a lower labeling cost, to ensure the efficiency and accuracy of the proposed active learning method.
Bacosone-agnostic performance. In all of the experiments in this paper, we use PV-RCNN as the backbone detection model. However, to show the invariance of our method towards change in the backbone, we perform experiments with SECOND [61] widely used 3D object detector. The results, as shown in Table 5 indicate that STONE achieves a 3.4% higher 3D mAP score at the HARD level and 2.43% higher mAP score at the HARD level in BEV detection compared to the state-of-the-an method, KECOR. This demonstrates the performance and generality of the proposed approach.
Conclusion
In this paper, we propose a novel approach called STONE, unified active 3D object detection methodology, based on submodular optimization. We provide a robust and compute-efficient solution by proposing a two-stage algorithm that first utilizes a submodular function based on the gradients using the Monte Carlo dropout [33 to select representative point clouds. We then apply a greedy search algorithm to balance the data distribution due to the possibility of having an imbalance across all the labeled point clouds. Extensive experiments on benchmark autonomous driving datasets, including KITTI [15] and Waymo Open [52 datasets, demonstrate the effectiveness and generalization of our method.
Limitations, One limitation of the proposed method that it does not reduce the running time of existing active 3D detection methods, which is primarily due to the large number of unlabeled point clouds. In future work, we will investigate more efficient methods for active 3D object detection.
Societal Impact. The proposed method will be important for several real-world applications such as autonomous driving and robotics, by reducing the cost of labeling.
Acknowledgements. We would like to thank the anonymous reviewers for their helpful comments. This project was supported by grant from the University of Texas at Dallas. This work is also supported by the National Science Foundation under Grant Numbers I1S-2106937, gift from Google Research, an Amazon Research Award, and the Adobe Data Science Research award.
10

---

References
[1] Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, Deep batch active learning by diverse, uncertain gradient lower bounds, arXiv preprint arXiv:1906.0367 (2019)
[2] Bilmes, J. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00132 (2022)
[3] Bodó, Z.., Minier, Z.,, Csató, L.: Active learning with clustering. In: Active Learning and
Experimental Design workshop In conjunction with AISTATS 2010. pp. 127-139. JMLR Workshop and Conference Proceedings (2011)
[4] Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribttionaavar margin loss. Advances neural information processing systems 32 (2019)
[5] Choi, J., Elezi, I., Lee, H.J., Farabet, C., Alvarez, J.M.: Active learning for deep object detection via probabilistic modeling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10264-10273 (2021)
[6] Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. Journal of artificial intelligence research 4, 129-145 (1996)
[7] Dasgupta, S.: Two faces of active learning. Theoretical computer science 412(19), 1767-1781 (2011)
[8] Deng, B., Qi, C.R., Najibi, M., Funkhouser, T., Zhou, Y., Anguelov, D.: Revisiting 3d object detection from egocentric perspective. Advances Neural Information Processing Systems 34, 26066-26079 (2021)
[9] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303-338 (2010)
[10] Feige, U.: A threshold of In for approximating set cover. Journal of the ACM (JACM) 45(4), 634-652 (1998)
[11] Feng, D., Wei, X., Rosenbaum, L., Maki, A., Dietmayer, K.: Deep active learning for efficient training of lidar 3d object detector. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 667-674. IEEE (2019)
[12] Fujishige, S.: Submodular functions and optimization. Elsevier (2005)
[13] Gal, Y., Ghahramani, Z.: Dropout as bayesian approximation: Representing model uncertainty
in deep learning. In: international conference on machine learning. pp. 1050-1059. PMLR (2016)
[14] Gal, Y., et al.: Uncertainty in deep learning (2016)
[15] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition. pp- 3354-3361 3361. IEEE (2012)
[16] Gissin, D., Shalev-Shwartz, S.: Discriminative active learning. arXiv preprint 1eXiv:1907.06.30 (2019)
[17] Gudovskiy, D., Hodgkinson, A., Yamaguchi, T., Tsukizawa, S.:
Deep active learning for biased datasets via fisher kernel self-supervision. In: Proceedings of the JEEE/CVF conference on computer vision and pattern recognition. pp. 9041-9049 (2020)
[18] Guo, C., Zhao, B., Bai, Y.: Deepcore: A comprehensive library for coreset selection in deep learning. In: International Conference on Database and Expert Systems Applications. pp. 181-195. Springer (2022)
[19] Haussmann, E., Fenzi, M., Chitta, K., Ivanecky, J., Xu, H., Roy, D., Mittel, A., Koumchatzky, N., Farabet, C., Alvarez, J.M.: Scalable active learning for object detection. In: 2020 IEFE intelligent vehicles symposium (iv). pp. 1430-1435 1435. IEEE (2020)

---

[20] Houlsby, N., Huszár, F., Ghahramani. Z., Lengyel, M.: Bayesian active learning for classification and preference learning. arXiv preprint urXiv:1112.5:7 (2011)
[21] Jordan, T., A., Surbhi, G., Akshay, K., Sham, K.: Gone fishing: Neural active learning with fisher embeddings. In: Annual Conference on Neural Information Processing (NeurIPS), pp. 8927-8939 (2021)
[22] Kao, C.C., Lee, T.Y., Sen, Liu, M.Y.: Localization-awwat aware active learning for object detection. In: Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part VI 14. pp. 506-522 Springer (2019)
[23] Kaushal, V., Iyer, R., Doctor, K., Sahoo, A., Dubal, P., Kothawade, S., Mahadev, R., Dargan, K., Ramakrishnan, G.: Demystifying multi-faceted video summarization: tradeoff between diversity, representation, coverage and importance. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 452-461. IEEE (2019)
[24] Killamsetty, K., Abhishek, G.S., Lnu, A., Ramakrishnan, G., Eyfimievski, A., Popa, L.., Iyer, R.: Automata: Gradient based data subset selection for compute-efficient hyper-parameter tuning. Advances in Neural Information Processing Systems 35, 28721-28733 (2022)
[25] Kim, K., Park, D... Kim, K.I., Chun, S.Y.: Task-aware variational adversarial active learning. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8166-8175 (2021)
[26] Kirsch, A., Van Amersfoort, J., Gal, Y.: Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems 32 (2019)
[27] Kothawade, S., Beck, N., Killamsetty, K., Iyer, R.: Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems
34, 18685-18697 (2021)
[28] Kothawade, S., Ghosh, S., Shekhar, S., Xiang, Y., Iyer, R.: Talisman: targeted active learning for object detection with rare classes and slices using submodular mutual information. In: European Conference Computer Vision. pp. 1-16. Springer (2022)
[29] Kothawade, S., Kaushal, V., Ramakrishnan, G., Bilmes, J., Iyer, R.: Prism: A rich class of pa- rameterized submodular information measures for guided data subset selection. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 10238-10246 (2022)
[30] Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017)
[31] Lewis, D.D.; A sequential ulgorithm for training text classifiers: Corrigendum and additional data. In: Acm Sigir Forum. vol. 29, pp. 19.19. ACM New York, NY, USA (1995)
[32] Lin, J.: Divergence measures based on the shannon entropy. IEEE Transactions on Information theory 37(1), 145-151 (1991)
[33] Liu, W., Anguelov, D., Erhan, D.,, Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: Computer EicV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14. pp. 21-37. Springer (2016)
[34] Luo, Y., Chen, Z., Fang, Z., Zhang, Z., Baktashmotlagh, M., Huang, Z.: Kecor: Kernel coding rate maximization for active 3d object detection. In: Proceedings of the IEFE/CVF International Conference Computer Vision. pp. 18279-18290 (2023)
[35] Luo, Y., Chen, Z., Wang. Z., Yu, X., Huang, Z., Baktashmotlagh, M.: Exploring active 3d object detection from a generalization perspective. In: The Eleventh International Conference on Learning Representations (2023)
[36] Ma. S., Zeng, Z., McDuff, D., Song, Y:: Active contrastive learning of audio-visual video representations. arXiv preprint arXiv:2009.0988 (2020)
[37] Nemhauser, G.I.,, Wolsey, L.A., Fisher, M.I.: An analysis of approximations for maximizing submodular set functions-2 i. Mathematical programming 14, 265-294 (1978)
12

---

[38] Nguyen, H.T., Smeulders, A.: Active learning using pre-clusterigg. In: Proceedings of the twenty-first first international conference on Machine learning. p. 79 (2004)
[39] Peng, H., Pian, W., Sun, M., Li, P.: Dynamic re-weighting for long-tailed semi-supervised learning. In: Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision. pp. 6464 6474 (2023)
[40] Pinsler, R., Gordon, J., Nalisnick, E., Hernández-Lobato, J.M.: Bayesian batch active learning as sparse subset approximation. Advances neural information processing systems 32 (2019)
[41] Roy, S., Unmesh, A., Namboodiri, V.P.: Deep active learning for object detection. In: BMVC. vol. 362, 91 (2018)
[42] Ruiyu, M., Ouyang, X., Yunhui, G.: Inconsistency-basee data-centric active open-set annotation. In: Proceedings of the AAAT Conference on Artificial Intelligence. vol. 38, pp. 4180-4188 (2024)
[43] Schmidt, S., Rao, Q., Tatsch, J., Knoll, A.: Advanced active learning strategies for object
detection. In: 2020 IEEE intelligent vehicles symposium (IV). pp. 871-876. IEER (2020)
[44] Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 1ee:::70800044 304890 (2017)
[45] Settles, B.: Active learning literature survey (2009)
[46] Settles, B., Craven, M., Ray, S.: Multiple-instance active learning. Advances in neural informa- tion processing systems 20 (2007)
[47] Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Proceedings of the fifth annual workshop on Computational learning theory. pp. 287-294 (1992)
[48] Shamnon, C.E.: A mathematical theory of communication. ACM SIGMOBILE mobile comput- ing and communications review 5(1), 3-55 (2001)
[49] Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-ronn: Point-voxel feature set abstraction for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10529-1000 -0538 (2020)
[50] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 1:Xiv:1409.11 (2014)
[51] Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: A rgb-d scene understanding benchmark
suite. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576 (2015)
[52] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 2446-2455 (2020)
[53] Tiwari, R., Killamsetty, K., Iyer, R., Shenoy, P:: Gcr: Gradient coreset based replay buffer selection for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 99-108 (2022)
[54] Tran, T., Do, T.T.,, Reid, I., Careiir, G.: Bayesian generative active deep learning. In: Interna- tional conference on machine learning. pp. 6295-6304. PMLR (2019)
[55] Wang, D., Shang, Y.: A new active labeling method for deep learning. In: 2014 International joint conference on neural networks (ICNN). pp. 112-119. IEEE (2014)
[56] Wang, J., Lan, S., Gao, M., Davis, L.S.: Infofocus: 3d object detection for autonomous driving with dynamic information modeling. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16. pp. 405-420. Springer (2020)
13

---

[57] Wang, M., Min, F., Zhang, Z.H., Wu, Y.X.: Active learning through density clustering. Expert systems with applications 85, 305-317 (2017)
[58] Wei, K., Iyer, R., Bilmes, J.: Submodularity in data subset selection and active learning. In: International conference on machine learning. pp. 1954-1963, PMLR (2015)
[59] Wei, K., Liu, Y., Kirchhoff, K., Bartels, C.,, Bilmes, J.: Submodular subset selection for large- scale speech training data. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3311 1-1113.. IEEE (2014)
[60] Wei, K., Liu, Y., Kirchhoff, K., Bilmes, J.: Unsupervised submodular subset selection for speech data. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4107-4111. IEEE (2014)
[61] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
[62] Yoo, D., Kweon, ISS.: Learning loss for active learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 93-102 (2019)
14

---

8 Appendix
8.1 Additional Ablation Studies
Impact of different submodular functions. We evaluate STONE with different submodular func- tions used in stage 1 i.c. GBMSS, and report the results in Table Submodular functions such as facility location have demonstrated effectiveness in modeling representativity and diversity [23], as We employ these functions in our GBMSS study to assess their
well as active learning 28 27 effectiveness. In terms of performance measured by 3D AP, we observe results similar to those of STONE. However, it comes with an extremely high labeling cost with 50.98% rise in bounding box annotations. With the max coverage submodular function, we see slight performance drops of 1.57%, 0.06%, and 0.65% for the FASY, MODERATE and HARD difficulty levels respectively. Overall, our method attains the best results in terms of 3D average precision (AP) while maintaining a low labeling budget.
Table 7: Sensitivity to thresholds I1, D2.
Table 6: Impact of different submodular functions.
3D AP
in
Bounding Boxes
Components
EASY
MOD.
HARD
Submodular functions
EASY
MDD:
HARD
T1: 400 12: 200
82.32
68.24
61.08
Facility location
82.91
70.64
65.98
11: 1'2:
70.33
STONE
coverage
83.03
81.46
70.33
69.47
64.68
65.33
1530
1674
T1: 500,
T1:500 500,
I2: 200
L: 200
$0.70
78.62
69.
67.44
61.01
63.77
T1: Fs: 200
82.53
68.90
63,82
F1: 600,
l'g 300
81.67
67.47
Sensitivity to thresholds I and 2- We conduct a sensitivity analysis of model performance to different variants of thresholds I' and r2, at all difficulty levels, to understand their importance in our method. We report the 3D AP (in %) across six different combinations in Table 7 with 500 point-cloud selections. For the MODERATE and HARD difficulty levels, we observe fluctuations within 2.89% and 1.56% (compared to the lowest). This suggests that our method is relatively stable with the best 11 and I22 being 400 and 300 respectively.
Table 8: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with PV-RCNN as the backbone architecture
Table 9: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with SECOND [6] as the backbone architecture
Method 3D AP % using 1% (bounding box)
2%
3%
Methool
using 1% (bounding box)
CRB KECOR
62.81
65.43
69.93
71.30
CRB
$1.00
55.67
57.01
STONE
64.05
66.83
70.86
KRNOR
STONE
58,75
$5.34
60.33
57.56
61.89
58.90
Effect of % labeled bounding boxes. To ensure a fair comparison with the previous state-of-the-art
methods CRB [35] and KECOR (34), we leveraged 1% of the labeled bounding boxes. Referring to Tables g and 9 as more labeled bounding boxes are added to the training, the results get better. It is worth noting that the KECOR method marginally surpasses STONE when using 2% and 3% of the bounding box. This is because the STONE method, when utilizing the PV-RCNN backbone, tends to select scenes with more objects. As a result, STONE reaches the bounding box limit very early in the active learning stage. The slightly better results achieved by KECOR are due to it querying more scenes and being trained over more epochs. In active 3D object detection, the goal is to label few bounding boxes as possible to achieve good performance. Therefore, it is critical to maintain high performance with a smaller number of labeled bounding boxes, as demonstrated by STONE
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made match the experimental results.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations have been discussed.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have discussed the details fully and the code will be attached in the supplementary.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will attached the supplementary.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included all the details.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: have included statistical significance.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We have included the information.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////earrppppcc/pp///aaaaaa
16

---

Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impacts.
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited the used code, data and models, etc.
13. New Assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
17


Reviews:
[{"review_id": "SUfz1854hl", "reviewer": "Reviewer_jaqa", "summary": "This paper proposed an active 3D object framework based on submodular optimization. It focuses on solving data imbalances and covering varying difficulty levels of the point cloud data by using submodular optimization. Extensive experiments show superior results with high computational efficiency.", "questions": "Why only use 1% of labeled bounding boxes? If using more labeled bounding boxes and unlabeled BBOX, is it possible to get better results?", "limitations": "The paper has discussed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed active learning pipeline is intuitive and reasonable. \n2. The experiments are thorough. They are conducted across various datasets and show superior results in terms of 3D detection. The ablation study shows the effectiveness of the proposed algorithm.", "weaknesses": "1. The paper needs a pipeline figure to illustrate the proposed active 3D object detection. A figure can help better understand the general steps of the proposed approach. \n2. Performance gains: Compared with the baseline KECOR, the performance gains seem marginal. L352 mentioned STONE achieves significant GPU memory savings. Why can STONE save significant memory?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments.\n\n#### **[W1] The paper needs a pipeline figure.**\n\nPlease see the attached PDF for the pipeline figure of the proposed approach. \n\n\n#### **[W2] Compared with the baseline KECOR, the performance gains seem marginal.**\nWe further use SECOND [1], a widely used 3D object detector, as the base model.  The results indicate that STONE achieves a **3.4%** higher mAP score at the hard level in 3D detection and **2.43%** higher mAP score at the hard level in BEV detection compared to the state-of-the-art method, KECOR, as shown in Table 1. This demonstrates the performance and generality of the proposed approach.\n\n#### Table 1: 3D mAP(\\%) of STONE and AL baselines on KITTI validation set with 1\\% queried bounding boxes with one-stage 3D detector backbone SECOND\n| **Methods** | **3D Detection mAP EASY \\%** | **3D Detection mAP MOD. \\%** | **3D Detection mAP HARD \\%** | **BEV Detection mAP EASY \\%** | **BEV Detection mAP MOD. \\%** | **BEV Detection mAP HARD \\%** |\n|----------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|\n| RAND | 69.33±0.62 | 55.48±0.42| 51.53±0.33 | 75.66±1.10 | 63.77±0.86  | 57.71±0.95  |\n| CORESET | 66.86±2.27 | 53.22±1.65 | 48.97±1.42 | 73.08±1.80| 61.03±1.98 | 56.95±1.53 |\n| LLAL           | 69.19±3.43                   | 55.38±3.63                   | 50.85±3.24                   | 76.52±2.24                   | 63.25±3.11                   | 59.07±2.80                   |\n| BADGE          | 69.92±2.90                   | 55.60±2.72                   | 51.23±2.58                   | 76.07±2.70                   | 63.39±2.52                   | 59.47±2.49                   |               |\n| CRB            | 72.33±0.35                   | 58.06±0.30                   | 53.09±0.31                   | 78.84±0.27                   | 65.82±0.07                   | 61.25±0.22                   |\n| KECOR          | 74.05±0.16                   | 60.68±0.13                   | 55.34±0.23                   | 80.00±0.12                   | 68.20±0.35                   | 63.26±0.25                   |\n| **STONE**      | **76.86±0.88**               | **64.04±0.27**               | **58.75±0.58**               | **82.14±0.90**               | **70.82±0.14**               | **65.68±0.42**               |\n\n#### **[W3]  L352 mentioned STONE achieves significant GPU memory savings. Why can STONE save significant memory**\nGiven the point cloud $\\mathcal{P}_i$, a typical 3D object detector first leverages an encoder $\\mathbf{g}(\\cdot; \\theta_g)$ for extracting the latent feature $\\mathbf{m}_i$, then a detector head $\\mathbf{h}( \\cdot; \\theta_h)$ is used to generate detection results $\\hat{\\mathcal{B}_i}$. KECOR needs to compute gradients of the output of the ROI head's fully connected shared layer, i.e., $\\mathbf{m}_i$, with respect to the encoder parameters. The gradients are a matrix of high dimensions.  In contrast, STONE only needs to compute the gradient of the output of the ROI head's classification loss layer and regression loss layer, i.e., $\\hat{\\mathcal{B}_i}$, with respect to the encoder parameters. Therefore, STONE can save significant memory compared with KECOR.\n\n#### **[Q1] Why only use 1% of labeled bounding boxes? If using more labeled bounding boxes and unlabeled BBOX, is it possible to get better results?**\nTo ensure a fair comparison with the previous state-of-the-art methods, we leveraged 1\\% of the labeled bounding boxes. Referring to Tables 2 and 3, as more labeled bounding boxes are added to the training, the results get better. It is worth noting that the KECOR method marginally surpasses STONE when using 2\\% and 3\\% of the bounding box. This is because the STONE method, when utilizing the PV-RCNN backbone, tends to select scenes with more objects. As a result, STONE reaches the bounding box limit very early in the active learning stage. The slightly better results achieved by KECOR are due to it querying more scenes and being trained over more epochs. In active 3D object detection, the goal is to label as few bounding boxes as possible to achieve good performance. Therefore, it is critical to maintain high performance with a small number of labelled bounding boxes, as demonstrated by STONE. When using SECOND [1] as the backbone detector, STONE is consistently better than other state-of-the-art AL baselines (Table 3) even when using more bounding boxes. \n\n#### Table 2: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with PV-RCNN as the backbone architecture.\n\n| Method | 3D AP \\% using 1% (bounding box) |  2\\% |  3\\% |\n|--------|---------------------------|--------------|--------------|\n| CRB    | 62.81                     | 65.43        | 69.93        |\n| KECOR  | 63.42                     | **67.25**    | **71.70**    |\n| **STONE** | **64.05**              | 66.83        | 70.86        |\n\n#### Table 3: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with SECOND [1] as the backbone architecture.\n\n| Method | 3D AP \\% using 1% (bounding box)    | 2\\% |  3 \\% |\n|--------|---------------------------|--------------|--------------|\n| CRB    | 53.09                     | 55.67        | 57.01        |\n| KECOR  | 55.34                     | 57.56        | 58.92        |\n| **STONE** | **58.75**              | **60.33**    | **61.89**    |\n\n[1] Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ycJUy8ANwJ", "reviewer": "Reviewer_4dDG", "summary": "This paper introduces a novel framework to reduce labeling costs in 3D object detection. Using submodular optimization, the framework addresses data imbalance and varying difficulty levels in LiDAR point cloud. It employs a two-stage algorithm: Gradient-Based Submodular Subset Selection (GBSSS) for selecting diverse and representative point clouds, and Submodular Diversity Maximization for Class Balancing (SDMCB) to ensure balanced label distribution. Experiments on KITTI and WOD datasets show that STONE outperforms existing methods, demonstrating high computational efficiency and state-of-the-art performance.", "questions": "While the method demonstrates effectiveness in 3D object detection, its generalizability to other domains or tasks is not sufficiently discussed. Is the STONE method applicable to other types of 3D data or different application scenarios?", "limitations": "Limitations have been adequately addressed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Reduces labeling cost with submodular optimization\n- Addresses data imbalance effectively\n- Achieves state-of-the-art performance on KITTI and Waymo datasets\n- Demonstrates generalizability to both 3D and 2D object detection tasks", "weaknesses": "- Generalizability of Submodular Functions: The paper uses specific submodular functions tailored to their problem, but it lacks a detailed analysis of how generalizable these functions are to other datasets or slightly different tasks (e.g., semantic segmentation). This limits the understanding of the robustness of the proposed method across diverse scenarios.\n- Handling of Data Imbalance: Although the paper proposes a two-stage algorithm to handle data imbalance, it does not provide a detailed comparison with other state-of-the-art methods specifically addressing data imbalance in 3D object detection. More in-depth comparative analysis would strengthen the claims regarding its effectiveness in balancing label distribution.\n- While the paper claims high computational efficiency for STONE, it does not provide specific data on computation time or resource usage. What are the experimental settings and resource usage details?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments.\n\n#### **[W1] Generalizability of Submodular Functions: The paper uses specific submodular functions tailored to their problem, but it lacks a detailed analysis of how generalizable these functions are to other datasets or slightly different tasks (e.g., semantic segmentation). This limits the understanding of the robustness of the proposed method across diverse scenarios.**\n\nWe implemented our method in the active learning domain for 3D semantic segmentation following [1], utilizing the nuScenes dataset [3] with MinkNet as the backbone. Even though our method is not particularly designed for 3D semantic segmentation, it still achieves better performance compared with traditional active learning methods like Entropy and obtains competitive performance compared to the state-of-the-art 3D semantic segmentation method Annotator [1].  Detailed per-class results are available in Table 1.\n\n#### Table 1: Per-class results of STONE and AL baselines on nuScene dataset with MinkNet as backbone detector using 5 voxel budgets.\n\n| **Model**    | **Vehicle** | **Person** | **Road** | **Sidewalk** | **Terrain** | **Man-made** | **Vegetation** | **mIoU** |\n|--------------|-------------|------------|----------|--------------|-------------|--------------|----------------|----------|\n| Entropy      | 86.2        | 0.0        | 88.1     | 38.1         | 64.8        | 72.8         | 67.8           | 59.7     |\n| Annotator [1]   | 88.1        | 44.2       | 91.9     | 56.7         | 67.1        | 75.5         | 69.5           | 70.4     |\n| **STONE**    | 85.5        | 34.5       | 81.9     | 41.85        | 65.48       | 72.94        | 70.26          | 64.63    |\n\n\n#### **[W2] Handling of Data Imbalance: Although the paper proposes a two-stage algorithm to handle data imbalance, it does not provide a detailed comparison with other state-of-the-art methods specifically addressing data imbalance in 3D object detection. More in-depth comparative analysis would strengthen the claims regarding its effectiveness in balancing label distribution.**\n\nWe adopted the class-balanced grouping [2] into our current pipeline, denoted as STONE-GROUP-BALANCE, which utilizes Focal Loss and dynamically adjusts its weights based on the class distribution within each group during training which won the nuScenes 3D Detection Challenge held in Workshop on Autonomous Driving (WAD, CVPR 2019). We have removed the class balance component of the STONE method accordingly to compare with STONE. Table 2 demonstrates that even with class balancing during training, a highly imbalanced label distribution still results in a performance drop compared to the proposed approach.\n\n#### Table 2: Performance comparisons of STONE and STONE-GROUP-BALANCE using 3D AP(%) scores on the KITTI validation set with SECOND as the backbone architecture.\n\n| **Method**              | **EASY** | **MOD.** | **HARD** |\n|-------------------------|----------|----------|----------|\n| STONE-GROUP-BALANCE     | 74.43    | 63.15    | 57.03    |\n| **STONE**               | **76.86**| **64.04**| **58.75**|\n\n\n#### **[W3] While the paper claims high computational efficiency for STONE, it does not provide specific data on computation time or resource usage. What are the experimental settings and resource usage details?**\nAll 3D detection experiments are run on a GPU cluster with 4 NVIDIA RTX A5000 GPUs. In terms of resource usage, STONE consumes only 10 GB of GPU memory compared to 24 GB of memory consumption by KECOR. This is an advantage of 140\\% in GPU memory. We will add the details in the final version of the paper. \n\n#### **[Q1]  While the method demonstrates effectiveness in 3D object detection, its generalizability to other domains or tasks is not sufficiently discussed. Is the STONE method applicable to other types of 3D data or different application scenarios?**\n\nPlease refer to weakness 1 for additional details.\n\n[1] Xie B, Li S, Guo Q, et al. Annotator: A generic active learning baseline for lidar semantic segmentation[J]. Advances in Neural Information Processing Systems, 2023, 36.\n\n[2]  Zhu B, Jiang Z, Zhou X, et al. Class-balanced grouping and sampling for point cloud 3d object detection[J]. arXiv preprint arXiv:1908.09492, 2019.\n\n[3]  Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving[J]. CoRR, abs/1903.11027, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jb6juwefyB", "reviewer": "Reviewer_nHfr", "summary": "This paper introduces a framework to reduce the labeling costs of 3D point cloud data in 3D object detection by using a submodular optimization approach. It tries to optimize for data imbalance the distribution of the data like the varying difficulty levels. By the combination of a transformer-architecture and an active learning approach I achieves SOTA results. To do so it optimizes two submodular functions. The first one represents the different difficulty levels and the second one ensures class balance.", "questions": "- What about this paper https://arxiv.org/pdf/2402.03235? They also apply active learning on 3D object detection. However, a comparison is not easy as they use the nuScenes and TUM Traffic intersection dataset (see 3rd point of weaknesses).", "limitations": "Were discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper mentions important related work and describes the necessary background well\n\n- The introduced submodular optimization framework is a novel contribution that can potentially be applied to other domains of active learning as well\n\n- The introduced components are well explained in detail\n\n- The training is described extensively", "weaknesses": "- If not intended, in line 113-144 it could be mentioned that the randomly selected number of point clouds D_L in the the beginning is labeled. This could be seen as obvious but it would make the description more logical as the unlabeled point clouds during training need to be annotated by a human annotator as well.\n\n- To allow a complete and fair comparison with other approaches, evaluations on additional datasets like nuScenes and/or TUM Traffic Intersection would be good\n\n- A better explanation of the Figure 2 would be helpful i.e. why is there not the same amount of datapoints for every method and why are the number of bounding boxes so different. Extrapolating some curves would suggest that some methods work better than STONE. However this can not be evaluated as the datapoints are missing\n\n- The claim that the results are SOTA does not sound tenable, insofar as the percentage improvements are only small. In addition, the diagrams indicate that other methods could deliver better results on closer inspection. In Figure 3, it appears that KECOR would provide better results if more data points and more bounding boxes were available. As these data points are missing, a more accurate comparison is not possible. Furthermore, in Table 1, the best values for cyclists in the moderate and hard cases are not highlighted in bold. This was not necessarily intentional, but should be noted\n\n- Some spelling/grammatical/contextual errors (no need to respond):\n\nLine 48: function -> functions\n\nLine 75-78: include BADGE -> like BADGE?\n\nLine 383 mAP -> AP?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. \n\n#### **[W1] If not intended, in line 113-144 it could be mentioned that the randomly selected number of point clouds D_L in the the beginning is labeled.**\n\nThanks for the suggestion! We will make it clear that the randomly selected point clouds, in the beginning, are labeled, in the final version. \n\n#### **[W2] To allow a complete and fair comparison with other approaches, evaluations on additional datasets like nuScenes and/or TUM Traffic Intersection would be good.**\n\nWith a more challenging dataset like nuScenes, which has 10 classes compared to KITTI and Waymo, both of which have 3 classes, our proposed method still demonstrates a performance advantage (Table 1), showing that it has good generalizability.\n\n#### Table 1: Performance comparisons of STONE and AL baselines on the NuScenes validation set with SECOND [1] as the backbone architecture.\n\n| **Method** | **nuScenes detection score (NDS)** | **mAP** |\n|--|--|--|\n| Random | 37.25 | 28.96|\n| KECOR | 46.92| 40.23|\n| **STONE** | **48.79** | **41.23** |\n\n#### **[W3] A better explanation of the Figure 2 would be helpful i.e. why is there not the same amount of datapoints for every method and why are the number of bounding boxes so different. Extrapolating some curves would suggest that some methods work better than STONE. However this can not be evaluated as the datapoints are missing**\n\nDue to some point clouds having more objects than others, we first fixed the budget to the maximum number of point clouds that can be queried, using only 1% of the labeled bounding boxes to ensure a fair comparison. This is consistent with previous works in this domain. Referring to Tables 2 and 3, as more labeled bounding boxes are added to the training, the results will improve. It is worth noting that the KECOR method marginally surpasses the STONE method when using 2\\% and 3\\% of the bounding box. This is because the STONE method, when utilizing the PV-RCNN backbone, tends to select scenes with more objects. As a result, STONE reaches the bounding box limit very early in the active learning stage. The slightly better results achieved by KECOR are due to it querying more scenes and being trained over more epochs. In active 3D object detection, the goal is to label as few bounding boxes as possible to achieve good performance. Therefore, it is critical to maintain high performance with a small number of labelled bounding boxes, as demonstrated by STONE. When using SECOND [1] as the backbone detector, STONE is consistently better than other state-of-the-art AL baselines (Table 3) even when using more bounding boxes.\n\n#### Table 2: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with PV-RCNN as the backbone.\n| Method | 3D AP \\% using 1% (bounding box) |2\\% |  3\\% |\n|-|-|--|-|\n|CRB| 62.8|65.43 | 69.93|\n|KECOR|63.42|**67.25**|**71.70**|\n|**STONE**|**64.05**|66.83| 70.86|\n#### Table 3: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the KITTI validation set (HARD level) with SECOND [1] as the backbone.\n| Method | 3D AP \\% using 1% (bounding box)|2\\% |3 \\%|\n|-|-|-|-|\n|CRB|53.09|55.67|57.01|\n|KECOR|55.34|57.56|58.92|\n|**STONE**|**58.75**|**60.33**|**61.89**|\n#### **[W4] The claim that the results are SOTA does not sound tenable, insofar as the percentage improvements are only small. In addition, the diagrams indicate that other methods could deliver better results on closer inspection. In Figure 3, it appears that KECOR would provide better results if more data points and more bounding boxes were available. As these data points are missing, a more accurate comparison is not possible. Furthermore, in Table 1, the best values for cyclists in the moderate and hard cases are not highlighted in bold.**\nWe further use SECOND [1], a widely used 3D object detector, as the base model.  The results in Table 4 indicate that STONE achieves a **3.4%** higher mAP score at the hard level in 3D detection and **2.43%** higher mAP score at the hard level in BEV detection compared to the state-of-the-art method, KECOR, as shown in Table 4. This demonstrates the performance and generality of the proposed approach. Missing data can be viewed in Tables 2 and 3. We will add the highlights in Table 1 in the final version of the paper.\n#### Table 4: 3D mAP(\\%) of STONE and AL baselines on KITTI validation set with 1\\% queried bounding boxes with one-stage 3D detector backbone SECOND [1]\n|Methods| **3D Detection mAP EASY \\%** | **3D Detection mAP MOD. \\%** | **3D Detection mAP HARD \\%** | **BEV Detection mAP EASY \\%** | **BEV Detection mAP MOD. \\%** | **BEV Detection mAP HARD \\%** |\n|-|-|-|-|-|-|-|\n|RAND| 69.33±0.62 | 55.48±0.42 | 51.53±0.33 | 75.66±1.10 | 63.77±0.86 | 57.71±0.95|\n|CORESET| 66.86±2.27| 53.22±1.65 | 48.97±1.42 | 73.08±1.80 | 61.03±1.98  | 56.95±1.53 |\n|LLAL  | 69.19±3.43 | 55.38±3.63 | 50.85±3.24 | 76.52±2.24 | 63.25±3.11| 59.07±2.80 |\n|BADGE| 69.92±2.90| 55.60±2.72| 51.23±2.58| 76.07±2.70| 63.39±2.52 | 59.47±2.49|\n|CRB | 72.33±0.35| 58.06±0.30| 53.09±0.31| 78.84±0.27 | 65.82±0.07| 61.25±0.22|\n|KECOR| 74.05±0.16  | 60.68±0.13 | 55.34±0.23 | 80.00±0.12  | 68.20±0.35| 63.26±0.25 |\n| **STONE**| **76.86±0.88**  | **64.04±0.27**  | **58.75±0.58** | **82.14±0.90**  | **70.82±0.14**  | **65.68±0.42** |\n\n#### **[Q1] What about this paper https://arxiv.org/pdf/2402.03235? However, a comparison is not easy as they use the nuScenes and TUM Traffic intersection dataset**\nPlease refer to the answer to [W2] for results on NuScenes. In https://arxiv.org/pdf/2402.03235, the authors consider of case of uniform label and point cloud distributions, however, our approach can be applied to real-world imbalanced label distribution which is more practical. \n\n[1] Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cb6f6QnCcP", "reviewer": "Reviewer_Lj2A", "summary": "The paper proposes a novel submodular-based active learning approach STONE for lidar-based 3D object detection. The method then does data balancing using a greedy search algorithm. The method achieves significant improvements on the KITTI and Waymo datasets.", "questions": "Please see the weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper proposes submodular optimization which is nice.\n+ The paper achieves good improvements on the two datasets.", "weaknesses": "- The paper relies solely on PV-RCNN as the baseline detector. Quantitaively benchmarking the results with recent detectors like FocalFormer [A] and PillarNet [B] on datasets would strengthen the evaluation.\n\n- Including quantitative evaluations of baselines and STONE on any two established leaderboards: KITTI, Waymo, or nuScenes would let us know the STONE's generalizability.\n\n- Figure 3 presents performance comparisons up to 50k boxes for the proposed STONE method but only 30k boxes for the baseline Kecor. Including results for Kecor at 50k boxes would provide a fairer comparison.\n\n- The paper focuses on active learning. Could the method be adapted for unlabeled data to detect moving objects? If so, a quantitative comparison against MODEST on KITTI and Lyft datasets would be beneficial.\n\n- Figure 1 could be enhanced by illustrating the entropy after step 1 and step 2 of SDMCB to 10 active learning rounds.\n\n- While the application on the Pascal VOC dataset is interesting, quantitative benchmarking of the MS-COCO dataset with base detectors like Faster R-CNN, YOLOv5, and DETR using ResNet-50/101 backbones would be a more comprehensive evaluation for 2D detection.\n\n- The authors mention that varying degree of difficulty comes from multiple sources. I would, therefore, want to compare the 2D detection performance of a baseline and KECOR on the KITTI-360 dataset [D] which has a heavily-truncated building category [D].\n\nReferences:\n- [A] Chen et al., FocalFormer3D: Focusing on Hard Instance for 3D Object Detection, ICCV 2023\n\n- [B] Shi et al., Pillarnet: Real-time and high-performance pillar-based 3d object detection, ECCV 2022\n\n- [C] You et al., MODEST: Learning to Detect Mobile Objects from LiDAR Scans Without Labels, CVPR 2022\n\n- [D] KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D, Liao et al., TPAMI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. \n#### **[W1] Quantitaively benchmarking the results with recent detectors like FocalFormer and PillarNet on datasets would strengthen the evaluation.**\nWe evaluated our method alongside other AL baselines using PillarNet (Table 1), and we also implemented the one-stage detector SECOND [1] to draw comprehensive and robust conclusions. Our method consistently outperformed other baselines by a significant margin. The results can be viewed in Table 1 and Table 2.\n#### Table 1: 3D mAP(%) of STONE and AL baselines on KITTI with PillarNet\n| Components | 3D Detection mAP EASY % | MOD. | HARD | BEV Detection mAP EASY % | MOD.| HARD |\n|-|-|-|-|-|-|-|\n| KECOR|28.98|24.79|23.71|31.00| 28.67| 27.26|\n| **STONE**| **29.42** | **27.09**| **24.35** | **32.14** | **30.12** | **29.23** |\n#### Table 2: 3D mAP(\\%) of STONE and AL baselines on KITTI with backbone SECOND\n| Methods | 3D Detection mAP EASY \\% |MOD.|HARD|BEV Detection mAP EASY \\%| MOD.|HARD|\n|-|-|-|-|-|-|--|\n|KECOR|74.05|60.68|55.34|80.00|68.20|63.26|\n|**STONE**|**76.86**|**64.04**| **58.75** | **82.14** | **70.82**  | **65.68** |\n\n#### **[W2] Including quantitative evaluations of baselines and STONE on any two established leaderboards: KITTI, Waymo, or nuScenes would let us know the STONE's generalizability.**\nWith a more challenging dataset like nuScenes, which has 10 classes compared to KITTI and Waymo, both of which have 3 classes, our proposed method still demonstrates a performance advantage, showing that it has good generalizability.\n#### Table 3: Performance comparisons of STONE and AL baselines on the NuScenes validation set with SECOND as the backbone architecture.\n\n|Method|nuScenes detection score (NDS)|mAP|\n|-|-|-|\n|Random|37.25|28.96|\n|KECOR|46.92|40.23|\n| **STONE** | **48.79** | **41.23** |\n#### **[W3] Figure 3 presents performance comparisons up to 50k boxes for the proposed STONE method but only 30k boxes for the baseline Kecor. Including results for Kecor at 50k boxes would provide a fairer comparison.**\nKECOR method slightly outperforms the STONE method with 30k, 40k, and 50k bounding boxes (Table 4) because it queries more scenes and undergoes more training epochs. In contrast, STONE, when using the PV-RCNN backbone, tends to reach its bounding box limit early due to its preference for object-rich scenes. In active 3D object detection, the goal is to label as few bounding boxes as possible to achieve good performance. Therefore, it is critical to maintain high performance with a small number of labeled bounding boxes, as demonstrated by STONE.  With the SECOND backbone, STONE consistently outperforms other active learning baselines (Table 5). \n\n#### Table 4: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the Waymo with PV-RCNN as the backbone.\n|Method|20k (bounding box)|30k|40k|50k|\n|-|-|-|-|-|\n| KECOR| 0.358 | **0.371** | **0.382** | **0.396** |\n| **STONE** | **0.361** | 0.365| 0.369| 0.375 |\n#### Table 5: Performance comparisons of STONE and AL baselines using 3D AP(%) scores on the Waymo with SECOND as the backbone.\n|Method|20k (bounding box)|30k|40k|50k|\n|-|-|-|-|-|\n| KECOR| 0.413| 0.439| 0.473| 0.488|\n| **STONE**| **0.428**| **0.457** | **0.489** | **0.503** |\n\n#### **[W4] Could the method be adapted for unlabeled data to detect moving objects? If so, a quantitative comparison against MODEST on KITTI and Lyft datasets would be beneficial.**\nThanks for pointing this out. The proposed method is a general active learning method for 3D detection and is not specifically designed for detecting moving objects. Additionally, since MODEST does not classify different object types, a direct comparison is difficult.\n\n#### **[W5] Figure 1 could be enhanced by illustrating the entropy after step 1 and step 2 of SDMCB to 10 active learning rounds.**\nThe results can be seen in Figure 2 of the attached PDF within the general response. The red line represents the predicted labels after step one, appearing more balanced compared to the entropy results after the SDMCB (step 2). However, since in step one, we only have access to predicted labels, which can be inaccurate, the estimated entropy cannot accurately reflect the label distribution. In step two, having access to the true labels of the previously selected point clouds allows the entropy to better reflect the label distribution. The drop in entropy value is observed because the total number of objects in the KITTI dataset is highly imbalanced (14,385 cars, 893 cyclists, and 2,280 pedestrians). As more cyclists are selected in the first few rounds, fewer are available for the last few rounds.\n\n#### **[W6] Quantitative benchmarking of the MS-COCO dataset with base detectors like Faster R-CNN, YOLOv5, and DETR using ResNet-50/101 backbones for 2D detection.**\nWe use Faster R-CNN as the detector and the MS-COCO dataset, and we leverage the codebase from \"Plug and Play Active Learning for Object Detection\" presented at CVPR 2024 for a fair comparison. The results demonstrate STONE is competitive with KECOR. Critically, STONE is more memory-efficient compared with KECOR which is practically important for resource-constraint settings. \n#### Table 6:Performance comparisons of STONE and AL baselines on MS-COCO with Faster R-CNN as the backbone architecture.\n|Method| 20% (training data)|25%|30%|35%|40%|\n|-|-|-|-|-|-|\n|KECOR| **27.36**|**29.41**| 30.21| 31.09| 31.87|\n|**STONE**| 27.13|29.36| **30.36** | **31.74** |**32.11**|\n#### **[W7] Compare the 2D detection performance of a baseline and KECOR on the KITTI-360 dataset [D] which has a heavily-truncated building category.**\nDue to time and computing resource limitations, we are unable to conduct this experiment, and we will include the results in the final version of the paper. \n\n[1] Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxsAgyk3ur", "reviewer": "Reviewer_qVYS", "summary": "The paper proposes an approach to address challenges in active learning for 3d object detection to select unlabeled point clouds for further labeling. The labeling criteria maximises representativeness of the chosen point cloud with respected to an unlabeled point cloud set and also making sure that classes are balanced and have similar entropy as the labeled dataset.\nThe paper proposes two submodular functions (GBSS, SDMCB) to achieve the criteria mentioned above. They lead to better performance compared to the current SOTA. GBSSS ensures that the selected point clouds are diverse and representative. SDMCB tries to reduce the bias that arises from imbalance in classes in the dataset.", "questions": "Q1. Considering Labeled Data for Optimization. The first submodular function maximises the representative capacity of the chosen point cloud compared to the unlabeled set. Why is already labeled data not being considered in the optimization objective? Is it because the pretrained network is a representative of labeled dataset?\n\nQ2. Difficulty enforcement during active learning. It is not clear how HARD, MODERATE AND EASY is enforced during active learning since the objective function is the same and we do not know about the unlabeled pool of point clouds. Is it possible to categorize unlabeled pointclouds?\n\nQ3. Objective Function. The objective in the equation tries to maximize the term while it is mentioned as minimizing the objective in line 142. This seems a mistake.\n\n=======\n\nThose could be answered during discussion phase. I am therefore raising my rating to \"weak accept\".", "limitations": "Yes, addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Point Cloud Choice. The paper proposes a way to choose point clouds based on varying ranges of difficulty which is useful.\n\nS2. Treating Data Imbalance. The paper addresses data imbalance by choosing the point clouds unlike previous approaches. A novel weighting scheme is proposed in classification and regression losses to avoid ignoring classes that occur less frequently.\n\nS3. Marginal Performance boost. The paper achieves better performance compared to other approaches.\n\nS4. Design Justification. Tha ablation studies clearly indicate the improvement with each proposed stage, reweighting factors, selected loss terms in submodular optimization.", "weaknesses": "W1. Performance boost is marginal. Although there is some improvement in the performance, it is marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.\n\n#### **[W1] Performance boost is marginal. Although there is some improvement in the performance, it is marginal.**\nWe further use SECOND [1], a widely used 3D object detector, as the base model.  The results indicate that STONE achieves a **3.4%** higher mAP score at the hard level in 3D detection and **2.43%** higher mAP score at the hard level in BEV detection compared to the state-of-the-art method, KECOR, as shown in Table 1. This demonstrates the performance and generality of the proposed approach.\n\n#### Table 1: 3D mAP(\\%) of STONE and AL baselines on KITTI validation set with 1\\% queried bounding boxes with one-stage 3D detector backbone SECOND\n| **Methods** | **3D Detection mAP EASY \\%** | **3D Detection mAP MOD. \\%** | **3D Detection mAP HARD \\%** | **BEV Detection mAP EASY \\%** | **BEV Detection mAP MOD. \\%** | **BEV Detection mAP HARD \\%** |\n|----------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|------------------------------|\n| RAND | 69.33±0.62 | 55.48±0.42| 51.53±0.33 | 75.66±1.10 | 63.77±0.86  | 57.71±0.95  |\n| CORESET | 66.86±2.27 | 53.22±1.65 | 48.97±1.42 | 73.08±1.80| 61.03±1.98 | 56.95±1.53 |\n| LLAL           | 69.19±3.43                   | 55.38±3.63                   | 50.85±3.24                   | 76.52±2.24                   | 63.25±3.11                   | 59.07±2.80                   |\n| BADGE          | 69.92±2.90                   | 55.60±2.72                   | 51.23±2.58                   | 76.07±2.70                   | 63.39±2.52                   | 59.47±2.49                   |               |\n| CRB            | 72.33±0.35                   | 58.06±0.30                   | 53.09±0.31                   | 78.84±0.27                   | 65.82±0.07                   | 61.25±0.22                   |\n| KECOR          | 74.05±0.16                   | 60.68±0.13                   | 55.34±0.23                   | 80.00±0.12                   | 68.20±0.35                   | 63.26±0.25                   |\n| **STONE**      | **76.86±0.88**               | **64.04±0.27**               | **58.75±0.58**               | **82.14±0.90**               | **70.82±0.14**               | **65.68±0.42**               |\n\n#### **[Q1]  The first submodular function maximises the representative capacity of the chosen point cloud compared to the unlabeled set. Why is already labeled data not being considered in the optimization objective? Is it because the pretrained network is a representative of labeled dataset?**\n\nThanks for pointing this out. Yes, the pretrained model is representative of the labeled dataset as it has already been trained on the labeled point clouds. Due to this reason, only the unlabeled point clouds are considered as we aim to select unlabeled point clouds that are representative of the whole unlabeled pool which can be further labeled to improve the model performance. \n\n#### **[Q2] Difficulty enforcement during active learning. It is not clear how HARD, MODERATE AND EASY is enforced during active learning since the objective function is the same and we do not know about the unlabeled pool of point clouds. Is it possible to categorize unlabeled point clouds?**\n\nSince we do not have access to the true labels of point clouds in the unlabeled pool, categorizing them by difficulty levels is highly challenging. To address this, we use Monte Carlo dropout to estimate the true labels. These estimated labels are then utilized in Gradient-Based Submodular Subset Selection (GBSSS). By employing a feature-based submodular function, we select representative samples from the unlabeled pool, aiming to cover samples of varying difficulty levels. The results in Table 1 of the paper demonstrate that our method achieves the best performance across all difficulty levels, highlighting the effectiveness of our proposed approach.\n\n\n#### **[Q3] Objective Function. The objective in the equation tries to maximize the term while it is mentioned as minimizing the objective in line 142. This seems a mistake.**\n\nSorry for the confusion. In line 142, we mentioned that we aim to minimize the absolute difference between $f_1(D_U )$ and $f_1(D_S )$. Since $D_S \\subset  D_U$, from the property of submodular function we have $f_1(D_S ) \\le f_1(D_U )$. In Equation 3, we maximize $[f_1(D_S) - f_1(D_U)]$, which is equivalent to minimizing the absolute difference between $f_1(D_U )$ and $f_1(D_S )$. We will clarify this further in the final version.\n\n[1] Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Figures for Rebuttal

Meta-review: This paper introduces a method for active object detection in 3D data based on submodular optimization, reducing labelling costs. It received favorable reviews from five experts, who appreciated the idea and novelty, SOTA performance on several datasets, breadth, presentation and writing.

Some weaknesses were raised, generality of submodular functions, significance of gains, missing information, requests for some additional experiments, a single baseline detector, minor doubts on experiments, some missing references.

The rebuttal provided by the authors could address several concern and provided additional experiments, including on semantic segmentation, which was satisfactory for all reviewers.

A consensus on publication was quickly reached, the AC concurs.
Decision: Accept (poster)