Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection

Authors: huang-yu  chen
jia-fong  yeh
jia-wei  liao
pin-hsuan  peng
winston h hsu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 19:45:07

Abstract: LiDAR-based 3D object detection is a critical technology for the development of autonomous driving and robotics. However, the high cost of data annotation limits its advancement. We propose a novel and effective active learning (AL) method called Distribution Discrepancy and Feature Heterogeneity (DDFH), which simultaneously considers geometric features and model embeddings, assessing information from both the instance-level and frame-level perspectives. Distribution Discrepancy evaluates the difference and novelty of instances within the unlabeled and labeled distributions, enabling the model to learn efficiently with limited data. Feature Heterogeneity ensures the heterogeneity of intra-frame instance features, maintaining feature diversity while avoiding redundant or similar instances, thus minimizing annotation costs. Finally, multiple indicators are efficiently aggregated using Quantile Transform, providing a unified measure of informativeness. Extensive experiments demonstrate that DDFH outperforms the current state-of-the-art (SOTA) methods on the KITTI and Waymo datasets, effectively reducing the bounding box annotation cost by 56.3% and showing robustness when working with both one-stage and two-stage models.

Main paper: 
 introduction: 
 Research on LiDAR-based 3D object detection [1, 2] has emerged due to its significant potential and applications. However, annotating LiDAR data requires locating multiple objects in threedimensional space, which is expensive and time-consuming. Therefore, obtaining annotated data more efficiently within limited time and resources has become a crucial and unavoidable issue. Several studies have attempted to reduce annotation costs through Auto Labeling [3, 4] or Domain Adaptation [5, 6] , which rely on a small number of annotated samples. However, unstable annotation accuracy and restricted domain shifts may limit their usage in various applications.Active Learning (AL), aiming to select the most informative samples from a large pool of unlabeled data for human annotation, is a significant solution to reduce the model's dependence on the amount of data. Although AL has been proven effective in mitigating annotation costs in various research domains [7, 8] , its application to LiDAR-based 3D Object Detection is underexplored, with three main challenges remaining unresolved: (1) Compared to 2D object detection, LiDAR-based object detection has additional geometric features (such as rotation and point density) that need to be considered. (2) While the detector's predictions are instance-level, the selection in AL is framelevel, making it challenging to propagate from instances to frames and estimate informativeness. (3) Aggregating multiple indicators under different units and scales is also challenging.Previous work has used general AL strategies such as entropy and ensemble methods to estimate uncertainty. However, they neglected the unique geometric features in LiDAR-based object detection (first challenge), and solely assessing the instance-level informativeness is insufficient to address the second challenge. A recent work, CRB [9] , proposed three heuristic methods to estimate label balance, representativeness, and point density balance through a staged filtering approach. Unfortunately, the filtering order significantly affects sampling results, impacting the fairness among indicators and failing to solve the third challenge. Another work, KECOR [10] , proposed a kernel coding rate maximization strategy. However, it also did not consider geometric features and used different weighted settings to aggregate multiple indicators in different datasets, affecting generalization.To address the above challenges, we propose the Distribution Discrepancy and Feature Heterogeneity (DDFH) method, as illustrated in Fig. 1 , where components (a), (b), and (c) are designed for the first, second, and third challenges, respectively. DDFH leverages model embedding and geometric information as features for informativeness estimation to address the first challenge. Moreover, we explores informativeness from instance-level and frame-level perspectives by considering intraclass Distribution Discrepancies (DD) and intra-frame Feature Heterogeneity (FH) to tackle the second challenge. Then, DDFH employs a Quantile Transform (QT) to normalize each indicator to the same scale, effectively aggregating the indicators to solve the third challenge. Finally, we propose Confidence Balance (CB) to evaluate the allocation of annotation resources. Unlike previous methods that solely count selected instances for each category, CB considers the summation of confidence levels for each instance within the same category.We verify the effectiveness of DDFH through experiments on real-world datasets, KITTI and Waymo Open Dataset. The results indicate that our DDFH method outperforms the existing SOTA, effectively reducing the data annotation cost by 56.3% and achieving an average improvement of 1.8% in 3D mAP with the same amount of data. From our extensive ablation studies, DDFH also demonstrates its generalization when used with both one-stage and two-stage detection models. 
related work: 
 LiDAR-based 3D Object Detection. LiDAR-based object detection techniques are primarily divided into two categories: point cloud direct processing and voxelization. Methods such as the PointNet series [11, 12] operate directly on point clouds, preserving the original spatial accuracy of the data but are less efficient in handling large-scale data. Recent research, like PointAugmenting [13] , introduces cross-modal augmentation, enhancing LiDAR point clouds with deep features extracted from pre-trained 2D object detection models, thereby improving 3D object detection per-formance. Voxelization methods, such as VoxNet [14] and SECOND [15] , convert point clouds into voxel grids to enable efficient 3D convolution, significantly increasing computational speed. The Voxel Transformer (VoTr) [16] architecture effectively expands the model's receptive field, enhancing the ability to capture large-scale environmental information. The PV-RCNN series [17, 18] improves detection accuracy and processing efficiency by fusing point cloud and voxel features. These detectors rely heavily on large volumes of high-quality training data; however, the labeling cost for LiDAR data is quite expensive.Active learning for object detection. Active learning selects the most informative samples for annotation, thus mitigating the model's dependence on the volume of data. Numerous general active learning strategies currently exist, such as those based on model uncertainty [4, 19, 20, 21, 22] , diversity [23, 24, 25] , or hybrid methods that combine both approaches [26] . Estimating in the gradient space is also a common approach (e.g., BADGE [27] , BAIT [28] ). Research applying these methods to object detection [29, 30, 31, 32] remains limited. Many studies directly use general strategies like maximum entropy [33] , bayesian inference [34] to estimate uncertainty in both bounding box and category. LT/C [35] introduces noise-perturbed samples and assesses tightness and stability based on the model's output. The estimation of information quantity through the output probability distribution is also a common approach [36, 37] . Research on LiDAR-based object detection is even scarcer, mainly due to the high computational cost of point cloud processing and the higher dimensionality of regression information. General AL methods (Shannon Entropy [38] , ensemble [29] ) do not consider geometric features and therefore are not well-suited for LiDAR. Recent work, such as CRB [9] , proposes three heuristic methods to incrementally filter samples. KECOR [10] identifies the most informative samples through the lens of information theory. However, these studies fail to effectively integrate multi-level information and do not consider the distribution of selected samples, leading to redundant annotation costs. Therefore, we propose DDFH, which estimates the informativeness based on the distributional differences of instances and intra-frame feature heterogeneity, and effectively aggregates multiple indicators using the quantile transform to estimate multi-level informativeness. 
active learning setup: 
 In the active object detection setup, the labeled set D L = {(P L , Y L )} contains a small amount of point clouds P L with annotations Y L , and D U = {P U } represents a large unlabeled set of raw point clouds P U . Initially, samples are randomly selected to form D L , and the detection model learns over multiple rounds r ∈ {1, ..., R}. The objective of active learning is to evaluate the informativeness of P U in each round and select the most informative samples to form a new subset D * s for human annotation. Then, the D * s is merged into D L to start a new round to retrain the model. This process repeats until the size of the labeled set reaches the annotation budget.We propose a novel active learning framework, Distribution Discrepancy and Feature Heterogeneity (DDFH) for LiDAR-based 3D object detection. As illustrated in Fig. 2 , we infer point clouds P ∈ {D U , D L } into the model and get model output containing embeddings and bounding boxes. However, estimating distributions in high-dimensional space is challenging, so we use t-SNE [40] to project embeddings into lower dimensions while retaining important information, denoted as f e * ∈ R 2 . The geometric features of LiDAR-based object detection (length, width, height, volume, rotation, and point cloud density) f g ∈ R 6 are also significant, as they convey direct information about objects such as occlusion, behavior, and morphology. So we use f = [f e * ⊤ f g ⊤ ] ⊤ ∈ R 8 as the input feature for DDFH, calculating multiple indicators to estimate informativeness. However, since the units of the indicators differ, normalization is required before aggregation. Score Normalization. DDFH evaluates informativeness based on multiple indicators, but their scales differ, making direct aggregation impossible. Thus, we use Quantile Transform ψ to normalize. ψ(., .) is a non-linear transform that converts the first input to follow a normal distribution and returns the transformed result of the second input. ψ spreads out the most frequent values and reduces the impact of outliers. Since the goal of active learning is to select the top k samples, the relative distance of scores is not particularly important. Instead, maintaining the ranks of various indicators and reducing outliers aids in aggregating the indicators, making ψ a crucial bridge in computing I total . Next, we will introduce the operating principles of DDFH sequentially.Since labeled set is much smaller than the unlabeled set, estimating the distribution is particularly important. Reducing the distribution gap between the labeled set and the unlabeled set will assist the model in inference. Inspired by [37] , we use a Gaussian Mixture Model(GMM) to estimate probability density. Unlike previous works, we consider both geometric features and embeddings, using dimensionality reduction to avoid overly sparse space. Intuitively, if an instance appears frequently in the unlabeled set but is rare in the labeled set, such an instance can help the model efficiently understand unlabeled samples. Therefore, for each class c ∈ C, we establishG L c fit on {f c,1 ...f c,N L c } and G U c fit on {f c,1 ...f c,N U c },where G U c is a GMM fit on unlabeled features, and N U c is the number of instances with class c in the unlabeled set. We estimate the probability density function of each instance in the unlabeled and labeled sets, and calculate the discrepancy score s dd i :EQUATIONP G U c (f i,c) represents the probability density of f i,c in the unlabeled set. However, considering S dd alone is insufficient, as very dense instances might overly influence the indicator, leading to frequent selection of dense but repetitive instances in the early stages. Hence, unlike previous works [7] , we also extract P G L c (f i,c ) to calculate the novelty score s nov :s nov c,i = -P G L c (f c,i ), i = 1, 2, ..., N U c .(2) s nov ensures the novelty of instances. If an instance has a high probability density in the labeled set, it indicates that the instance has already been selected, thus the s nov score will decrease, effectively reducing redundant annotation costs. Following these two indicators, I dd (P j ) can be calculated as:EQUATIONwhere N is the number of instances in P j . S dd is the set of all instances' s dd . Ablation studies show that I dd enables the active learning model to learn rapidly with a small amount of data. However, I dd does not consider frame-level information, which may overlook similar instances within the same frame. Therefore, we introduce a second component to address this issue.Object detection is a multiple-instance problem. Considering only instance-level information can lead to redundant annotations. Complex scenes usually contain numerous objects that share the same lighting and environmental factors, making their features highly similar or following linear variations. Such samples are costly to annotate but offer limited assistance to the model. Therefore, we propose frame-level Feature Heterogeneity (FH). As shown in Fig. 1b , we decompose heterogeneity into correlation and variance. DenoteF U j,c = [f U j,c,1 ...f U j,c,m ] as the feature vector of all instances with class c in j-frame, where m represents the number of instances. FH ensures F j,c can maximize the feature heterogeneity of the labeled instance vector F L c . Specifically, we combine the two intoFj,c = [F U j,c F L c ] ∈ R 8×Nwhere N as the number of all sample. We use covariance cov and variance σ 2 to calculate the Pearson correlation ρ, ensuring non-linear variations among features. The correlation ρ is calculated as:EQUATIONwhere F k j,c and F ℓ j,c represent the k-th and ℓ-th feature dimension of matrix Fj,c , andF k j,c is the mean of F k j,c . σ 2 ( F k j,c ) = 1 N ∥ F k j,c -F k j,c ∥ 2 2, N f is the number of feature dimensions of matrix Fj,c . The smaller the value of ρ, the less linear the correlation, enabling the model to learn more feature combinations. However, as shown in Fig. 1 , considering only correlation overlooks the information about the distance between features. Thus, we also consider σ 2 to ensure sufficient variation among features. Based on these two indicators, we calculate s cor and s var :EQUATIONThe closer ρ is to 0, the less linear the correlation between features. Conversely, values far from 0 indicate positive or negative correlations. Therefore, we take the absolute value of correlation and subtract it from 1, making the s cor j,c indicator larger. Based on these two scores, we calculate the feature heterogeneity informativeness I f h (P j ) for P j :EQUATIONwhere S var is the set of all s dd . I f h ensures instances maintain low correlation and high variance, calculated through multiplication. This introduces the core components of the DDFH approach, exploring informativeness from both instance-level and frame-level perspectives.Balancing annotation costs across classes has always been crucial in active learning. Previous works [10, 9] calculate entropy by the number of all categories and the classification logit from the classifier, termed Label Balance (LB). However, these method is limited in imbalanced datasets, as imbalanced classes usually have lower confidence, resulting in more false-positive instances. The actual quantity is often less than expected. Therefore, we propose Confidence Balance (CB) I cb , replacing instance quantities with the sum of confidences in each category to better reflect the true class distribution in the frame, enhancing the number of minority classes. I cb can be calculated as follows: where p j,c represents the sum of confidences of all instances of class c in the j-th frame. We compare the effectiveness of LB and CB sampling in Fig. 4(c-d ), demonstrating the importance of I cb for imbalanced data.EQUATIONAs described in 3.2, DDFH combines multiple indicators to explore informativeness comprehensively. Through QT, all indicators are normalized to compute a unified informativeness indicator I total , ensuring that the top-k frames are efficient, novel, heterogeneous, and balanced, identifying the optimal selected sets D * s , formulated as: D * s = arg max Ds⊂D U I total (P U ), I total (P j ) = (I dd (P j ) + I f h (P j )) • I cb (P j ).I dd and I f h evaluate informativeness, and multiplying by I cb ensures balanced annotation costs across classes while considering informativeness. 
experimental settings: 
 3D Point Cloud Datasets. We tested our method on two real-world datasets: KITTI [43] and Waymo Open Dataset [44] . Baselines. We comprehensively evaluated 6 general active learning (AL) methods and 6 AL methods for object detection. RAND selects samples randomly. ENTROPY [45] and LLAL [42] are uncertainty-based methods. CORESET [24] is a diversity-based method. BAIT [28] and BADGE [41] are hybrid methods. MC-MI [38] and MC-REG [9] utilize Bayesian inference. CONSENSUS [29] employs an ensemble to calculate the consensus score. LT/C evaluates instability and local- ization tightness. CRB [9] progressively filters based on three heuristic methods. KECOR [10] identifies the most informative sample through the lens of information theory.Evaluation Metrics. We follow the work of KECOR [10] Implementation Details. We strive to avoid excessive parameter tuning by using a unified set of hyper-parameters across all experiments. We set the perplexity for t-SNE to 100. For Gaussian Mixture Model, set number of components to 10, reg_covar to 1e-2 to increase generalization and initialize parameters by k-means++.Since the most of baseline method do not provide initial selection samples, to present the results more fairly, we reproduce most of the baseline methods using the same initial settings and provided the official reported performance of two recent methods (CRB and KECOR) in our experiments.DDFH with Two-Stage Detection Model. For the KITTI dataset, We ran each method three times. As show in Fig. 3 , our method outperforms all baseline methods. Compared with the CRB and KECOR, the annotation cost is reduced by 51.7% and 33.2%, respectively, especially when the number of annotations is small, the growth rate is particularly fast. In Table 1 , we follow the KECOR setting, showing the performance with 800 (1%) bounding box annotations. Under the same initialization conditions, our average performance surpasses the SOAT by 2.23%, especially improving by 4.17% in the imbalanced class (Cyclist). For the Waymo Dataset, in Fig. 4(a-b ), the Level-1 and Level-2 performance are reported, respectively. Compared to KECOR and CRB, DDFH improves APH by 1.8% and 3.8%, respectively, and reduces the annotation cost by 56.3% and 66.4%, respectively, demonstrating the effectiveness of DDFH in more diverse and complex scenarios.DDFH with One-Stage Detection Model. Table 2 reports the 3D and BEV mAP scores with 1% annotation bounding boxes. Compared to the SOTA, it improves by approximately 2.8% in 3D mAP and 2.28% in BEV mAP. In Fig. 5 , we further report the performance growth trend for each category in different levels of difficulty in MOD. DDFH, especially in the car category, which often leads to excessive annotations, can quickly improve performance with the most streamlined annotation cost.For the pedestrian category, the uncertainty-based method performs exceptionally well with a small number of annotations. However, the lack of consideration for diversity limits the performance. Efficacy of Confidence Balance. As shown in Table 3 , CB effectively improves the 3D mAP of the imbalanced class (Cyclist) by 4.3% compared to LB. From Fig. 4c , it is evident that the performance using LB decreases by an average of 2.8% 3D mAP in each round. In Fig. 4d , we further demonstrate the label entropy of the selected samples. DDFH can effectively allocate annotation resources in each round, while KECOR sacrifices balance while considering informativeness.Efficacy of Distribution Discrepancy and Feature Heterogeneity. The results in Table 3 show that DD can significantly enhance overall performance, especially in categories with fewer instances, such as pedestrians and cyclists, by focusing more on the selection of these categories due to their larger distribution differences. Using FH alone to estimate informativeness without considering the labeled distribution would be too limiting. The experiments demonstrate that DDFH effectively combines the advantages of both components, resulting in improvements across all categories.Efficacy of Geometric Features. In Fig. 4c , the results show that after considering geometric features, DDFH improves the average performance by 1.6% in 3D mAP, confirming that diverse geometric features help the model capture a wider variety of objects. 
conclusion: 
 We The results show that DDFH has a significant advantage in each active round. We present more Considering that the distribution of objects in real environments is often uneven, common objects 40 tend to occupy the majority (e.g. cars). This leads to the underestimation of less frequent categories 
 
 Appendix:  


Decision: Summary: This paper proposes DDFH, an active learning method for LiDAR-based 3D object detection that reduces annotation costs by 56.3% by simultaneously considering geometric features and model embeddings, ensuring feature diversity and efficient learning with limited data, outperforming current state-of-the-art methods on the KITTI and Waymo datasets.

Strengths:
* Proposes a novel active learning method (DDFH) for LiDAR-based 3D object detection, addressing challenges by enhancing instance distributional discrepancy and frame-feature heterogeneity.
* Outperforms state-of-the-art methods on KITTI and Waymo datasets.
* Includes extensive experiments and comparisons with multiple recent baselines, validating the proposed approach.

Weakness:
* Some key aspects, such as GMM-based distribution discrepancy and feature heterogeneity calculations, are not clearly explained and need more theoretical and empirical justification.
* Inconsistencies in the number of maximum selected boxes across different approaches in graphs make comparisons less clear.
* The method's performance on imbalanced classes like Cyclist and Pedestrian shows inconsistencies, with certain combinations decreasing performance.
* The paper does not discuss potential limitations or future research directions.

----

Post rebuttal: The reviewers are happy with the authors' replies and have reached a consensus on accepting the paper.

Reviews: review: title: title: Review of submission 329
paper_summary: paper_summary: This paper proposes an active learning method for LiDAR-based 3D object detection, by enhancing instance distributional discrepancy and frame-feature heterogeneity. The final information score is normalized to find the best data to annotate in future training for the largest data efficiency. Experiments have been conducted on different datasets to evaluate the proposed results.
main_review: main_review: The paper is well-organized and the main idea is easy to follow. Active learning is important to data efficiency, LiDAR-based 3D object detection is crucial for autonomous driving, and the ground truth bounding boxes are challenging to obtain in the real world. So the motivation is sound. The experiment is extensive, involving different datasets and comparing multiple recent baselines. I have some concerns about the method part.

- For the GMM-based distribution discrepancy, with the labeled GMM fitting $G_c^L$ and unlabeled GMM fitting $G_c^U$, how can you correctly align each component of $G_c^L$ and $G_c^U$ to find the distribution discrepancy correspondingly? i.e. in Eq (1), maybe the distribution of $\textbf{f}$ in $G_c^L$ is totally different from $G_c^U$ as they are based on different and misaligned Gaussian components. Also, how to dynamically specify the number of components in GMM fitting in the active learning process?  

- For the feature heterogeneity, the covariance and correlation in Eq. (4) are for different dimensions of the mixed features of labeled  $F^L$ and unlabeled $F^U$, but there may be some noise correlation within labeled  $F^L$ or unlabeled $F^U$. Why not directly calculate the correlation between labeled  $F^L$ and unlabeled $F^U$ for each dimension? Also, Eq. (5) seems to be too heuristic and lacks math formulation, and I wonder if Mahalanobis distance between features can be adopted to measure heterogeneity. More theoretical and empirical justifications are needed.

- The Quantile Transform $\psi$ is not formally defined, when it is mentioned in Eq (3) and (6). It is expected to mathematically define the nonlinear transform $\psi$. Besides, the introduction of score normalization is not clear and confusing, especially the right part in Figure 2. For the left part of Figure 2, is the bounding box information also incorporated into the input feature of DDFH? If so, for the unlabeled samples, how do we obtain the input feature since there are no bounding boxes?

- It is not clear why LB is limited in 3D object detection and the formulation of CB seems to be trivially modified from LB. In the experiment, Table 3 shows that CB performs better than LB without DD and FH for imbalanced classes like Cyclist, but after DD and FH are added, it seems that the performance of imbalanced classes like Cyclist decreases even with CB. Does it mean that CB does not work well together with DD and FH? It is expected to show how LB works together with DD and FH.

- Some minors: Line 82 "Estimating ...[36] [37]" is not a complete sentence. Line 152, notation $N$ is not defined and seems to be confusing. The notation of $\hat{F}$ in Eq (4) is not consistent with $\tilde{F}$ on line 154-155.
questions: questions: See Review part.
limitations: limitations: 2
review_summary: review_summary: Concerns about the soundness of the proposed method, some incomplete experiments
title: title: Review of Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection
paper_summary: paper_summary: The paper introduces a novel active learning method called Distribution Discrepancy and Feature Heterogeneity (DDFH), which considers both geometric features and model embeddings to assess information from instance-level and frame-level perspectives. DDFH generates several scores, including discrepancy score, novelty score, correlation, and variance score. These multiple indicators are then aggregated using Quantile transform to provide a unified measure of informativeness. Extensive experiments demonstrate the benefits of the proposed approach, as it outperforms the current state-of-the-art methods on the KITTI and Waymo datasets.
main_review: main_review: Strength:
1.  The paper is well-written, easy to comprehend, and logically organized into sections.
2.  The figures effectively complement the text, facilitating easy understanding of the tasks.
3. The proposed approach is well desgined according to address the challenges of active learning for LiDAR-based 3D Object Detection
4.  The proposed approach outperforms existing state-of-the-art methods on both KITTI and Waymo dataset.

Weakness:
1. Clarification on Graph Interpretation: It is unclear why a specific number of maximum selected boxes was chosen for the graphs individually for each approach. Why not use a consistent number, such as 4000 selected boxes, across all approaches? From the plot, the reported approaches (for 4000) seem to saturate by the time they reach 4000 selected boxes. Using a consistent number of selected boxes, for all approaches would provide clearer comparisons on their efficacy. For example, DDFH, KECOR, and CRB show growth, but it is uncertain if they would saturate if 4000 boxes were used. Given that obtaining 4000 annotations is relatively insignificant compared to the total of 80000 annotations, using this number would enhance the comparability and clarity of the results.

2. In my opinion, the main table should also include the results of the PV-RCNN and SECOND approaches using the full dataset. This addition would provide a clearer understanding of how close AL methods are to the upper limit of performance.

3. The figure placements are a bit odd. In the main results section, Figure 5 is referred to on page 7 after Figure 3, and Figure 4 is referred to on page 8. Given that Figure 5 contains the main results, the figures should be rearranged for a smoother flow. Additionally, Figure 5 reports different results for two different datasets. The caption should clearly mention that parts (c) and (d) are for the KITTI dataset.

4. Could you provide some intuition for the results in Table 3? Transitioning from LB to CB improves the average mAP score due to a significant increase in the Cyclist category, but the performance on pedestrians decreases. Adding FH further decreases performance for both Pedestrians and Cyclists. Why do you think this happens? Similarly, adding DD results in a drop for Cyclists. Do you have any speculation on why this occurs? Additionally, I recommend presenting results for the combination of DD+FH+LB.
questions: questions: Please address the points listed under weaknesses, especially points 1 and 4.
limitations: limitations: 1
review_summary: review_summary: I recommend a weak accept, as there is a definite use case for the proposed work. However, if there were an option for a borderline recommendation, I would have chosen that until clarity is provided for weaknesses 1 and 4.
title: title: Review of DDFH for Active 3D Object Detection
paper_summary: paper_summary: The paper proposes a metric for active learning to choose a lidar frame for human annotation to maximize informativeness of the new labeled data with respect to existing labeled data. The proposed method uses three metrics to estimate informativeness: instance-level distribution discrepency, frame-level feature homogeneity, and confidence balance.
main_review: main_review: The authors present a framework for active learning over segmented lidar data from the Waymo and KITTI datasets. The proposed method minimizes the cost of human annotations while maximizing prediction accuracy using frame-level heterogeneity, instance-level distribution discrepancy, and confidence balance for imbalances classification data.

The paper validated the proposed methodology against 12 state-of-the-art approaches and demonstrates improvements over all methods for birds-eye view and 3D mAP evaluation metric as well as minimizes the annotation cost compared to these methods. Overall, the authors have present their work clearly, concisely, and convincingly, and provide an ablation study to validate their choices. I would note that the authors do not discuss limitations of their work, or even future research directions. I would be interested to see these elements included. Other than some minor grammatical errors and adding this limitations section, I see no additional required edits.
questions: questions: Fix minor grammatical errors and add a limitations section.
limitations: limitations: 1
review_summary: review_summary: Great presentation and compelling results. Overall an excellent paper.