Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the task of 3D object detection with only a small portion of labeled data. Conventional active learning-based approaches show a promising solution, but they fail to balance the trade-off between point cloud informativeness and box-level annotation costs. To overcome this limitation, this paper jointly investigates three novel criteria for point cloud acquisition, i.e., label conciseness, feature representativeness and geometric balance. Extensive experiments are conducted on KITTI and WOD benchmarks to demonstrated the superiority of the proposed method.
strength_and_weaknesses: **Strength**
1. The motivation of this paper is solid, and the proposed framework CRB is novel.
2. The paper is well written.
3. Extensive experiments are conducted to demonstrate the superiority of the proposed method.

**Weakness**
1. In Sec. 2.1, I am very confused how to annotate a point cloud $\mathcal{P}$ with a bounding box $\mathcal{B}$ and its associated box label $\mathcal{Y}$?
2. From Fig. 2 in the main paper and Fig. 5 in the Suppl., the proposed method seems to get inferior results when a low budget is required. Considering the scale of Waymo is bigger than KITTI, what is the performance on WOD under the same setting as Tab.1?
3. In B.2, there are several hyper-parameters about active learning protocols. So how to set values of them? From Tab.3, results are sensitive to the setting of $\mathcal{K}1$ and $\mathcal{K}2$, does both thresholds need to be carefully tuned when different number of annotated bounding boxes are available?

clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to follow and understanding. Codes are available in the submission. The proposed method is novel to me.
summary_of_the_review: This paper proposes a novel CRB framework for 3D object detection with active learning. The proposed method is novel, and achieves superior performance on both KITTI and WDO benchmarks. Although there are some questions mentioned above, I think the overall quality of this paper is good.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed three new criteria for 3D box-level active learning in LiDAR object detection. The first part(Eq. 4) encourages selecting more class-balance point clouds. The second part advocates selecting point clouds with more diverse gradients w.r.t. pseudo classification and regression labels. The las thet part favors point clouds with similar object point density as of the labelled set. Empirical studies showed the proposed three components are mostly effective and out-perform existing active learning methods on some datasets.
strength_and_weaknesses: Pros:
1. The authors gave a good summary of existing active learning methods for LiDAR object detection, which highlighted the significance and popularity of the problem under study.
2. The authors developed an active-3D-det toolbox which could be quite helpful for setting fair and rigorous baselines for future researches.
3. Experiments on KITTI showed promising results.

Cons:
1. There are quite some factual errors in the method part of this work.
    1. PointNet series did not encode free-point cloud into a 3D tensor. On the bottom of Page 2, “Mainstream 3D object detectors encode free-form point clouds P into a regular feature map x ∈ R W ·L·F directly (Qi et al., 2017) or by voxelization (Shi et al., 2020), with W, L, F representing width, length, and channels of the feature map.” 
    2. Wrong assumptions about loss function. The authors assumed that “The loss functions for classification and regression are assumed to be Lipschitz continuous”, but a simple binary cross entropy loss is not Lipschitz continuous for p in (0, beta]. Also I failed to connect the Lipschitz continuous of loss function with anything in Section 2.2.
2. Section 2.2 is quite vague to me, I don’t how Theorm 2.1 is connected to Eq 1 and what are Ps in Eq 1
3. A uniform prior of target labels as assumed in Eq. 2 is too far from the observed reality in LiDAR object detection 
4. Stage 2 in section 2.3 is quite hard to understand. Each step in Eq. 7 seems too big a leap for me and proper discussion are not made in the text part.
5. Only the KITTI experiments showed promising results. For the Waymo experiments in Figure 2, CRB is out-performed by LLAL and ENTROPY in the low data regime while almost on par with RAND baseline in the high data regime. I find there are implementations for other datasets like nuScenes and Lyft in the supplementary codes, which makes me wonder if the method generalizes well across different datasets.
clarity,_quality,_novelty_and_reproducibility: Questions:
1. Why does GPDB alone in Table 2 has so wide an error bar and underperform the baseline?
2. Why different lines in Figure 2 have different sample point w.r.t Number of Selected Bounding Boxes ?
summary_of_the_review: It is easy to see that the authors put a lot of efforts in this work and tried to connect the proposed selection criteria with some theoretical supports. Although there are some errors and unclear writings in the method part, but the proposed methods are backed up by at least solid experiments on the KITTI datasets, so I am leaning to recommend for its acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores how activate learning can help reduce annotation costs in the field of object detection of 3D point clouds. Specifically, to achieve efficient active learning with limited fixed annotation budgets, three selection criteria termed CRB are proposed to learn better sample acquisition of the 3D boxes annotation. Extensive experiments on KITTI and Waymo validate the effectiveness of the proposed method. The paper is well-written and easy to follow. The experimental results demonstrate the effectiveness of the proposed hierarchical active learning scheme in active learning for 3D object detection.

strength_and_weaknesses: ### Srength: 
- Activate learning in point cloud object detection is interesting and novel. I believe this paper can provide insights and inspiration to the related works in the 3D object detection area.

- The theoretical analysis is sufficient and helpful to understand the method, and also provide support for the final performance validation.

- The experiments are extensive and can validate the effectiveness of the proposed framework.


### Weaknesses: 
- I would suggest the authors clearly clarify the major difference of the paper with existing works on object detection and point clouds processing. It seems that this paper is analyzed in an general optimized view, but it will be better if the author can explicitly illusrate the innovation and the fundamental difference with exisiting techniques.

- For related work, I would suggest that the author add literature on 2D Detection and some of the latest works in 3D point cloud detectors. Just name a few: 

	1. QBox: Partial Transfer Learning With Active Querying for Object Detection [2021, TNNLS]
	2. Towards Dynamic and Scalable Active Learning with Neural Architecture Adaption for Object Detection [2021, BMVC]: Add NAS into the AL loops.
	3. Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint [2022, CVPR]
	4. Weakly Supervised Object Detection Based on Active Learning [2022, NPL]
	5. Active Learning Strategies for Weakly-Supervised Object Detection [2022]
	6. Label-Efficient Point Cloud Semantic Segmentation: An Active Learning Approach [2021, CVPR]
	7. ReDAL: Region-based and Diversity-aware Active Learning for Point Cloud Semantic Segmentation [2021, ICCV]
	8. Active Learning for Point Cloud Semantic Segmentation via Spatial-Structural Diversity Reasoning [2022]

- As for the experimental results in Table 1, it is a little bit counterintuitive: the proposed CRB with 1% labels can perform better than the general full-label training model. In my view, the active learning methods can achieve the performance very close to the full-label training model. The authors are suggested to explicitly clarify why the performance is even better than fully supervised model.

-More experiments on various detectors (e.g., SECOND, PV-RCNN) are required to illustrate the effectiveness of the proposed criteria for general active 3D object detection. Additionally, indoor scenarios such as ScanNet should also be taken into consideration.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, and seems solid.

There are a number of literatures on active learning in 2D object detection. Referring to the development of general 2D detectors to 3D detectors, this paper seems adapt data based on 3D point cloud data and 3D bounding box labeling, and the novelty is not well-clarified.
summary_of_the_review: I'm not quite familiar with this topic, but this paper looks very comprehensive and solid, so I'm inclined to borderline accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the problem of active 3D object detection from point clouds. The main problem addressed in this paper is how to design a proper criterion for cost-efficient sample acquisition. In this work, the authors jointly investigate three novel criteria in the framework CRB for point cloud acquisition (label conciseness, feature representativeness and geometric balance) which hierarchically filters out the point clouds of redundant 3D bounding box labels. 

In particular, (1)To alleviate the issues of label redundancy and class imbalance, the entropy of bounding box label predictions are used to pick top *K*1 point clouds. (2)Monte Carlo dropout and the hypothetical labels by averaging predictions from multiple stochastic forward passes are used for considering the impact of classification and regression objectives on gradients. (3) Finally, to maintain the *geometric balance* property, we minimize the KL divergence between the marginal distributions of point cloud density of each predicted bounding box. 

strength_and_weaknesses: Strengths:

The paper is well written, the experiments are convincing, and the proposed method is simple to implement. Figure 1 nicely illustrates the overall framework of the method.

The work presents an exhaustive experimental evaluation which demonstrates the effectiveness of the presented approach on KITTI and Waymo datasets. 

Ablation study (Table 2) and complexity analysis  (Table 4) support the experimental well.

Weaknesses:

Although the complexity analysis is given in the article, no specific time cost is given.

These experiments are all carried out under the autonomous driving dataset. Does this structure work for the 3D detection task of the indoor dataset?
clarity,_quality,_novelty_and_reproducibility: Heading 3.2 (line 3): I would suggest to use "300 and 200 for KITTI" instead of "300, 200 for KITTI" since the latter is easily mistaken for 300200.
summary_of_the_review: This paper is overall a goog paper, my main concern is the above weakness .
The author's response addressed my concerns well, so my final recommendation is accept
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agree that the paper is solid, novel, and well-written. They agree that the experiments are correct and show the superiority of their method. They also point out that the toolbox they have created will be helpful for future research.

There were concerns regarding notation, results of some experiments that required further details, factual errors on some claims, more details on literature review, additional experiments, specific computation cost, and questions regarding whether this method can be used indoors.

The authors carefully addressed most of the reviewer's concerns and submitted a reviewed manuscript version. Accordingly, I recommend accepting this article.
justification_for_why_not_higher_score: I recommended it as a spotlight. The method is good, but I think there will be more interesting papers for oral given the constraints of time. Maybe something of a more general purpose.
justification_for_why_not_lower_score: The paper is good. The authors did a great work addressing the reviewers comments.
Decision: Accept: notable-top-25%