Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to address object detection from LiDAR point clouds. The key idea is to train a densify network that can densify 3D features in a sparse point cloud so that they match the quality of the 3D features trained on dense point clouds. This densify network can be plugged into different object detection networks and enhance their performance - this is demonstrated in experiments.

strengths_and_weaknesses: + The S2D network is a lightweight, plug and play network, that can enhance the performance of different object detection networks
+ Lots of good design and engineering choices, e.g., voxelizing dense point cloud, design of S2D network, point cloud reconstruction network, etc
+ The experiments are done thoroughly and show improvements on a number of different object detection networks

- Lacks discussion of the reconstruction quality - it would help readers to understand if S2D only learns the feature in latent space, or actually starts to learn 3D dense geometry

questions: Will the code be released after publication?

limitations: This is a well written paper. Good idea. Good execution and good experiments. I do not see strong limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a knowledge-distillation approach to learning densified 3D features for outdoor 3D object detection. Concretely, the authors propose to train two networks DDNet and SDNet for 3D detection. During the training, DDNet will take densified 3D point cloud as inputs~(through multiple frame aggregation) while SDNet only takes single-frame Lidar point cloud. During the training, features computed from SDNet are matched with the corresponding features computed through DDNet at multiple different levels. This feature mimicking encourages the SDNet network to learn densified 3D features even with single frame input. The final model is evaluated on the Waymo 3D Detection and Domain adaptation dataset where the method considerably outperforms multiple popular baselines while maintaining similar latency. 
strengths_and_weaknesses: S1: The idea of learning densified 3D features is interesting and widely applicable to a wide range of 3D detection problems where the point cloud is sparse due to the physical constraints of the Lidar sensor. 

S2: The design of the feature matching process is novel. The feature mimicking or knowledge distillation is performed at multiple levels including latent features and raw point clouds~(through a point cloud completion task). 

S3: The method is effective and the experimental validation is solid. 

W1: Some important references and discussions with previous work are missing. For instance, [1] also proposed a knowledge distillation approach to learn densified features with single frame input. While it seems that the current method performs considerably better than [1], the authors need to add more detailed discussions to show the difference/improvements.

W2: In section 3.4, the authors claim that they propose a novel voxel-level reconstruction scheme (occupancy + inner voxel offset regression). To my knowledge, this is already explored in [2]. Please add proper references. 

W3: All Waymo results are now evaluated with 20% training data + validation set. A comparison with published SOTA methods on the test set (and 100% training data) is needed to verify generalization. 

[1] Wang, Yue, et al. "Multi-frame to single-frame: knowledge distillation for 3d object detection." arXiv preprint arXiv:2009.11859 (2020).

[2] Xu, Qiangeng, et al. "Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.


questions: Q1: Please address the weakness part I listed above. I will adjust my rating mainly based on the author's response to this part. 

Q2: In section 3.2, the authors mention that the model filters out the outlier points after aggregating the multi-frame point cloud. How is done exactly? 

Q3: line 162: more dense -> denser

limitations: The authors adequately addressed the limitations and potential societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to boost the performance of object detectors in 3D LiDAR scans by learning an auxiliary task to densify them (in a self-supervised way, by distilling intermediate dense features and regressing densified point cloud in a DAE fashion). It is fairly detector-agnostic and can thus be applied to different base SOTA detectors such as CentrePoint [1] or SECOND [15]. It improves their performance by 2–5 p.p. at a cost of adding 15–20% to the runtime, which is a reasonable trade-off.
strengths_and_weaknesses: I think the paper proposes a simple, practical, and universal method, which is sufficiently evaluated, and the paper is well written. I vote to accept it, although I am not very confident as 3D detection is not really my area.

Clarity:
* (+) the paper is generally well written and has sufficient details,
* (−) I think the paper will benefit from explaining the background on how anchor-free and anchor-based methods work; related to that: what is L_reg, specifically? Is it regressing centres of bounding boxes, or corners, or else?
* (=) the hyperparameters are anisotropic, like grid cell size in l. 146; the lateral bounds in l. 230 are 75.2 m which is both large and weirdly specific – is it really so? then why?

Method:
* (+) reasonable design; quite simple – (almost) all added complexity is motivated by ablation,
* (+) the proposed improvement is orthogonal to the design of the underlying detector, so can be universally used;
* (+) if I understand correctly, the densified point cloud is usually obtained from scan sequences, which is probably a common setting, so the method is applicable to a wide range of practical problems,
* (−) PCR module predicts opacities and offsets – if the point cloud in that cell approximates a surface, those offsets are ambiguous in 2 dimensions; is it useful to predict them at all? I suggest to run an ablation where offsets are dropped; 
* (=) for PCR: as a future work you can try predicting a different parametrisation like an SDF predicted with an MLP in each cell; it can be naturally supervised with point clouds;

Experiments:
* (+) comparison to SOTA seems sufficient (although I am not an expert);
* (+) ablation study has all necessary baselines (except for dropping offsets in PCR),
* (−) however, ablation results are only reported on 2 out of 6 categories (no pedestrians at all);
* (+) Table 3 shows that the benefit of densification increases with the distance to the object,
* (±) latency analysis in Section 4.6 is nice, however I am not confident 100 samples are sufficient – can you compute confidence intervals?

==========================

Typos / wording:
* l. 18: missing reference,
* l. 40: semantic points – what are these?
* l. 76: should not be there,
* l. 163: what is BEV?
* ll. 207, 267: adopt ← adapt.


questions: The only substantial issue to address is missing ablations: L1 and pedestrian category, and skipping predicting the offset.

limitations: They are addressed in conclusions; I cannot think of anything beyond it that is relevant to the proposed extension.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new framework to efficiently boost 3D detection performance by learning to densify point clouds in latent space. And experiments on Waymo Open Dataset show promising results.
strengths_and_weaknesses: 1. By learning the feature representation of the point cloud in the dense point cloud space and then transferring it to the sparse point cloud space, this idea is relatively novel and feasible.

2. Although the transition from dense to sparse is a good idea, there is still a large gap from dense to sparse. Especially when there is severe occlusion, this gap can be particularly large. And this case is a very common problem in the large practical application of point cloud target detection. So what about the performance in this bad case? 

3. The process of learning the feature representation of the point cloud in the dense point cloud space is actually an upsampling of the point cloud, and the method of voxelization is actually not an optimal way. Have you tried any other way? And have you compared the effects of different upsampling methods?

4. The method in this paper is mainly used for experiments on the waymo dataset, so how about the performance on other datasets? Such as KITTI, nuScenes.

5. The method in this paper is only compared with a few methods in the experimental results. In fact, there are many excellent works in the point cloud target detection task, but the comparison with them is missing.
questions: see part ‘Strengths And Weaknesses’.
limitations: see part ‘Strengths And Weaknesses’.
review_title: null

Meta-review: metareview: This paper proposes to utilize point cloud completion tools to densify sparse point clouds which could subsequently improve the performance of point cloud detection methods. After rebuttal, reviewers agree on the novelty of the method and its effectiveness on the Waymo open dataset. AC recommends this paper for acceptance following the unanimous opinion.
confidence: Certain
Decision: Accept