Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a non-learning convolution operator that simply use magnitude of feature as cue to remove redundant background points in the sparse convolution operation. The authors validated the proposed operator by combining it to popular sparse convolutional network based object detection and achieve more than 50% reduction in GFLOPs without compromising the performance on benchmark datasets.
strengths_and_weaknesses: Strengths:
- Simplicity of the proposed method

Weaknesses
- It is not clearly stated why magnitude is the key to discriminate foreground and background points. In the experimental section, the authors combines the operator with existing object detection networks CenterPoint and Voxel R-CNN. But the effectiveness of the methods seems to be more related to feature/representation of the input point-cloud and the authors didn’t discuss that part in the paper.
questions: Is it possible that the effectiveness of the propose method comes from the fact that background points has been filtered out by magnitude, thanks to the characteristics of the input point cloud feature representation used in the experiment?
limitations: I suggest the author to add visualization point cloud that belongs to important set to help with understanding.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the spatial redundancy in LiDAR-based 3D object detection. The authors propose spatial pruned sparse convolution (SPS-Conv) that skips computation for voxels with lower activation values. The authors have evaluated their proposed method on KITTI and nuScenes, achieving a 50% reduction in #MACs without loss of accuracy.
strengths_and_weaknesses: **[Strengths]**

The paper is well-written and easy to follow. The proposed SPS-Conv is well-motivated, technically sound and achieves good empirical performance on two large-scale benchmarks. The authors have provided sufficient implementation details, which could facilitate the reproduction. The authors have also promised to release the code upon acceptance.

**[Weaknesses]**

The authors use #GFLOPs as the primary efficiency metric throughout the paper. However, the reduction in #MACs does not necessarily translate into the measured speedup on the actual hardware (due to other costs such as data movement). I would highly recommend the authors report the measured latency on GPUs. As 3D object detectors contain more than sparse convolutions (since there are extra computations in the BEV decoder), it is completely acceptable to just present the latency of the sparse LiDAR encoder.

The inference engine has a substantial impact to the final inference speed. It would be great if the authors could try out the three available sparse convolution inference engines, including SpConv, TorchSparse and MinkowskiEngine, to see whether the proposed method can achieve consistent speedup with all of them.

The authors claim that the activation magnitude serves as a good indicator of selecting important voxels. It would be more interesting to see how random voxel dropout works (in additional to comparing with the inverse magnitude baseline).

Finally, the authors have evaluated their proposed method on KITTI and nuScenes. It would be great if the authors could also evaluate their method on Waymo since the point cloud data on Waymo is much denser (and could be more redundant).
questions: I would love to see the answer to the following questions in the rebuttal:
* What is the latency reduction achieved with the proposed SPS-Conv?
* How does random voxel dropout work compared with magnitude-based pruning?
* How does the proposed SPS-Conv work on Waymo with much denser point clouds?

The authors could refer to the previous section for more detailed comments.

limitations: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for dynamically determining which features on which to perform convolutions in a sparse convolutional network. In particular, the authors apply a sigmoid on top of the magnitude of each feature to determine a score, on which they threshold to determine active sites in each feature map. The motivation is that features with higher magnitude are more important to the network. In the forward pass, only features with a score above a set threshold have convolutions applied, while low score features are passed through via a skip connection. Features are also weighted by this score. Experiments are performed by replacing of a regular sparse convolutional network with the proposed dynamic sparse convolution blocks, where minimal performance regressions are seen despite significant reductions in GFLOPs. Ablations are also provided demonstrating that large score thresholds can be used with minimal impact on performance.
strengths_and_weaknesses: The authors provide a relatively simple solution to the dynamic sparse convolution problem, given the intuition that higher feature norms correlate with more important features. The method can be dropped into any existing sparse convolution network, and is demonstrated to have significant reductions in computation cost. The sparsity can also be controlled at test time by tuning the score threshold. The method is well explained with nice figures outlining the pipeline, and the results are compelling, with good coverage across two datasets. 

Firstly, the intuition that features with higher norms are more important mostly makes sense, but some experimental validation would be very helpful. In particular, as the convolution operator applies the dot product between features, it's possible that a feature has relatively low magnitude compared to others, but high magnitude in a channel that also has high magnitude in the convolutional kernel. This may not be captured by the current approach which looks at the overall norm. 

A more concerning issue I see is the ablation in Table 6, where experiments are performed by only keeping the lowest scored features. While there is a significant regression in the medium cases, the difference for easy and hard cases is actually very minimal. The text describes this as a dramatic performance drop, but it's not convincing that this is actually the case. If the network is able to still achieve similar performance on hard examples when using what should be completely irrelevant features, what does that say about the foundational assumption that high norm features are the most important?

Also, in (4), the features are further weighted by the score for that feature. It's unclear why this is necessary, compared to only using the scores to select sites for convolutions. Does this weighting improve performance or provide some theoretical guarantees? Some ablations for this would be helpful. Otherwise, if the base assumption is not always correct, this weighting may serve to further remove the effect of some potentially important features.

This weighting also serves to provide a gradient on the feature norms (perhaps this is the intention?). This seems like it could be useful, but might also interfere with the final detection task. Some explanations/ablations would be useful.

The pruning ratio in the experiments is also not explained. It sounds like it is the top_k features, but could also be the score threshold. 

L159: should this be absolute magnitudes instead of absolute means?
questions: Please see the strengths and weaknesses section. My main concern regards the ablations in Table 6, which would be alleviated with more explanations/exploration of the underlying assumption of this work.
limitations: No limitations section is provided. Some discussion of the validity of the base assumption would help future readers.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors study a previously under-explored direction in 3D deep learning acceleration: spatial redundancy. The major observation from the authors is that most points in the large scale LiDAR scans are background points. Thus, it is possible to reduce the computation on these points. The authors then design spatial pruned (submanifold/regular) sparseconv layers using magnitude-based importance masks. Experiment results show 2x FLOPs reduction without accuracy loss.
strengths_and_weaknesses: Strengths:

- As I have mentioned in the summary, this work explores spatial redundancy in 3D point clouds, a new direction in efficient 3D deep learning.
- The paper is clearly written and most of the figures look good to me.
- Experiment results are promising: 2x FLOPs reduction with no accuracy loss on nuScenes. Note that the baseline is a representative network, CenterPoint.

Weaknesses:

- The biggest question related to this work is whether such reduction in FLOPs can be translated to latency improvement. Especially when you consider latest sparse conv implementations in spconv 2.x which uses highly customized computation kernels, I'm a little bit concerned whether such pruning will lead to load imbalance between different CUDA threads and limit the speedup. Also, I'm curious what is the overhead introduced to calculate the importance masks.
- It will be great if the authors can add more experiments on larger-scale datasets, such as Waymo. Waymo is much denser than nuScenes, it will be interesting to see the compression ratio and speedup. If the paper is eventually accepted, I'm also interested in seeing some application in sensor fusion methods on nuScenes.
- [Minor] I would suggest the authors to make Section 3 more compact or move some details to the appendix as the target readers can be very familiar with sparse conv.
questions: Please address my comments in "Weaknesses".
limitations: See my comments in "Weaknesses". The major concern is on latency reduction and performance on larger scale datasets such as Waymo.
review_title: null

Meta-review: metareview: The paper shows that it is possible to obtain a good saving in both terms of FLOPS and latency using sparse convolutions for 3d object detection by leveraging the magnitude of features. After a strong rebuttal all 4 reviewers vote for acceptance of the paper with high-confidence.

I suggest that the authors incorporate the comments from reviewers and some of the results from the rebuttal to make the paper more immediately convincing upon a first reading.
confidence: Certain
Decision: Accept