Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a novel Mixed-scale Sparse Voxel Transformer to capture both types of information simultaneously by the divide-and-conquer philosophy. Experiments on waymo show promising results.
strengths_and_weaknesses: 1. The general idea of this article is to extract features through multi-scale transformers in point cloud voxels, and then perform regression through 2D CNN. First of all, the overall process and idea are exactly the same as PointPillar. Second, this multi-scale idea is the same as Voxel-FPN[2], but the processing backbone is replaced by a transformer. So from this point of view, the author needs to think carefully about the novelty of this article.

[1] Lang A H ,  Vora S ,  Caesar H , et al. PointPillars: Fast Encoders for Object Detection From Point Clouds[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.

[2] Wang B ,  An J ,  Cao J . Voxel-FPN: multi-scale voxel feature aggregation in 3D object detection from point clouds[J].  2019.

2. This article is poorly written and needs careful polishing, such as line 5, line 32.
questions: Refer to part Strengths And Weaknesses.
limitations: Refer to part Strengths And Weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a backbone, dubbed MsSVT, for exploiting mixed-scale sparse voxel transformer for LiDAR-based 3D detection. It devises chessboard sampling and balanced multi-window sampling strategies to obtain queries and a set of keys from different scales and then uses multiple attention heads for different scale-aware groups. The mixed-scale features are aggregated for multi-scale scene understanding. Experiments demonstrate the efficacy of the proposed method and the qualitative analysis with expected attention weights shows the role of the mixed-scale design.
strengths_and_weaknesses: Strengths:
- The motivation for designing mixed-scale voxel feature extraction is reasonable.
- The proposed method not only shows impressive performance compared to other methods but also turns out to be efficient with the customized sampling designs. It is important for the practical use of LiDAR-based 3D detectors.
- The illustration, such as Fig. 2 and 5, clearly show the design methods and the functionality of mixed-scale voxelization feature extraction.
- It shows great potential for improving the detection performance of small objects, such as pedestrians and cyclists, in the Waymo experiments.

Weaknesses:
- The motivation for the necessity of mixed-scale voxelization is a little inaccurate.

Large-scale outdoor scenes do not feature significant variance in instance scales, from my perspective, at least compared to 2D object detection in images. A more reasonable explanation should be related to the ununiform distribution of LiDAR point clouds? Due to the sparsity and irregularity of LiDAR points distribution, a voxel-based 3D detector needs customized designs for voxelization to tackle the difficulty of detecting small objects and far-away objects. Of course, I think the scale variance of different instances is also a problem in 3D although it is just one reason for the necessity of mixed-scale voxelization.

Here also needs a literature review for different voxelization methods in previous works, for example:

Every View Counts: Cross-View Consistency in 3D Object Detection with Hybrid-Cylindrical-Spherical Voxelization, NeurIPS 2020

Reconfigurable Voxels: A New Representation for LiDAR-Based Point Clouds, CoRL 2020

It's All Around You: Range-Guided Cylindrical Network for 3D Object Detection

Context-Aware Dynamic Feature Extraction for 3D Object Detection in Point Clouds (Pillar-in-Pillar)

Voxel-FPN: Multi-Scale Voxel Feature Aggregation for 3D Object Detection from LIDAR Point Clouds, Sensors 2020

- Technical contribution: This paper contributes more engineering techniques, including mixed-scale voxelization, multi-scale heads, and efficient sampling implementation. From the perspective of novel technical contributions and new insights to the community, the contribution is just above the borderline.

- Experiments: There are only results on the validation set both on Waymo and KITTI. Although the results on the validation set of Waymo are commonly used for comparison, the validation results on the KITTI are usually not convincing enough. Additional results on the test set would strengthen the experiment part.

- Minor typos: The paper needs polishing and fixing some minor typos, such as line 123 prsent -> present, line 125 regrard -> regard, line 197 wrt -> w.r.t or with respect to. The methodology part can be also improved. For example, the two sections about sampling are not put together, which makes the reader confused about their relationship and the main philosophy of different sampling methods for queries and keys (It is clarified until the beginning of Sec. 3.2). 
questions: - Does the implementation only try the mentioned two-scale case? How about the results for other alternative designs? For example, using the same sampling strategy for both queries and keys, and using more scales for mixing?

limitations: The author analyzes the limitations and societal impacts adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new transformer architecture for Lidar-based 3D object detection. Previous windows-based transformers utilize large windows to capture the long-range context information but ignore the negative effect of blurring fine-grained details. This is especially true for 3D point cloud cases where the point downsampling is performed and sparse and distant points are selected (further blurring fine-grained details). To alleviate this, the authors propose to design a multi-scale transformer architecture and set up key windows of different scales in which different windows contain the same amount of downsampled points. Scale-aware head attention is also proposed to divide the attention head into multiple groups and aggregate after inner group computation with the query. Additionally, the authors further propose a chessboard sampling strategy to iteratively sample different query voxels in order to reduce the computational cost while maintaining similar accuracy performance. In the end, the final models are evaluated on the KITTI and Waymo datasets and achieve SOTA results. 
strengths_and_weaknesses: S1: The overall design is sound and relatively novel. This work nicely extends the Single-Stride Transformer [1] to work with voxel data. Concretely, SST only works with pillar (a long elongated voxel) representation to maintain efficiency. Extending windows-based transformers to the voxel representations come with an efficiency challenge due to the larger amount of voxels inside a key window. The authors nicely address this by proposing multi-scale window attention and chessboard query sampling strategy. The overall approach significantly improves the SST baseline and sheds light on efficient transformer design with 3D point cloud data. 

S2: The approach is well ablated on two popular datasets and achieves SOTA results compared to published methods. 

S3: The paper is generally well written and easy to follow. 

W1: There is no ablation on the number of groups for multi-scale attention (currently set to two). To further verify the motivation, the authors need to investigate the accuracy/speed tradeoff with a larger number of groups.

W2: Current model only works with single-frame data. Could it scale to multi-frame? 

W3: Experiments with more two-stage architecture will be interesting (minor, this could be done later). 
questions: Q1: On line 180, the authors mentioned that the model selected one of the four characters as the sampling location. How are these chessboards generated? Is it purely random? The authors need to provide more details. 

Q2: Please address the weakness I listed above. 
limitations: Partially. More discussion of the potential negative societal impact like the earlier adoption of not-yet validated autonomous vehicles (or misleading 3D detection performance) will be useful.
review_title: null

Meta-review: metareview: After the rebuttal and discussion two reviewers are positive, one remains negative. The reviewers liked the overall approach, the writing, and the core experimental results. Some reviewers asked for additional broader experiments and comparisons, which the authors were able to provide. The main concern of reviewer QRYf is the close relation to PointPillar's, which the authors are able to clarify sufficiently in their rebuttal.
There is thus sufficient evidence to accept this submission.
confidence: Certain
Decision: Accept