MsSVT: Mixed-scale Sparse Voxel Transformer for 3D Object Detection on Point Clouds
Authors: Shaocong Dong
Lihe Ding
Haiyang Wang
Tingfa Xu
Xinli Xu
Jie Wang
Ziyang Bian
Ying Wang
Jianan Li
Venue: NeurIPS 2022 Accept (2022)

Abstract: 3D object detection from the LiDAR point cloud is fundamental to autonomous driving. Large-scale outdoor scenes usually feature significant variance in instance scales, thus requiring features rich in long-range and fine-grained information to support accurate detection. Recent detectors leverage the power of window-based transformers to model long-range dependencies but tend to blur out fine-grained details. To mitigate this gap, we present a novel Mixed-scale Sparse Voxel Transformer, named MsSVT, which can well capture both types of information simultaneously by the divide-and-conquer philosophy. Specifically, MsSVT explicitly divides attention heads into multiple groups, each in charge of attending to information within a particular range. All groups' output is merged to obtain the final mixed-scale features. Moreover, we provide a novel chessboard sampling strategy to reduce the computational complexity of applying a window-based transformer in 3D voxel space. To improve efficiency, we also implement the voxel sampling and gathering operations sparsely with a hash map. Endowed by the powerful capability and high efficiency of modeling mixed-scale information, our single-stage detector built on top of MsSVT surprisingly outperforms state-of-the-art two-stage detectors on Waymo. Our project page: https://github.com/dscdyc/MsSVT. 

MsSVT: Mixed-scale Sparse Voxel Transformer for 3D Object Detection on Point Clouds
Shaocong Dong' ; Ding""", Haiyang Wang", Tingfa Xu Xinli Xu', Ziyang Bian', Ying Wang', Jie Wang', Jianan Lil-t Beijing Institute of Technology, Peking University. (dsc1010295799, dean. dinglihe, xxlbigbrother, j com, wanghalyang@stu, pku. aku.edu.com cn, (lijianan, ciom_ xt.__xfii (afffl.bbiteeem
Abstract
3D object detection from the LiDAR point cloud is fundamental autonomous driving. Large-scale outdoor scenes usually feature significant variance in instance scales, thus requiring features rich in long-range and fine-grained information support accurate detection. Recent detectors leverage the power of window- based transformers model long-range dependencies but tend to blur out ane-grained details. To mitigate this gap. we present a novel Mixed-scale Sparse Voxel Trans- former, named MsSVT, which can well capture both types of information simul- taneously by the divide-and-come conquer philosophy. Specifically, MsSVT explicitly divides attention heads into multiple groups, each charge attending to infor- mation within particular range. All groups' output is merged to obtain the final mixed-scale features. Moreover, we provide a novel chessboard sampling strategy to reduce the computational complexity of applying window- hindow-bbsed transformer in 3D voxel space. To improve efficiency, we also implement the voxel sampling and gathering operations sparsely with hash map. Endowed by the powerful capability and high efficiency of modeling mixed-scale information, our single-stage detector built on top of MsSVT surprisingly outperforms state-of of of-the-art two-stage detectors on Waymo. Our project page: https: h // giggtu...b..... com www/dso/eeccccccccooo dscdyc/MsyVT
Introduction
3D object detection has received increasing attention due its successful autonomous driving applications. Unlike 2D images with regular structure of pixels, LiDAR point clouds are naturally irregular and unordered. Hence directly applying CNN-like operations difficult. To solve this, many researchers have rasterized point clouds into regular voxel grids [24] and employed 3D CNNs to extract high-dimensional voxel features. With the recent rise of vision transformer (ViT) 42 on 2D images, some attempts have been made to generalize global or more efficient window-bar based transformers 3D voxels 23 or pillars 77. These methods successfully seek long-range context by utilizing transformers' powerful abilities in modeling long-range information. However, they ignore that blindly increasing receptive fields would easily blur fine-grained details, especially in sparse 3D space, crucial to accurate object recognition and localization.
them can be
Standard window-based transformers update the features of queries a local window by attending to keys from the same window. Hence simultancously aggregating long- range context and fin-grraine details require enlarging the window size to embrace local and distant voxels. Nevertheless, directly gathering all the voxels within the window as keys suffers a cubical growth of the computational load with the window size. Some attempts alleviate this by sampling only a certain number of key
Equal contribution. 'Coreepondeecce to: Jianan Li and Tingfa Xu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
(b)
(c)
(d)
Figure I: Top: contrast to sampling key voxels from b) a single-scale 3D window in a) raw point clouds, MsSVT samples key voxels from e) multi-scale windows, thus keeps finer granularity on the target object while covering large-scale neighborhood. Bottom: Different head groups accept keys sampled from windows of different scales, and respectively responsible for obtaining d) fine-grained details and e) long-range context (reflected by higher attention weights), thus together contribute accurate object detection collaboratively.
While a trivial sampling strategy quickly leads to the sparse sampling of local voxels (Fig. bb). thus bias mainly on long-range context. To mitigate the above, try to set up multiple key windows of varying sizes centered on a query window and sample the same number of local and distant key voxels separately from the smaller and larger windows. As result, we can keep finer granularity in the local region to retain fine-grained details while collecting distant voxels roughly to enlarge the receptive field (Fig. (c)).
voxels 23
With the sampled voxels ready, the next question is how to effectively attend to voxels from different windows and simultaneously capture long-range context and fineggriined details. We argue that the sivide-anddcooquer philosophy can satisfactorily resolve this issue. Specifically, inspired by the recent findings [42 (59 26 that transformers learn different levels of self-attention by different heads, we propose a novel Mixed-scale Sparse Voxel Transformer (MsSVT), which explicitly divides the transformer heads into multiple groups. Different head groups accept voxels sampled from windows of different sizes, so they are each in charge of capturing information of a particular scale. Combining the outputs from all the head groups, we can capture mixed-scan scale information. i.e., long- long-range context, and fine-gra ine-ggrined details. We also design a novel scale-aware relative position encoding strategy to adaptively adjust the position encoding used in each head group according to the range of the keys. We provide some resulting attention maps by two different head groups (Fig. (d), (e)). It is also worth mentioning that the mixed- scale attention enables information exchange across local windows, making MsSVT more compact by saving additional shift window operation commonly required by window-based transformers 21
Moreover, to improve the efficiency of applying transformers in 3D voxel space, we strive to reduce computational costs in two ways. First, we propose novel chessboard sampling (CBS) strategy to reduce the number of query voxels that need to be sampled within the query window, to reduce computational costs without losing information. Specifically, we partition the query window into chess-like spaced, and termed as "x". "O", "Á", ","""" "O" positions separately. During each attention layer, only one specific position of voxels is sampled and updated by serving as queries, and the updates of the other voxels can be obtained by interpolation. Four positions are selected in the circular. Thus, we can update all the voxels without introducing deviation. Second, we take advantage of non-empty voxels' sparsity by performing mixed-scale window-based attention solely on non-empty sites in 3D space. Furthermore, we parallelize the search and feature gathering for non-empty voxels using hash mapping for further acceleration.

---

We build a 3D detector by replacing the original sparse 3D CNN backbone in SECOND 150 with our MsSVT and conduct extensive experiments on the large-scale Waymo open dataset 38 Benefiting from the powerful capability of abstracting mixed-scale voxel features, our single-stage detector based on MsSVT surprisingly outperforms offte-of-the-are two-stage detectors. We summarize our contributions as follows:
We present novel Mixed-scale Sparse Voxel Transformer (MsSVT), which simultaneously abstracts voxel features with long-range context and fine-graine ane-grained details.
We design an efficient chessboard sampling strategy to vastly reduce the computational cost of applying a voxel-based transformer in 3D space and sparsely implement all operations to improve efficiency.
Our MsSVT-based single-stage detector outperforms state-of satte-of-the-eee two-stage detectors on Waymo.
Related work
3D detection on point clouds. The mainstream 3D object detectors are based voxels 3 51 52
37 [1] 60) (50 [44] [43 or pillars [16] 32). VoxeINet [60] utilizes PointNet [27 32 to aggregate features within each voxel and then apply sparse 3D convolution to generate detection results. SECOND (50) investigates improved sparse convolution to further improve speed. Pointpillar [16] converts the point cloud into pillars such that 2D CNNs can be applied to trade off between accuracy and efficiency. Rapoport-Lavie et al. (8 introduce the Cylindrical Coordinates to leverage the natural scanning pattern from LiDAR sensors. Chen el al. [2] further operate on both Bird Eye View (BEV) and Range View (RV) in the unified
Cylindricall-Shherical (HCS) voxel representation. Reconfigurable Voxels 45 improves the local neighbors searching of each voxel using random walk scheme. Voxel-FPN 15 and Pillar-in-Pillar 441 adopt a multi-scale voxelization strategy to rasterize the input point cloud into multi-sized voxels and incorporate FPN 20 aggregate multi-resolution feature maps. Two-stage detectors [5] 36 31 refine the bounding boxes output by a single-stage detector by aggregating raw point clouds or voxel features, yielding state-of-the-art performance.
Vision transformer. Transformer 42 5
has recently achieved great success in computer vision 6
221 18
61 (33 46). Swin-transformer 21 restricts self-attention to non-overlapping local windows while allowing wridsww connection to improve efficiency. SSA [29] divides attention heads into multiple groups to aggregate image features with different granularities. Guo al. and Zhao et al. [58] make the first step towards introducing the transformer for point cloud analysis.
apply local self-attention to learn richer 31) feature representation. Our work extends the window-bant attention on 3D voxels by introducing scale-aware attention learning equipped with novel sampling strategies for the queries and the keys to improve both accuracy and efficiency.
Recently, many approaches 47 57
10
22
49
Voxel transformer for 3D detection. VoTr 26 introduces a voxel- transformer backbone that performs self-attention on sparse voxels with local and dilated attention mechanisms. Our work improves VoTT by introducing window-basee based attention and optimizing sparse operation. The recent SST 7 follows a single-stride design and the tranffrrnsformer architecture, which performs well on small objects. Nevertheless, SST is implemented based on pillars. The single window size is not conducive to capturing multi-scale features, resulting in unsatisfactory performance on Vehicle when simultaneously detecting multiple categories. In comparison, our MsSVT can capture mixed-scale information to boost the detection of objects of various scales.
3 Method
This section first gives details of the MsSVT block, then its efficient sparse implementation, and finally the 3D detector based on MsSVT.
3

---

Voxel Feature Encoder
MsSVT (3D Backbone)
Detection Head
MsSVT Block
CBS
Attention Head
Query Window
Attention Head
FPS
Group
FFN
Key Window
Atlention Head
Atlention Head
Group
Key Window 2
Balanced Multi-window
Scale-aware Attention
Sampling
Figure 2: Top: overall architecture of our detection network. Bottom: We gather the non-empty voxels within the query window and apply Chessboard Sampling (CBS) to sample the queries. For the keys, we gather the non-empty voxels from the key windows of different sizes separately, and get multiple sets of keys through Balanced Multi-window Sampling, with each set representing information of specific scale. Keys from windows of different sizes assigned to different head groups to perform scale-aware attention learning, thus simultancously capturing both long-range context and fineggrained details.
details of the MsSVT block.
3.1 Mixed-scale Sparse Voxel Transformer
Fig. illustrates the overall architecture of the MsSVT block. We first gather the query and the key voxels via chessboard sampling and balanced multi-window sampling, respectively. The obtained queries and keys are then fed into multiple head groups capture mixed-scale information through scale-aware attention learning. We further incorporate scale-aware relative position encoding to make better use of position information different head groups
3.1.1 Balanced Multi-window Sampling
Let {sxl8k E Jo denote series of window sizes, where 8n is the size the query window and $ the sizes of M successively larger key windows. Let V {viie, (x,, fi) be the input voxel set, with zyz coordinates æ; 2 and feature vector f. RC for voxel i. We first partition the voxel set into non-overlapping 3D windows each size Sp. and find the non-empty ones query windows with their centers denoted by {cilc, 23) where is the total number of query windows. To get query voxels Ve,.to for the query window centered on C, one can simply gather all the non-empty voxels within the window the queries. While keeping efficiency in mind, we present a novel chessboard sampling strategy, which will be detailed in Section 3.1.2
As for the key voxels, instead of sampling within single large window once as previous meth- ods (23), which inevitably biases on either local or distant voxels, we simultaneously search for the neighbors for each center c; within multiple key windows of different sizes. For the key window of size Sk. gather
than Np non-empty voxels Ve..e {v, s æj C ) == where Np is pre-set number. Furthermore, to reduce computational cost and keep balanced sam- pling, we adopt the farthest point sampling (FPS) algorithm to uniformly sample NR voxels from Ve,.s; to obtain the final key voxels Vira k = 1, ..., M different scales, where Nx a pre-set maximum number of sampled voxels. Benefitting from the multi-window strategy equipped with uniform sampling by FPS, we can achieve balanced sampling key voxels at various scales, which crucial to capturing mixed-scale information.

---

3.1.2 Chessboard Sampling
The prerequisite for to sample the keys is that do not have to preserve all the key voxels and only need to select some representative ones to reduce computational redundancy. While things different for the queries. On the one hand, we should retain and update every query voxel after an attention layer; otherwise, the information could be irreversibly lost. But on the other hand, the computational cost and memory footprint grow dramatically with the window size, making implementation impractical if the queries are not reduced.
Since the positions of non-empty voxels remain changed after a series of attention blocks, an effec- tive solution is to sample subset of query voxels each time for feature updating, and then use the sampled voxels to update those unsampled. Moti- vated by the above, we present Chessboard Sam- pling (CBS) as illustrated in Fig. 3] Each voxel in the query window is marked with "x". "O", "A". or "L"", in an evenly spaced pattern. In each MsSVT block, we sample queries from the non-empty voxels with one specific type of the marks, and update the queries' features through attention learning. Then for each unsampled non-empty voxel, we update its feature by finding the K-nearest query voxels (by default K = 3) and by linearly interpolating their updated features. The four masks are used in the
A sampling
sampling
O
A O O
OA
A
O X
O
X O X X
X
A O O
A
A
O
O X X
X X
O O A O
A
DAOL
A
A
X
R X
X
O X
O A A
A
sampling
sampling
window range
non-emply vaxel
selected voxol
Figure Diagram of chessboard sampling.
circular to sample query voxels in stacked blocks. This way, we retain the original structure and cover all voxels comprehensively
possible. Note that we can apply the interval sampling any of the a.y.2 axes to achieve a sampling rate of 1/2, 1/4, or 1/8. Generally, we apply it in the horizontal r-y plane (as illustrated in Fig. 3).
3.1.3 Scale-aware Head Attention
Given the query voxels Ve,,o = (Xo, Fb) with voxel coordinates Xo E NN and feature vectors Fo E R-N and the multi-scale key voxels v ci.ac (Xx, Fx), = 1,... M with voxel )
coordinates X₁ Nx and feature vectors F2 NCC first get queries Q, keys {K))
and values (Vi k=1
Q, Kr, V FOWO, F.W", F.WV, k: k=1,.., M
where Wº, WQWWK W WV E CxC are linear projections. To achieve scare-aware attention learning, we divide multiple attention heads into M groups and assign the keys from windows of different sizes to different head groups. Accordingly, we also split the feature channels of the queries Q into M groups. C/M),
The k-th channel group of Q, denoted by Qx Ql: (k- 1) C/M is fed into the k-th head group. As a result, each head group is responsible for learning to attend information specific scale. The attended feature for the k-thh head group is
..., M,
YK MHA(Q KR. Vt. (PE(X,, Xx)),
where MHA(-) denotes multi-hean group attention and RPE(1 represents the newly proposed relative position encoding which will be detailed in Section 3.1.4 a head group with one or more attention heads. We concatenate the output from all heads groups (Y%)M to Y E N which is further fed into a foed-forward network (FFN) implemented by multi-layer perceptron (MLP) to obtain the final mixed-scale feature Y:
Each window size corresponds to
Y = CAT(Y,.... Ynr):
(3)
Y= MLP(LN(Ý))) + Y.
(4)
where (-)() represents layer normalization.
5

---

3.1.4 Scale-aware Relative Position Encoding
Relative position encoding is necessary for transformer-base networks because fine-graing position information may be lost in high-level features with the deepening the network. To make better use of position information to facilitate multi-scale feature learning in our case, we adopt scale-aware adaptive relative position encoding strategy inspired by (30 48 55 which can generate the positional bias dynamically with scales for different head groups.
Specifically, we establish learnable embedding table T₃ RDx for the k-th head group according to the size of the largest key window [2I]. where R is the number of possible relative position pairs and D the feature dimension. We get the relative positional bias for the queries as:
B GQQ,T Ib) R R
(5)
where J E ZLXNoxNN represents the table indices corresponding to the actual relative position between the queries and the keys, G(-) is the operation of gathering features according the indices. Similarly, we can ge the relative positional bias _ E 4x xNx for the keys. Finally, the biases act directly on the attention weights, and Eq. 2 be rewritten
YK o QiKe VD
+ B + VE:
66)
where (-) represents softmax. This way, position embeddings can be adjusted adaptively according to different scales to guide the scale-ware head attention better.
3.2 Sparse Implementation
To leverage the natural sparsity of point clouds and further improve efficiency, sparsely implement all our window center searching, window gathering, and balanced window sampling into CUDA operations. These operations are mainly based on hash map that establishes the mapping from coordinate space to voxel index (23]. Taking the window gathering operation as an example, we query each possible position w.r.t. the given center within the window, and retrieve the corresponding
features if the position is a valid key in the pre-built hash map. More details are available in the supplementary materials.
3.3 Detector Establishment
We build our 3D backbone by stacking multiple MsSVT blocks, as shown in Fig. Noted that we set both the query and the key window size in the last MsSVT block as 11, to compress the 3D voxels into a 2D feature map, where the query is the average voxel features within the pillar window. We replace the 3D backbone in SECOND [50] with MsSVT and keep the other network components unchanged. We discard the down sampling process because MsSVT can already capture features at different scales. The input point cloud is first converted into regular voxels and fed into MsSVT backbone to get mixed-scale voxel features. The features are then compressed vertically and sent to the subsequent 2D RPN and detection head to get detection results.
4 Experiments
In this section, we first provide architectural details of MsSVT, then compare our model with recent state-of-the-an detectors on Waymo Open (38) and KITTI (8 datasets. Thorough ablation studies and in-depth analysis are further provided to validate our design choices.
4.1 Architectural Details
MsSVT comprises four regular MsSVT blocks with query window size of (3,3,5) and key window sizes of (3, 3, 5) and (7, 7, 7), followed by a specialized MsSVT block where the windows are set
as 1 x pillar as mentioned in 3.3. We divide 8 attention heads into 2 head groups. The sampling
rate of the chessboard sampling 1/4 and the maximum number of sampled keys Nx 32. We
use the center head [56] to generate single-stage bounding boxes. In addition, we also provide two-stage version with CT3D RT. More details can be found in OpenPCDet 40 since we conduct
all experiments with this toolbox.

---

Table I: Results WOD validation set (train with 20% Waymo data). SS: Stngle-stage model, TS: Two-stage model, SF: Single frame input. Note that some priors only report results of single-class training, which is generally simpler than multi-class training.
VeLL1
Vel.
122
PoLL1
Pol
Cye_L1
Cye L.2
Motbod
Reference
mAP
MAPH
MAP
MAPH
mAP
MAPH
mAP
MAPH
mAP
MAPH
mAP
mAPH
Single-Stage Methods
SECOND 50
Semsrrs 2018
70.96
70.34
62.58
62.02
65.23
51.21
57.22
47.49
57.13
55.62
51,97
53.53
PointPillar
16
CVPR 2019
70.43
69.83
62.18
61.64
66.21
46.32
58.18
40.64
55.26
51.75
53.18
49.80
ConterPoint
%6
CVPR 2021
72.76
72.25
64.91
64.42
74.19
67.96
66.03
60.34
71.04
69.
68 49
67.28
VOTR-SS
35
ICCV 2021
68.99
68.39
60.22
$9.69
RSN-SP
CVPR 2021
75.10
74.60
66.00
65.50
McSVT-SS (ours)
77.18
76.67
68.75
68.28
80.25
73.05
72,88
66.14
73.75
72.53
70.96
69.79
Two-Stage
Methods
Part-A2 35]
TPAMI 2020
74.66
74.12
65.82
65.32
71.71
62.24
62,46
54.06
66.53
65.18
64.05
62.75
PV-RONN 35
CVPR 2020
75.95
75.43
66.00
67.54
75.94
69,40
67.66
61.62
70.18
68.
67.73
66.57
Vosa-RRCNN 1
AAAI 2021
76.13
75.66
68.18
67.74
78.20
71.98
69.29
63.59
70.75
69.68
68. 25
67.21
PV-RCNN++ 36
ARXIV 2021
77.61
77.14
69:18
68.75
79.42
73.31
70.88
65.21
72,50
71.39
69 84
68.77
VOTR-TS 23
ICCV 2021
74.95
74.25
65.91
65.29
CT3D RIT
ICCV 2021
76.30
69.04
MsSVT-TS (ours)
78.41
77.91
69.74
69.17
82.34
76.77
74.71
69.36
75.74
74.65
73.72
72.64
Table 2: Results on WOD validation set (train with 100% Waymo data).
VeLLI
VeLL2
PadLI
Ped_12
Cyc_Li
Cyc_L2
Methol
Reference
mAP
MAPH
MAP
MAPH
mAP
MAPH
mAP
MAPH
mAP
MAPH
mAP
mAPH
Single-Stage Methods
SECOND 50
Semsors 2018
72:27
71.69
63.85
63.33
68.70
58.18
60.72
51.31
60.62
59.28
58.34
$7.05
PoinPillar 16
CVPR 2019
71.57
70.99
63.06
62.54
70.61
56.70
62,85
50.24
64.36
62.27
61.95
59.93
SST-S8-SH 7
CVPR 2022
73.57
64
64.80
80.01
71.66
70.72
68
MsSVT-SS (ouns)
77,83
77.32
69.53
69.06
80.39
73.61
73,00
66.65
75.17
73.99
72.37
71.24
Two-Stage Methods
Part-2 311
TPAMI 2020
77.05
76.51
68.47
67.97
75.24
66.87
66.18
58.62
68.60
67.36
66.13
61.93
PV-RCNN 355
CVPR 2020
78.00
77.50
69.43
68.98
79.21
73.03
70.42
64.72
71.46
70.27
68.95
67.79
PV-RENN++ 16
ARXIV 2021
79.25
78.78
70.61
70.18
81.83
76.28
75.17
08.00
73.72
72.66
71
70.19
M-SVT-T (ours)
79.35
78.86
70.65
70.23
82.41
77.04
74.74
69.57
77.12
76.01
74.98
74.07
4.2
Results on Waymo
Setups. We first evaluate our model on large-scale Waymo Open Dataset (38). The input is single-frame point cloud with 150mx 150m detection range. We set the detection range in the horizontal and the vertical direction [-75.2m, 75.2m] and [-2.0m, 4.0m), respectively. The voxel size is (0.4m, 0.4m, 0.6m). We follow the same training strategy as in (3 Specifically, we train our
model for 80 epochs on 20% Waymo data using the Adam optimizer [3]. We apply the eyclic decay scheme [50]. by which the learning rate is increased from le-4 to le-3 during the first 40% epochs and further decreased to le-5 the remaining epochs. We also report the results of training for 30 epochs on 100% Waymo data using the same optimizer and learning rate scheme. The evaluation metric the 3D mean Average Precision (mAP) for difficulty levels of LEVEL I and LEVEL 2.
Main results. We compare our model with state-of-the-art priors in Table I and Table It is worth noting that our model performs simultaneous detection of three object categories, which more challenging than detecting single category. Using 20% training data, our single-stage MsSVT-SS significantly surpasses other single-stage counterparts, even though some of them are trained specially for one specific category. Noticeably, MsSVT-SS performs on par with the state-of-the-uu two-stage PV-RCNN+ (36] on Vehicle, and even better (0.8-1.5 mAP higher) on Pedestrian and Cyclist. In addition, our two-stage MsSVT-TS performs the best on all categories, and significantly outperforms previous best-performing PV-RCNN+ by 2.9 and 3.2 mAP on Pedestrian and Cyclist, respectively.
Similarly. when using 100% data, both our single-stage MsSVT-SS and two-stage MsSVT-TS achieve the best results compared to their corresponding counterparts. Noticeably, our M&SVT-SS surpasses detector SST 7 dedicated to small object detection by large margins of 4.3 and 4.5 mAP on Vehicle and Cyclist, respectively. This clearly evidences the superiority of our MsSVT in capturing mixed-scale information over conventional transformer designs.
the recent transformer-basee
7

---

Visualization. We provide some qualitative results in Fig. 4 Our MsSVT can still predict accurate bounding boxes in scenes beyond the range of 50m where the points become extremely sparse (Fig. (a)). This evidences that MsSVT successfully captures valuable contextual information in the absence of fine-gu ain-ggained details for long-range object detection. In addition, our model also performs well in complex scenes in the presence of dense objects with large-scale variations (Fig. (b). It should be owed to the flexibility and robustness of MsSVT.
Analysis. We further visualize some attention maps which indicate what has been focused on by MsSVT to understand its behavior better. As shown in Fig. the head group with smaller key window focuses on the local foreground information, while the head group with larger key window attends more to the longer-range context. As result, the two head groups complement each other, well capturing mixed-scale information to support the detection objects of various scales.
Table 3: Results on KITTI validation set.
Car (loU=0.7)
3D
(loU=0.55)
Cye. (IoU=0.5)
Method
Reference
Easy
Mod
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
Single-Stage Methods
SECOND 50
Sensors 2018
86.46
77.28
74.65
61.63
56.27
52.60
80.10
62.69
59.71
PoindPillar 16
CVPR 2019
88.61
78.62
77.22
56.55
52.98
47.73
80.59
67.16
63.11
3DSSD 54
CVPR 2020
88.55
78.45
77.30
58.18
54.32
49.56
86.25
70.49
65.32
VoTr-55
2
ICCV 2021
87.86
78.27
76.93
MsSVT-SS (ours)
89.08
78.75
77.35
63.59
57.33
53.12
88.57
71.70
66.29
Two-Stage
Methods
PointRCNN 33
CVPR 2019
89.03
78.78
77.86
62.50
55.18
50.15
87.49
72.55
66.01
Part-A2 ++
TPAMI 2020
88.48
78.96
78.36
70.73
64.13
57.45
88.18
73.35
70.75
PV-RCNN
CVPR 2020
89.35
83.69
78.70
63.12
54.84
51.78
86.06
69.48
64.50
VoIr-TS 23
ICCV 2021
89.04
84.04
78.68
MsSVT-TS (ours)
89.32
84.66
78.94
66.11
58.94
53.86
92.49
73.60
69.34
4.3
Results on KITTI
Setups. We also evaluate our model on KITTI 8. Given an input point cloud, we reserve the points within the range of [0m, 70.4m), [-40.0m. 40.0m] and [-3.0m, 1.0m] on the x.y.a axes, respectively. The voxel size is set to (0.32m, 0.32m, 0.4m) and other settings remain the same as in the experiments on Waymo. We train the model for 100 epochs with the Adam optimizer. The learning rate is 0.003, decayed by the cyclie scheme (50). The evaluation metric is 3D mAP for three difficulty levels (easy, moderate, and hard).
Main results. Table 3 depicts that our model achieves competitive performance on all three categories. Specifically, Car, single-stage MsSVT-SS surpasses the superior VoTr (3) by 0.5 mAP. Meanwhile, on Pedestrian and Cyclist, MsSVT-SS even outperforms some preeminent two-stage detectors. Moreover, two-stage MsSVT-TS further inereases the lead. These results demonstrate that our MsSVT can be well generalized to various datasets.
4.4 Ablation Study
All ablation models are trained for 12 epochs on 20% Waymo data. More implementation details and ablations on hyperparameters are available in supplementary materials.
Balanced multi-window sampling. We first validate our balanced in Table 4 The base model listed in the first row employs a 3D version of standard window- based attention [2I] with window size (3, 5) and without shift window scheme, adopts dilated key sampling [23), and gathers all non-empty voxels within the window as queries. We build model variant by simply replacing the dilated key sampling with our proposed balanced multi- window sampling, Results in the second row demonstrate that sampling key voxels from multiple
sampling strategy

---

(a)
(b)
Figure 4: Qualitative results on Waymo. The red and green boxes denote ground truths and predic- tions, respectively. Our MsSVT performs impressively in scenes (a) beyond the range of 50m, and (b) in the presence of dense objects with large scale variations.
Figure 5: Visualization of attention maps. Pink dot denotes the query position. Positions with high and low attention weights are in red and blue, respectively.
windows different sizes brings noticeable performance gains, i.e., 71.66->73.44 on Pedestrian and
on Cyelist.
Scale-aware head attention. We also validate the effectiveness of our proposed scale-aware head attention Table By comparing the results in the second and the third rows, one can find that enabling multiple head groups to capture informa- tion at different scales clearly boots the performance.
Table Ablations on different components of MsSVT. BMS: Balanced Multi-window Sampling. SHA: Scale- aware Head Attention, SRPE: Relative Position Encoding.
BMS
SHA
SRPE
Veh / Ped Cyc
69.51 /71 .66 63.31
71.24/ /3////3.444446//
71.96 75.07//6..1111 /
Scale-aware relative position encoding. Table 4 shows that incorporating scale- aware relative position encoding (listed in the last row) further improves the per-
04.37775.99/67.00
formance compared to the model variant that adopts scale-agnostic position encoding. It well supports design motivation that the position encoding should vary with different scales.
Chessboard sampling. T reports
the performance of the model variants that apply chessboard sampling with differ-
Table 5: Ablations on sampling strategy.
sampling rates and of the model with-
Strategy
Veh / Cyc
Mem (G)
Lat (ms)
out performing any sampling. We mea- sure the latency on Tesla-V 100 GPU and Unbuntu- 16.04, Python 3.7, Cuda-10.2, and Pytorch-1.88 1.8. Our model not sensi-
w/o
72.58/75.74/66
18.3
167
1/2
72.44/76.03/ /66/7/8811
14.5
138
1/4
72.37 72.37/75.99/677 /6.99/67.90
12.2
121
1/8
11.4
113
tive to varying sampling rates. Incorporat- ing chessboard sampling with a sampling
rate of 1/4 results in only negligible performance degradation compared to the model variant without sampling, yet largely improves the computational efficiency by a significant reduction memory footprint (up to 33% decrease) and in latency (up to 28% decrease).

---

Conclusion
This paper presents MsSVT, novel sparse voxel transformer that is capable capturing mixed-scale information to boost 3D object detection. We begin with sampling the queries and the keys at different scales through the novel chessboard and the balanced multi-window sampling strategy, respectively. Then, explicitly divide attention heads into multiple scale-aware groups, each in charge of capturing information at a specific scale. Extensive experiments have shown that our MsSVT is superior in detecting objects of various scales and granularities thanks to the learned mixed-scale information.
Limitations. MsSVT can capture mixed-scale information from multiple local windows and achieves promising performance. Yet, the sizes of the windows need to pre-set manually. We will explore the waidwwve-window version of MsSVT in future work.
Acknowledgements
This work was financially supported by the National Natural Science Foundation of China (No. 62101032), the Postdoctoral Science Foundation China (Nos. 2021M690015, 2022T150050), and Beijing Institute of Technology Research Fund Program for Young Scholars (No. 304001 1182111).
References
[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[2] Qi Chen, Lin Sun, Ernest Cheung, and Alan Yuille. Every view counts: Cross-view consistency in 3d object detection with sybrid-cylinddrcal--phere voxelization. Advances in Neural Information Processing Systems, 33:21224-21235 2020.
[3] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, pages 1907-1915, 2017.
[4] Jajjun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. arXiv preprins arXiv:2012.15712. 1(2)44, 2020.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0440 04805, 2018.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
[7] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. arXiv preprint arXiv:2112.063755 2021.
[8] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11):1211-1237 2013.
[9] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-19,,200 2021.
[10] Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel transformer: A set-to-set approach to 3d object detection from point clouds. arXiv preprint arXiv:2203. /03/4, 2022.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
10

---

[12] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[14] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In IROS, pages 1-8. IEEE, 2018.
[15] Hongwu Kuang, Bei Wang, Jianping An, Ming Zhang, and Zehan Zhang. Voxel-fpn. Multi- scale voxel feature aggregation for 3d object detection from lidar point clouds. Sensors, 20(3): 704, 2020.
[16] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[17] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint arXiv:1608. 07996, 2016.
[18] Yanghao Li, Chao- Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint arX:v:2112.01526, 2021.
[19] Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cnn: An efficient and universal 3d object detector. In CVPR, 2021.
[20] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR. 2017.
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[22] Anas Mahmoud, Jordan SK Hu, and Steven L Waslander. Dense voxel fusion for 3d object detection. arXiv preprint arXiv:2203.0088 2022.
[23] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV, 2021.
[24] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real- time object recognition. In 2015 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 922-928. IEEE, 2015.
[25] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park. Fast point transformer. arXiv preprint 12.00222 2021.
[26] Namuk Park and Songkuk Kim. How do vision transformers work? arXiv preprint arXiv:2202.06700 06709, 2022.
[27] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[28] Meytal Rapoport-Lavin and Dan Raviv. It's all around you: Range-guided cylindrical network for 3d object detection. In ICCV, 2021.
[29] Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self- attention via multi-scale token aggregation. arXiv preprint arXiv:2111.15/93. 2021.
[30] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv:1803.02155 2018.
[31] Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, and Min-Jian Zhao. Improving 3d object detection with channel-wise transformer. In ICCV, 2021.

---

[32] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillamet: Real-time and perefperrormacee pillar- based 3d object detection. arXiv preprint arXiv:2205.07400 2022.
[33] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[34] Shaoshuai Shi, Zhe Wang, Xiaogang Wang, and Hongsheng Li. Part-a 2 net: 3d part- aware and aggregation neural network for object detection from point cloud. arXiv preprint arXiv:1907.03360, 2(3) 2019.
[35] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. CVPR, 2020.
[36] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature abstraction with local vector representa- tion for 3d object detection. arXiv preprint arXiv:2102.00466 2021.
[37] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In ICCV, pages
2015.
[38] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[39] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov. Rsn: Range sparse net for efficient, accurate lidar 3d object detection. In CVPR, 2021.
[40] OpenPCDet Development Team. Openpodet: An open-source toolbox for 3d object detection from point clouds. http::://gttttbb.ccmmrppaaaa-ae wwtpeemmmmmaaaapeemmeeeeeeee 2020.
[41] Yonglin Tian, Lichao Huang, Xuesong Li, Yuan Li, Zilei Wang, and Yue Wang. Pillar in pillar: Multi-scale and dynamic feature extraction for 3d object detection in point clouds. arXiv preprint 192.04775 2019.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[43] Haiyang Wang, Lihe Ding, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, and Liwei Wang. CAGroup3d: Class-award grouping for 3d object detection on point clouds. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL wwwpaarwwweee.eeoourmmmmiii.....
[44] Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11 1110-1119, 2022.
[45] Tai Wang, Xinge Zhu, and Dahua Lin. Reconfigurable voxels: A new representation for baser point clouds. arXiv preprint 2XXi:22004.027224 2020.
[46] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568-578, 2021.
[47] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191. PMLR, 2022.
[48] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021.
[49] Xinli Xu, Shaocong Dong, Lihe Ding, Jie Wang, Tingfa Xu, and Jianan Li. Fusionrenn: Limera fusion for two-stage 3d object detection. arXiv preprint arXhv:2209.10733
2022. 2022.
12

---

[50] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
[51] Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection.
In Conference on Robot Learning, pages 146-155. PMLR, 2018.
[52] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In CVPR, pages 7652-7660, 2018.
[53] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprini arXiv:2107.00642
2021.
[54] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
[55] Zetong Yang, Li Jiang, Yanan Sun, Bernt Schiele, and Jiaya Jia. A unified query-based -aaed paradigm for point cloud understanding. arXiv preprint arXiv:2203.01252, 2022.
[56] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021.
[57] Cheng Zhang, Haocheng Wan, Shengqiang Liu, Xinyi Shen, and Zizhao Wu. Pvt: Point-voxel transformer for 3d deep learning. arXiv preprint arXiv:2108.0667 2021.
[58] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021.
[59] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. arXiv preprint arXiv:2204.124 12451, 2022.
[60] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. CVPR, 2018.
[61] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arX1v:2010.04159, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work is only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Our implemen- tation details are posted on Section 4.1 and the supplemental materials. The data are
public datasets and our code will be available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and setup in and 4.3
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report average performance of 5 trials by default.
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Waymo 38 and
KITTI g
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We don't oon't include any new assets.
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [Yes] Waymo (38) and KITTI [8] are two standard benchmarks. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Waymo 38 and KITTI g are two standard benchmarks.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We didn't use crowdsourcing conduct research with human subjects...
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] We didn't use crowdsoureing or conduct research with human subjects...
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We didn't use crowdsourcing or conduct research with human subjects...
14


Reviews:
[{"review_id": "Ro09zGEyvHw", "reviewer": "Reviewer_QRYf", "summary": "This paper presents a novel Mixed-scale Sparse Voxel Transformer to capture both types of information simultaneously by the divide-and-conquer philosophy. Experiments on waymo show promising results.", "questions": "Refer to part Strengths And Weaknesses.", "limitations": "Refer to part Strengths And Weaknesses.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The general idea of this article is to extract features through multi-scale transformers in point cloud voxels, and then perform regression through 2D CNN. First of all, the overall process and idea are exactly the same as PointPillar. Second, this multi-scale idea is the same as Voxel-FPN[2], but the processing backbone is replaced by a transformer. So from this point of view, the author needs to think carefully about the novelty of this article.\n\n[1] Lang A H ,  Vora S ,  Caesar H , et al. PointPillars: Fast Encoders for Object Detection From Point Clouds[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.\n\n[2] Wang B ,  An J ,  Cao J . Voxel-FPN: multi-scale voxel feature aggregation in 3D object detection from point clouds[J].  2019.\n\n2. This article is poorly written and needs careful polishing, such as line 5, line 32.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Paa1Sju5CNJ", "reviewer": "Reviewer_G2ui", "summary": "This paper proposes a new transformer architecture for Lidar-based 3D object detection. Previous windows-based transformers utilize large windows to capture the long-range context information but ignore the negative effect of blurring fine-grained details. This is especially true for 3D point cloud cases where the point downsampling is performed and sparse and distant points are selected (further blurring fine-grained details). To alleviate this, the authors propose to design a multi-scale transformer architecture and set up key windows of different scales in which different windows contain the same amount of downsampled points. Scale-aware head attention is also proposed to divide the attention head into multiple groups and aggregate after inner group computation with the query. Additionally, the authors further propose a chessboard sampling strategy to iteratively sample different query voxels in order to reduce the computational cost while maintaining similar accuracy performance. In the end, the final models are evaluated on the KITTI and Waymo datasets and achieve SOTA results. ", "questions": "Q1: On line 180, the authors mentioned that the model selected one of the four characters as the sampling location. How are these chessboards generated? Is it purely random? The authors need to provide more details. \n\nQ2: Please address the weakness I listed above. ", "limitations": "Partially. More discussion of the potential negative societal impact like the earlier adoption of not-yet validated autonomous vehicles (or misleading 3D detection performance) will be useful. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The overall design is sound and relatively novel. This work nicely extends the Single-Stride Transformer [1] to work with voxel data. Concretely, SST only works with pillar (a long elongated voxel) representation to maintain efficiency. Extending windows-based transformers to the voxel representations come with an efficiency challenge due to the larger amount of voxels inside a key window. The authors nicely address this by proposing multi-scale window attention and chessboard query sampling strategy. The overall approach significantly improves the SST baseline and sheds light on efficient transformer design with 3D point cloud data. \n\nS2: The approach is well ablated on two popular datasets and achieves SOTA results compared to published methods. \n\nS3: The paper is generally well written and easy to follow. \n\nW1: There is no ablation on the number of groups for multi-scale attention (currently set to two). To further verify the motivation, the authors need to investigate the accuracy/speed tradeoff with a larger number of groups.\n\nW2: Current model only works with single-frame data. Could it scale to multi-frame? \n\nW3: Experiments with more two-stage architecture will be interesting (minor, this could be done later). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NzTfeGbhSkd", "reviewer": "Reviewer_aXhD", "summary": "This paper proposes a backbone, dubbed MsSVT, for exploiting mixed-scale sparse voxel transformer for LiDAR-based 3D detection. It devises chessboard sampling and balanced multi-window sampling strategies to obtain queries and a set of keys from different scales and then uses multiple attention heads for different scale-aware groups. The mixed-scale features are aggregated for multi-scale scene understanding. Experiments demonstrate the efficacy of the proposed method and the qualitative analysis with expected attention weights shows the role of the mixed-scale design.", "questions": "- Does the implementation only try the mentioned two-scale case? How about the results for other alternative designs? For example, using the same sampling strategy for both queries and keys, and using more scales for mixing?\n", "limitations": "The author analyzes the limitations and societal impacts adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation for designing mixed-scale voxel feature extraction is reasonable.\n- The proposed method not only shows impressive performance compared to other methods but also turns out to be efficient with the customized sampling designs. It is important for the practical use of LiDAR-based 3D detectors.\n- The illustration, such as Fig. 2 and 5, clearly show the design methods and the functionality of mixed-scale voxelization feature extraction.\n- It shows great potential for improving the detection performance of small objects, such as pedestrians and cyclists, in the Waymo experiments.\n\nWeaknesses:\n- The motivation for the necessity of mixed-scale voxelization is a little inaccurate.\n\nLarge-scale outdoor scenes do not feature significant variance in instance scales, from my perspective, at least compared to 2D object detection in images. A more reasonable explanation should be related to the ununiform distribution of LiDAR point clouds? Due to the sparsity and irregularity of LiDAR points distribution, a voxel-based 3D detector needs customized designs for voxelization to tackle the difficulty of detecting small objects and far-away objects. Of course, I think the scale variance of different instances is also a problem in 3D although it is just one reason for the necessity of mixed-scale voxelization.\n\nHere also needs a literature review for different voxelization methods in previous works, for example:\n\nEvery View Counts: Cross-View Consistency in 3D Object Detection with Hybrid-Cylindrical-Spherical Voxelization, NeurIPS 2020\n\nReconfigurable Voxels: A New Representation for LiDAR-Based Point Clouds, CoRL 2020\n\nIt's All Around You: Range-Guided Cylindrical Network for 3D Object Detection\n\nContext-Aware Dynamic Feature Extraction for 3D Object Detection in Point Clouds (Pillar-in-Pillar)\n\nVoxel-FPN: Multi-Scale Voxel Feature Aggregation for 3D Object Detection from LIDAR Point Clouds, Sensors 2020\n\n- Technical contribution: This paper contributes more engineering techniques, including mixed-scale voxelization, multi-scale heads, and efficient sampling implementation. From the perspective of novel technical contributions and new insights to the community, the contribution is just above the borderline.\n\n- Experiments: There are only results on the validation set both on Waymo and KITTI. Although the results on the validation set of Waymo are commonly used for comparison, the validation results on the KITTI are usually not convincing enough. Additional results on the test set would strengthen the experiment part.\n\n- Minor typos: The paper needs polishing and fixing some minor typos, such as line 123 prsent -> present, line 125 regrard -> regard, line 197 wrt -> w.r.t or with respect to. The methodology part can be also improved. For example, the two sections about sampling are not put together, which makes the reader confused about their relationship and the main philosophy of different sampling methods for queries and keys (It is clarified until the beginning of Sec. 3.2). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept