Spatial Pruned Sparse Convolution for Efficient 3D Object Detection
Authors: Jianhui Liu
Yukang Chen
Xiaoqing Ye
Zhuotao Tian
Xiao Tan
XIAOJUAN QI
Venue: NeurIPS 2022 Accept (2022)
Keywords: efficient sparse convolution
3D detection
spatial pruning

Abstract: 3D scenes are dominated by a large number of background points, which is redundant for the detection task that mainly needs to focus on foreground objects. In this paper, we analyze major components of existing sparse 3D CNNs and find that 3D CNNs ignores the redundancy of data and further amplifies it in the down-sampling process, which brings a huge amount of extra and unnecessary computational overhead. Inspired by this, we propose a new convolution operator named spatial pruned sparse convolution (SPS-Conv), which includes two variants, spatial pruned submanifold sparse convolution (SPSS-Conv) and spatial pruned regular sparse convolution (SPRS-Conv), both of which are based on the idea of dynamically determine crucial areas for performing computations to reduce redundancy. We empirically find that magnitude of features can serve as an important cues to determine crucial areas which get rid of the heavy computations of learning-based methods. The proposed modules can easily be incorporated into existing sparse  3D CNNs without extra architectural modifications. Extensive experiments on the KITTI and nuScenes datasets demonstrate that our method can achieve more than 50% reduction in GFLOPs without compromising the performance.

Spatial Pruned Sparse Convolution for Efficient 3D Object Detection
Jianhui Liu!* Yukang Chen
Xiaoqing Ye
Zhuotao Tian?
Xiao Tan
Xiaojuan Qi
'The University of Hong Kong TThe Chinese University of Hong Kong Baidu
Abstract
3D scenes are dominated by large number of background points, which re- dundant for the detection task that mainly needs focus foreground objects. In this paper, we analyze major components of existing sparse 3D CNNs and find that 3D CNNs ignore the redundancy of data and further amplify it in the down-sampling process, which brings a huge amount of extra and unnecessary computational overhead. Inspired by this, propose named spatial pruned sparse convolution (SPS-Conv), which includes two variants, spatial pruned submanifold sparse convolution (SPSS-Conv) and spatial pruned regular sparse convolution (SPRS-ConV), both of which are based on the idea dynamically determining crucial areas for redundancy reduction. We validate that the magnitude can serve as important cues to determine crucial areas which get rid the extra computations of learning- based methods. The proposed modules can easily be incorporated into existing sparse 3D CNNs without extra architectural modifications. Extensive experiments on the KITTI, Waymo and nuScenes datasets demonstrate that our method can achieve more than 50% reduction in GFLOPs without compromising the performance. Code and models available this link
convolution operator
Introduction
3D object detection has always been a research field of great interest due to its wide applications in autonomous driving, virtual reality, and robotics. However, compared with the 2D detection task, LiDAR- based 3D detection is more challenging due the inherent characteristics of LiDAR points such as disorders, irregularity, and non-umi non-uniformity.. Recently, voxel- based methods [23 10, 37
[8] with deep 3D sparse convolutional neural networks (CNNs) have become one of the major research streams to tackle 3D object detection problem due to its effectiveness and simplicity. Nevertheless, the improved accuracy is often accompanied by increased computational costs (30 36 3
limiting
its applicability in practical systems. This motivates us to investigate potential redundancies in the detection model that can be safely avoided improve efficiency without sacrificing accuracy.
When delving into the task 3D detection, we found that 3D data itself has high redundancy as shown in Fig. aa). a.). Notably, compared with background points in each stage of the sparse CNN, the proportion of foreground points in the entire scene is extremely low (around 5%), which shows that less-informative background points occupy the major areas of scene. However, the existing 3D sparse CNNs are applied uniformly to the whole scene, causing a considerable amount of computation on the background areas, which might be potentially redundant. Intuitively, if such areas can be identified and selectively skipped, the computational costs have the potential to be dramatically reduced without deteriorating the performance.
Besides the redundancy of the data, the model design itself also brings redundancy. To avoid aggressive down-sampling, 3D sparse CNNs adopt dilation-like design when applied to perform
EEqual Contribution. i Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Reeguad Badkground
1000
Sparse CNN
SP Sporse CNN
255
0.105
2.52
Q,002
1.66
0,0802
15
1.32
2/1167
1.429
166167
0.011 52.
0.57
0.7268
Stage
Stag-2
Stage 3
Stagn 4
Stage 1
Sbage 2
Stgee
Stage
(a)
(b)
Figure I: (a) Comparison of the foreground and background ratios. (b) By replacing SPRS- PRSS-Cov with its counterpart sparse CNN, unnecessary positions are effectively suppressed from being activated.
the down-sampling operation. Specifically, it will compute features for adjacent empty voxels as shown in Fig. 2 (b): the yellow voxels which are initially empty will be assigned with computed features. Consequently, after convolutional (stride 1) down-sampling the number of non-empty voxels might be increased rather than decreased as shown in Fig. [ (b): the number of non-empty voxels even doubled (see stage compared the input (see stage This will undoubtedly increase unnecessary computational costs for follow-up stages.
The above observations prompt us ask whether there is way identify redundant computations that can be pruned to improve efficiency. A solution that has been attempted the 2D image domain 29 11 is add auxiliary learnable module to predict soft mask [21 28 that locates areas to be skipped for computational efficiency. The module often requires additional post-fine- tuning, auxiliary costs for integration, and incurs nog-neglig computational overheads.
With aforementioned considerations, we propose a new simple and efficient sparse convolution operator named Spatial Pruned Sparse Convolution SPPS-Con),, to alleviate redundancies caused by data and model design. The core idea of SPS-Conv is to find redundancy in the model dynamically. To avoid the overheads of learning- based methods for simplicity and efficiency, we investigate whether the features from the model itself contain useful cues that can be leveraged to identify redundancies. Fortunately, find that the magnitude of features could be robust signal reflect the importance where locations with smaller magnitudes are more likely to be redundant areas. This leads us to magnitude-based sampling module which is incorporated into 3D convolution layers reduce redundancy in data and model.
Specifically, to address data redundancy, the submanifold variant of SPS-Conv, termed spatial pruned submanifold sparse convolution (SPSS-Conv) can adaptively calculate important positions in the light of the magnitude of features and perform convolutions these locations, leaving features of redundant locations unchanged. As for model redundancy in the down-sampling process, another variant named spatial pruned regular sparse convolution (SPRS-Conv), which can dynamically determine the position that needs to be inflated. As Tab. [b) shows, our method effectively reduces the meaningless expansion caused by convolution (around 50% in stage2). In general, SPS-Conv only keeps the essence and discards the irrelevant ones for pursuing higher efficiency without compromising performance.
In summary, we propose an efficient convolution operator for spatial redundancy pruning, which can be easily incorporated into existing sparse 3D CNNs for 3D detection. We conduct extensive experiments on both KITTI, Waymo, and nuScenes datasets. The result shows that we can achieve comparable performance with SOTA methods while enjoying 52.4%, 62.46%, and 46.5% GFLOPs reduction.
Related Work
Sparse Convolution. Depending on how it is used, sparse convolution 15 can be subdivided into regular sparse convolution and submanifold sparse convolution. Regular sparse convolution is

---

Cuepet
Thkeed bwaze
Figure 2: Illustration of conventional sparse convolution. (a) Submanifold sparse convolution (b) Regular sparse convolution.
often used for the down-sample layer, which dilates all input features to its kernel-size neighbors, sacrificing computational efficiency in exchange for a wider interaction of information. contrast, the submanifold convolution is more like simplified version of the regular sparse convolution, frequently used in residual blocks[[7] only calculates on valid positions, avoiding meaningless expansion and achieving efficient computation.
Voxel-based Detectors. In order to process irregular point cloud data, voxel-based methods first convert the point cloud into regular voxel, and then use mature CNNs for feature extraction. However, the computational cost and memory requirement both increase cubically along with the voxel resolution. Thus, it is infeasible to train a voxel-based model with high-resolution inputs. Benjamin al. [15] proposed novel convolution operator named submanifold sparse convolution, by reducing calculation on invalid position, which greatly improves the computing efficiency and memory.
Approaches for voxel-based detection methods can be grouped into two categories, i.e., single- st 16 34
25 22 24]. Single-stage detectors relatively simple, and directly predict final bounding boxes based on the features extracted by sparse CNN. VoxeINet[4[] utilizes Sparse CNN to extract voxel features from a dense grid. DE proposes 3D sparse convolutions to efficiently extract voxel features. HVNet designs convolutional network that attentively aggregates and projects the multi-scale feature maps to achieve better performance. In contrast, two-stage detectors
35
39 40 and two-stage detectors[IO)
complex but get higher performance. Part-A (24 proposed a part-aware and aggregation module to exploit the intra-object [24]
part location. PV-RCNN 23 uses key points to extract voxel features for better box refinement.
Dynamic Kernel Shape Design. Much literature work on dynamic kernel design for adapting various tasks. Deformable convolution [9] predicts offsets for feature sampling. For 3D scene understanding, КPСопу (27) constructs dynamic graph for kernel points. Deformable PV-RCNN 3 applies offset prediction for feature sampling in 3D object detection. Focals Conv [8] learns a dynamic spatially sparsity which enhances the network for spatial modeling capabilities,
Background and Motivation
In this section, we will introduce the mechanism of sparse convolution, then analyze its redundancy and introduce our motivation.
3.1 Preliminary of Sparse Convolution
To better illustrate sparse convolution, we first introduce its general expression. We denote xp as an input feature with Cin dimension at position p. E k xx.xxx is the weight of convolution kernel, where and Kd refers to the dimension of the spatial space and spatial size of the kernel respectively. Thus can formulate the convolution process as follows.
Yp E Poul
Σ
Wk Xpe, where (p, Pin) = f1 KE Pm, k E Kd),
(1)
(p.
3

---

Here, Pm and Pour refer to the input and output feature space, k is the kernel offset that corresponds to all the valid locations kernel space K Ps p + ls denotes all non-empty neighbor voxels around center Due to the discrete data distribution in 3d space, we define K (p, Pin) as subset of K', leaving out the empty position, which is constrained by position p and input feature space Pin-
There are two types of sparse convolutions, namely, regular sparse convolution [14] and submanifold sparse convolution (15), their major difference lies the output position of the convolution. For regular sparse convolution, as shown in Fig. 2 the position related to the input will be activated during the convolution process. Specifically, these positions are assigned value and combined with the Pin to form a new Pour. This process can be easily formulated as
Pout =
U (p, where Q (p, KJ) = =D+tRIEEK) Poam
(2)
where Pawny is a subset of P.n. representing the position where the convolution needs to be performed. Q (p, K2) refers to the position related to Paom which is restricted by kernel size.
Regular sparse convolution can effectively expand the receptive field like 2D convolution, which is beneficial for 3D sparse data. However, it also brings a computation burden due to the generation of too many active locations, which can lead a decrease in speed in subsequent convolution layers due to a large number of active points. In order to achieve a better trade-off between efficiency and receptive field, regular sparse convolution is often adopted in the down-sample layer in sparse CNNs.
In contrast, submanifold sparse convolution [5] restricts an output location to be active if and only the corresponding input location is active, which avoids the dilation in regular sparse convolution, although the receptive field is limited. With a simple modification on Eq. Paut = Pin. regular mode can be convert to submanifold mode.
3.2 Redundancy Analysis and Motivations
Despite the wide adoption of sparse CNNs 3D object detection due to their generality and efficiency, spatial redundancy caused by sparse CNNs for this task is still under-explored. Considering the large number pask-independent background points in the 3D scene, a lot of redundant computation will be generated in each stage. In addition, due to the expansion of regular convolution, the background points will become denser, bringing more computational burden to the following stages. Therefore, in response to these problems, we have targeted the improvement of the sparse convolution operator. Details are shown in Sec.
Spatial Pruned Sparse Convolution
In this section, we propose new efficient sparse convolution operator, named spatial pruned sparse convolution (SPS-Conv), Intending to effectively bring down the unnecessary computation costs caused by the 3D spatial redundancy, two variants of SPS-Conv are investigated, i.e., spatial pruned submanifold sparse convolution (SPSS-Con' and spatial pruned regular sparse convolution (SPRS- Conv), and both of them are based on the same idea that the redundancy should be dynamically ameliorated in 3D space according to the specific contexts different individuals. In the following. the magnitude- based sampling strategy is presented in Sec. 4.1 followed by the introduction of SPSS Conv and SPRS-Com in Sec. 4.2 and Sec. 4.3 respectively. Then, Sec, 4.4 manifests the generalization ability by showing how our method is applied to generic sparse CNNs.
4.1 Magnitude-guided Spatial Sampling
In previous works 29 5l. sampling module is often designed in a learnable manner, i.e., convolution layer is trained to yield soft masks for sampling, while, at the time of inference, the soft masks are converted hard ones where elements with lower responses will be simply discarded. Although these methods have been empirically shown effective in 2D tasks, the following issues still exist: 1) Additional supervision is required on the prediction branch, otherwise it easy to fall into trivial solutions, i.e., the values in the soft mask are all close to 1. 2) The proportion of pruning is hard to be fixed, and is instead automatically determined throughout the learning process, making it hard to satisfy the specific requirement with certain limited computation budget. 3) Two-stage

---

Outgut
importent
Inpurr
Idaitity
sbortout
Magnitade
Inpur feonure
Impor tove
Unimportous
Murtjely
Concat
Figure 3: Illustration of magnitude-guided spatial sampling and spatial pruned submanifold sparse convolution CoPS)-Conv. Notably, during the process SPSS-Conv, "conv on important" refers to performing convolution only at the positions with high magnitude value, and the neighborhood of the convolution still contains the unimportant features.
fine-tuning required to compensate for the performance loss of converting from soft mask hard mask, consuming additional time and resources.
Recent literature has shown that the attention mechanism is able reveal what the model cares about, Following [8 33 the feature magnitude is adopted for yielding the model's "attention" to highlight the informative elements. For the purpose of getting the magnitude- segnitude-based attention mask, we first calculate the channel-wist absolute mean values on different voxels, and then the sigmoid function is applied to get the normalized output, which can be formulated
C
1 M(F) = Sigmoid where G(F) = C
Fel
(3)
where F' and C' denote the input feature and its channel. F. is the feature of the c-th dimension. G and M refer the spatial magnitude map and magnitude mask.
4.2 Spatial Pruned Submanifold Sparse Convolution
Considering the fact that the background regions often take the majority 3D scenes, directly performing the submanifold convolution everywhere of the input volume inevitably involves sub- stantial unnecessary calculations, leading to computational redundancy. Therefore, with a focus on alleviating this issuc, propose the spatial pruned submanifold sparse convolution (SPSS-Conv) that dynamically examines the regions of interest.
Specifically, with the guidance of the magnitude-basee attention mask, separate the elements P. into two disjoint subsets, i.e., the important P. the unimportant set Paim. where P.n. Pumm Pm. We note that various methods can be incorporated for accomplishing the division, such as using fixed threshold simply taking the elements with top-k scores. Then we re-weight the input feature map by multiplying it with the magnitude mask, applying submanifold convolution on important positions Pim according to Eq. and concatenate it with the features unimportant positions Paim as residual, as shown in Fig. To this end, the reconstructed output feature can be written as:
(Xp, M ((xx)) Wky DEE. Pm
Ny E Pout
(p.
(4)
Xpx M(xp)) p E Paim
Since Pim is sparser than Pm. convolution enables aak-beneficcall locations to be highlighted. In con- trast, those unimportant positions have relatively small activation values, thus the skip connection can not only reduce the computational overhead but also suppress the diffusion of redundant information.
5

---

Important
Dilate on important
Conv
Down sample
Unimportant
Ouipat
Dilated
Valid Position
Stride Mask
Figure 4: Illustration of spatial pruned regular sparse convolution (SPRS- Cons-C.nv.... The input of SPRS- Conv is generated by magnitude-guine spatial sampling. The figure shows the case of stride=2.
Irput
Subm blokk
SPSS blcc
SPAc Mock
SPSS slock
CPSS Bock
SPSS block
SPSS Neck
Stage
Stage
Stage
Stage
Siben Siert
Aeplice
SP55 Mudk
Rig Muck
Aeciice
SPRS Blokk
Figure 5: Framework overview. We substitute all the submanifold blocks and regular blocks with SPSS blocks and SPRS blocks, except the block in the stem layer. Each block contains a convolution layer followed by batch normalization and ReLU activation.
4.3 Spatial Pruned Regular Sparse Convolution
The "dilation" effect of the regular sparse convolution inevitably causes a large number adjacent positions to be activated, which brings greater computational burden to subsequent layers. Recent work [ shows that the spatially dynamic sparsity in sparse convolution is essential for sophisticated 3D object detection. Motivated by this, SPRS-Conv designed magnitude-based dynamic prediction down-sampling module, which can effectively suppress the amplification effect of spatial redundancy.
Similar to SPSS-Conv, Pin first divided into two disjoint subsets, iee. Pim and Paim- Unlike the regular sparse convolution that dilates all possible neighborhood locations, leading sub-optimal efficiency, we instead dynamically choose whether to expand for each position based on the magnitude mask yielded by Eq. 3 Further, define Pastate, which is set composed of all P. positions and the positions within their kernel range. The equation can be written as:
U U(s U.P. PEP.
Pailate
(5)
Considering the stride during down-sampling, we designed a stride mask E (P) to ignore those that should be skipped by the stride, which can be formulated as below:
E(P)
(p. %1s (m.
Σ
0 O
(6)
where refers to the stride of the convolution specific dimension, % denotes mod operation. With E (P), we can further modify Eq 2 as:
P E (Pallae) U E (Pmm)
As shown in Fig. we operate at the specified positions according to Eq. to get the output feature map. Then we down-sample the correct size according to the stride. Compared to the regular sparse convolution, method dynamically prunes Pour conditioned on its specific content. As

---

Method
mAP
NDS
Car
Truck
Bus
Trailer
C.V.
Ped
Mol
Bye
T.C.
Bar
PointPillars
18
30.5
45.3
68.4
23.0
28.2
23.4
4.1
59.7
27.4
1.1
30.8
38.9
3DSSD 32
42.6
56.4
81.2
47.2
61.4
30.5
12.6
70.2
36.0
8.6
31.1
47.9
CBGS 42
52.8
63.3
81.1
48.5
54.9
42.9
10.5
80.1
51.5
22.3
70.9
65.7
HotSpotNet 7
59.3
66.0
83.1
50,9
56.4
53.3
23.0
81.3
63.5
36.6
73.0
71.6
CVCNET 6
58.2
66.6
82.6
49.5
59.4
51.1
16.2
83.0
61.8
38.8
69.7
69.7
CenterPoint [37
58.0
65.5
84.6
51.0
60.2
53.2
17.5
83.4
53.7
28.7
76.7
70.9
CenterPoint 39.9
67.3
84.1
52.3
65.1
53.5
19.6
84.6
58.3
31.3
78.0
72.4
SP-CenterPoint 59.7
67.2
84.2
51.1
61.8
54.5
20.1
84.3
60.7
30.4
78.1
72.0
Table I: Comparison with other methods on nuScenes test split. CenterPoints denotes our own
re-implementation result for CenterPoint.
Method
mAP
NDS
GELOPS
Car
Truck
Bus
Trailer
C.V.
Ped
Mot
Bye
T.C.
Bar
CenterPoint 37
58.6
66.2
62.9
85.0
58.2
35.7
15.5
58.8
70.0
SP- SP-CenterPoint
58.5
66.1
34.3
84,9
58.1
70.2
34.7
17.2
85.3
58.7
38.6
69.7
67.0
Table 2: Comparison with base method on nuScenes val split. GFLOPs only include the floating-point operations in sparse CNNs.
shown in Fig. Tb), SPRS-Conv greatly reduces the number of voxels in each stage, especially stage2 (about 50%), thus reducing the computational burden caused by background points.
4.4 Spatial Pruned Convolution Network
Our method model-agnostic thus can be easily plugged into any existing sparse CNNs. regular frame of sparse CNN is composed of stem layer and four stages, each of which contains a down- sampling layer and two submanifold sparse convolution blocks. demonstrate the effectiveness and generalization ability, we replace all regular sparse convolutions and submanifold sparse convolutions in sparse CNN, except those in the stem layer, with our proposed SPRS- Conv and SPSS-Conv, respectively. Detailed experimental results and analysis presented in Sec.
Experiments
5.1 Experimental Setting
3D Object Detection Datasets. We evaluate our method on three challenging benchmarks KITTI 13
Waymo (26 and nuScenes [4. KITTI dataset consists of 7,481 samples and 7.518 testing samples, where the training samples are generally divided into the train split (3,712 samples) and the val split (3,769 samples). Waymo dataset contains 1,000 sequences in total, with 798 for training and 202 for validation. As the Waymo dataset is really large-scale, we use 5 data for training. Results in Waymo are evaluated in difficulty LEVEL_1 1 (L1) and LEVEL (L.) objects. nuScenes dataset is large-scale autonomous driving dataset, which contains 1,000 driving sequences in total. It is split into 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. It is collected using a 32-beam synced LIDAR and 6 cameras with the complete 360 environment coverage.
Model Configurations. iee, VoxeIR-CNN I0,, PV-RCNN 23 and SECOND 31 for KITTI, CenterPoint [37 for nuScenes and Waymo. We replace the submanifold convolution and regular convolution in sparse CNNs with SPSS-Conv and SPRS-Conv respectively, other experimental
hyperparameters flowing the default settings of the baseline methods 10 23 37 3T]
Training. Unlike existing 2D approaches 29 5 that design the sampling module in a learnable manner which needs a two-stage fine-tune. We train the model in an end-to-end manner. Our SPS-Conv is purely embedded into sparse CNNs without any modifications on network architectures or parameters (i.e., feature dimensions). Our modules contain one hyperparameter, i.e., pruning ratio and we choose use top-k to divide the pruning part. For spatial pruned submanifold sparse convolution (SPSS-Conv) it refers to the proportion positions each stage that can be ignored for calculation, which can be symbolized as (r Fel. T22, ra3 We set it as (0.3, 0.3, 0.3, 0.3
7

---

SPSS
SPRS
GFLOPs
Vec L1
Vec
Le
Ped L2
Cyc L1
L2
76.7
12.76772.25
64.917 64.42
74.19767.96
66.03 60.34
71.0766.77 69.79
68.497 67.28
43.5
72.46 / 71.91
64.35 63.85
73.71 /67.53
65.81 60.13
70.85
68 67.22
55.2
/2.80/71.90 71.96
64.36/ 64.07
66.05
70.42/ 69.69
68.85
28.8
72.68
72.13
64.66/ 64.16
74.39/ 68.02
66.00/ 60.36
71.88 69.84
68.47/
Table 3: Comparison CenterPoint baseline method Waymo val split, trained with 20% data.
GFLOPs only include the floating-pone operations in sparse CNNs.
Method
Easy
Mod.
Hard
Point R-CNN 22
88.88
78.63
77.38
Method
GFLOPs
Easy
Mod.
Hard
Part-A-
24
89.47
79.47
78.54
STD 1
89.70
79.80
79.30
Second 31
88.36
78.75
77.57
SA-SSD 32
90.15
79.91
78.78
SP-Second
3.6
88.57
78.64
77.42
PV-RCNN 23
89.35
83.69
78.70
PV-RCNN [23
89.35
83.69
78.70
VoTr-TSD 20
89.04
84.04
78.68
SP-PV-RCNN
89.27
83.22
78.87
Pyramid-PY
89.37
84.38
78.84
Voxel R- R-CNN [10
89.41
84.52
78.93
Voxel R-CNN 10
89,41
84.52
78.93
SP-Voxel R-CNN
89.69
84.57
78.89
SP-Voxel R-CNN
89.69
84.57
78.89
Table 5: Comparison with base methods on KITTI val
Table 4: Comparison with other methods on split in AP (R11) for Car. GFLOPs only include the floating-pon maating-point operations in sparse CNNs.
KITTI val split in 3D (R11) for Car.
for nuScenes and {0.5, 0.5, 0.5, 0.5} for Waymo and KITTI. As for spatial pruned regular sparse convolution (SPRS-Conv), the pruning ratio used for controlling the amount that needs to be inflated when downsampling. We symbolized it as frai, Taz, Tazl, which are set as {0.5, 0.5, 0.5 in nuScenes and Waymo and {0.7, 0.5, 0.3) KITTI.
5.2 Main Results
nuScenes. For the sake of fairness, we choose LIDAR-only methods for comparison, model en- sembling and additional augmentation are not included during inference. Our method achieves competitive performance on both mAP and NDS on nuScenes test split, in Tab. [ Besides, We apply the method the base model for a more detailed comparison, shown in Tab, 2 our method can help the model to preserve the original performance while skipping redundant computation, specifically, SP- CenterPPint's GFLOPs are only 54.5% of the base model.
KITTI. We make comparisons with recent state-of-the-art methods KITTI val split. As shown in Tab, 4 Our Spatial Pruned Voxel R-CNN achieves competitive results with other methods. To better demonstrate the effectiveness of our method, we validate our method on popular voxel-based detectors, i.e., Second, PVRCNN, and VoxcI-RCNN, Tab shows that with our method, the GFLOPs of the sparse CNN are reduced by more than 50%, and the matching performance can still be obtained on KITTI val split in AP3p in recall 11 positions.
Waymo. We present SPS-Conv based on the CenterPoint on the outdoor dataset Waymo 266 in Tab. The results demonstrate the generality of SPS- Conv on large-scale datasets with dense point clouds: SPS-conv still maintains high performance while significantly reducing FLOPs.
Method
PR.
GFLOPs
mAP
NDS
Method
PR.
GFLOPs
mAP
NDS
CenterPoint
62.9
58.59
66,20
CenterPoint
62.9
58,59
66.20
0.1
56.6
58.64
66.25
0.1
59.1
58.57
66.01
0.3
45.2
58.48
66.11
0.3
55.4
58.61
66.07
SPSS-Conv
0.5
33.5
58.19
66,08
SPRS-Conv
0.5
47.2
58.58
66.24
0.7
21.6
57.80
65.69
0.7
32.3
58.18
65.93
0.9
9.7
55.21
64.23
0.9
9.9
52.09
61.89
Table 6: Ablations on pruning ratio in SPSS-Conv Table 7: Ablations pruning ratio in SPRS-Conv on nuScenes val split.
on nuScenes val split.

---

Method
mAP
NDS
Car
Truck
Bus
Trailer
C.V.
Ped
Mot
Byc
T.C.
Bar
SPSS-Conv
58.48
66.11
58.2
69.5
35.7
15.5
85.3
40.9
70.0
68.1
SPSS-Conv random drop
56.02
64.72
84.4
56.3
67.3
34,3
13.8
84.1
54.3
34.9
63.2
65.6
SPSS-Con inverse
55.84
64.49
84.3
56.7
68.2
33.3
14.4
83,5
54.4
34.2
63.3
66.1
UPRS-Conv
58.58
66.24
84.9
57.5
69.1
35.2
15.5
85.3
59.6
40.2
70.6
67.8
SPRS- Cons random drop
56.58
64.77
82.7
54.8
69.4
34.1
14.9
83.6
54.0
38.6
67.5
66.0
SPRS-Com inverse
16.72
39.29
29.6
9.4
16.3
1.7
0.6
24.2
11,8
3.6
29.0
41.1
Table 8: Ablations on the importance of position with high magnitude on nuScenes val split.
@a) Visualization in Lidar View
(b) Vissalization in BEV
Scene 1 Pruned
Scene Pruned
Scene Raw
Scene Raw
Feature magnitude
Figure 6: Visualization of SPS-Conv in lidar view and BEV. (a) Comparison before and after pruning in lidar view. (b) Visualization of magnitude in BEV.
5.3 Ablation Study
SPS-Conv Pruning Ratio. The pruning ratio (PR.) of the SPS-Conv is used control the proportion of unimportant positions selected in each block, the higher the proportion, the fewer positions are involved in the calculation. We evaluate nuScenes val split upon CenterPoint. As shown in Tab. 6] and Tab. 7 there is relatively obvious and proportional decrease in GFLOPs, in contrast, the performance drop not severe, except when the pruning ratio increases to 0.9. This also reveals that 3D Scenes contain spatial redundancy, and when we selectively skip these redundant positions, will not affect the performance.
Sampling and Interpolation Strategy in SPSS-Conv: In order to verify the effectiveness of our method, we conduct a combination of several sampling and interpo- lation methods, where the pruning ratio limited to 0.5. Results are shown in Tab. 9 it is not difficult to find that the learnable method and magnitude method have com- parable results which makes us think that magnitude another manifestation of network attention, so we select important regions according to this logic, which conforms to the inherent performance of the network. In addition, for the interpolation operation, compared to the average pooling operation, a simple skip connection can play a good role. possible explanation is that these predicted unimportant regions not task-essential, even using the original values would not have much impact on the results.
Method
mAP
NDS
magnitude + skip
58.19
66.08
magnitude + avg pooling
58.03
65,99
conv3x3 + skip conv3x3 avg pooling
57.29
65.37
57.57
65.61
linear avg pooling
57.67
65.54
Ablations on sampling and interpo- Table 9:
lation in SPSS-Conv on nuScenes val split.
Importance of Position with High Magnitude. To verify that the location chosen by this mechanism friendly,, we exchanged the calculation method of positions with high magnitude and other positions. As shown in Tab. 8 for SPSS-Conv, it can be seen that when we reverse the important position, there considerable performance drop in categories. This phenomenon is more obvious in small objects, especially for the traffic cone and bicycle categories (even reaching about 7% performance drop). Large object categories (such as vehicles) with more points are less sensitive to the sampling method but still have notable performance drop. Compared with the experimental

---

results of SPSS-Conv, the inversion experiment of SPRS has more exaggerated performance loss. When inversion is not performed, SPRS will suppress the irrelevant features with low magnitude which reduced in the downsample part, showing that there is no obvious performance loss in the result. However, when we choose to suppress positions with large magnitudes, the spatial redundancy is amplified, and important features cannot be effectively expanded, and even discarded due to downsampling. After the above features are converted to Bird's Eye View (BEV), since the number of foreground points from the input is weakened, the effective features extracted by BEVbackone are very limited, resulting in a great degree of performance degradation.
Magnitude-base Pruning vs Random Drop. Compared to magnitude- pruning, we observe using random drop as an indicator will lead Tab.8 This is caused by the randomness, part of the foreground is discarded, resulting in performance degradation. However, the important part still has a high probability of being selected, which also guarantees performance to certain extent. Despite its degraded performance, the random drop method also has a certain degree of randomness. This is not desirable in practical applications as may have the chance to lose some safety-critical areas which will cause problems in safety-critical applications.
certain loss in performance (around 2%). shown in
5.4 Visualization
Visual Analysis. To better understand points pruned by our magnitude criterion, we visualize point clouds before and after pruning. The comparison results are shown in Fig. 6 a). the original images and the pruned images are provided respectively. We observe that most of the foreground points are preserved. For the background areas, points that fall in vertical structures, such as light, poles, and trees, are also preserved as they tend to be hard negatives, and easily confused with foreground objects. These points require a deep neural network with a certain capability to process in order to recognize them background. In contrast, background points in flat structures such as road points are largely removed because they are easily identifiable redundant points.
Why foreground points with high feature magnitude? We visualize magnitude in BEV for clearer comparison, as shown in Fig. (b)), is not difficult see that the feature magnitude of most foreground points is relatively large (vehicles). In contrast, the simple background points (ground) show in small magnitude, which are easier be distinguished by the network. To gain more insights into why high feature magnitude corresponds to the above patterns, we conjecture that this is caused by the training objective in 3D object detection. When training a 3D object detection model, the focal loss is adopted as default in 3D object detection. When we look closer at the focal loss, it will incur loss on positive samples and hard negatives while easy negatives are removed from the loss. Thus, this will generate gradients in the direction that can incur an update of features for areas with positive samples and hard negatives. This can eventually make a difference in their feature magnitudes in comparison with areas for easy negatives which are less frequently considered in the optimization objective.
6 Concluding Remarks
In this paper, investigate the spatial redundancy in sparse CNNs and propose new simple and efficient convolution operator SPSS-Conv with two variants, i.e., SPSS-ConV and SPRS-Conv Extensive experiments have proved the following conclusions: 1) Magnitude can serve indicator for sparse CNNs to dynamically identify the informative points in diverse scenes. 2) The overwhelming background region in the 3D scene results in spatial redundancy, which can be pruned by our method without adversely affecting the performance. 3) Selectively inflating the elements near the region of interest during the down-sampling process not only saves computation but also keeps the necessary information intact. 4) The proposed SPSS-Com can be effortlessly applied to generic sparse CNNs without specific structural constraints. We hope our work can provide new thoughts for inspiring future research in the community.
good
7 Acknowledgement
This work has been supported by Hong Kong Research Grant Council- Early Career Scheme (Grant No. 27209621), HKU Startup Fund, and HKU Seed Fund for Basic Research.
10

---

References
[1] 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2. 2019, 2019. IEEE. ISBN https://ieeexploo. ieee httpa//gggg//cccooo//////7722200
URL
[2] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning a general surface descriptor for 3d point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11753-1 1766,, 2021.
[3] Prarthana Bhattacharyya and Krzysztof Czarnecki. Deformable PV-RCNN: improving 3d object detection with learned deformations. CoRR, abs/2008.08766 2020. URL. https
//arxiv. barg/abs.. 2008 0000//000000006666
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[5] Shijie Cao, Lingxiao Ma, Wencong Xiao, Chen Zhang, Yunxin Liu, Lintao Zhang, Lanshun Nie, and Zhi Yang. Seernet: Predicting convolutional neural network feature-may sparsity through low-bit quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11216-18 1225, 2019.
[6] Qi Chen, Lin Sun, Ernest Cheung, and Alan L Yuille. Every view counts: Cross-view consistency in 3d object detection with hybrid-cy syliidiical-ppherttt Information Processing Systems,
voxelization. Advances in Neural 2020.
[7] Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan L. Yuille. Object as hotspots: An anchor- free 3d object detection approach via firing of hotspots. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI, volume 12366 of Lecture Notes in Computer Science, pages 68 84. Springer, 2020. doi: org 10. 1007 3 030-58589--------- 1_5
URL heLLhhtps://doo /tsttp::/ddi..or/11.....
[8] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection, 2022. URL https://: arxiv org/ abs 2204. 12463
[9] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages pages 764-77777 773. IEEE Computer Society, 2017.
doi: 10.1 00.100//CC.22017..80 URL https:/do/// org/ /trgg/10000 10. 1109/ICCV 2017.0112 .89
[10] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. arXiv preprint arXiv:2012.15712 1(2)44, 2020.
[11] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[12] Liang Du, Xiaoqing Ye, Xiao Tan, Jianfeng Feng, Zhenbo Xu, Errui Ding, and Shilei Wen. Associate-3dee 3ddet: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13329, 2020.
co-oceeetulooo association for 3d point cloud object detection. In
[13] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11): 2013.
[14] Benjamin Graham. Spatially-sparre convolutional neural networks. CoRR, abs//449.6077
2014. URL
6070
[15] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv preprint
2017.

---

[16] Chenhang He, Hui Zeng, Jianqiang Huang,
Hua, and Lei Zhang. Structure aware
single-stage 3d object detection from point cloud. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1870-
11879. Computer Vision Foundation IEEE, 2020. doi: 04100//CVP4422200200000000000 URL
htt.0//oaia/eeesuuuuuuurreee htttps:/ thecvf com http:/ooteeetttteeee CVRR__V2RRR//cc_ Stt.agaeeeaaageooooooooooottetaa Point wwwooodd__eR__2000000002
html
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12697-12705, 2019.
[19] Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid R- R-CNN: towards better performance and adaptability for 3d object detection. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Montreal, QC. Canada. October 10-17, 2021, pages 2712-2712. IEFE, 2021. doi: 10. 109/1CCV48922.2200 2002.00092...0.00022..... .00272. URL https: /ttp:://doo...o org/ 10. Iww.0///CC9992.... 00.0221002770
[20] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 3 3144-3153, IEEE, 2021. doi: 10.1 1199/CCCV88922.00222 1.0015.. URL httph::///oo.. org/10. 1 1109 ICCV48922 2021 00315
[21] Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks net- work for fast inference. CoRR, 011/1801.0210000 2018. URL URL.ttpp///axxv..rrr/aab////////
02108
[22] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770-779, 2019.
[23] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10538, 2020.
[24] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and augreaggregttion network. IEEE transactions on pattern analysis and machine intelligence, 43(8):2647-2666 2020.
[25) Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Py-renn++: Point-voxel feature set abstraction with local vector representa- tion for 3d object detection. arXiv preprint arXiv:2102.0. 004600663, 2021.
[26] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuming Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages 2443-2451, 2020.
[27] Hugues Thomas, Charles R. Qi, -amanull Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In 2019 IEEF/CVF International Conference on Computer Vision ICCV 2019, Seoul, Korea (South), October 27 November 2019, pages 6410-6419, IEEE, 2019. doi: 10.1 109/ ICCV.2019.0066 URL https://doi. org/ 110gg/10.or 1000..199900 1119/110CIII 2019. 00651
12

---

[28] Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2317-2326. Com-
puter Vision Foundation / IEEE, 2020. doi: 10. N011000/CVPP444500000.00222.
URL Dynaaii...
htttss:/oppeacceest www.eeaaaaccsss.eeeeeeeeeeeeeeee thecvf www.c//.cccm//ooooooooooo/oooooo wttp://w..ooooootetttttoo content_ttttttttcecc http:/cccPPP.22///aaeeeeeeeeeee http://co//ooooooooo//////////// hig.ratrrrrtorPaattteeeeeeeeeeeo
2020_paper. html
[29] Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin. Spatially adaptive inference with stochastic feature sampling and interpolation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes Computer Science pages 531 158.. Springer, 2020. doi: //tp:::/ddoiooooo 10. 098000777--00 3- 030 58452-8_33
_31.
[30] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Bin Zhou, and Liangjun Zhang. Fusion- painting: Multimodal fusion with adaptive attention for 3d object detection. In 24th IEEE International Intelligent Transportation Systems Conference, ITSC 2021, Indianapolis, IN, USA, September 19-22, 2021, pages 3047-3054. IEEE, 2021. doi: 10. 109/TSSC48933322199599999 URL htttt:://oool. org/ 10. 1109/ IO000/ITTC999988 2021 00021.9564911
[31] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
[32] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object de- tector. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 11040-1 11048, 2020.
[33] Zhendong Yang. Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.1183 2021.
[34] Maosheng Ye, Shuangjie Xu, and Tongyi Cao. Hvnet: Hybrid voxel network for lidar based 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1631-1640) 2020.
[35] Hongwei Yi, Shaoshuai Shi, Mingyu Ding, Jiankai Sun, Kui Xu, Hui Zhou, Zhe Wang, Sheng Li, and Guoping Wang. Segvoxelnet: Exploring semantic context and depth-aware features for 3d vehicle detection from point cloud. In 2020 JEEE International Conference on Robotics and Automation (ICRA), pages 2274-2280. IEFE, 2020.
[36] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detec- tion. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021. December 6-14, 2021, virtual, pages 16494-16507, 2021. URL https: / /ttp://///oc/eeeee////oe neur ips ccs Sapper/221/haah//aee 45d.4/40000004444000000444440444 wwwfs544433001110 w Awstract. html
[37] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and track- ing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11784-11793, 2021.
[38] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. CoRR, abs/1612.0399 2016. http://///arxii org/ /oop////111... 03928
[39] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. Sess: Self-ensembling semi-supervised 3d object detection. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 11079-11087, 2020.
[40] Wu Zheng, Weiliang Tang, Sijin Chen, Li Jiang, and Chi-Wing Fu. Cia-ssd: Confident iou-aware single-stage object detector from point cloud. arXiv preprint arXiv:2012.03013 2020.
13

---

[41] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4490-4499, 2018. 8.
[42] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint
2019.
Checklist
1. For all authors...
(a) the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Code will be
released upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "qRj2gV6q4L-", "reviewer": "Reviewer_TzZm", "summary": "This paper proposed a non-learning convolution operator that simply use magnitude of feature as cue to remove redundant background points in the sparse convolution operation. The authors validated the proposed operator by combining it to popular sparse convolutional network based object detection and achieve more than 50% reduction in GFLOPs without compromising the performance on benchmark datasets.", "questions": "Is it possible that the effectiveness of the propose method comes from the fact that background points has been filtered out by magnitude, thanks to the characteristics of the input point cloud feature representation used in the experiment?", "limitations": "I suggest the author to add visualization point cloud that belongs to important set to help with understanding. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simplicity of the proposed method\n\nWeaknesses\n- It is not clearly stated why magnitude is the key to discriminate foreground and background points. In the experimental section, the authors combines the operator with existing object detection networks CenterPoint and Voxel R-CNN. But the effectiveness of the methods seems to be more related to feature/representation of the input point-cloud and the authors didn’t discuss that part in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o2yFdphOjcS", "reviewer": "Reviewer_cuTj", "summary": "This paper studies the spatial redundancy in LiDAR-based 3D object detection. The authors propose spatial pruned sparse convolution (SPS-Conv) that skips computation for voxels with lower activation values. The authors have evaluated their proposed method on KITTI and nuScenes, achieving a 50% reduction in #MACs without loss of accuracy.", "questions": "I would love to see the answer to the following questions in the rebuttal:\n* What is the latency reduction achieved with the proposed SPS-Conv?\n* How does random voxel dropout work compared with magnitude-based pruning?\n* How does the proposed SPS-Conv work on Waymo with much denser point clouds?\n\nThe authors could refer to the previous section for more detailed comments.\n", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strengths]**\n\nThe paper is well-written and easy to follow. The proposed SPS-Conv is well-motivated, technically sound and achieves good empirical performance on two large-scale benchmarks. The authors have provided sufficient implementation details, which could facilitate the reproduction. The authors have also promised to release the code upon acceptance.\n\n**[Weaknesses]**\n\nThe authors use #GFLOPs as the primary efficiency metric throughout the paper. However, the reduction in #MACs does not necessarily translate into the measured speedup on the actual hardware (due to other costs such as data movement). I would highly recommend the authors report the measured latency on GPUs. As 3D object detectors contain more than sparse convolutions (since there are extra computations in the BEV decoder), it is completely acceptable to just present the latency of the sparse LiDAR encoder.\n\nThe inference engine has a substantial impact to the final inference speed. It would be great if the authors could try out the three available sparse convolution inference engines, including SpConv, TorchSparse and MinkowskiEngine, to see whether the proposed method can achieve consistent speedup with all of them.\n\nThe authors claim that the activation magnitude serves as a good indicator of selecting important voxels. It would be more interesting to see how random voxel dropout works (in additional to comparing with the inverse magnitude baseline).\n\nFinally, the authors have evaluated their proposed method on KITTI and nuScenes. It would be great if the authors could also evaluate their method on Waymo since the point cloud data on Waymo is much denser (and could be more redundant).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dbnbi7oRQq", "reviewer": "Reviewer_B4Lc", "summary": "This paper presents a method for dynamically determining which features on which to perform convolutions in a sparse convolutional network. In particular, the authors apply a sigmoid on top of the magnitude of each feature to determine a score, on which they threshold to determine active sites in each feature map. The motivation is that features with higher magnitude are more important to the network. In the forward pass, only features with a score above a set threshold have convolutions applied, while low score features are passed through via a skip connection. Features are also weighted by this score. Experiments are performed by replacing of a regular sparse convolutional network with the proposed dynamic sparse convolution blocks, where minimal performance regressions are seen despite significant reductions in GFLOPs. Ablations are also provided demonstrating that large score thresholds can be used with minimal impact on performance.", "questions": "Please see the strengths and weaknesses section. My main concern regards the ablations in Table 6, which would be alleviated with more explanations/exploration of the underlying assumption of this work.", "limitations": "No limitations section is provided. Some discussion of the validity of the base assumption would help future readers.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a relatively simple solution to the dynamic sparse convolution problem, given the intuition that higher feature norms correlate with more important features. The method can be dropped into any existing sparse convolution network, and is demonstrated to have significant reductions in computation cost. The sparsity can also be controlled at test time by tuning the score threshold. The method is well explained with nice figures outlining the pipeline, and the results are compelling, with good coverage across two datasets. \n\nFirstly, the intuition that features with higher norms are more important mostly makes sense, but some experimental validation would be very helpful. In particular, as the convolution operator applies the dot product between features, it's possible that a feature has relatively low magnitude compared to others, but high magnitude in a channel that also has high magnitude in the convolutional kernel. This may not be captured by the current approach which looks at the overall norm. \n\nA more concerning issue I see is the ablation in Table 6, where experiments are performed by only keeping the lowest scored features. While there is a significant regression in the medium cases, the difference for easy and hard cases is actually very minimal. The text describes this as a dramatic performance drop, but it's not convincing that this is actually the case. If the network is able to still achieve similar performance on hard examples when using what should be completely irrelevant features, what does that say about the foundational assumption that high norm features are the most important?\n\nAlso, in (4), the features are further weighted by the score for that feature. It's unclear why this is necessary, compared to only using the scores to select sites for convolutions. Does this weighting improve performance or provide some theoretical guarantees? Some ablations for this would be helpful. Otherwise, if the base assumption is not always correct, this weighting may serve to further remove the effect of some potentially important features.\n\nThis weighting also serves to provide a gradient on the feature norms (perhaps this is the intention?). This seems like it could be useful, but might also interfere with the final detection task. Some explanations/ablations would be useful.\n\nThe pruning ratio in the experiments is also not explained. It sounds like it is the top_k features, but could also be the score threshold. \n\nL159: should this be absolute magnitudes instead of absolute means?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aaJwubmzNIB", "reviewer": "Reviewer_aQxF", "summary": "In this paper, the authors study a previously under-explored direction in 3D deep learning acceleration: spatial redundancy. The major observation from the authors is that most points in the large scale LiDAR scans are background points. Thus, it is possible to reduce the computation on these points. The authors then design spatial pruned (submanifold/regular) sparseconv layers using magnitude-based importance masks. Experiment results show 2x FLOPs reduction without accuracy loss.", "questions": "Please address my comments in \"Weaknesses\".", "limitations": "See my comments in \"Weaknesses\". The major concern is on latency reduction and performance on larger scale datasets such as Waymo.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- As I have mentioned in the summary, this work explores spatial redundancy in 3D point clouds, a new direction in efficient 3D deep learning.\n- The paper is clearly written and most of the figures look good to me.\n- Experiment results are promising: 2x FLOPs reduction with no accuracy loss on nuScenes. Note that the baseline is a representative network, CenterPoint.\n\nWeaknesses:\n\n- The biggest question related to this work is whether such reduction in FLOPs can be translated to latency improvement. Especially when you consider latest sparse conv implementations in spconv 2.x which uses highly customized computation kernels, I'm a little bit concerned whether such pruning will lead to load imbalance between different CUDA threads and limit the speedup. Also, I'm curious what is the overhead introduced to calculate the importance masks.\n- It will be great if the authors can add more experiments on larger-scale datasets, such as Waymo. Waymo is much denser than nuScenes, it will be interesting to see the compression ratio and speedup. If the paper is eventually accepted, I'm also interested in seeing some application in sensor fusion methods on nuScenes.\n- [Minor] I would suggest the authors to make Section 3 more compact or move some details to the appendix as the target readers can be very familiar with sparse conv.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept