DBQ-SSD: Dynamic Ball Query for Efficient 3D Object Detection
Authors: Jinrong Yang
Lin Song
Songtao Liu
Weixin Mao
Zeming Li
Xiaoping Li
Hongbin Sun
Jian Sun
Nanning Zheng
Venue: ICLR 2023 poster (2023)

Abstract: Many point-based 3D detectors adopt point-feature sampling strategies to drop some points for efficient inference. These strategies are typically based on fixed and handcrafted rules, making it difficult to handle complicated scenes. Different from them, we propose a Dynamic Ball Query (DBQ) network to adaptively select a subset of input points according to the input features, and assign the feature transform with a suitable receptive field for each selected point. It can be embedded into some state-of-the-art 3D detectors and trained in an end-to-end manner, which significantly reduces the computational cost. Extensive experiments demonstrate that our method can reduce latency by 30%-100% on KITTI, Waymo, and ONCE datasets. Specifically, the inference speed of our detector can reach 162 FPS on KITTI scene, and 30 FPS on Waymo and ONCE scenes without performance degradation. Due to skipping the redundant points, some evaluation metrics show significant improvements.

Published as a conference paper at ICLR 2023
DBQ-SSD: DYNAMIC BALL QUERY FOR EFFICIENT 3D OBJECT DETECTION
Jinnong Yang! Lin Song?* Songtao Liu Zeming Li
Weixin Mao
Xiaoping Li
Hongbin Sun
Jian Sun
Nanning Zheng*
'Huazhong University of Science and Technology
"Tencent AI Lab yangjinrongghust. edu. cn, ronnysongetencent.. com
MMEGVII Technology
aXi' Jiaotong University
ABSTRACT
Many point-based 3D detectors adopt point-feature sampling strategies drop some points for efficient inference. These strategies are typically based on fixed and handerafted rules, making it difficult to handle complicated scenes. Different from them, we propose a Dynamic Ball Query (DBQ) network to adaptively select a subset of input points according to the input features, and assign the feature transform with a suitable receptive field for each selected point. It can be embedded into some state-of-the-ar 3D detectors and trained in an end-to-end manner, which significantly reduces the computational cost. Extensive experiments demonstrate that our method can increase the inference speed by 30%-100% on KITTI, Waymo, and ONCE datasets. Specifically, the inference speed of our detector reach 162 FPS on KITTI scene, and 30 FPS on Waymo and ONCE scenes without performance degradation. Due to skipping the redundant points, some evaluation metrics show significant improvements.
INTRODUCTION
3D object detection, as a fundamental task in the 3D scene, has made significant progress. It aims to recognize and localize objects from point clouds and paves the way for several real applications like autonomous driving (Geiger al., 2012), robotic gystem(Yang al., 2020b), and augmented reality (Park et al., 2008).
The structure of point clouds is sparse, unordered, and semantically deficient, making it difficult to encode point features like highly structured image features. To eliminate this barrier, voxel-based methods are proposed to organize the overall point cloud as neatly distributed voxels. Therefore, the naive 3D convolution or its efficient variant, i.e., 3D sparse convolution ( (Yan et al., 2018), 8), can be used to extract voxel features like image processing manner. Although the voxel-based methods bring convenience to the processing of point cloud, they are prone to drop detail information, making inescapable suffer from suboptimal performance. Another stream of methods are point-based methods (Yang et al., 2020b; Chen al., 2022; Zhang al., 2022) inspired from PointNet++ (Qi et al., 2017). They employ a series of operations, i.e., farthest point sample (FPS), query, and grouping, to directly extract informative features from the naive point cloud. However, the straightforward pipeline is cumbersome and costly. 3D-SSD (Yang et al., 2020b) first propose a single stage architecture, i..., using encoder-only architecture, which replaces the geometric Distance-based FPS (D-FPS) with the Feature similarity-based FPS (F-FPS) to recall more foreground point features and further discard feature propagation (FP) layers for reducing latency. Although eliminating the overhead of FP layer, F-FPS operations still occupy vast latency. IA-SSD (Zhang et al., 2022) further proposes a contextual centroid prediction module to replace F-FPS, which directly predicts the classification scores of each point and adopts efficient top-k operation to further recall more foreground, and further cut computation overhead.
The current advanced designs mainly credit to efficient foreground recall, but they may still have redundancy the other parts like the background points or the network structure. In this paper, we
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2023
conduct several empirical analyses of IA-SSD (Zhang al., 2022) two representative benchmarks, i.e. KITTI (Geiger et al., 2012) and Waymo (Sun et al., 2020), to uncover the full picture of its inference speed bottleneck. We first calculate the latency distribution of all detector modules to figure out which parts are the main speed bottlenecks. Then we count the ratio of background points and the scale distribution of all objects further explore potential redundancy clues over the cumbersome modules. Our study reveals three valuable points: (1) MLP network occupies over half latency. (2) Tremendous spatial redundancy exists in background point features that appear in each stage of the detector. (3) The size of each object is varying, making it unusable to align each receptive field of the conventional multi-scale grouping (MSG) and suffering from branch redundancy in MSG.
The above valuable finding motivates us to further build a more efficient detector with higher speed by reducing the redundant background points and blocking useless MSG branches. As shown in Fig. 2, we propose Dynamic Ball Query (DBQ) to replace the vanilla ball querying module, where the vanilla ball querying means the sampling technique proposed by PointNet++ (Qi et al., 2017). It dynamically activates useful and compact point features, and blocks redundant background point features for each branch of MSG. For each point feature sampled by FPS or top-k classification score, we design a dynamic query multiplexer to determine which branch to go through. Specifically, apply a light- MLP network for point features to predict N masks corresponding to N branches of MSG, where the value of the mask is (0, 1} corresponding blocking and activating states. The overall dynamic router procedure is data-driven so that the point features are adaptively activated or blocked with a suitable combination among all receptive fields of MSG. Ultimately, we introduce a resource budget loss for DBQ to learn a trade-off between latency and performance.
To verify the efficiency of our method, we conduct extensive experiments on three typical datasets, Le., KITTI (Geiger et al., 2012), Waymo (Sun et al., 2020), ONCE (Mao et al.. 2021b). Our Dynamic Ball Query enables the 3D detector to cut the latency from 9.85 ms (102 FPS) to 6.172 ms (162 FPS) on KITTI scene and speed up the inference speed from 20 FPS to 30 FPS on Waymo and ONCE scene while keeping the comparable performance. In particular, some evaluation metrics show significant improvements.
RELATED WORK
2.1
3D DETECTORS
The task of 3D detection is predict 3D bounding boxes and class labels for each object in a point cloud scene. The detection algorithms can be split into voxel-based (Zhou & Tuzel, 2018; Yan al., 2018; Lang et al., 2019; He et al., 2020) and point-based (Shi al., 2019; Yang et al., 2020b; Chen et al., 2022: Zhang et al., 2022) methods. Voxel- based methods convert the point cloud into regular voxels or pillars, making natural to apply 3D) convolution or its sparse variant (Yan et al., 2018) for feature extraction. This regular partition method may lead to detailed information lost, so point-based methods are proposed to directly process vanilla point cloud. Inspired by PointNet++ (Qi et al., 2017) and Faster R-CNN (Ren et al., 2015), PointRCNN (Shi et al., 2019) employs SA and FP layers to extract feature for each point and designs a region proposal network (RPN) to produce proposals, and utilizes an extra stage of the module to predict bounding boxes and class labels. In addition, PV-RCNN (Shi et al., 2020a) integrates voxel and point features to the RPN for generating higher-quality proposals. Pyramid R-CNN (Mao al., 2021a) la) introduces a pyramid Rol head with learnable radii to boost accuracy at the expense of latency overhead. In contrast, our method aims to achieve more efficient inference by adaptively selecting the network branches for each input point. Other efforts are similar to single-stage 2D detectors (Lin et al., 2017: Tian et al., 2019; Wang et al., 2021; Song et al., 2019b: Zhang et al., 2019). 3DSSD (Yang et al., 2020b) and IA-SSD (Zhang et al., 2022) discard the region proposal network and use encoder-only architecture to localize 3D objects. Our work focus on dynamically dropping the redundant background point features for angle-stage point-based detector, which is rarely researched in previous works. Our method endows the detector with super inference speed.
2.2 EFFICIENT 3D POINT-BASEE DETECTORS
The point-based based methods need to process large-scale vanilla point features, which requires to build cumbersome models and suffers expensive computation costs. Therefore, several works (Yang et al.,
2

---

Published as a conference paper at ICLR 2023
XITTI
7100
Warse
PNELP
600
KITTIM
Inpat
18
200
Nays2
Stage!
Stope
Ban
(1415))
042g)
0x44
[ddel
Reacher
Oum
Boadee
Oer
(a) Latency
(b) Background ratio
(c)
Scale distribution
Figure I: Statistics of latency, background ratio, and size distribution on both KITTI val (Geiger al., 2012) and Waymo val (Sun al., 2020) sets. (a) reveals that the MLP network occupies the largest latency. "Q & G" means query and grouping operation. (b) reflects that redundant background points significantly dominate the input points of each stage. (c) means the distribution on varying object sizes (measuring in Vvolume, where volume the volume of ground truth).
2020b; Chen et al., 2022; Zhang et al., 2022) aim at designing a lightweight and effective point-based detector. 3DSSD (Yang et al., 2020b) proposes feature-based fastest point sampling (F-FPS) strategy by measuring the similarity of point features to replace the geometric distance of D-FPS (Qi et al., 2017). The policy makes reasonable to discard FP layers since F-FPS can recall more foreground points instead of compensating for foreground information by cumbersome FP layers. Even so, the F-FPS operation suffers from high complexity bottleneck. IA-SSI) introduces a contextual centroid prediction module for replacing the F-FPS operation to efficiently recall more foreground points. It predicts the classification score for each point feature and applics the efficient top-k operation to all points ordered by category independent scores. Instead of focusing on the foreground part by efficient sampling policies, we explore the spatial redundancy of background points. We introduce a dynamic network mechanism adaptively reduce useless background points in data-dependent regime.
2.3 DYNAMIC NETWORK
Dynamic networks target adaptively change the structure of networks and parameters in data- driven manner. In the field of model automation designing, dynamic network mechanism is used to drop blocks (Wu al., 2018; Huang et al., 2017; Mullapudi et al., 2018; Wang et al., 2018), pruning channels (Lin et al., 2018; You et al., 2019; Jiang et al., 2018), adjusting layer-level scales (Li et al., 2020b; Yang et al., 2020a) and changing spatial typologies (Song et al., 2019a; 2020b). DRNet (Li et al., 2020b) aims at learning an automatic scale transformation for a feature pyramid network in the semantic segmentation scene. Switch Transformer (Fedus et al., 2021) employs the Mixture of Experts (MoE) (Shazeer et al., 2017) model to choose different parameters for each input data. Deformable convolution (Dai et al., 2017) learns a convolution of arbitrary shapes by predicting an offset for each parameter. Dynamic Grained Encoder (Song et al., 2021) adopts dynamic gate mechanism to select the spatial granularity of input query in the transformer network. Dynamic Convolution (Verelst & Tuytelaars, 2020), Dynamic Head (Song et al., 2020a) focus on learning a sparse mask for network to discard redundant image features. In this paper, we focus on 3D detection on point cloud scenes and aim designing a point-wise dynamic network drop redundant background points.
ANALYSIS AND MOTIVATION
To explore what hampers the higher speed for 3D detection task, we conduct several experiments on both KITTI val and Waymo val sets with the state-of-the-art point-based 3D-detector (Zhang et al., 2022). To establish strong baseline, we split the point cloud into four parallel parts to speed up the first FPS operation. We first measure the latency each detector module. As shown in Fig. 1(a), the MLP network occupies the major part of the overall time overhead, i.e., 6.44 ms (65.4%) and 28.56 ms (56.5% on KITTI and Waymo scenes, respectively. Therefore, optimizing the cumbersome and costly MLP network is a top priority for building an efficient detector.
Decreasing the input scale of points network parameters is natural and empirical manner for reducing the latency but the policies are prone to damage the performance. To avoid both notorious drawbacks, attempt to analyze the redundancy of background points in each backbone stage. As
3

---

Published as a conference paper at ICLR 2023
Query Muttipeesrr
Group
9
Vanilla Ball Query
MLP Max Pool Aggregation
Remap
Quantization
Query Mulliplexes
Sparse Select
Group K
Vanilla Ball Query
MLP Mux Pool Aggregation
Remap
Query
Query (gr(C+3)
Gating Mask
(Qx(C+3)
Ng
Figure 2: The pipeline of dynamic ball query in abstraction layer. 'NS' indicates the nearest sampling, which samples the query features from the input features. The query multiplexer generates gating masks to adaptively select subset of imput queries for each group. The remap operator is used to map the sparse features to the dense form.
shown in Fig. 1(b), it indicates that the number of background points (point features) dominates the proportion over 70% at any network level. The phenomenon reveals that significant redundancy exists in background points which may be discarded for speeding up the detection procedure.
Going one step further, we point out that the conventional multi-scale grouping (MSG) operation (Qi et al., 2017) of the set abstraction (SA) layer may be also redundant. As shown in Fig. 1(c), it reflects that the size of each object varies in either KITTI or Waymo scenes. Therefore, the given receptive field of MSG may not entirely align with the size of objects. In this regime, some grouping branches of MSG are useless. The valuable observation motivates us to choose optimal grouping branches, which can further make the detector efficient.
4 DYNAMIC BALL QUERYING
Our DBQ-SSD framework is established on the efficient LA-SSD framework, which adopts set abstraction (SA) layers to encode point features. To achieve a better halance between effectiveness and efficiency, introduce the dynamic network mechanism into the IA-SSI) framework. Specifically, we propose Dynamic Ball Querying (DBQ) to replace the vanilla ball querying in each set of abstraction layer, which is shown in Fig. 2. It is able adaptively select subset of input points as queries and extract local features in suitable spatial radius. Dynamic ball querying is generic module which can be easily embedded into the encoder blocks in mainstream point-basee based 3D detection frameworks.
Given an input point sequence X Nx3 with the corresponding features Xy R. NxC, N denotes the length of sequence and C' indicates the number of feature channels. Besides, the coordinate sequence of input queries is denoted as qe N33 where N, the number of queries. For efficiency, most previous point-based methods (Yang et al., 2020b; Zhang et al., 2022) use Farthest Point Sampling (FPS) or its variants (e.g., dividing point cloud by multiple parallel parts to reduce computational complexity) to generate the query points. For each query, the vanilla ball querying samples a predefined number of local points within a specific spatial radius. Since different object instances require different receptive fields, in a set abstraction layer, the previous 3D detectors (Yang et al., 2020b; Zhang et al., 2022) typically adopt multiple groups of vanilla ball querying with different radii increase feature diversity.
Specifically, for one set abstraction layer. we define the set of predefined radii as R = (rx) } and the number of sampling points & (ox) } K where K indicates the number of groups. Based on these, we establish a set of K vanilla ball querying blocks. As shown in Fig. 2, our dynamic ball querying vanilla ball querying blocks. Similar to the gate-based dynamic networks (Song et al., 2021; Wang et al., 2018; Li et al., 2020b), the query multiplexer adopts a fine-gra fine-grained routing process to select suitable combination of vanilla ball querying blocks for each query.
made up of query multiplexer and
4.1 INFERENCE
For an input query E 21, ..,,..,,,, with coordinate (i), we first obtain the corresponding query feature according its coordinate for the query multiplexer. The query feature can be used as the
4

---

Published as a conference paper at ICLR 2023
basis for dynamic decisions. Specifically, we adopt the nearest sampling technique to obtain its representation from the features of imput points. Albeit the sampling process with more sample points, e.g., top-k sampling, and ball gathering, can lead to slight performance gain, it causes significant decrease in efficiency. The top-k sampling indicates choosing a set of points with the smallest distance under specific metric. Additionally, ball gathering means sampling and concatenating features according to the coordinates of points. The previous variants of ball querying employ heuristic and hand-designed rules. Different from them, the routing process of our query multiplexer is performed in data-dependent way. To achieve it, we aggregate input features for each query from the nearest input point and predict the gating logits for each query by a linear projection:
h ()) = xr(g min qq X W + b RIXK
where W E RCCK and b E R x k denotes the weight and bias, respectively. Moreover, the binary gating masks for the i-th query and the k-th group are generated by quantizing the gating logits:
1, if h(i,k) 20 0
m(i, k) st)p(h(i, where step(h(i, =
(2)
0, if otherwise
The gating masks control which ball querying group is enabled, i.e., the group with a positive mask value is enabled and vice versa. Based this, we can adaptively reduce the number of queries and obtain the coordinates of sparse queries for each group:
(k( = {ee (i) m(i, k) * O, Vi E 1,2, NaJ. Ak E (1.2,.., K},
(3)
Furthermore, the sparse coordinates are used to guide the vanilla ball querying with corresponding settings to generate the sparse query features, Following the conventional protocols in the PointNet-
like methods, the sparse query features are then transformed by the predefined MLP layers and max pooling operator:
2(k) = MaxPool(MLP. MaxxPoolMLLP (VanillaBalllQuerryy... Xe, Xr, Yk,, (u)));
To fuse the sparse transformed features in different groups, we remap them into the dense form according to the gating mask. The remap operator is similar to the unpooling process, which projects each enabled feature to the original position and fills zero to the disabled positions. The output feature of the SA layer are then blended by summation:
Yr
Σ
z(k), where z(k) = Aggreaation,(RRmmp([([] (Remap(2(k); h)):
(5)
where the Aggregation; is a linear layer, which is used to transform the features in different groups into the same dimension.
4.2 TRAINING
Since the sparse selection in Eq. 3 is non non-diffcrentiable, it is nontrivial for the dynamic ball querying enable fully end-to-end training. To achieve it, we replace the determined decisions in Eq. 3 with stochastic sampling process. Concretely, we consider the gating logits unnormalized log probabilities under the Bernoulli distribution. To this end, with noise samples g and g' drawn from standard Gumbel distribution (Gumbel, 1954), a discrete gating mask can be yielded:
m (i, k) = step(h(i, k) + 9 - g')),, where g.g' ~ Gumbel(O, 1).
(6)
To enable end-to-end training, motivated by the previous dynamic networks, we use the Gumbel- Sigmoid technique (Gumbel, 1954) to give a differentiable approximation for the Eq. 6 by replacing the hard step function with the soft sigmoid function. The likelihood of the i-th query in k-th group being selected is:
exp((h(i, + 9)/T)
(i, k) exp((h(i, k) + + - exp(g'/7) E |0, 1],
where 7 is the temperature coefficient. In the training phase, we use the Eq. 6 as the gating mask to select the sparse queries and employ straight- through estimator (Bengio et al., 2013; Verelst & Tuytelaars, 2020) to obtain the gradients of gating logits:
2(1,, k)
forward
yr(1)= =
(8)
F(i, k) k(i
backward
5

---

Published as a conference paper at ICLR 2023
4.3 LATENCY CONSTRAINT
Without latency constraint, dynamic ball querying typically enables more queries for each group to obtain high accuracy. To achieve better balance between effectiveness and efficiency, we introduce the latency constraint as training target to reduce the inference time. Different from the computational complexity employed in many previous dynamic networks (Song et al., 2021; Wang al., 2018; Li et al., 2020b), the latency end, we first establish latency map for each group in each SA layer, which records the latency with regard the number of queries. Based this, with dynamic ball querying:
represent the actual runtime in specific devices. To this
calculate the latency ratio of all the SA layers
E.EN VIA m (i, EE (
where I indicates the latency map of k-th group in (-th layer. Finally, the latency is constrained by using the Euclidean distance design budget loss, thus the total loss
L = Lrasks + Audgeet where Luudget = I 7/-
(10)
For simplicity, the objective latency budget y is set to 0 in all experiments. The hyper-parameter is used to scale the budget loss. In addition, if the input is batched format, v needs to be averaged along the batch dimension estimate the average overhead of the network.
5
EXPERIMENT
5.1 SETTING
Dataset We evaluate our detector on two representative datasets: KITTI dataset (Geiger et al., 2012) and Waymo dataset (Sun et al., 2020). KITTI dataset includes 7,481 training point clouds/images and 7,518 test point clouds/images. The KITTI scene contains three classes, i.e., Car, Pedestrian, and Cyclist. Waymo scene contains 798 training, 202 validation, and 150 testing sequences with three classes of Vehicle, Pedestrian, and Cyclist. Each sequence includes nearly 200 frames with 360-degree lidar point cloud.
Evaluation metrics. For KITTI scene, we report the performance of all classes by measuring the average precision (AP) metric. Follow most of state-of-the-art methods, adopt 0.7, 0.5, and 0.5 of the loU thresholds for Car, Pedestrian, and Cyclist, respectively. In addition, three levels of difficulty ("Easy", "Moderate". and "Hard") are also reported. To evaluate Waymo, official metrics, Average Precision (AP) and Average Precision weighted by Heading (APH), and report the performance LEVEL (LI)) and LEVEL 2 (1.2) difficulty levels.
the
Implementation Details Following the stream of single-stage point-based methods, use encoder- only architecture like (Yang et al., 2020b; Chen et al., 2022: Zhang et al., 2022). Specially, we split the point features into four parallel fan parts speed up D-FPS of the first sampling layer, which will be acted as the "Efficient Bascline". Other sampling layers follow the default setting of (Zhang al., 2022). We employ two groups of each MSG with different radius ((0.2, 0.8], [0.8. 1.6), [1.6, 4.8], [4.8, 6.4)) to aggregate point features. We set the temperature 1 for all the experiments. All experiments are implemented by OpenPCDet framework.
5.2 EVALUATION ON KITTI DATASET
We randomly sample 16,384 points from the overall point cloud per single view frame. We train our model by ADAM (Kingma Ba, 2014) optimizer with onecycle learning strategy (Smith & Topin, 2019). The batch size is set to 16 with 8 GPUs. The initial learning rate is 0.01 and is decayed by 0.1 35 and 45 epochs.

---

Published as a conference paper at ICLR 2023
81
AP (3D moderate Can)
3D-5DD
2-0.0
79
APPRA'2
Baoklin
875 2e0
Laancy
0.4
77
X
227 8PS
Query & Grouping MLP Network
10
30
50
70
90
110
130
150
170 190 210 230
1
Speed (FPS)
Layer index set abstraction
(a) Performance & Latency
(b) Layer (A = 0.1)
Small
Large
Small & Large
0.9
propetion
Activate
0.7
0.6
0.4
0.5
0.4
Layer set abstraction
Layer index
(c) Activation (A 0.1)
(d) Group (A 0.1)
Figure 3: Illustration of the effects on Dynamic Ball Query. All experiments are evaluated on KITTI val set. À is the scale parameter resource budget loss Eq. 10. Latency here evaluated by single RTX2080Ti GPU with batch size of 16. (a) reports the comparison on both accuracies of Car class and overall latency distribution. (b) indicates the latency reduction of query grouping operation and MLP network in different SA layers. (c) reflects the activation distribution of point features in different SA layers. (d) shows the proportion of point features go through different groups of MSG. "Small" and "Large" means activating on group branches with small and large radii respectively. "Kill" represents blocking all groups, while "Small Large" means going through all scales of groups.
Table Performance of dynamic gating with different routing manners on KITTI val set. The scale parameter À is set to 0.1. "Layer" indicates controlling an entire SA layer. "Share" means whether to share masks to all groups. "Point" indicates using point- wise routing instead of layer-wise routing.
Dynamic
Layer
Routing
Share
Easy
3D Car (loU=0.7)
Easy
Podestrian (loU=D.5)
Hard
Easy
3D Cyclist (loU=0.5)
Modentte
Latency (TPS)
x
88.89
79.17
77.96
61.01
58.11
52.62
84.83
69.29
65.72
102
89.19
79.33
79.31
61.32
58.60
52.75
85.81
70.77
66.04
120
87.28
77.
76.29
60.43
56.79
51.45
84.83
68.88
65.12
168
Dynamic Static To verify the efficiency of our method, we first report the performance and latency on KITTI scene. As shown in Fig. 3(a), our detector achieves super speed while maintaining comparable performance with other detectors. When À is set 0.1, the performance surpasses the efficient baseline. Therefore, we set the scale parameter to 0.1 for all experiments by default. As increasing the supervision of resource budget loss, the speed is further improved to 223 FPS. The impressive results show that our method endows point-based 3D detector with efficient detection capability. Going one step further, we report the reduction of latency on two revenue modules, i.e., query & grouping operation, and MLP network. As shown in Fig. 3(b), method cut the considerable overhead of both in different SA layers. It verifies that adaptively turning off useless points by DBQ can speed up the computation of both modules.
7

---

Published as a conference paper at ICLR 2023
Origin point cloud
Smsall (1)
Smull (2)
Small (3)
Small (4)
Spoeess
(l)
Large (2)
Large (3)
Large (4)
Visualization results on KITTI val set. The 3D boxes in the figures are the prediction boxes. Green, cyan, and yellow represent Car, Pedestrian, and Cyclist. Red and white points represent activation and blocking points, respectively. "Small" and "Large" means the scale of group in MSG, and the digital in parentheses is the index of SA layer.
Figure 4:
Activation and Blocking Points To explore which points are activated or blocked, we conduct quantitative and qualitative experiments using visualization and statistics respectively. Fig. 3(c) counts the activation ratios of point features in different layers. It indicates that a considerable large part of points is discarded. Next, we dip into visualization results figure out the reason for the quantitative results. As shown in Fig. 4, the deeper the network goes, the blocking point ratio larger. This is because it learns rich semantic information to judge useful foreground and redundant background points when going deeper into the network. As for the early layer, high-level semantic knowledge is difficult to extract. Therefore, the model is hard to make decisions to block out background points. In general, it reflects clearly that the most activating points (red points) in the deeper layer are distributed around objects, while other points away from the objects are blocked. reveals that our method discards most redundant background points which may be useless for the localization and classification of objects, making reasonable for the detector to speed up inference. The remaining foreground points and surrounding points are used to support the structure of objects and enrich context information. Therefore, our method does not damage the performance of detection. It echoes our analysis in Sec. 3.
Routing Manner and Branch Redundancy Tab. I compares the effectiveness between layer-wise and point-wise routing manner. During inference, the layer-wise routing manner only speeds up by 18 FPS with small performance gains, while our point-wise manner not only achieves significant performance improvement but also reduces considerable latency. In addition, the contrast on whether to predict split mask or share identity mask for each group of MSG indicates that the former is the optimal policy. Generating two masks for different scales of group allow each point to reach an optimal combination of the receptive field. Corresponding statistical and visualization results can be seen in Fig. 3(d) and Fig. 4, which reveals that more points go through both groups in early layers while most points are only keen on single branch or even be killed. The phenomenon agrees well with the empirical analysis in Sec. 3.
As illustrated in Tab. 2, our method outperforms the efficient baseline on all cate- gories, while achieving higher inference speed (162 vs 102). It verifies that our method can not only drop redundant points for speeding up inference, but extract more useful information for localization and classification. By comparing with other state-of-the-an methods, our detector outperforms their speed with a large margin while maintains comparable accuracy.
Main Results
5.3 EVALUATION ON WAYMO DATASET
To verify the generalization of our method, we further evaluate the performance Waymo (Sun et al., 2020) dataset. Because Waymo scene is made up of the 360-degree point cloud whose scale is larger than KITTI scene, we increase the input number of points by sampling from 16,384 to 65,536. The batch size is set to for each GPU. We train 30 epochs with 8 GPUs. Other settings are the same as the experiments of KITTI scene.
8

---

Published as a conference paper at ICLR 2023
Table 2: Comparison with the state-of-the-art methods on the KITTBI test set. Bold font is used to indicate the best performance. The speed is tested on single GPU with with batch size of 16 and measured by FPS.
3D Car (loU=0.7)
3D Ped. (loU=0.5)
3D Cye. (IoU=0.5)
Method
Type
Speed
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
Voxel-based Methods
VoxelNet (Zhou & Tuzel, 2018)
I-stage
77.47
65.11
57.73
39.48
33.69
31.5
61.22
48.36
44.37
4.5
SECOND (Yan et 2018)
1-stage
84.65
75.96
68.71
45,31
35.52
33.14
75.83
60.82
53.67
20
PointPillars (Lang et al., 2019)
I-slage
82.58
74.31
68.99
51.45
41.92
38.89
77.10
58.65
51.92
42.4
3D toUl Loss (Zhou et al., 2019)
I-stage
86.16
76.50
71.39
12.5
Associate-3Ddet (Du ol., 2020)
I-stage
85.99
77.40
70.53
20
SA-SSD He et al. (2020)
I-stage
88.75
79.79
74.16
25
CIA SSD (Zheng et al., 2020)
I-slage
89.59
80.28
72.87
32
TANet Liu et al. (2020)
2-stage
84.39
75.94
68. 82
53.72
44.34
40.49
75.70
59.44
52.53
28.5
Part-A
2-stage
87.81
78.49
73.51
53
43.35
40.06
79.17
63.52
56.93
12.5
Point-Vovs Methods
Fast Point R-CNN Chen al. (2019)
2-stage
89.29
77.40
70.24
16.7
STD (Yang et al., 2019)
2-stage
87.95
79.71
75.09
53,29
42.47
38.35
78.69
61.59
55,30
12.5
PVERONN (Shi al., 2020a)
stage
90.25
81.43
76.82
52.17
40.29
57.65
12.5
VIC-Net (Jiang et al., 2021)
I-stage
88.25
80.61
75.83
43,82
37.18
35.35
78.29
63.65
57.27
17
HVPR (Noh al., 2021)
stage
86.38
77.92
73.04
52.47
40.64
36.1
Point-based Methods
PoinRCNN (Shi al., 2019)
2-stage
86.96
75.64
70:70
47.98
39.37
36.01
74.96
58.82
52.53
10
3D loU-Net (Li al., 2020a)
2-stage
87.96
79.03
72.78
10
Point-GNN (Shi & Rajkumar, 2020)
I-stage
88.33
79.47
72.29
51.92
43.77
40.14
78.60
63.48
57.08
1.6
3DSSD (Yang et al., 20200)
stage
88.36
79.57
74.55
54.64
44,27
40.23
82.48
64.10
56.90
25
IA-SSD (Zhang al., 2022)
I-stage
88.34
80.13
75.04
46.51
39.03
60
78.35
61.94
55.70
83
IA SSD (Reproduced)
stage
87.67
79.40
74.22
46.16
38.29
35.61
78.26
61.53
55.48
83
DBQ-SSD
I-stage
87.93
79.39
74.40
47.59
38.08
35.61
78.18
62.80
55.70
162
Table 3: Comparison with the state-of-the-art methods on the Waymo val set. The bold font is used to indicate best performance. The speed tested on a single GPU with batch size of 16 and measured by FPS.
Method
Vahicle (LEVEL mAPH
Vehicl mAP
(LEVELL
IAAPH
mAP
(LEVEL
MAPH
Pod. (LEVEL
mAP
MAPS
Cyc. (LEVEL
mAPH
Cye. (LEVELL mAP
mAPH
speed
PoiniPallars (Lang et oll, 2019)
60.67
59:79
52.78
52.01
43.49
23.51
37:32
20.17
35.94
28:34
34.60
27.29
SECOND (Yan et al., 2018)
68.03
67.44
59.57
39.04
61.14
50.33
53.00
43.56
54.66
53.31
52.67
$1.37
A² (Shi et al.,, 2020b) 20200)
21.82
71.29
66.33
63.82
63.15
54.96
$4.24
47.11
65.23
63.92
62.61
61.35
PV-RCN (Shi et al.. 2020g)
74.06
73.38
64.99
64.38
62.66
52.68
53.50
45.14
63.32
61.71
60:72
59.18
IA-SSD (2hing al. 2022)
70,53
09.67
61.55
00.80
69.38
58.47
60.30
50.75
57.67
65.30
64.95
62,71
14
Elicient Baseline
21.15
70.30
62.49
61.73
68.38
58.21
75
50.80
68.64
66.23
66.09
63.78
20
DBQ SSD (1=0.1)
69.82
68.39
63.99
64.22
30
DRQ-5SD
71.08
27
As shown in Tab 3, we report the performance and inference speed of our DBQ-SSD. As carrying out suitable supervision (i.e., A 0.05) for the detector, it achieves impressive performance in all classes. Especially, some categories achieve state-of-the-ait accuracy in evaluation metrics or levels of difficulty. Meanwhile, it also improves the inference speed of the efficient baseline from 20 FPS to 27 FPS. The results show that adaptively blocking redundant point features and activating high highquuaitty point features are the key to endow our detector with efficient performance. When conducting a larger scale of supervision for DBQ-SSD, it is capable of detecting objects with real-time speed (30 FPS) and achieves comparable accuracy. It can provide flexible configuration for practical applications to realize the best trade-off between accuracy and overhead cost.
6 CONCLUSION
In this paper, we point out the existing spatial redundancy on background points and useless receptive field groups in MSG. This redundancy impedes the inference efficiency improvement of basnt-based 3D detectors. To eliminate the dilemma, we propose a dynamic ball query, which can dynamically generate gate masks for each group of MSG to process useful points and block redundant background points. The extensive experiments demonstrate our analysis and show the effectiveness of our method. In short, we launch new view to focus redundant background points instead of the limited foreground part, which further deepens the understanding of the sparsity of point cloud. We hope this work can shed the light to the research of efficient point-based models and inspire future works.

---

Published as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through
stochastic neurons for conditional computation. 2013. 5
Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics-augmentee abstraction for point-based 3d object detection. arXiv preprini arXiv:2201.01977, 2022. 1, 1,2,3,1 3,6
Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast point r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9775-9784, 2019. 15
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks, In IEEE International Conference on Computer Vision, 2017. 3
Liang Du, Xiaoqing Ye, Xiao Tan, Jianfeng Feng, Zhenbo Xu, Errui Ding, and Shilei Wen. Associate- 3ddet: Perceptuullto--ooneepuall association for 3d point cloud object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13329-13338, 2020. 9, 15
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021. 3
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp- 2354-3361. IEEE, 2012. 1, 2, 3, 6
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Shenp Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1187-118822 2020. 2, 9, 15
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint 1703 2017. 3
Jianwen Jiang, Yu Cao, Lin Song, Shiwei Zhang, Yunkai Li, Ziyao Xu, Qian Wu, Chuang Gan, Chi Zhang, and Gang Yu. Human centric spatio-temporal action localization. In ActivityNet Workshop on CVPR, 2018.
Tianyuan Jiang, Nan Song, Huanyu Liu, Ruihao Yin, Ye Gong, and Jian Yao. Vic-net: Voxelization information compensation network for point cloud 3d object detection. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13408-13414. IEEE, 2021. 9, 15
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14X2::1412.6980, 2014.
Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12697-12705, 2019. 2, 9. 15
Jiale Li, Shujie Luo, Ziqi Zhu, Hang Dai, Andrey S Krylov, Yong Ding, and Ling Shao. 3d iou-net: Iou guided 3d object detector for point clouds. arXiv preprint XXXx:20044.0462 2020a. 9, 15
Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In IEEE Conference Computer Vision and Pattern Recognition, 2020b. 3,
Shaohui Lin, Rongrong Ji, Yuchao Li, Yongjian Wu, Feiyue Huang, and Baochang Zhang. Accelerat- ing convolutional networks via global & dynamic filter pruning. In IJCAI, 2018.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017. 2
10

---

Published as a conference paper at ICLR 2023
Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11677-11684, 2020. 9, 15
Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid r-cnn: Towards better performance and adaptability for 3d object detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2723-2732, 2021a.
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2106. 1037, 2021b.
Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Special- ized dynamic architectures for efficient inference. In JEEE Conference on Computer Vision and Pattern Recognition, 2018. 3
Jongyoun Noh, Sanghoon Lee, and Bumsub Ham. Hypr: Hybrid voxel-point representation for single-stage 3d object detection. In Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14605-14614, 2021." 9
Youngmin Park, Vincent Lepetit, and Woontack Woo. Multiple 3d object tracking for augmented reality. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 117-120. IEEE, 2008. 1
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017. 1, 3,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 2
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-expee layer. arXiv preprint /rXiv: .065388, 2017.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 770-779, 2019. 2, 15
Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng
Li. Pv-rean: Point feature set abstraction for 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. (0529-10538, 2020a. 2, 9, 15
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions pattern analysis and machine intelligence, 2348):2647-2666 2020b. 9, 15
Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1711-1719, 2020. 9, 15
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for mulli-domar operations applications, volume 1 1006, pp. 1100612. International Society for Optics and Photonics, 2019. 6
Lin Song, Yanwei Li. Zeming Li. Gang Yu, Hongbin Sun, Jian Sun, and Nanning Zheng. Learnable tree filter for ceunctur-ppreeeviin feature transform. In Advances in Neural Information Processing Systems, 2019a. 3
Lin Song, Shiwei Zhang, Gang Yu, and Hongbin Sun. Tacnet: Transition-awee aware context network for spatio-temporal action detection. In JEEE Conference on Computer Vision and Pattern Recognition, 2019b.
11

---

Published as a conference paper at ICLR 2023
Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. Fine-grained dynamic head for object detection. Advances in Neural Information Processing Systems, 2020a. 3
Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, and Nanning Zheng. Rethinking learnable tree filter for generic feature transform. Advances in Neural Information Processing Systems, 2020b.
Lin Song, Songyang Zhang, Songtao Liu, Zeming Li, Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. Advances in Neural Information Processing Systems, 34, 2021. 3, 4, 6
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2446-2454, 2020. 2, 6,
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9627-9636, 2019. 2
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 3,5 5
Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. End-to-end object detection with fully convolutional network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 2
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. European Conference on Computer Vision, 2018. 3,
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In IEEE Conference
Computer Vision and Puttern Recognition, 2018. 3
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18
(10):3337, 2018 1, 9, 15
Le Yang. Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In IEEE Conference Computer Vision and Pattern Recognition, 2020a.
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Spanse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1951-1960, 2019. 9, 15
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11040-11048, 2020b. 1, 2, 3, 4, 6. 15
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint arXiv:1909.00 08174, 2019. 3
Shiwei Zhang, Lin Song. Changxin Gao, and Nong Sang. Ginet: Global local network for weakly supervised action localization. IEEE Transactions on Multimedia, 2019.
Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-basee based detectors for 3d lidar point clouds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022. 1, 3, 4, 6, 9, 15
Wu Zheng, Weiliang Tang, Sijin Chen, Li Jiang, and Chi- ChigWing Fu. Cia-ssd: Confident iou-aware single-stage object detector from point cloud. arXiv preprint arXiv:2012.030/5, 2020. 9, 15
12

---

Published as a conference paper at ICLR 2023
Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. lou loss for 2d/3d object detection. In 2019 Interaational Conference on 3D Vision (3DV), pp. 85 94. IEEE, 2019.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490-4499,
2018. 2, 15
13

---

Published as a conference paper at ICLR 2023
A LIMITATION AND FUTURE WORK
With increasing the supervision on resource budget (increasing the value of X.), the performance will decrease accordingly. We suspect that dropping too many point clouds may eliminate the part of the useful point cloud. Therefore, this paper targets to achieve a hetter trade-off between accuracy and inference speed, which maintaining or even achieving gain in accuracy, and significantly speeding up inference. In this paper, we can not specifying what point to drop but our detector equips strong ability to eliminate redundancy. Therefore, we look forward to inspire future works for focusing on dropping more redundant point cloud without performance degradation.
Table Illustration of the architecture of DBQ-SSD, npoint denotes the number of sampled points, [radii] denotes the grouping radii, [nquery] denotes the number of grouping points, [dimension] denotes the feature dimensions. Aggregation indicates aggregation operation and MLP size.
Module
npoint
[radii]
[nqueryl
[dimension]
Aggregation
SA layer
409
[0.2, 0.8]
[16, 32]
[[16, 16, 32), [32. 32, 64]]
MLP 64)
(64 64)
SA layer
1024
[0.8, 1.6]
[16.
[[64, 64, 128). [64, 96, 128]]
MLP (128 128) MLP (128 128)
SA layer
512
[.8)
[16,
[[128, 128, 256]. [128, 256, 256]]
MLP (256 - 256) MLP (256 - 256)
Vote layer
256
MLP (256 -> 128 -
SA layer
256
[4.8, 6.4]
(2)
[1256,26,,122., (256,512.1024]
MLP (512-+ 512) MLP (1024 512)
B DETAILED DETECTOR ARCHITECTURE
We report the detailed architecture of our DBQ-SSD, DBQ-SSD is single-stage point-based detector that consists of three Set Abstraction (SA) layers for extracting point features, and one SA layer for aggregating centroid-based instances. Each SA layer has two different groups for the spherical neighbor query. In addition, vote layer is used generate candidate points. A light-weight head is attached the backbone to predict final results. The detailed architecture for KITTI reported in Tab. 4.
The head consists of two parallel branches, i.e., classification and regression branches. The corre- sponding architecture:
Classification branch: FC(512) FC(256) -> FC(256) FC(3)
Regression branch: FC(512) + FC((56) + FC(256) - FC(30)
where the classification branch predicts 3 classes, while the regression branch predicts 12 classes of equally angle bins and their corresponding angle offsets, and the distance (dz, dz)dz) to its corresponding instance, well as the size (d), du: dz). For Waymo scene, we use the same model setting and just adjust the scale of input point cloud to 16.384, 4,096, 2,048 and 1,024.
C EXPERIMENTS ON TITTI val AND ONCE val SET
To verify the generalization, evaluate method on both KITTI test set and ONCE val set.
KITTI val set. As shown in Tab. 5, DBQ-SSD achieves comparable performance compared with IA-SSD, while showing super inference speed nearly two times than LA-SSD.
ONCE val set. Because the official configuration file of IA-SSD is not released with respect to ONCE dataset, we reproduce the results according to the paper. As shown in Tab. 6, our method significantly improves the inference speed 33 FPS (2.4x speedup). while maintaining compara- ble performance with IA-SSD. When adjusting the y to 0.1, our method achieves nearly mAP performance improvement while gaining 1.7x speedup.
14

---

Published as a conference paper at ICLR 2023
Table 5: Comparison with the state-of-the-art methods the KITTI val The average precision is measured with 11 recall positions. The bold font is used to indicate the best performance. The speed tested on single GPU with a batch size 16 and measured by FPS.
Car Mod.
Pedestrian Mod.
Cyclist Mod.
Speed (FPS)
Method
References
Type
(loU=0.7)
(loU=0.5)
(loU==0.5)
VoxeINet (Zhou & Tuzel. 2018)
CVPR 2018
I-stage
65.46
53.42
47.65
4,5
SECOND (Yan et al., 2018)
SENSORS 2018
1-stage
76.48
20
PointPillars (Lang et al., 2019)
CVPR 2019
I-stage
77.98
42.4
TANet (Liu 2020)
2020)
AAAI 2020
1-stage
77.85
63.45
64.95
28.5
Associate-3Ddet (Du et al., 2020)
CVPR 2020
1-stage
79.17
20
SA-SSD (He et ul., 2020)
CVPR 2020
1-stage
79.91
25
CIA-SSD (Zheng
2020)
2021
1-stage
32
Purt-A² (Shi 2020b)
TPAMI 2020
2-stage
79.47
63.84
73.07
12.5
Fust Point R-CNN (Chen al., 2019)
ICCV 2019
2-stage
79.00
16.7
STD (Yang et al., 2019)
ICCV 2019
2-stage
79.80
12.5
PV-RCNN (Shi et ul., 2020a)
CVPR 2020
2-stuge
83,90
12.5
VIC-Net (Jiang
2021)
ICRA 2021
1-stage
79.25
17
PuimRCNN (Shi et al., 2019)
2-stage
10
3D loU-Net (Li al., 2020a)
Arxiv 2020
2-stage
79.26
10
Point-GNN (Shi & Rajkumar, 2020)
CVPR 2020
1-stage
78.34
1.6
3DSSD (Yang et al., 20206)
CVPR 2020
1-stage
79.45
IA-SDD (Zhang al., 2022)
CVPR 2022
1-stage
79.57
58.91
71.24
83
Efficient Baseline DBQ-SSD
ICLR 2023
1-stage
79.17
58.11
69.29
102
ICLR 2023
1-stage
79.56
58.82
71.03
162
Table 6: Comparison with the state-of oatte-of-ie tatt--of-the-art methods on the ONCE val set. Bold font is used indicate the best performance. The speed is tested on single GPU with batch size of 16 and measured by FPS.
Metbod
Overall
0-30m
30-50m
>50m
Overall
0-30m
30- 30-50m
>550m
Overall
0-30m
Cyclist
30-50m
>550m
mAP
Speed
PointPollars
68.57
$0.86
62.07
67.01
17.63
19.74
15:15
10.23
<6.81
58.33
40.32
25.86
44.34
SECOND
71.19
34.04
63.02
47.25
26.44
29.33
24.05
18.05
58.04
09:96
52.43
34,61
51.89
PV-CUNN
77.77
$9.39
72.55
58.64
23.50
25.61
22.84
17.27
59.37
71.66
52.58
36.17
53.55
PoinRCNN
$2.00
74.45
4.78
2.40
29.84
20.94
28:72
IA-SSD
73.78
56.31
LA-SSD (Repeoduced)
70.48
84.16
63.77
49.27
38.22
44.14
33.10
20.41
61.90
73.94
55.44
38.37
56.87
14
DBQ SSD (==1.05)
72.05
$4.63
64.66
50.13
38.3
43.35
32.97
21.22
62.16
73.94
$6.65
38.20
57.51
DBQ SSD (-+0.10)
72.14
$4.81
64,27
50.22
37.83
43.38
32.18
20.29
62.99
75.13
56.65
38.91
57.65
24
DBQ SSD ()=0.20)
71.63
$4.38
64,06
37.27
41.90
33.59
20.95
62.77
74.94
$7.14
38.67
57.22
27
DBQ-SSD (2-0.30)
70.06
37.46
42.35
32.94
74.46
D VISUALIZATION
As shown in Fig. 5, Fig. 6, and Fig. 7, we provide the detailed visualization of predicted results for Waymo val and KITTI val set. The conclusion is the same KITTI scene. As the network depth increases, the foreground points are retained for classification and regression, while redundant background points are dropped. reveals that our method can adaptively discard useless points for speeding up inference. It's worth noting that the discarding behavior of point clouds significantly differs between KITTI and Waymo scenes, which verifies that our method equips generalization.
15

---

Published as a conference paper at ICLR 2023
Origin point cloud
Small (1)
Large (1)
Small (2)
Amplifier1
Large (2)
Amplifier2
Small
Amplifier3
Large (3)
Amplifier4
Small
Amplifier5
Large (4)
Amplifieró
EAmplifier
Amplifier3
SA pimpfiieer
Amplifiers
Ampfffiero.
Figure 5: Visualization results on Waymo vol set. The red and green 3D boxes in figures are ground truth and prediction boxes. Green, cyan, and yellow represent Car, Pedestrian, and Cyelist. Red and white points represent activation and blocking points, respectively. "Small" and "Large" means the scale of group in MSG, and the digital in parentheses is the index of SA layer.
16

---

Published as a conference paper at ICLR 2023
Origin
Small (1)
Small (2)
Small (3)
Small (9)
Ampiliert
Asspliiaz
Arquitfert
Large
Large
(4)
Anghifert
Ampliiie
Asspliier
Angliie
Ampidie
Angliiir
Ampliaa
Aiquiter
00
Figure 6: Another visualization results on Waymo val set.
Origin point
Small
Small
Amplifierl
Amplifier'
Amplifiers
Amplifier?
Large (1)
Large
Large (3)
Large (4)
Ampltfier2
Amplifierd
Amplifieri
Ampitfierk
Ampiltier
Amplitter?
Implitier
Impiifeer
Amplittert
Amplitiero
Amplttter?
Amplitieri
Figure 7: Detail visualization results on KITTI val set. The red and green 3D boxes in figures are
ground truth and prediction boxes. Green, cyan, and yellow represent Car, Pedestrian, and Cyclist. Red and white points represent activation and blocking points, respectively. "Small" and "Large" means the scale of group in MSG, and the digital in parentheses is the index of SA layer.
17


Reviews:
[{"review_id": "sbD0odYNuDK", "reviewer": "Reviewer_Bins", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes an extension to point-cloud processing networks that use nearest neighbor queries. Instead of fixing the search radii for those queris before training, the radii are selected dynamically on a per-point basis, and the selection criteria is learned. An additional loss is used to penelize the computational cost induced by selecting multiple radii. The experiments are well-designed and show that this approach significantly improves the cost-vs-performance tradeoff compared to prior art.", "summary_of_the_review": "The paper presents a novel, well thought of and motivated idea that is fairly and thoroughly evaluated and exceeds state of the art on some metrics. Its general design has the potential to improve several 3D point cloud processing networks that use range queries.", "strength_and_weaknesses": "- The overall **idea** is novel, well motivated and well thought off. Especially the parameterization of speed vs. accuracy using an additional loss is very interesting and allows tuning that trade-off without changing the architecture. The building block is generic can can likely be introduced into any network that uses ball-like queries on 3D (or 2D) data.\n\n- The paper is **well written**, the method description is easy to follow, all steps are well motivated and the method can likely be reproduced.\n\n- The **experiments** are thorough and fair. They are performed on large state-of-the-art datasets and the method is compared to many prior art methods. Additional experiments shed more insight into the model's inner workings and performance, further showing the internal impact of the proposed dynamic radii selection.\n\n- The **results** are good and show that the proposed method improves the baseline and exceeds state of the art in the cost-vs-performance tradeoff.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hV7BSLQirJ", "reviewer": "Reviewer_ATJW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors pointed out the issue of the spatial redundancy of existing point-based object detection frameworks. To handle this issue, the authors proposed the dynamic ball querying module, which is adaptively activated or blocked with a suitable combination among pre-defined group set. With this module and a resource budge loss, the model can be trained for achieving a better balance between effectiveness and efficiency. In experimental section, the authors provided various comparison to demonstrate that the proposed method can reduce effectively latency by a large margin on popular point cloud detection benchmarks.", "summary_of_the_review": "I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper.", "strength_and_weaknesses": "[+] First of all, the authors explained the motivation of the paper very clearly and convincingly through the analysis of the various aspects in the existing point-based object detection model. Motivated by routing process of dynamic network, the proposed ball querying module seems to be well designed to adaptively select a subset of input points as queries and extract local features in a suitable spatial factor. Lastly, the proposed model can be easily used into the existing point-based detection model framework to replace the set abstraction layer.\n\n[-] One concern is how well routing procedure works. There seems to be a lack of detailed explanation about whether it works well if it is divided more finely than 2 groups (large and small), and what kind of points is when multiple selections are made. \n\nFor efficiency, the authors used the nearest sampling, but it seems that the density change in terms of the distance from the LiDAR sensor does not take into account. Object nearby sensor has many points and farthest objects have few points, so nearest sampling can collect points that are completely unrelated to the farthest object.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N9jUf-YUAs", "reviewer": "Reviewer_FVXp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to learn the radius in ball query with gumble-sigmoid trick. Experiments on both KITTI and Waymo show the algorithm is more efficient and effective than previous state-of-the-art point-based 3d object detectors.", "summary_of_the_review": "See above", "strength_and_weaknesses": "However, the core contribution of this paper, the dynamic ball query, is not new. In Pyramid R-CNN[1], the authors proposed an almost the same algorithm. Both of these two papers use gumble-sigmoid trick to learn the radius, and they are all applied on the 3d objection detection task.\nI believe that the authors haven't seen [1] before. But based on the existence of [1], I still vote to reject this paper.\n\n[1] Mao, J., Niu, M., Bai, H., Liang, X., Xu, H., & Xu, C. (2021). Pyramid r-cnn: Towards better performance and adaptability for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2723-2732).\n\nAfter reading the rebuttal, I still think that the idea of learning the radius of a ball query is not particularly novel, even though there are some differences in the implementation details and the position of the module. However, I missed another key point: the authors argue that the radius can be learned to reduce the latency of the network. Given this contribution, I am inclined to give a borderline recommendation.\nAdditionally, I noticed that the authors keep mentioning \"latency\" in the introduction and method sections, but only \"FPS\" is evaluated in the experiments. In an autonomous driving scenario, \"latency\" is a more important metric than \"FPS\", since point clouds are typically processed frame by frame with a batch size of 1. The authors should consider replacing all mentions of \"FPS\" with \"latency\" to better align with real-world applications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HY2JHrj4K4t", "reviewer": "Reviewer_iTND", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out the spatial redundancy on background points and useless group operation in MSG for the inappropriate receptive field. The authors present an algorithm named dynamic ball query, which dynamically generates gate masks for each group of MSG to process useful points and block redundant background points. The method is validated on three detection datasets, where a noticeable improvement in inference speed is observed.", "summary_of_the_review": "Unlike existing point-based detectors that focus on recalling more foreground points with efficient sampling policies, the main contribution of this paper is choosing optimal grouping branches to process point features. I appreciate the approach, but more abundant quantitative and qualitative results are necessary to demonstrate the effectiveness of the proposed method. On the other hand, the authors should correct the typos and clarify some statements.", "strength_and_weaknesses": "Strengths:\n\nS1: The motivation of this paper is clearly presented. The paper analyzes the latency of existing point-based detectors and reveals the spatial redundancy existing in features of background points in Sec. 3. The observation may inspire studies of efficient point-based detectors.\n\nS2: The idea of using a dynamic network to drop redundant background points is novel. In order to block redundant background points, the authors propose to predict the gating logits, which are used to select a subset of input queries for each group in MSG operation. Therefore, the method chooses optimal grouping branches to process point features and avoids useless receptive field groups.\n\nS3: The experimental evaluations are comprehensive to support the claim. The proposed method presents an impressive improvement in inference speed over state-of-the-art and achieves comparable accuracy on KITTI, Waymo, and ONCE datasets.\n\nWeakness:\n\nW1: The authors claim the method achieves higher inference speed (162 vs 102). My concern is whether the setting of the baseline is reasonable enough. As shown in Fig. 3(d), no points go through the large-scale group in the fourth SA layer. The baseline is expected to achieve faster speed if it discards the useless grouping operation in SA layers.\n\nW2: In Table. 1, the paper lists the performance of dynamic gating with different routing manners. What does the layer-wise routing mean? Please supplement a clear definition for this in the text.\n\nW3: The visualization results on KITTI val set are not informative enough; please show local details and make an analysis about why most points are blocked at the large-scale group in fourth SA layer.\n\nTypos:\n1. Page 1, “INTRODUCATION” ⇒ “INTRODUCTION”\n\n2. Page 1, “Yang et al. (2020b) first proposes a single stage architecture”⇒ “Yang et al. (2020b) first propose a single stage architecture”\n\n3. Page 3, \"The valuable observation motivates us to block and choose optimal grouping branches\" => \"The valuable observation motivates us to choose optimal grouping branches\".\n\n4. Page 6, “need to be averaged along”⇒ “needs to be averaged along”\n\n5. Page 8, “point cloud which scale is” ⇒”point cloud whose scale is”\n\n6. Page 9, “with with batch” ⇒ “with batch”\n\n7. Page 9, “the inference speed of efficient the baseline”⇒ “the inference speed of the efficient baseline”\n\n8. Page 9, “When conducting larger scale”⇒ “When conducting a larger scale”\n\n9. Page 9, “which further deepen the understanding”⇒”which further deepens the understanding”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster