Fully Sparse 3D Object Detection
Authors: Lue Fan
Feng Wang
Naiyan Wang
Zhaoxiang Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Autonomous Driving
LiDAR
3D Object Detection

Abstract: As the perception range of LiDAR increases, LiDAR-based 3D object detection becomes a dominant task in the long-range perception task of autonomous driving. The mainstream 3D object detectors usually build dense feature maps in the network backbone and prediction head. However, the computational and spatial costs on the dense feature map are quadratic to the perception range, which makes them hardly scale up to the long-range setting. To enable efficient long-range LiDAR-based object detection, we build a fully sparse 3D object detector (FSD). The computational and spatial cost of FSD is roughly linear to the number of points and independent of the perception range. FSD is built upon the general sparse voxel encoder and a novel sparse instance recognition (SIR) module.  SIR first groups the points into instances and then applies instance-wise feature extraction and prediction. In this way, SIR resolves the issue of center feature missing, which hinders the design of the fully sparse architecture for all center-based or anchor-based detectors. Moreover, SIR avoids the time-consuming neighbor queries in previous point-based methods by grouping points into instances. We conduct extensive experiments on the large-scale Waymo Open Dataset to reveal the working mechanism of FSD, and state-of-the-art performance is reported. To demonstrate the superiority of FSD in long-range detection, we also conduct experiments on Argoverse 2 Dataset, which has a much larger perception range ($200m$) than Waymo Open Dataset ($75m$).  On such a large perception range, FSD achieves state-of-the-art performance and is 2.4$\times$ faster than the dense counterpart. Codes will be released.

Fully Sparse 3D Object Detection
Lue Fan 1,2,3,4
Naiyan Wang* Zhaoxiang Zhang 1,2.3,6, b- Institute of Automation, Chinese Academy of Sciences
Feng Wang"
University of Chinese Academy of Sciences
National Laboratory of Pattern Recognition, CASIA
*School of Future Technology, UCAS
STuSimple
CCenter for Artificial Intelligence and Robotics, HKISI_CA,
ffanlue2019, zhaoxiang-zhange zhang) (feng.wff, winaty)
Abstract
As the perception range of LiDAR increases, LiDAR-based 3D object detection becomes a dominant task in the long-range perception task of autonomous driving, The mainstream 3D object detectors usually build dense feature maps in the network backbone and prediction head. However, the computational and spatial costs on the dense feature map are quadratic the perception range, which makes them hardly scale up to the long-range setting. To enable efficient long-range LiDAR-baad -based object detection, we build a fully sparse object detector (FSD). The computational and spatial cost of FSD is roughly linear to the number of points and independent of the perception range. FSD built upon the general sparse voxel encoder and novel sparse instance recognition (SIR) module. SIR resolves the issue of center feature missing, which hinders the design of the fully sparse architecture. Moreover, SIR avoids the time-consuming neighbor queries in previous point- based methods. We conduct extensive experiments on the large-scale Waymo Open Dataset to reveal the inner workings, and state-of-the- the-art performance is reported. To demonstrate the superiority of FSD in long-range detection, we also conduct experiments on Argoverse 2 Dataset, which has much larger perception range (200m) than Waymo Open Dataset (75m). On such a large perception range, FSD achieves state-of-the-art performance and is 2.4x faster than the dense com/TuSimple/SSS
counterpart. Our code is released at wtthtts://giii
Introduction
Autonomous driving systems are eager for efficient long-range perception for downstream tasks, espe- cially in high-speed scenarios. Current 3D LiDAR- based object detectors usually convert sparse features into dense feature maps for further feature extrac- tion and prediction. For simplicity, we name the detectors utilizing dense feature maps as dense de- tectors. Dense detectors perform well current popular benchmarks [I 7 2], where the percep- tion range is relatively short (less than 75 meters). However, it is impractical to scale the dense detec- tors to the long-range setting (more than 200 meters, Fig. 2). In such settings, the computational and spa- tial complexity on dense feature maps is quadratic to the perception range. Fortunately, the sparsity of LiDAR point clouds also increases as the perception range extends (see Fig. 2. and the calculation on the
Figure 1: Illustration of center feature miss- ing and feature diffusion on dense feature maps from Bird's Eye View. The empty in- stance center (red dot) is filled by the features diffused from occupied voxels (with LiDAR points), after several convolutions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

unoccupied area is essentially unnecessary. Given the inherent sparsity, an essential solution for efficient long-range detection is to remove the dense feature maps and make the network architectures fully sparse.
However, removing the dense feature map is non-trivial since it plays a critical role in current designs. only extract the features on the non-emply voxels for efficiency. So without dense feature maps, the object centers are usually empty, especially for large objects. We name this issue as "Center Feature Missing (CFM)" (Fig. D. Almost all
Commonly adopted sparse voxel encoders [8
5 29
in this field adopt center- based or anchor- based assignment since the center feature is the best representation of the whole object. However, CFM significantly weakens the representation power of the center voxels, even makes the center feature empty in some extreme cases like super large vehicles. Given this difficulty, many previous detectors 38
popular voxel or pillar based detectors [2
5
30
38
28 5 have to convert sparse voxels to dense feature maps in Bird's Eye View after the sparse voxel encoder. Then they resolve the CFM issue by applying convolutions on the dense feature maps to diffuse features to instance centers, which we name as feature diffusion (Fig.
To properly remove the dense feature map, we then investigate the purely point-based detectors because they are naturally fully sparse. However, two draw- backs limit the usage of point-based methods in the autonomous driving scenario. (1) Inefficiency: The consuminn neighborhood query (2 is the long- standing difficulty to apply it to large-scale point cloud (more than 100K points). (2) Coarse represen- tation: To reduce the computational overhead, point- based methods aggressively downsample the whole scene to a fixed number of points. The aggressive downsampling leads to inevitable information loss and insufficient recall of foreground objects 440 43). As a result, very few purely point- based detectors have reached cent benchmarks with large-scale point clouds.
200
performance in the re-
Figure 2: Short-range point clouds (red, from KITTI (7]) long- raage point clouds (blue, from Argoverse 2 (37)). The radius of the red circle is 75 meters. The sparsity quickly increases as the range extends.
In this paper, we propose Fully Sparse Detector (FSD), which takes the advantages of both sparse voxel encoder and point-based instance predictor. Since the central region might be empty, the detector has to predict boxes from other non-empty parts of instances. However, predicting the whole box from individual parts causes a large variance on the regression targets, making the results noisy and inconsistent. This motivates us to first group the points into an instance, then further extract the instance-level feature and predict a single bounding box from the instance feature. To implement this
principle, FSD first utilizes the sparse voxel encoder 38 29 [] to extract voxel features, then votes object centers based on these features as in VoteNet 23 Then the Instance Point Grouping (IPG) module groups the voted centers into instances via Connected Components Labeling. After grouping, a point-based Sparse Instance Recognition (SIR) module extracts instance features and predicts the whole bounding boxes. As a point-based module, SIR has several desired properties. (1) Unlike previous point-based modules, SIR treats instances as groups, and does not apply the time-consuming neighborhood query for further grouping. (2) Similar to dynamic voxelization 46 SIR leverages dynamic broudcast/pooling for tensor manipulation to avoid point sampling or padding,
(3) Since SIR covers the whole instance, it builds a sufficient receptive field regardless of the physical size of the instance. We list our contributions as follows.
We propose the concept of Fully Sparse Detector (FSD), which is the essential solution for efficient long-range LiDAR detection. We further propose Sparse Instance Recognition (SIR) to overcome the issue of Center Feature Missing in sparse feature maps. Combining SIR with general sparse voxel encoders, we build an efficient and effective FSD implementation.
FSD achieves state-of-the performance on the commonly used Waymo Open Dataset. Besides, we further apply our method to the recently released Argoverse dataset to demonstrate the superiority of FSD in long-range detection. Given its challenging 200 meters perception range, FSD showcases state-of-the-aat performance while being 2.4x faster than state-of-the-e art dense detectors.

---

Poir: wise Classtication & Center Voting
Predict pooposal
X
Prediction
SiR. Madia Midde
SRZ Modhle
Spane Vouel Fazure Extractic
Tratance
Curasaal instarae
Connected
Poedict peoposal
Add mixiing
puints
liput Point Cloud
Iiatance
Curostel instarac. 2
Instance Point Gropiig CCL
Instunce-wwce feature
Group Correction
Iustoncu-niise fenttor
entraction and prodiction
extraction and prediction
Figure 3: Overall architecture FSD, For simplicity, only use two instances illustrate the pipeline. Red dots are the voted centers from each LiDAR point (blue dots). The SIR module and the SIR2 module all contain SIR layers.
2 Related Work
uses dense convolution for voxel feature extraction. Although it achieves competitive performance, it is inefficient to apply dense convolution to 3D voxel representation. PIXOR 39] and PointPillars [3] adopt 2D dense convolution in Bird's s Eye View (BEV) feature map achieving significant efficiency improvement. We name such detectors as dense detectors since they convert the sparse point cloud into dense feature maps.
Voxel-based dense detectors Pioncering work VoxeINet [45]
Voxel-based semi-dense detectors Different from the dense detectors, semi-dense detectors incorporate both sparse features and dense features. SECOND 388 adopts sparse convolution to extract the sparse voxel features in 3D space, which then are converted to dense feature maps in
BEV to enlarge the receptive field and integrate with 2D detection head [9 25 44). Based on SECOND-stE -tyle semi-dense detectors, many methods attach second stage for fine- fine-grained feature extraction and proposal refinement [29] 28, 30 3]. It is noteworthy that the semi-dens. detector is hard to be trivially lifted to the fully sparse detector since it will face the issue of Center Feature Missing, as we discussed in Sec. T
Point-based sparse detectors The purely point-based detectors are born to be fully sparse. PointR- CNN 27 is the pioneering work to build the purely point-based detector. 3DSSD [40] accelerates point-based method by removing the feature propagation layer and refinement module. VoteNet [ first makes a center voting and then generates proposals from the voted center achieving better precision. Albeit many methods have tried to accelerate the point-based method, the time-consuming neighborhood query is still unaffordable in large-scale point clouds (more than 100k points per scene). So current benchmarks 312 with large-scale point clouds are dominated by voxel-based
dense/semi-dense detectors II 30
Methodology
3.1 Overall Architecture
Following the motivation of instances groups, we have four steps to build the fully sparse detector (FSD): We first utilize a sparse voxel encoder 22 38 extract voxel features and vote object centers(Sees Sec. 3.2). 2) Instance Point Grouping groups foreground points into instances based on the voting results (Sec. 3.2). 3) Given the grouping results, Sparse Instance Recognition (SIR) module extracts instance/point features and generates proposals (Sec. (.3). 4) The proposals are utilized to correct the point grouping and refine the proposals via another SIR module (Sec. 3.4).
(Sec.3.3)..
3.2 Instance Point Grouping
Classification and Voting We first extract voxel features from the point cloud with a sparse voxel encoder. Although FSD is not restricted to a certain sparse voxel encoder, we utilize sparse attention block SST due its demonstrated effectiveness. Then we build point features by concatenating voxel features and the offsets from points to their corresponding voxel centers. These point features are passed into two heads for foreground classification and center voting. The voting is similar to VoteNet (23), where the model predicts the offsets from foreground points to corresponding object

---

H4 Ny N
2
3
Nx3 NXC
2
Nx
N X.C
Point wise sabinssion
Group 1. X,
Par Eaaaa octectio)
Dynomic Pooling
Instenor
Oupet point
Dynamic Brondcast
Group
Graup
Broadusted
Bruadiunted
Inpur
Gropping nenily
group
eusealinntes
features
festurer
group featores
Figure 4: Illustration of building instance-level point operators with dynamic broadcast/pooling, Best viewed in color. Left: calculating center-to-neighbor offsets given point clouds. Right: updating point features. Note that the operation is parallel among all instances.
centers. LI loss 25 and Focal Loss
18
are adopted as voting loss Loote and semantic classification
loss Lsem-
Connected Components Labeling (CCL) To group points into instances, we regard all the predicted centers (red dots in Fig. 3 as vertices in a graph. Two vertices are connected if their distance is smaller than a certain threshold. Then a connected component this graph can be viewed as an instance, and all points voted to this connected component share group ID. Unlike the ball query in VoteNet, our CCL-based grouping greatly avoids fragmented instances. Although there are many claborately designed instance grouping methods 32)34[10 [T0, opt for the simple CCL because it is adequate in our design and can be implemented by the efficient depth-first search.
3.3 Sparse Instance Recognition
3.3.1 Preliminaries: Dynamic Broadcast/Pooling
Given N points belong to M groups, we define their corresponding group ID array as / in shape of [N,] and their feature array as F in shape of [N, CJ. where C is the feature dimensions. F(8) is the feature array of points belonging to the i-th group. Dynamic pooling aggregates each F(1) into one group feature 9, of shape [c.]. Thus we have g; = p( ((),, where is a symmetrical pooling function. The dynamic pooling on all group features G of shape [M, C] formulated G = p(F, I). The dynamic broadcast can be viewed as the inverse operation to dynamic pooling, which broadcasts 9; to all the points in the i-th group. Since the broadcasting essentially an indexing operation, we use the indexing notation [] to denote it as G[T]. which is in shape of [N, Cj. Dynamic broadcast/pooling is very efficient because can be implemented with high parallelism on modern devices and well fits the sparse data with dynamic size.
The prerequisite of dynamic broadcast/pooling is that each point uniquely belongs group. In other words, groups should not overlap with each other. Thanks to the motivation of instances as groups, the groups 3D space do not overlap with each other naturally.
3.3.2 Formulation of Sparse Instance Recognition
After grouping points into instances in Sec. 3.2 we can directly extract instance features by some basic point-based networks like PointNet, DGCNN, etc. There are three elements define a basic point-based module: group center, pair-wise feature and group feature aggregation.
Group center The group center is the representative point of a group. For example, in the ball query, is the local origin the sphere. In SIR, the group center is defined as the centroid of all voted centers in a group.
Pair-wise feature defines the input for per point feature extraction. SIR adopts two kinds of features: 1) relative coordinate between group center and each point, 2) feature concatenation between group and each neighbor point. Taking feature concatenation example and using the notations in 3.3.1 the wisr feature can denoted as CAT(F, G[T)), where CAT is channel concatenation.
Group feature aggregation In a group, pooling function is used to aggregate neighbor features. SIR applies dynamic pooling to aggregate feature array F. Following the notations in 3.3.1 we have G p(F,1), where G is the aggregated group features.

---

Integration Combining the three basic elements, we could build many variants of point-based operators, such as PointNet [22]. DGCNN [3]. Meta-Kernel 4 etc. Fig. f illustrates the basic idea of how to build an instance-level point operator with dynamic broadcast/pooling. In our design, we adopt the formulation of VFE 45 as the basic structure of SIR layers, which is basically two-layer PointNet. In the 1-th layer of SIR module, given the input point-wise feature array Fi- point coordinates array X, the voted center X' and group ID array I, the output of (-th layer can be formulated as:
F LinNormAct (CAT (Pi, - Pavg (X' (ID)), Fi+1 LinNormAct (CAT (Pi, P +)
(2)
where LinNormAct is a fully-connected layer followed by a normalization layer (33] and an activation function [9]. The Pavg and the Pmax are average-pooling and max-pooling function, respectively. The output Fi+1 can be further used as the input of the next SIR layer, so our SIR module is stack of couple of basic SIR layers.
3.3.3 Sparse Prediction
With the formulation in Eqn. and Eqn. 2 SIR extracts features of all instances dynamically in parallel. And then SIR makes sparse prediction for all groups. In contrast to two-stage sparse prediction, our proposals (i.e., groups) do not overlap with each other. Unlike one-stage dense prediction, we only generate a single prediction for group. Sparse prediction avoids the difficulty of label assignment in dense prediction when the center feature is missing, because there is need to attach anchors or anchor points to non-empty voxels. It is noteworthy that the fully sparse architecture may face severe imbalance problem: short-range objects contain much more points than long-range objects. Some methods [ 4 use hand-crafted normalization factors to mitigate the imbalance. Instead, SIR avoids the imbalance because it only generates single prediction for a group regardless of the number of points in the group
Specifically, for each SIR layer, there is Gi Pmoz (F), 1) in Eqn. 2 which can be viewed as the group features. We concatenate all G, from each SIR layer in channel dimension and use the concatenated group features predict bounding boxes and class labels via MLPs. All the groups whose centers fall into ground-truth boxes are positive samples. For positive samples, the regression branch predicts the offsets from group centers ground-truth centers and object sizes and orientations. LI loss (25) and Focal Loss [8) are adopted as regression loss Lyeg and classification loss Los., respectively.
3.4 Group Correction
There is inevitable incorrect grouping in the Instance Point Grouping module. For example, some foreground points may be missed, or some groups may be contaminated by background clutter. So we leverage the bounding box proposals from SIR correct the grouping. The points inside proposal belong to corrected group regardless of their previous group IDs. After correction, we apply an additional SIR to these new groups. To distinguish it from the first SIR module, we denote the additional SIR module as SIR2.
SIR2 predicts box residual from the proposal to its corresponding ground-truth box, following many two-stage detectors. To make SIR2 aware of the size and location of a proposal, we adopt the offsets from inside points to proposal boundaries extra point features following [16). The regression
loss is denoted as Lyes = LI(Ares) Aree), where Aree the ground-truth residual and A is the predicted residual. Following previous methods [2 the 3D Intersection over Union (IoU) between the proposal and ground- tround-truth truth serves as the soft classification label in SIR2. Specifically, the soft label is defined and corresponding ground-truth. Then cross entropy loss is adopted to train the classification branch, denoted as Liou- Taking all the loss functions in grouping (Sec. 3.2 and sparse prediction into account, we have
min(1. 11 max(0, 21oU - 0.5)), where IoU is the IoU between proposals
Ltotal = Lsee + Livate + Lieg + Lects + Liees + Liou:
(3)
where we omit the normalization factors for simplicity.
5

---

3.5 Discussion
The center voting in FSD is inspired by VoteNet 23 while FSD has two essential differences from VoteNet.
After voting, VoteNet simply aggregates features around the voted centers without further feature extraction. Instead, FSD builds a highly efficient SIR module taking advantage of dynamic broadcast/pooling for further instance-level feature extraction. Thus, FSD extracts more powerful instance features, which is experimentally demonstrated in Sec. 4.6
VoteNet is a typical point-based method. As we discussed in Sec. it aggressively down- samples the whole scene to a fixed number of points for efficiency, causing inevitable information loss. Instead, the dynamic characteristic and efficiency of SIR enable fine- grained point feature extraction from any number of input points without any downsampling. In Sec. 4.66 we showcase the efficiency of our design in processing large-scale point clouds and the benefits from fine-graing point representation.
4 Experiments
4.1 Setup
Waymo Open Dataset (WOD) We conduct our main experiments on WOD BI] WOD is currently the largest and most trustworthy benchmark for LiDAR-based 3D object detection. WOD contains 1150 sequences (more than 200K frames), 798 for training, 202 for validation, and 150 for test. The detection range in WOD is 75 meters (cover area of 150m x 150m).
Dataset:
Dataset: Argoverse (AV2) We further conduct long-range experiments on the recently released Argoverse 2 dataset (37 to demonstrate the superiority of FSD in long-range detection. AV2 has similar scale to WOD, and it contains 1000 sequences in total, 700 for training, 150 for validation, and 150 for test. In addition average precision (AP), AV2 adopts composite score as evaluation metric, which takes both AP and localization errors into account. The perception range in AV2 is 200 meters (cover area of 400m x 400m). which is much larger than WOD. Such large perception range leads to a huge memory footprint for dense detectors.
Model Variants To demonstrate the generality of SIR, we build two FSD variants. FSD sst adopts the emerging single stride sparse transformer [5] as sparse voxel feature extractor. FSD eSDepc is built upon sparse convolution based U-Net in PartA2 29 Unless otherwise specified, we use FSD in the experiments.
Implementation Details We use sparse regional attention blocks [5] in SST as our voxel feature extractor. The SIR module and SIR2 module consist of 3 and 6 SIR layers, respectively. A SIR layer defined by Eqn. [and Eqn. Our SST-based model converges much faster than SST, so we train models for epochs instead of the 2 schedule (24 epochs) in SST. For spoose in addition to the 6-epoch schedule, we adopt a longer schedule (12 epochs) for better performance. Different from the default setting in MMDetection3D, we decrease the number of pasted instances in the CopyPaste augmentation, to prevent FSD from overfitting.
4.2 Comparison to State-of-thed Methods
We first compare FSD with state-of-the-an detectors and our baseline in Table I FSD achieves the state-of-ther -ft-----tttttttt performance among all the mainstream detectors. Thanks to the line-grained feature extraction in SIR, FSD also obtains exciting performance Pedestrian class and Cyclist class with single-frame point clouds.
4.3 Study of Treatments to Center Feature Missing
In what follows, we conduct experiments on WOD to elaborate the issue of Center Feature Missing (CFM). We first build several models with different characteristies. Note that all the following models adopt the same voxelization resolution, so they face the same degree of CFM at the beginning.
plain: After the sparse voxel encoder, FSDplain directly predicts the box from each voxel. The voxels inside ground-trun truth boxes are assigned positive. Although FSDplain uses the most

---

Table 1: Performances on the Waymo Open Dataset validation split. All models only take single- frame point cloud as input without any test-time augmentations or model ensemble. All classes are trained in a single model in FSD. Different from the default CopyPaste in MMDeteetion33D we decrease the number of pasted instances to prevent overfitting. t: Longer schedule (12 epochs).
mAP/mAPH
Vehicle APIAPH
Pedestrian 3D AP/APH
Cyclisr 3D AP/APH
Methods
L2
L1
L2
LI
L2
SECOND
61.0/5722
72.3/71.7
63. 33.9/63.3
7857/88.2
60.7/51.3
60.6/59.3
58.3/57.0
MVF 46
62.9/
65.3/-
AFDet 6
+/-
63.7/-
+/-
-/-
+/- +/-
Pillar-DD
36
-/-
69.8/-
-/-
72.5/-
-/-
-/-
RengeDet 4
65 05.0/63.2
72.9/72.3
64.0/63.6
75.9/719
67. 67.6/63.9
65.7/64.4
63. 63.3/62.1
PointPillars
62.8/57.8
72.1/71.5
63.6/63.1
70.6/56.7
62.8/50.3
64.4/62.3
61.9/59.9
Voxel RCNN
75.6/
66.6/-
RCD 0
+/-
69.0/68.5
-/-
+/-
+
VoTr-TDD
21
+/-
74.9/74.3
65.9/65.3
-/-
+/-
+ 68.6/66.9
+
LiDAR-RCNN 16
65.8/61.3
76,0/75.5
68.3/67.9
71.2/58.7 +/-
63.1/5177
66. 1/644
Pyramid RCNN 20
+/-
76.3/75.7
67.2/66.7
Voxel-to- Siceltt--Pintt
77.21-
69.8/-
3D-MAN 41
-/-
74.5/74.0
6/67/
71.7/67.7
62.6/59.0
4/-
-/- 62,7/61.5
M3DETR 8
8158/88.7 7
75.7/75.1
66 66.6/66.0
65.0/56.4
56.0/48.4
65.4/64.2
Part-A2-Net 29
66.9/63.8
77.1/76.5
08.5/
75.2/66.9
66.2/58.6
68.6/67.4 -/-
66.1/64.9
CenterPoint- Pillar
-/-
76.1/75.5
68.0/67.5
1/6./
1857/
CenterPoint Voxel
69.8/67.6
76.6/76.0
68.9/68.4
79.0/73.4
71.066..8
72. 1/71/7100
69.5/68.5
IA-SSD 43
62.3/58.1
70.5/69.7
61 6166/6100
69.4/58.5
60.3/50.7
67.7/65.3
65.0/62.7
PV-RCNN
66.8/63.3 +
77.5/76.9
69.0/68.4
75.0/65.6 6
66.0/57.6
67.8/66.4 +
65.4/64.0 +
RSN 19
75.1/74.6
66.0/65.5
77.8/72.7
68.3/63.7
TS
+/-
76.2/75.8
68.0/67.6
81.4/74.0
72.8/65.9
SST
67.8/64.6
74.2/73.8
65.5/65.1
78.7/69.6
70.0/617
70.7/69.6
68.0/66.9
AFDetV2 2 11I
71.0/68.8
77.6/77.1
69.7/69.2
80.2/74.6
72.2/67.0
73.7/72.7
71.0/70.1
PillarNet-34 26
71.0/68.5
79. 177.77
70.9/70.5
80.6/74.0
72.3/66.2
72.3/71.2
69. 69.7/68.7
PV-RCNN+
68.4/64.9
78. 8/78/7
70.3/69.7
76.7/67.
68.5/59.7
69.0/67.6
66.5/65.2
PV-RCNN
71.7/69.5
79.3/78.8
70.6/70.2
81.3/76.3
73.2/68.0
73.7/72.7
71,2/70.2
FSD FSDapcame spcome (ours)
71.9/69.7
77.8/77.3
68. 9/98568
81.9/76.4
73.2/68.0
76.5/75.2
73.8/72.5
ES (ours)
71.5/69.2
76.8/76.3
27.9567.5
81.3/75.3
72.5/67.0
77.2/76.0
74.4/73.2
FSD, FSD spcome (ours)
72.9/70.8
79.2/78.8
70.5/70.1
82.6/77.3
73.9/69.1
77.1/76.0
74.4/73.3
straightforward solution for CFM, it suffers from the large variance of regression targets and low-quality predictions from hard voxels.
SST center: replaces the anchor- head in SST with CenterHead 44 Based on sparse
voxel encoder, SST center converts sparse voxels into dense feature maps and applies several convolutions diffuse features to the empty object centers in Fig. Then makes predictions from the diffused center feature.
noge: Poge
FSD
It removes the group correction and SIR2 module FSD.
CenterPoint-PP: does not resort any sparse voxel encoders. Instead, it applies multiple dense convolutions soon after voxelization for feature diffuusio, greatly eliminating CFM. It also is equipped with CenterHead avoiding large variance of regression targets.
Experiments and analyses There is usually quite large unoccupied area around the cen- ters of large vehicles. Thus the performance of large vehicles an appropriate indicator that reveals the effect of CFM. So we build customized evaluation tool, which breaks down the object length following the COCO evaluation [17]. Then we use evaluate the performance of vehicles with different lengths. Table shows the results, and we list our findings as follows.
Table 2: Vehiele detection with vehicle length breakdown. re- re-implemented ourselves. * official Waymo L2 overall metric. Arrows indicate the performance changes from SST center.
Vehicle longth (m)
Methods
(0,4)
14, 8)
12)
[12. -oc)
Official"
CenterPuint PP
34.3
69.3
42.0
43.6
66.2
FSDate
SSTuwe
S
36.0
69.4
33.7
30.5
66.3
FSDwer FSD
33,5 1.5
68.
L 1.2
47.7 14.0
47.9 17.4
65.2
36.7 0.7
71.0
$1.3
17.6
53.7
23.2
23.2
69.3 1

---

800
FSD
714
Inference Lateney (ms)
Training Menory (GIB)
15.3
24.0
SST center
434
400
ConterPoini-P
400
13.0
300
CenterPoon
11.0
11.8
10.5
250
232
9.0
200
6.3
164
5.5
4.9
150
105
3.6
3,4
81
90
94
97
3.0
1.4
1.0
50
100
150 200
50
100
150
200
Perception Range
Perception Range (m)
Figure Memory footprints and inference latency in different perception ranges. We use FSD (Sec. 4.1 here. Statistics are obtained on single 3090 GPU with batch size 1. Inference latency is evaluated by the standard benchmark script in MMDetection3D without any test-time optimization. CenterPoint-PP PP and SST center are defined in Sec. 4.3 Best viewed in color.
Comparing FSDplain with SST center. they share the same attention-based sparse voxel encoder. However, the trend is totally opposite w.r.t vehicle size. With feature diffusion, SST SST attains much worse performance than PSDplain on large vehicles. It suggests feature diffusion is sub- optimal solution for CFM in the case of large objects. For those large objects, the features may not be diffused the centers the diffused features are weak make accurate predictions.
However, FSD plain obtains the worst performance among detectors on vehicles with normal sizes. Note that the CFM issue is minor for the normal size vehicles. So, in this case, the center- based assignment in SSTcenter shows its superiority to the assignment FSDptain- It suggests the solution for CFM in FSD plain is also sub-optimal, even if it achieves better performance in large objects.
Comparing FSDnoge with SSTcenters they share the same sparse voxel encoder while FSD noge replaces the dense part in SSTeenter with SIR. The huge improvements of FSDnoge on large vehicles fairly reveal that SIR effectively resolves CFM and is better than feature diffusion.
CenterPoint-PP suffers much less from CFM because it leverages dense feature maps from very beginning of the network. It is also equipped with the advanced center-based assignment. Even so, FSDnoge and FSD still outperform CenterPoint-PPi especially on large vehicles.
Table 3: Performance Argoverse 2 validation split. t: provided by authors AV2 dataset. $:
Weak Caste augmentation for preventing overfitting (one instance per class). re-implemented by ourselves. C-Barrel: construction barrel. MPC-Sign: mobile pedestrian crossing sign. A-Bus: articulated bus. C-Cone: construction
V-Trailer: vehieular trailer. We omit the results of dog, wheelchair and message board trailer because these categories contain very few instances. The average results take all categories into account, including the omitted categories. We mark the categories attaining notable improvements in bold.
Vehcle
Avenuee
Padestrion
Box Toukk
Billad
C.Barol
MPC-S1g
Monocyele
A-hos
C-Oons
V-Tnaerr
SmnWee
Bus
Methods
Prectsion
CesterPoint
13.5
61.0
36.0
33.0
28.0
2600
23.0
22.5
16.0
1600
12.5
9.5
8.5
7.5
8,0
8.0
7.0
2.0
CenterPoint
22.0
67.6
38.9
46.5
37.4
40.1
32.2
25.6
27.4
33.4
24.5
8.7
25.8
22.6
29.5
22.4
0.3
20.1
RSD
21.0
39.5
51.4
38.3
28.1
30.0
25.6
38.1
15.6
30.0
20.1
38.9
21.9
57.
RSD gesert
41.2
Compunit Souve
CenterPoin
33.0
pere
4.4 Long-range Detection
Several widely adopted 3D detection benchmarks 31l 2 have relatively short perception range. To unleash the potential of FSD, conduct Long-range detection experiments the recently released

---

Argoverse 2 dataset (AV2), with a perception range of 200 meters. In addition, AV2 contains objects in 30 classes, facing the challenging long-tail tail issue.
Main results We first list the main results of FSD on AV2 in Table 3 The authors of AV2 provide baseline CenterPoint model, but the results are mediocre. To make fair comparison, we re-implement stronger CenterPoint model the AV2 dataset. The re-implemented CenterPoint adopts the same training scheme with FSD, including ground-truth sampling to alleviate the long-tail issue. FSD outperforms CenterPoint in the average metric. It is noteworthy that FSD significantly outperforms CenterPoint in some tiny objects (e.g., Pedestrian, Construction Cone) as well as some objects with extremely large sizes (e.g., Articulated Bus, School Bus). We owe this to the virtue of instance-level fine-grained feature extraction in SIR.
Range Scaling To demonstrate the efficiency FSD long-range detection, we depict the trend of training memory and inference latency of three detectors when the perception range increases in Fig. 5 Fig. F shows dramatic latency/memory increase when applying dense detectors larger perception ranges. Designed to be fully sparse, the resource needed for FSD is roughly linear to the number of input points, so its memory and latency only slightly increase as the perception range extends.
4.5 More Sparse Scenes
Argoverse dataset provides a highly reliable HD map, which could be utilized as prior to remove uninterested regions making the scene more sparse. Thus proceed with experiments removing some uninterested regions to show the advantages of FSD in more sparse scenarios. The results are summarized in Table FSD has significantly lower memory footprint and latency with acceptable precision loss after removing the uninterested regions. On the contrary, the efficiency improvement of CenterPoint minor. reveals that FSD benefits more from the increase of data sparsity, which is another advantage of the fully sparse architecture.
Table 4: Performance with different detection ar- cas. t: Region of Interest is defined by the HD map in AV2 dataset.
CosterPoant Laieneytin)
Mon
Latznoyiim))
mAP
Man.
mAP
GlI only Rult
5.9
97
24,0
10.4
232
22.0
3.. 43.88
KLL 16 995
21.3
9.00
0.85
227 2.05
31.5
wil ground
233 6L.0%%
74.
25, $5,996
21.0
9.73
6.7%
217.
6.4%
1931
4.6 Ablation Study
Effectiveness of Components In addition to
Table 5: Ablation of design factors in SIR. Perfor- mances are evaluated on Waymo validation split.
FSD and FSD noge (Sec. (icc.4.3) 4.3), we also de- grade FSD to FSDagg to understand the mecha- nism of FSD. In FSDagn we aggregate grouped point features by dynamic pooling and then di- rectly make predictions from the pooled features, after Instance Point Grouping. FSD agg is simi- lar to the way in VoteNet (23 we discussed in Sec, 3.5 Thus, FSDago agg can explicitly leverage instance-level features other than the point-level not take advantage of further point feature extraction in SIR. As can be seen in Table 5 the improvement is limited if we only apply grouping without SIR. The combination of grouping and SIR attain notable improvements.
Group Correction
L2 APII Pedestrian
Grouping
SIR
Vehicle
Cyclist
PSD FSD
62.29
64.31
64.19
63.13
64.52
features FSD plain However, FSDagg
Downsampling in SIR The efficiency of SIR makes it feasible extract fine- ane-ggained point features without any point downsampling, This another notable difference between FSD and
Table 6: Performances with different representa- tion granularity. f: Latency of SIR module.
AP Bicyclist
VoteNet. To demonstrate the superiority, we apply voxelization on the raw points before SIR module and treat the centroids of voxels as downsampled points. We conduet experiments on AV2 dataset because it contains couple of categories tiny size, which may be sensitive to downsampling. As expected, small objects have notable performance loss when adopting downsampling, and we list some of them in Table 6 We
Voxal
Bollard
Stop Sign
Latency (ms))
30cm
35.4
36.5
24.6
18.3
3.5
20cm
37,3
37.3
26.4
20.0
27.0
Point
39.3
38.6
27.1
21.5
6.3

---

also evaluate the inference latency of the SIR module on 3090 GPU. As can be seen, compared with the overall latency (97ms, Fig. 55. the SIR module is highly efficient.
Conclusion
This paper proposes FSD, a fully sparse 3D object detector, aiming for efficient long-range object detection. FSD utilizes a highly efficient point-based Sparse Instance Recognition module to solve the center feature missing in fully sparse architecture. FSD achieves not only competitive performance on the widely-used Waymo Open Dataset, but also sate-of-the-aa performance in the long-range Argoverse 2 dataset with much faster inference speed than previous detectors.
Limitation A more elaborately designed grouping strategy may help with performance improve- ments. However, it is beyond our design goal in this paper, and we will pursue it in future work.
6 Acknowledgements
This work was supported in part by the Major Project for New Generation of AI NN022018AAA00004000, the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231), and in part by the TuSimple Collaborative Research Project.
References
[1] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu. Range Condi- tioned Dilated Convolutions for Scale Invariant 3D Object Detection. arXiv preprint arXiv:2005.09927, 2020.
[2] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[3] Jiajun Deng, Shaoshuai Shi, Peiwei Li. Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: Towards High Performance bosel-based 3D Object Detection. In AAAI, 2021.
[4] Lue Fan, Xuan Xiong, Feng Wang. Naiyan Wang, and ZhaoXiang Zhang. RengeDet: In Defense of Range View for LiDAR- R-sed 3D Object Detection. In ICCV, 2021.
[5] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing Single Stride 3D Object Detector with Sparse Transformer. In CVPR, 2022.
[6] Runzhou Ge. Zhuangzhuang Ding, Yihan Hu, Yu Wang, Sijia Chen, Li Huang, and Yuan Li. AFDet: Anchor Free One Stage 3D Object Detection. arXiv preprint arXiv:2006.1267) 2020.
[7] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012.
[8] Tianrui Guan, Jun Wang, Shiyi Lan, Rohan Chandra, Zuxuan Wu, Larry Davis, and Dinesh Manocha. M3DETR: Multi-Representee In WACV, 2022.
Multi-Scale, Rulual-Relation 3D Object Detection With Transformers.
[9] Dan Hendrycks and Kevin Gimpel. Gaussian 16Xiv:1606.08... 08415, 2016.
Error
Linear
Units (GELUs).
arXiv preprint
[10] Fangzhou Hong. Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. baser-based panoptic segmentation via dynamic shifting network. In CVPR, 2021.
[11] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. AFDetV2: Rethinking the Necessity of the Second Stage for Object Detection from Point Clouds. arXiv preprint 1xX.v:21/200920: 2021.
[12] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In CVPR, 2020.
[13] Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. PointPillars: Fast Encoders for Object Detection from Point Clouds. In CVPR, 2019.
10

---

[14] Jiale Li, Hang Dai, Ling Shao, and Yong Ding. From Voxel to Point: IoU-geided 3D Object Detection for Point Cloud with Voxel-to-Point Decoder. In ACM-MM, 2021.
[15] Yingwei Li, Adams Wei Yu, Tianjian Meng. Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen. Bo Wu, Yifeng Lu, Denny Zhou, et al. DeepFusion: Lidar-Camess Deep Fusion for Multi-Modul 3D Object Detection. In CVPR, 2022.
[16] Zhichao Li, Feng Wang, and Naiyan Wang. LiDAR R-CNN: An Efficient and Universal 3D Object Detector. In CVPR, 2021.
[17] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European conference on computer vision. Springer, 2014.
[18] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal Loss for Dense Object Detection. In ICCV, 2017.
[19] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Sheng-Yang, Fu, and Alexander Berg. SSD: Single Shot Multihox Detector. In ECCV, 2016.
[20] Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection. In ICCV, 2021.
[21] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng. Xiaodan Liang. Hang Xu, and Chunjing Xu. Voxel Transformer for 3D Object Detection. In ICCV, 2021.
[22] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In CVPR, 2017.
[23] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep Hough Voting for 3D Object Detection in Point Clouds. In ICCV, 2019.
[24] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++:: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In NeurlPS, 2017.
[25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks. NeurlPS, 28, 2015.
[26] Guangsheng Shi, Ruifeng Li, and Chao Ma. PillarNet: High- figh-Perffrmance Pillar-based 3D Object Detection. arXiv preprint arXiv:2205.074000 2022.
[27] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. In CVPR, 2019.
[28] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: Point- Voxel Feature Set Abstraction for 3D Object Detection. In CVPR, 2020.
[29] Shaoshuai Shi, 7he Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[30] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN+ Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection. arXiv preprint arXiv:2 2021.
[31] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla. Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In CVPR, 2020.
[32] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Smin- chisescu, and Dragomir Anguelov. RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection. In CVPR, 2021.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NewrlPS, 2017.
[34] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. SGPN: Similarity Group Proposal Network for 3d Point Cloud Instance Segmentation. In CVPR, 2018
11

---

[35] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein. and Justin M Solomon. Dynamic graph for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[36] Yue Wang. Alireza Fathi, Abhijit Kundu, David Ross, Caroline Pantofaru. Tom Funkhouser, and Justin Solomon. Pillar-based Object Detection for Autonomous Driving. In ECCV, 2020.
[37] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. In NeurIPS Datasets and Benchmarks 2021, 2021.
[38] Yan Yan, Yuxing Mao, and Bo Li. SECOND: Sparsely Embedded Convolutional Detection. Sensors, 18 (10), 2018.
[39] Bin Yang, Wenjie Luo, and Raquel Urtasun. PIXOR: Real-time 3D Object Detection from Point Clouds. In CVPR, 2018.
[40] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3DSSD: Point- baintbbsed 3D Single Stage Object Detector. In CVPR, 2020.
[41] Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3D-MAN: 3D Multi-Frame Attention Network for Object Detection. In CVPR, 2021.
[42] Tianwei Yia, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3D Object Detection and Tracking. arXiv preprint arXiv:2006. 11.006.1122 11275, 2020.
[43] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds. In CVPR, 2022.
[44] Xingyi Zhou, Dequan Wang, and Philipp Kräähebüühl. Objects as Points, arXiv preprint 1w0.iv:904..79900 2019.
[45] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-Ent Learning for Point Cloud Based 3D Object Detection. In CVPR, 2018.
[46] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang. James Guo, Jiquan Ngiam, and Vijay Vasudevan, End-to-Fnd Multi-Vies Fusion for 3D Ohject Detection LiDAR Point Clouds, In CoRI, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [N/A]
We are not releasing new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "OSLHuaIcuO6", "reviewer": "Reviewer_2M9Y", "summary": "This paper studies efficient long-range LiDAR-based 3D object detection. The authors propose a fully sparse 3D object detector (FSD) to eliminate the dense BEV feature map that scales poorly with the perception range. Similar to VoteNet, FSD first applies the sparse encoder to extract point/voxel-wise features, then groups the points into instances, and finally predicts the boxes based on the instance-wise features. The authors have evaluated their proposed FSD on Waymo and Argoverse 2. FSD achieves superior performance on both benchmarks, especially Argoverse 2 where the perception range is much larger.", "questions": "I would love to see the answer to the following questions in the rebuttal:\n* What are the pros and cons of DETR and SIR in dealing with long-range perception?\n* How well does FSD scale with denser point clouds (*e.g.*, from temporal fusion)?\n* Where does the efficiency boost of FSD come from?\n* Could FSD still achieve the same level of speedup with highly-optimized inference engines (*e.g.*, TensorRT, TVM)?\n* What is the accuracy of SST on Argoverse 2?\n\nThe authors could refer to the previous section for more detailed comments.", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strengths]**\n\nThis paper is well-written and easy to follow. The authors clearly explain the missing center feature problem with a good illustration (Figure 1). Generally, the figures in this paper have pretty good quality and clarity, which helps readers understand the proposed algorithm easily.\n\nThe problem studied in this paper is very important as long-range perception is very critical in high-speed driving scenarios. The proposed FSD is technically sound and achieves good empirical results on large-scale benchmark datasets.\n\n**[Weaknesses]**\n\nThe technical novelty of this paper is a bit limited as the proposed FSD is very similar to VoteNet. The authors have clarified the differences between FSD and VoteNet in Section 3.5. However, both of them are more of engineering improvements than technical contributions. That said, the authors apply this idea to solve an important real-world problem, which could still provide valuable insights to other researchers in the community.\n\nResearchers have spent considerable efforts on transformer-based detection heads (DETR) besides anchor-based and center-based heads. Lately, there have been several follow-ups in 3D object detection, such as Object DGCNN, DETR3D, and TransFusion. DETR head provides a potential workaround for large dense BEV feature maps. The authors could discuss the pros and cons of DETR and SIR in their rebuttal.\n\nAlthough the computational cost of FSD does not grow with the perception range, it does scale linearly with the point cloud density. This could be problematic for denser point clouds (either with temporal fusion or simply with more laser beams). In contrast, dense detectors suffer less from this since the BEV feature map will always have the same resolution regardless of the point cloud size. In all experiments, the authors use single-frame point clouds for evaluation. It would be necessary to conduct experiments (or at least measure the memory footprints and inference latency) with denser multi-frame point clouds.\n\nAs this paper focuses on efficiency, it would be essential to understand where the efficiency boost comes from. For example, the authors could provide a detailed latency breakdown of different components in FSD. Besides, different inference backends could lead to very different latency results. The dense BEV encoder is composed of regular 2D convolutions, which can be easily accelerated by existing inference libraries (such as TensorRT and TVM), while the proposed SIR module is harder to be optimized. Therefore, it is unclear whether FSD could still achieve such significant speedups with highly-optimized inference engines.\n\nFinally, there are a few aspects that could potentially be improved in the experimental evaluation:\n* As FSD is based on SST, it would be necessary to report the accuracy of SST on Argoverse 2 for an apples-to-apples comparison.\n* The authors could consider conducting experiments on nuScenes, where there are many competitive LiDAR baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JMaXyxyiibv", "reviewer": "Reviewer_WF16", "summary": "This paper targets the problem of high computational complexity in LiDAR-based long-range 3D object detection. It proposes a fully sparse 3D object detection paradigm, i.e., from the feature extraction to the proposal generation and bounding box prediction, all are conducted based on sparse voxels or points. Specifically, built upon the commonly used sparse voxel encoder, it devises a novel sparse instance recognition (SIR) module to perform 3D detection based on local points, which is efficient in implementation and resolves the \"center feature missing\" problem. Experiments demonstrate the efficacy of this proposed method, especially for large vehicles/tiny objects and long-range cases.", "questions": "(Minor) I notice the model converges faster than SST and wonder whether there is any potential reason/discussion about it.", "limitations": "The author discusses the limitations and societal impact adequately in the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The studied problem is valuable for practical use. The motivation that dense prediction can bring high computational complexity for long-range cases is reasonable. A fully sparse 3D object detector is suitable given the sparse property of LiDAR points distribution.\n- The analysis of the difference between the proposed method and voxel-based semi-dense detectors/conventional voting in VoteNet makes the contribution of this paper clear.\n- Illustrations are clear and concise, making the proposed method much easier to understand.\n- Experiments support the main claims in this paper. The improvement in the detection performance of small objects/huge vehicles is impressive.\n- The efficiency analysis in terms of different perception ranges shows the main advantage of the proposed paradigm.\n\nWeaknesses:\n- The methodology part is hard to follow. In particular, so many new concepts are unnecessarily defined in the presentation, for example, dynamic broadcast/pooling, sparse instance recognition, group feature aggregation, integration, sparse prediction, etc. For some of them, the author would have better expressions/names to explain them for easier understanding. There is no need to package some easy operations or commonly used modules with totally new names, which can only make the implementation hard to understand.\n\nIn contrast, the illustration is clear and concise to represent the overall pipeline and main modules of the proposed framework.\n\n- Experiments: Although the proposed method can address the center feature missing problem to some extent and improve the detection performance of huge vehicles, it still performs a little worse than state-of-the-art methods (e.g., PV-RCNN++) on the vehicle detection of Waymo. The better mAP over 3 classes is mainly contributed by better performance on small objects. Maybe need some analysis on this phenomenon.\n\n- The analysis of the center feature missing problem is a little vague. Specifically, is there any way that we can observe the effect of addressing this problem more straightforwardly? From my perspective, the detection performance of huge vehicles can not directly support the claim strongly, maybe at least we need a more detailed analysis of the center localization accuracy or some qualitative analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DJ_59eLceC9", "reviewer": "Reviewer_UjRj", "summary": "The authors propose a novel 3D lidar point cloud detection method. the method is based on \na sparse instance recognition module which groups points into features, making the feature\nextraction and bounding boxes regression for detection more efficient than other point-based\nmethods.", "questions": "However, it would be nice to have a quick explanation for the metric used as well as some qualitative\nvisual results, especially for the \"failing\"/less good results compared to other methods.\nAlso perhaps for the memory and latency performances, having more of the SOTA methods for comparison\nwould bring more value to the discussion.\nFinally, even if the paper is more focused on long and sparse lidar data compared to other methods.\nIt would be interesting for a small comparison on the KITTI dataset, to see if the method\nstill performs correctly on shorter-range point clouds.", "limitations": "The limitation mentioned by the authors is found in the grouping strategy employed. \nI agree that a comparison with another grouping method would be out of the scope of the paper, but then perhaps an extra comparison with the grouping method used by another method would be interesting? (pointnet++ multi-scale grouping module for example).", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The motivation and goals of the work are well explained and clear. \nThe experimentation section is good and the ablation study brings interesting conclusions.\n\nThe paper has some little missing information listed in the questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept