Sparse2Dense: Learning to Densify 3D Features for 3D Object Detection
Authors: Tianyu Wang
Xiaowei Hu
Zhengzhe Liu
Chi-Wing Fu
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Object Detection
Sparse2Dense

Abstract: LiDAR-produced point clouds are the major source for most state-of-the-art 3D object detectors. Yet, small, distant, and incomplete objects with sparse or few points are often hard to detect. We present Sparse2Dense, a new framework to efficiently boost 3D detection performance by learning to densify point clouds in latent space. Specifically, we first train a dense point 3D detector (DDet) with a dense point cloud as input and design a sparse point 3D detector (SDet) with a regular point cloud as input. Importantly, we formulate the lightweight plug-in S2D module and the point cloud reconstruction module in SDet to densify 3D features and train SDet to produce 3D features, following the dense 3D features in DDet. So, in inference, SDet can simulate dense 3D features from regular (sparse) point cloud inputs without requiring dense inputs. We evaluate our method on the large-scale Waymo Open Dataset and the Waymo Domain Adaptation Dataset, showing its high performance and efficiency over the state of the arts.

Sparse2Dense: Learning to Densify 3D Features for 3D Object Detection
Tianyu Wang
12.3, 12.3, Xiaowei Hu Zhengzhe Liu', Chi-Wing Ful.2
1 The Chinese University of Hong Kong
The Shun Hing Institute of Advanced Engineering
Shanghai Al Laboratory
(wangty, ,eliu,,eccccm ccse cuhk. edu. hk, sxxiaowei@pjlate org. cn
Abstract
LiDAR-produce point clouds are the major source for most state-of-the-art 3D object detectors. Yet, small, distant, and incomplete objects with sparse or few points are often hard to detect. We present Sparse2Dnnse, a new framework to efficiently boost 3D detection performance by learning to densify point clouds in latent space. Specifically, we first train dense point 3D detector (DDet) with dense point cloud as input and design a sparse point detector (SDet) with a regular point cloud input. Importantly, we formulate the lightweight plug-in S2D module and the point cloud reconstruction module SDet to densify 3D features and train SDet to produce 3D features, following the dense 3D features in DDet. So, in inference, SDet can simulate dense 3D features from regular (sparse) point cloud inputs without requiring dense inputs. We evaluate the large-scale Waymo Open Dataset and the Waymo Domain Adaptation Dataset, showing its high performance and efficiency the state of the arts. The code is available at https:////// http//ggtttuu..../ com/ http://ttuuu..co///eeeeeronoeoe/
method
Introduction
3D object detection is important task for supporting autonomous vehicles sense their surroundings. Previous works [H4]] design various neural network structures improve the detection perfor- mance. Yet, it remains extremely chal- lenging to detect small, distant, and in- complete objects, due to the point sparsity, object occlusion, and inaccurate laser re- flection. These issues hinder further im- provements in the precision and robust- ness of 3D object detection.
(a) Detection results of baseline
(b) Our detection results
Sparse features visualization from baseline
(d densified 3D features visualization from S2D
To improve the detection performance, some works attempt to leverage additional information, e.g., images [5, image
Figure I: Our approach is able to produce dense and good-quality 3D features (d) from regular (raw) point clouds, enabling better detection of small, distant, and incomplete objects (b) vs. [I (a,c). Red boxes are detec- tion results and green boxes are the ground truths.
segmentations [10] I1. and multi-frame
information I2 13 By fusing the in-
formation with the input point cloud in
"Corresponding author wwwpiaawwe@@jjlahoorr.oooo
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

physical or latent space, the 3D detector can obtain enhanced features for small and distant objects improve the 3D detection performance. However, the above works require additional data pre-processing and fusion in inference, thereby unavoidably increasing the computational burden and slowing down the overall detection efficiency in practice.
More recently, 14] associates incomplete perceptual features of objects with more complete features of the corresponding class-wise conceptual models via an incomplete-aware re-weighting map and a weighted MSE loss. However, this network still struggles to deal with sparse regions with limited points, due to the difficulty of generating good-quality features in these regions. Recently, [T5] proposes to generate semantic points on the predicted object regions and then train modern detectors, leveraging both the generated and original points. However, as the generated points in sparse regions could be incomplete, the generation quality in these regions is still far from satisfactory. Also, takes a long time to generate the semantic points in large scenes.
In this work, we present new approach address the point sparsity issue in 3D object detection. Specifically, we design the Sparse2Dense framework with two detectors: (i) the Dense point 3D Detector (DDet), which is pre-trained with dense point clouds for 3D detection, and (ii) the Sparse point 3D Detector (SDet), which is trained with regular (raw) point clouds as input. Very importantly when we train SDet, we use DDet to teach SDet to simulate densified 3D features, so that it can learn to produce good-qualite 3D features from regular point clouds to improve the detection performance. Unlike previous approaches, we design two effective modules to further help densify the sparse features from regular point clouds in latent space. Also, unlike previous multi-modal approaches ( 10), which require extra information, like image segmentation and images, in both training and inference, our approach needs dense point data only in training but not in inference.
Our framework is trained in two stages, First, we prepare dense point clouds by fusing multi-frame point clouds for training DDet. Then, we transfer the densified 3D features derived from DDet for embedding into the SDet features when training SDet, such that it can learn generate densified 3D features. even from regular normal point clouds. To facilitate SDet to simulate dense 3D features, we design the lightweight and effective (S2D) module to densify sparse 3D features in latent space. Also, to further enhance the feature learning, we formulate the point cloud reconstruction (PCR) module to learn to reconstruct voxel-level point cloud as an auxiliary task. Like DDet, we need this PCR module only in training but not in inference.
Furthermore, our framework is generic and compatible with various 3D detectors for boosting their performance. We adopted our framework to work with three different recent works [16 [7 [ on the large-scale Waymo [18] Open and Waymo Domain Adaptation Datasets, showing that the detection performance of all three methods are improved by our approach. Particularly, the experimental results show that our approach outperforms the state-of-the-an art 3D detectors on both datasets, demonstrating the effectiveness and versatility of our approach.
Below, we summarize the major contributions of this work.
(i) We design a new approach to address the point sparsity issue in 3D detection and formulate the Searse2 framework transfer dense point knowledge from the Dense point 3D Detector (DDet) to the Sparse point 3D Detector (SDet).
(ii) We design the lightweight plug-in S2D module to learn dense 3D features in the latent space and the point cloud reconstruction (PCR) module to regularize the feature learning.
(iii) We evaluate our approach on the large-scale henchmark datasets, Waymo [8] open and Waymo domain adaptation, demonstrating its superior performance over the state of the arts
2 Related Work
3D Object Detection. Recent years have witnessed the rapid progress of 3D object detection, existing works [9] have gained remarkable achievements on 3D object detection. SECOND [16] employs sparse convolution (22 and PointPillars [17] introduces pillar representation to achieve a good trade-off between speed and performance. Recently, CenterPoint [ proposes a center-based anchor-free method to localize the object and VoTr (30) joins self-attention with sparse convolution to build a transformer-basee 3D backbone. Besides, PV-RCNN 44) fuses deep features by Rol-grid pooling from both point and voxel features, and LiDAR R-CNN
(26] presents PointNet-brat based

---

second-stage refnnment to address size ambiguity. In this work, we adopt our approach to three recent methods as representatives, i... [T6 [7 showing that our approach can effectively enhance the performance of all three methods, demonstrating the compatibility of our framework.
Sparse/Dense Domain Transformation for 3D Point Cloud. Raw point clouds are typically sparse and incomplete in practice, thereby limiting the performance of many downstream tasks. To address this issue, several approaches, including point cloud upsampling [4] and comple- tion (37:40) have been proposed to densify point cloud and complete the objects to improve the 3D performance.
segmentation 42
and detection 43
14 15
44
Here, we review some works on sparse/dense domain transformation on 3D object detection. 45 first propose a multi-frame to single-frame distillation framework, which only uses five adjacent frames to generate the dense features as the guidance, thus limiting the performance for distant objects. 43 present a self-contained method to first extract complete cars at similar viewpoints and high-density regions across the dataset, then merge these augmented cars at the associated ground-truth locations for conceptual feature extraction. Later, [15] introduces semantic point generation to address the missing point issue. Recently, 44 presents two-stage framework: first predict 3D proposals then complete the points by another module, which employs an attention- baased GNN to refine the detection results with completed objects. The above works 43 14
14
44 conduct various operations the 15
point cloud explicitly, e-g., object extraction and matching however, the explicit point cloud operations lead two issues. First, it is challenging to conduct the above operations in distant and occluded regions, due to the high sparsity of points, thus severely limiting their performance in these regions. Second, it typically takes a very long time to conduct these operations explicitly, especially for large scenes.
43 14], and point generation [55 44
Beyond the prior works, we present an efficient sparse-to-denss approach learn densify 3D features in the latent space. instead of explicitly generating points in the point cloud. Notably, our approach needs dense point clouds only training. In inference, it takes only a regular (sparse) point cloud as input for 3D detection. Quantitative experiments demonstrate that our approach outperforms existing ones in terms of both accuracy and efficiency.
Methodology
3.1 Overall Framework Design
Figure 2 gives an overview of our Sparse2 framework, which consists of the Dense point 3D Detector (DDet) top and the Sparse point 3D Detector (SDet) bottom. Overall, DDet takes a dense point cloud as input, whereas SDet takes a regular (sparse) point cloud as input. Our idea to transfer dense point knowledge from DDet to SDet and encourage SDet to learn to generate dense 3D features, even with sparse point clouds as inputs, to boost its 3D detection performance. The workflow of our framework design is summarized follows:
(i) Dense object generation: we prepare dense point clouds for training DDet using raw multi- frame point cloud sequences. Particularly, we design a dense object generation procedure by building voxel grids and filling the voxels with object points (Section 3.2 Then, we replace the object regions in the sparse point cloud ps with the corresponding dense object points obtain the dense point cloud p?
(ii) From dense detector to sparse detector:
The training has two stages. First, as Figure (a) shows, train DDet with dense point cloud pd with region proposal network (RPN) to extract region proposals and multiple heads to perform object classification and regression, following VoxelNet- based methods [6] or Pillar-based based method [17]. Second, Figure (b) shows, we initialize SDet with the weight of the pre-trained DDet, and train SDet with regular sparse point cloud ps as input. Meanwhile, we freeze the weights of DDet and adopt an MSE loss to reduce the feature difference (FD & FS) between DDet and SDet.
(iii) Dense feature generation by S2D module: MSE loss itself is not sufficient to supervise SDet to effectively simulate dense 3D features like DDet. To complement the MSE loss, we further design the S2D module learn dense 3D features of objects in latent space. In detail, we feed dense object point cloud Pg. which is the object region of the dense point cloud PD, to DDet and then extract the dense object features FP; After that, we further encourage feature F enhanced by S2D to simulate to the dense object feature F??
3

---

Dense point 3D Detector (DDet)
Fusion
Masked MSElass
x
Cut in inference
Centers/
Multi- Heads
(a)
Dense point cloud pD0
boxes
Training stage: One
VFE/PFN 3D Backbone
Dense object point cloud F
Training stage: Two
Centers/ Classes
RPN
Multi- Heads
(b)
Voxel mask
point cloud p
S2D
Reconstruction Module
offset
Sparse point 3D Detector (5Det)
The overall framework of our proposed Sparse2Dense. Our framework contains two training stages: (a) in the first stage, we train the Dense point 3D Detector (DDet) by taking the dense point cloud as the input (dark arrows); and (b) in the second stage, train the Sparse point 3D Detector (SDet) by using the dense features from DDet as the supervision signals (gray and pink arrows). In testing, only need SDet for 3D object detection on the raw point cloud input (pink arrows), without the DDet and the point cloud reconstruction module.
Figure 2:
(iv) Feature enhancement by point cloud reconstruction (PCR) module: further, we adopt the PCR module to promote the S2D module to simulate better dense 3D features; as auxiliary task, PCR takes the feature from S2D module and predicts the voxel mask and point offset for reconstructing the voxel-level dense object point cloud P!?
3.2 Dense Object Generation
To prepare dense point clouds to train the DDet network, we design offline pre-processing pipeline to process raw point cloud sequences, each with around 198 frames (see Figure
(i) First, for each annotated object, we fuse the points inside its bounding box from multiple frames and then filter out the outlier points by using the radius outlier removal algorithm from Open3D [46], as shown in Figure (b).
(ii) To keep the object points and obtain a voxel grid (see Figure (c)) granularity (0.1m, 0.1m, 0.15m). which is the same size for point cloud voxelization. Specifically, we sort the frames in descending order of the object point number, as shown Figure voxel grid with object points starting from the beginning of the sorted frames until more than 95% voxels have been filled (see Figure (d)) Note that we stop filling single voxel when the number of points in the voxel has reached five this voxel has been filled by the previous frames to obtain enough points for training.
patterns and reduce the point number, we voxelize the fused
Then, we
(iii) For the vehicle category, whose shape is often symmetric, we flip and copy the denser side each object about its axial plane to further improve its density (see Figure (c)).
3.3 Dense Feature Distillation with the S2D Module
To transfer dense point knowledge from DDet to SDet, a straightforward solution is pair up associated features in DDet and SDet and minimize the distance between each pair of features with an MSE loss. However, MSE loss itself is struggling to achieve satisfying feature transfer, as the inputs of DDet and SDet differ lot from each other, especially for objects far from the LiDAR sensor. Also, the backbone structure [1 | built up with VoxeINet consists of SPConv, which processes only non-empty voxels. Hence, it cannot generate features empty voxels.

---

(a) soneet
Vanelin points
La How dloud requeecce
[b] Funed pbject points
(old) Filled vemah Denus pedestriar
Ruse
Voosllze
points
Mimor
polnt cloud
Denolon
Rused object points
Vovel
Filled
Completed vehicle point
object
(okc) vomels (e) cydiat
dloud sequence
Figure 3: Dense object generation pipeline. Note that we use the annotated 3D bounding box to
extract points from multiple frames and then fuse the points together with the help of voxel grid.
H
GBLU
3x3 BN GE.
H/2
3x3 BN GEL
3e3, BN GILL
H/4
Com/Not Block
Comoleut Black
Con/Noct Block Block
H/4
4x2 INN GILL
BN GELU
H/2
CELU
PS
0717
256
6x2 BLBN
256
256
256
Densited feoture
feature
+ 1024
256
Skp
Conw2D
Carv2D
Transposed
Concatenation
H
Height
Channel
Figure Left: the architecture of the S2D module. Right: ConvNeXt block
47
To better densify the 3D features, we formulate the S2D module that takes SDet's backbone feature input and learns output denser features for 3D object detection. Figure shows its architecture, which we first project sparse 3D feature to obtain BEV feature F: so that we can employ efficient 2D convolution operations in the BEV space. Then, we down-sample the feature maps to 1/4 size of the input sparse features by using convolution layers with stride Inspired by the efficient design of convolution block in 477. embed three ConvNeXt [47 residual blocks to aggregate the object information. Each block contains a X 7 depth-wise convolution, followed by a layer normalization, 1 x conv with GELU, and 1 X 1 conv. As shown the right of Figure 4 the first 1 x 1 conv increases the number of feature channels from 256 to 1024 and the second x conv reduces the channel number back to 256. Next, we upsample the features via 2D transposed concatenate the result with the previous features. After that, we feed the concatenated feature to conv layer and upsample the features to obtain the final densified feature E Note that each conv, except convs in the ConvNeXt blocks, is followed by a batch normalization and a GELU non-linear operation. With the densified feature 12 we fuse F and F by feeding each of them into x conv layer and add them together the final output feature F.S. as shown in Figure
and
To train the S2D module, we consider two kinds of supervision. The first one is on the high-level features in the DDet network, where FP and F.P are the features obtained on the dense point cloud with and without background information, respectively. We minimize the feature difference between F$ and FD well between P/P and F The second supervision comes from the point reconstruction module to be presented in the next subsection.
3.4 Point Cloud Reconstruction Module
To encourage the S2D module produce good-quality dense 3D features, further design the point cloud reconstruction (PCR) module with an auxiliary task. short, the PCR module reconstructs voxel-level dense object point cloud Pd from feature F.S. i.e., the output of the S2D module. Yet, is extremely challenging to directly reconstruct large-scale dense object points (15). Thus, propose voxel-level reconstruction scheme to prediet only the average of the input points in each non-empty voxel. Specifically, we decouple this task into two sub-tasks: first predict a soft voxel-occupancy mask indicating the probability that the voxel is non-empty; further predict point offset Pof for each non-empty voxel, i.e., an offset from voxel center V. to the averaged input points of this voxel.
5

---

F shows the architecture of the PCR mod- ule. The densified feature F% is first projected back to 3D view and fed to two 3D convolution layers and one transposed 3D convolution up-
H/8
Predicted point aftset
Point offeet
H/4
scale to 1/4 scale of the original input. Then, we
predict the voxel-occupancy mask Vmakk by us-
Point offeet
ing one 1 x 3D convolution with sigmoid func-
H/2
Veswl mask Predicted vave! mack
tion and predict point offset faet by using one 1 3D convolution. Next, repeat the same steps to further predict voxel mask mask and point offset Po fset at 1/2 scale. Then, the reconstructed point P. is predicted as
Conw3D
Transposed ComaD
Dense Object Point Cloud Reconetruction
Figure 5: The architecture of the point cloud re- construction (PCR) module.
Pe (Pofseet Ve) x
Vmask
and Pe is optimized to reconstruct the voxelized dense object point cloud Po-
3.5
Training Loss
Our Searse2Dense framework is trained in two stages. First, train DDet with the following loss:
CDDet
Lreg + Lhmills
(2)
where we adopt an L1 Loss as the regression loss Lieg followed [6
heatmap loss Cam is variant of the focal loss [48] for center-based methods [1] 17 and classification loss Late the focal loss for anchor-based method [16]. Here, Lhintets means we use Lhm or Leks, depending which method that we adopt our framework to. Second, we train SDet with the following overall loss function:
SSDE
Lreg + Lhmichs + LS2D + Lmusk + Colfset + Lhan_dis:
(3)
where Ls2D is for optimizing S2D; Lmssk and Loftset for training PCR; and Lam dis the distillation loss like Lhm. but its input is the predicted heat maps of SDet and DDet. In detail, LsDD helps to optimize SDet learn to densify 3D features based the associated features in DDet and is an MSE Loss with the masks to indicate the empty and non-empty elements in the feature maps:
Ñ
LsDD 3
1 [N]
Σ
Σ
-P2))
IN
(4)
M
8
1 M
1 M
Σ
C - P.:
where N and N are numbers of non-zero and zero values, respectively, on FD, while M and M are numbers of non-zero and zero values, respectively, on F/P Also, we empirically set 3=10 and y=20 to balance the loss weight on non-empty and empty features. Lmask and Lollset are for training PCR:
Cyask Σ
Nb N
yj log(p,) (1 1og p,)
(5)
N Σ
and
Loffset
(Posssot; + Ves) - -Pst
(6)
where N₂ and N2 are numbers of background and foreground voxels, respectively; p, and yj are the prediction and ground-truth values of the voxel mask; and indexes the voxels in Vmask- Note also that Eq. 3 includes Lum dis, only when adopting our method to the center-based methods
Experiments
4.1 Datasets and Evaluation Metrics
We employ the Waymo Open Dataset and the Waymo Domain Adaptation Dataset 18 under the Waymo Dataset License Agreement, to evaluate framework. Waymo Open Dataset is
which are

---

Table 1: Comparisons on the Waymo Open Dataset on 202 validation sequences with existing works. i means re- re-produced by 28 * means re- pr-prduued by [4]. I means re-produced by us. Note that our re-produced models and most of other re- produced models were trained on 20% data of Waymo Open Dataset following the strategy of [28 See more details of implementation in Sec. [42] and comparison in Sec. 4.3 Note that [30 29 were trained only on the vehicle category, so they can largely focus on learning features for vehicles, [5] was trained on the vehicle and pedestrian categories, and others were trained on all three categories.
Vehicle L.
Pedestrian L.
Cyclist-LI
Vehicle 1.2
Pedestrian L.2
Cyclist-
Methods
mAP
MAPH
MAP
mAPH
mAP
mAPH
mAP
MAPH
MAP
MAPH
mAP
MAPH
Part-A2-Net
28
71.82
71.29
63.15
54.96
65.23
63.92
64.33
63.82
54.24
47.11
62.61
61.35
VoTr-SSD
68.99
68.39
60.22
59.60 99.69
VoTr-TSD
30
74.95
74.95
65.91
65. 25.29
PyramidPV
76.30
75.68
67.23
66.68
Densified:
PV-RCNN 4
74.06
73.38
62.
52.68
63.32
60.72
64.99
64.38
53.80
45.14
60.72
59.18
PV-RCNN SPG
75.27
66.93
SECOND' 16
67.40
66.80
57.40
47.80
$3.50
52.30
58.90
58,30
49.40
41,10
51,80
50.60
SECOND+AGO Net'
4
69.20
68.70
59.30
48.70
55.30
54,20
60.60
60.10
51.80
42.40
53.50
$2.50
SECOND [6
67.49
66.06
55,59
44.66
57.32
54.54
57.92
47.99
38.50
52.51
SECOND (ours)
71.94
70.47
58.78
48.29
59.24
56.76
63.49
62.17
51.12
41.92
57.03
$4.64
CenterPoint- Pillar!
[+
72.36
71.73
69.16
59, 16
62. 11
60.42
64.12
63. 54
61.14
52.13
59.76
58. 58.14
76.10
75.53
74.29
65.20
67.81
66.22
67.58
66.41
58.06
65.28
63.74
CenterPoint T
73.70
72.96
74.73
69.07
68.85
67.73
65.52
65.01
66.30
61.09
66.32
65,24
CenterPoint(ours)
76.09
75.52
78.22
72.50
71.95
70.83
68.21
67.68
70.07
64.72
69.31
68.23
the largest and most informative 3D object detection dataset, which includes 360 LiDAR point cloud and annotated 3D bounding boxes. The training set contains 798 sequences with around 158K LiDAR frames and the validation set includes 202 sequences with around 40k LiDAR frames. The dataset is captured across California and Arizona. The labeled object categories include vehicle, pedestrian, and cyclist. All the objects in sequences are named with a unique ID that can be used to generate the dense object point cloud in our method. Also, we perform unsupervised domain adaptation on the Waymo Domain Adaptation dataset without re- re-training our model to show the generalization capability of our method. The Waymo Domain Adaptation dataset contains 20 sequences with 3933 frames for evaluation. This dataset is captured in Kirkland and most frames are captured rainy day [15], which means the point cloud is sparser and more incomplete than the point cloud in Waymo Open Dataset. The labeled object categories include vehicle and pedestrian in the Waymo Domain Adaptation dataset. Following prior works 4 we adopt Average Precision weighted by Heading (APH) and Average Precision (AP) as evaluation metries.
4.2 Implementation Details
In the first training stage, following [.. we train DDet from scratch using Adam with a learning rate of 0.003 and one-cycle learning rate policy with dividing factor of 0.1 and percentage of the cycle of 0.3. We set the detect range as [-7522m, 75.2m] for the X, axes and set [-2m, 4m] for the Z axis, and the size of each voxel grid as (0.1m, 0.1 0.15m). We apply global rotation around the Z-axis, random flipping, global scaling, and global translating as the data augmentation. We train the DDet four Nvidia RTX 3090 GPUs with batch size of four per GPU for 30 epochs.
In the second training stage, we adopt DDet's weights to initialize SDet, then optimize SDet by adopting the same hyperparameters as the first stage with DDet frozen. Following 4 [4] 30), we adopt 20% subset of the Waymo Open Dataset to train our models. Note that we adopt the second stage for CenterPoint-basee baselines [7] by following [.]
4.3 Comparison with State-of-the-art Methods on the Waymo Open Dataset
We compare our methods with multiple sate-of-theeare 3D object detectors 28 4 30 29
15
on three categories, i.e., pedestrian, vehicle. and cyclist, and on two difficulty levels, i.e., level 1 (L1) and level (L.2). Note that our method is plug-and-play module and can be easily adopted to work with various beep-leerning-base 3D object detectors. We obtain the results of the existing works by
copying from their papers and GitHub (8
14] 30, 29 15 or by re- re-producing their methods using
the public code with recommended parameters [6]. We cannot compare with [44), as the authors
7

---

(a) SECOND 16
(b) CenterPoint Piilll 17
CenterPoint Ponteoooint-oxxll
(d) SECOND (Ours)
(e) CenterPoint- (Ours)
(I) CenterPoint- Voxel (Ours)
Figure 6: Visual comparison of 3D object detection results and 3D features produced by (a,b,c) baselines [16 17 1 and (d,e,f) our methods (our approach + corresponding baseline), where our approach successfully densifies the object features and help the baseline methods produce more accurate detection results than the three baselines. Note that red boxes show the detection results and green boxes show the ground truths. Orange boxes highlight the improvement brought by our approach.
did not report the performance of their method on the Waymo Open Dataset and did not release code. Among the existing works, [15][4 are the state-of of-theart methods that adopt densified operations explicitly in point clouds and can be adopted to work with other methods.
Table I reports the comparison results, where our method clearly improves all three baseline methods (SECOND, PointPillarr-enter and CenterPoint) for all categories on all evaluation metrics. Notably, our approach achieves more performance gain in comparison than 114 when working with [16]. and outperforms [5] when working with [17 D. even though [15 is built upon a stronger model [4]. Also, level (L.) contains more challenging samples than level (LJ), since point cloud L22 are much sparser. Despite that, our method consistently shows greater improvement on L2, demonstrating its effectiveness to deal with the sparse point clouds. Furthermore, Table 2 shows the performance gain for three different distant ranges the Waymo validation set. Our method also achieves significant improvements over all three baseline methods for all distance ranges, including the long-range, showing again that method effectively learn to densify 3D features in challenging cases. See more detailed comparison results and the performance our model trained on the full Waymo Open Dataset in Appendix A. We further provide the visual comparisons in Figure 6 where we can see that: (i) our method successfully detects more objects that contain a few points and compensates sparse features of these objects; (ii) our method generates more accurate 3D bounding boxes, which are consistent with the ground truth boxes; see the orange boxes in the first row; (iii) method generates more dense and robust features in sparse distant regions. Please see more visual comparisons in Appendix B.
4.4 Quantitative Comparison on the Waymo Domain Adaptation Dataset
We further evaluate our methods on the Waymo Domain Adaptation Dataset and compare it with the state-of-the-art methods. Table 3 shows that our method able to consistently improve the performance of all three methods [16 17 for all categories and all difficulty levels. Note also

---

Table 2: Performance gain over baseline approaches on Waymo validation set (level-2) in different ranges. Evaluated on three categories.
Methods
All Range
Range [0, 30)
Range [30, 50)
Range [50, +inf)
mAP
mAPH
MAP
MAPH
mAP
mAPH
mAP
mAPH
SECOND [6
54,20
49.65
70.58
66.17
52.33
46.95
32,64
28.25
SECOND (ours)
57.21 +3.01
52.91 +3.26
72.94 +2.36
68.91 +2.74
55.30
+2.97
50.28 +3.33
36.32
+3.68
31.96 +3.71
CenterPoint PaiterPuiat-Plllar [7
61.67
57.94
74.18
70.55
61.63
57.88
42.91
38.83
66.60
+4,93
63.13+5.110
77.90
74.60
+4.05
66.72
+5.09
63.32
49.42 +6.51
45.41 +6.58
CenterPoint Voxel T
66.04
63.78
80.80
78,86
64.24
61.66
45,37
42.35
CenterPPintt Voxel(Ours)
+3.15
66.88
+1.2
80.77
+1.91
67.60
64.96
+3.30
49.45 +4.08
46.28
Table 3: Comparisons on the Waymo Domain Adaptation Dataset on 20 validation sequences with existing works. means re-produced by (15). means re-produced by us. Still, our re-produced models and most of other re-produced models were trained on 20% data of Waymo Open Dataset following the strategy of [28]; see more details in Sec. 4.2 15 was trained on two categories (vehicle
and pedestrian) while ours were trained on all the three categories.
Vohiclo-L1
Podestrian-LI
Vehiclo-L2
Podostrian-L22
Methods
mAP
mAPHI
mAP
MAPH
mAP
mAPII
mAP
mAPHI
SPG 15
58.31
30.82
48.70
22.05
SECOND
51,56
49.55
13.96
12.14
42.90
41,22
9.83
8.54
SECOND(Ours)
55.49
53,96
17.45
15:25
46.25
44.95
12.23
10.68
CenterPoinl Pillar 0
54.15
53,26
12.50
10.36
45.33
44.57
8.80
7.29
59.18
58,52
18.95
16.26
50,12
49,55
13.31
11.42
CenterPoint- Voxel 1
57.54
56.99
30.21
28.30
48.36
47,88
21.16
19.82
CenterPoint- Voxeliours)
60.54
59.87
37.15
38.21
51.01
50.43
26.03
24.66
that the most recent state-of-the of-the--rt method SPG [1] is trained only on the vehicle and pedestrian
categories. yet our method can achieve better performance even when trained on three categories.
4.5 Ablation Study
We conduct experiments to evaluate the key components in our Sparse2Dense framework. Here, we adopt our approach to work with CenterPoint (one stage version) [. Then, we conduct the feature distillation ("+ Distillation") to distill the 3D features from F. in DDet to F in SDet, and adopt heat map distillation Lum dis between the heat maps of SDet and DDet, as discussed in Sec Sec3.5 Next, further add the S2D module ("+ S2D") in SDet to enable the feature distillation between the 7/? in DDet and F in SDet. In addition, construct our full pipeline by further adding the point cloud reconstruction module ("+ PCR"). Also, we conduct an additional experiment ("- Distillation") by ablating the feature loss LswD and heat map distillation loss Lam_dis from our full pipeline.
The results are shown Table 4 First, feature distillation ("+ Distillation"") helps to moderately improve the performance of both categories. By adopting our S2D module ("+ S2D"), we can largely improve the quality of the densified features, boosting the performance of 3D object detection by around 2% on the vehicle and 1% on the cyclist, compared with "+ Distillation". Next, our point cloud reconstruction module (*+ PCR") further enhances the performance on both categories and metrics consistently by providing additional supervision to regularize the feature learning. Finally, even without distillation ("- Distillation"), "), our approach can still improve the baseline performance by more than 2.5% on the vehicle, demonstrating the effectiveness of our S2D and PCR modules.
4.6 Latency Analysis for S2D module
In our framework, both DDet and PCR are used only in the training stages. In inference, we only add the lightweight S2D module into the basic 3D object detection framework. To evaluate the efficiency of the S2D module, we employ the Waymo Open Dataset validation set and report the average processing time with and without the S2D module in inference. Fable/3Jreport the results, showing that S2D only brings around extra 10 ms latency to detectors, thus demonstrating our approach's 's high efficiency. Also, we show the latency of SPG [55 reported in their paper, í.e., 16.9 ms, which is evaluated on the KITTI dataset with much smaller number of points and objects than the dataset we employed, i.e., Waymo. The latency analysis results manifest the superior efficiency of our S2D in comparison to the state-of-tee the-art approach [15].

---

Table 4: Ablation studies on the Waymo Open Dataset validation set.
Vehiele-12
Pedestrian-L2
Cyclist-1_2
Methods
mAP
mAPH
mAP
mAPH
mAP
mAPH
Baseline
63,03
62.53 63.72
58,03
65.03
63.90
+ Distillation
63.84
63.32 67.04
61.21
66.44
S2D
65.75
65.22
67.62
61.65
68.50 67.34
PCR
66.12
65.58
67.47
61.59
68.69
67.54
Distillation
65.61
65.08
64.75
58.80
65.79
64.62
Table 5: Latency analysis on our S2D module. We evaluate each model with the batch size of 1. The latency is averaged over the Waymo validation set. As reference, we include SPG [[5] (evaluated on KITTI). Our method needs only ~10 ms vs. 16.9 ms by SPG.
Detectors
CenterPoint Pillar
17
CenterPoint Pillar-S2D CenterPoint Voxel 1
CenterPoint Voxe++S2D
Inference time (ms)
42.7
53.1 (+10.4)
53.0
62,8 (49.8)
Detectors
PV-RCNN
PVP
[S]
Inference time (ms)
140.0
156.9 (+16.9)
5 Discussion and Conclusion
This paper presents the novel Sparse2Dense framework that learns densify 3D features to boost 3D object detection performance. Our key idea is to learn to transfer dense point knowledge from the trained dense point 3D detector (DDet) to the sparse point 3D detector (SDet), such that SDet can learn to densify 3D features in the latent space. With the trained SDet, we only need the core component of SDet to detect 3D objects regular point clouds, so we can enhance the detection accuracy without degrading the speed. Further, to enhance the transfer of dense point knowledge, we design the S2D module and the point cloud reconstruction module in SDet to enhance the sparse features. Last, we adopt our framework to various 3D detectors, showing that their performance can all improved consistently on multiple benchmark datasets. In the future, we will apply our framework to more point cloud applications that require dense features, such as 3D segmentation and object tracking, to boost their performance while maintaining high computational efficiency.
Limitations. First, objects far from the LiDAR sensor in the point cloud sequence contain only few points. is still difficult to generate dense features for these objects with our DDet. Second, the training time of our framework is longer than traditional training, as we need multi-stage training to pre-train DDet and then transfer knowledge from the pre-trained DDet SDet. Third, our models trained on Waymo Open Dataset need inputs containing specific point features like intensity and elongation, which limits our models evaluating on different datasets, like KITTI. We will explore removing the specific point features to make the model more general in future works.
Societal Impacts. Our proposed framework can provide better 3D object detection performance for autonomous vehicles. However, like most existing 3D detectors, it may produce errors in some edge cases, due the limited data, so further research is still needed to improve its robustness.
Acknowledgements. Thank all the co-authors, reviewers, and ACs for their remarkable efforts. This work was supported by the project #MMT-p2--21 of the Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong, and the Shanghai Committee of Science and Technology (Grant 1 100100).
References
[1] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3D object detection and tracking. In CVPR, 2021.
[2] Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, and Min-Jian Zhao. Improving 3D object detection with channel-wise transformer. In ICCV, 2021.
[3] Weijing Shi and Raj Rajkumar. Point-GNN: Graph neural network for 3D object detection in point cloud. In CVPR, 2020.
[4] Shaoshuai Shi, Chaoxu Guo, Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng
Li. PV-RCNN: voint-voxel feature set abstraction for 3D object detection. In CVPR, 2020.
10

---

[5] Xiaozhi Chen, Huimin Ma. Ji Wan, Bo Li, and Tian Xia. Multi-view 3D object detection network for autonomous driving. In CVPR, 2017.
[6] Charles Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum PointNets for 3D object detection from RGB-D data. In CVPR, 2018
[7] Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3D-CVF: Generating joint camera and lidar features using cross-view spatial feature fusion for 3D object detection. In ECCV, 2020.
[8] Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3D object detection. In CVPR, 2021.
[9] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. In NIPS, 2021.
[10] Sourabh Vora, Alex H. Lang, Bassam Helou, and Oscar Beijbom. PointPainting: Sequential fusion for 3D object detection. In CVPR, 2020.
[11] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. PointAugmenting: Cross-modal augmentation for 3D object detection. In CVPR, 2021.
[12] Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam, 3D-MAN: 3D multi-frame attention network for object detection. In CVPR, 2021.
[13] Charles R. Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3D object detection from point cloud sequences. In CVPR, 2021.
[14] Liang Du, Xiaoqing Ye, Xiao Tan, Edward Johns, Bo Chen, Errui Ding, Xiangyang Sue, and Jianfeng Feng. AGO-Net: ssoociation-uuieel 3d point cloud object detection network. IEEE TPAMI, 2021.
[15] Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R. Qi, and Dragomir Anguelov. SPG: Unsu- pervised domain adaptation for 3D object detection via semantic point generation. In ICCV, 2021.
[16] Yan Yan, Yuxing Mao, and Bo Li. SECOND: Sparsely embedded convolutional detection. Sensors, 2018.
[17] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. PointPillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[18] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[19] Martin Engelcke, Dushyant Rao, Dominic Zeng Wang, Chi Hay Tong, and Ingmar Posner. Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks. In ICRA, 2017.
[20] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-end learning for point cloud based 3D object detection. CVPR, 2018.
[21] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3D object proposal generation and detection from point cloud. In CVPR, 2019.
[22] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. STD: Sparse-to-dense 3D object detector for point cloud. ICCV, 2019.
[23] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3DSSD: Point-based 3D single stage object detector. In CVPR, 2020.
[24] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: Towards high performance voxel-based 3D object detection. arXiv:2012. 157 2020.

---

[25] Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. SE-SSD: Self-ensembling single-stag stage object detector from point cloud. In CVPR, 2021.
[26] Zhichao Li, Feng Wang, and Naiyan Wang. LiDAR R-CNN: An efficient and universal 3D object detector. In CVPR, 2021.
[27] Hongwu Kuang, Bei Wang, Jianping An, Ming Zhang, and Zehan Zhang. Voxel-FPN: Multi- scale voxel feature aggregation for 3d object detection from lidar point clouds. Sensors, 20(3), 2020.
[28] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network. IEEE TPAMI, 43(8): 2021.
[29] Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid R-CNN: Towards better performance and adaptability for 3D object detection. ICCV, 2021.
[30] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3D object detection. In ICCV, 2021.
[31] Qiangeng Xu, Yiqi Zhong, and Ulrich Neumann. Behind the curtain: Learning occluded shapes for 3d object detection. In AAAI, 2022.
[32] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
[33] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv preprint arXiv: 1706.0000.00 1006.000077777 00370, 2017.
[34] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. PU-Net: Point cloud upsampling network. In CVPR, 2018.
[35] Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. PU-GAN: point cloud upsampling adversarial network. In CVPR, 2019.
[36] Ruihui Li. Xianzhi Li, Pheng- Ann Heng, and Chi- Chi-Wing Fu. Point cloud upsampling via disentangled refinement. CVPR, 2021.
[37] Liang Pan, Xinyi Chen, Zhongang Cai, Junzhe Zhang, Haiyu Zhao, Shuai Yi, and Ziwei Liu. Variational relational point completion network. In CVPR, 2021.
[38] Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. SnowflakeNet: Point cloud completion by snowflake point deconvolution with skip-transformer. In CVPR, 2021.
[39] Chulin Xie, Chuxin Wang. Bo Zhang, Hao Yang, Dong Chen, and Fang Wen. Style-based point generator with adversarial rendering for point cloud completion. In CVPR, 2021.
[40] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. PoinTr: Diverse point cloud completion with geometry-aware transformers. In CVPR, 2021.
[41] Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In AAAI, 2021.
[42] Li Yi, Boqing Gong, and Thomas Funkhouser. Complete label: domain adaptation approach to semantic segmentation of LiDAR point clouds. In CVPR, 2021.
[43] Liang Du, Xiaoqing Ye, Xiao Tan, Jianfeng Feng, Zhenbo Xu, Errui Ding, and Shilei Wen. Associate-3D Perceptual-to-cooceptuaa association for 3D point cloud object detection. In CVPR, 2020.
[44] Yanan Zhang, Di Huang, and Yunhong Wang. PC-RGNN: Point cloud completion and graph neural network for 3D object detection. In AAAI, 2021.
12

---

[45] Yue Wang, Alireza Fathi, Jiajun Wu, Thomas Funkhouser, and Justin Solomon. Multi-fran frame to Baagle-framee Knowledge distillation for 3d object detection. In ECCVW, 2020.
[46] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv: wwXiv:/801.00000 008.09947, 2018.
[47] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[48] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
13


Reviews:
[{"review_id": "lYsG8ZFgAwy", "reviewer": "Reviewer_4BxU", "summary": "This paper aims to address object detection from LiDAR point clouds. The key idea is to train a densify network that can densify 3D features in a sparse point cloud so that they match the quality of the 3D features trained on dense point clouds. This densify network can be plugged into different object detection networks and enhance their performance - this is demonstrated in experiments.\n", "questions": "Will the code be released after publication?\n", "limitations": "This is a well written paper. Good idea. Good execution and good experiments. I do not see strong limitations.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The S2D network is a lightweight, plug and play network, that can enhance the performance of different object detection networks\n+ Lots of good design and engineering choices, e.g., voxelizing dense point cloud, design of S2D network, point cloud reconstruction network, etc\n+ The experiments are done thoroughly and show improvements on a number of different object detection networks\n\n- Lacks discussion of the reconstruction quality - it would help readers to understand if S2D only learns the feature in latent space, or actually starts to learn 3D dense geometry\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "k2nVnSah5Rs", "reviewer": "Reviewer_6Pvq", "summary": "The paper proposes a method to boost the performance of object detectors in 3D LiDAR scans by learning an auxiliary task to densify them (in a self-supervised way, by distilling intermediate dense features and regressing densified point cloud in a DAE fashion). It is fairly detector-agnostic and can thus be applied to different base SOTA detectors such as CentrePoint [1] or SECOND [15]. It improves their performance by 2–5 p.p. at a cost of adding 15–20% to the runtime, which is a reasonable trade-off.", "questions": "The only substantial issue to address is missing ablations: L1 and pedestrian category, and skipping predicting the offset.\n", "limitations": "They are addressed in conclusions; I cannot think of anything beyond it that is relevant to the proposed extension.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper proposes a simple, practical, and universal method, which is sufficiently evaluated, and the paper is well written. I vote to accept it, although I am not very confident as 3D detection is not really my area.\n\nClarity:\n* (+) the paper is generally well written and has sufficient details,\n* (−) I think the paper will benefit from explaining the background on how anchor-free and anchor-based methods work; related to that: what is L_reg, specifically? Is it regressing centres of bounding boxes, or corners, or else?\n* (=) the hyperparameters are anisotropic, like grid cell size in l. 146; the lateral bounds in l. 230 are 75.2 m which is both large and weirdly specific – is it really so? then why?\n\nMethod:\n* (+) reasonable design; quite simple – (almost) all added complexity is motivated by ablation,\n* (+) the proposed improvement is orthogonal to the design of the underlying detector, so can be universally used;\n* (+) if I understand correctly, the densified point cloud is usually obtained from scan sequences, which is probably a common setting, so the method is applicable to a wide range of practical problems,\n* (−) PCR module predicts opacities and offsets – if the point cloud in that cell approximates a surface, those offsets are ambiguous in 2 dimensions; is it useful to predict them at all? I suggest to run an ablation where offsets are dropped; \n* (=) for PCR: as a future work you can try predicting a different parametrisation like an SDF predicted with an MLP in each cell; it can be naturally supervised with point clouds;\n\nExperiments:\n* (+) comparison to SOTA seems sufficient (although I am not an expert);\n* (+) ablation study has all necessary baselines (except for dropping offsets in PCR),\n* (−) however, ablation results are only reported on 2 out of 6 categories (no pedestrians at all);\n* (+) Table 3 shows that the benefit of densification increases with the distance to the object,\n* (±) latency analysis in Section 4.6 is nice, however I am not confident 100 samples are sufficient – can you compute confidence intervals?\n\n==========================\n\nTypos / wording:\n* l. 18: missing reference,\n* l. 40: semantic points – what are these?\n* l. 76: should not be there,\n* l. 163: what is BEV?\n* ll. 207, 267: adopt ← adapt.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cEebHkVF-n1", "reviewer": "Reviewer_vmwQ", "summary": "This paper presents a new framework to efficiently boost 3D detection performance by learning to densify point clouds in latent space. And experiments on Waymo Open Dataset show promising results.", "questions": "see part ‘Strengths And Weaknesses’.", "limitations": "see part ‘Strengths And Weaknesses’.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. By learning the feature representation of the point cloud in the dense point cloud space and then transferring it to the sparse point cloud space, this idea is relatively novel and feasible.\n\n2. Although the transition from dense to sparse is a good idea, there is still a large gap from dense to sparse. Especially when there is severe occlusion, this gap can be particularly large. And this case is a very common problem in the large practical application of point cloud target detection. So what about the performance in this bad case? \n\n3. The process of learning the feature representation of the point cloud in the dense point cloud space is actually an upsampling of the point cloud, and the method of voxelization is actually not an optimal way. Have you tried any other way? And have you compared the effects of different upsampling methods?\n\n4. The method in this paper is mainly used for experiments on the waymo dataset, so how about the performance on other datasets? Such as KITTI, nuScenes.\n\n5. The method in this paper is only compared with a few methods in the experimental results. In fact, there are many excellent works in the point cloud target detection task, but the comparison with them is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "58w803SWyX", "reviewer": "Reviewer_oGcw", "summary": "This paper proposes a knowledge-distillation approach to learning densified 3D features for outdoor 3D object detection. Concretely, the authors propose to train two networks DDNet and SDNet for 3D detection. During the training, DDNet will take densified 3D point cloud as inputs~(through multiple frame aggregation) while SDNet only takes single-frame Lidar point cloud. During the training, features computed from SDNet are matched with the corresponding features computed through DDNet at multiple different levels. This feature mimicking encourages the SDNet network to learn densified 3D features even with single frame input. The final model is evaluated on the Waymo 3D Detection and Domain adaptation dataset where the method considerably outperforms multiple popular baselines while maintaining similar latency. ", "questions": "Q1: Please address the weakness part I listed above. I will adjust my rating mainly based on the author's response to this part. \n\nQ2: In section 3.2, the authors mention that the model filters out the outlier points after aggregating the multi-frame point cloud. How is done exactly? \n\nQ3: line 162: more dense -> denser\n", "limitations": "The authors adequately addressed the limitations and potential societal impact. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The idea of learning densified 3D features is interesting and widely applicable to a wide range of 3D detection problems where the point cloud is sparse due to the physical constraints of the Lidar sensor. \n\nS2: The design of the feature matching process is novel. The feature mimicking or knowledge distillation is performed at multiple levels including latent features and raw point clouds~(through a point cloud completion task). \n\nS3: The method is effective and the experimental validation is solid. \n\nW1: Some important references and discussions with previous work are missing. For instance, [1] also proposed a knowledge distillation approach to learn densified features with single frame input. While it seems that the current method performs considerably better than [1], the authors need to add more detailed discussions to show the difference/improvements.\n\nW2: In section 3.4, the authors claim that they propose a novel voxel-level reconstruction scheme (occupancy + inner voxel offset regression). To my knowledge, this is already explored in [2]. Please add proper references. \n\nW3: All Waymo results are now evaluated with 20% training data + validation set. A comparison with published SOTA methods on the test set (and 100% training data) is needed to verify generalization. \n\n[1] Wang, Yue, et al. \"Multi-frame to single-frame: knowledge distillation for 3d object detection.\" arXiv preprint arXiv:2009.11859 (2020).\n\n[2] Xu, Qiangeng, et al. \"Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept