HEDNet: A Hierarchical Encoder-Decoder Network for 3D Object Detection in Point Clouds
Authors: Gang Zhang
Chen Junnan
Guohuan Gao
Jianmin Li
Xiaolin Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D object detection; encoder-decoder structure

Abstract: 3D object detection in point clouds is important for autonomous driving systems. A primary challenge in 3D object detection stems from the sparse distribution of points within the 3D scene. Existing high-performance methods typically employ 3D sparse convolutional neural networks with small kernels to extract features. To reduce computational costs, these methods resort to submanifold sparse convolutions, which prevent the information exchange among spatially disconnected features. Some recent approaches have attempted to address this problem by introducing large-kernel convolutions or self-attention mechanisms, but they either achieve limited accuracy improvements or incur excessive computational costs. We propose HEDNet, a hierarchical encoder-decoder network for 3D object detection, which leverages encoder-decoder blocks to capture long-range dependencies among features in the spatial space, particularly for large and distant objects. We conducted extensive experiments on the Waymo Open and nuScenes datasets. HEDNet achieved superior detection accuracy on both datasets than previous state-of-the-art methods with competitive efficiency. The code is available at https://github.com/zhanggang001/HEDNet.

HEDNet: A Hierarchical Encoder-Decoder Network for 3D Object Detection in Point Clouds
Gang Zhang', Junnan Chen', Guohuan Gao², Jianmin Li', Xiaolin Hu4.4.5+
*Deparmeent of Computer Science and Technology, Institute for AI,
BNRist, THU-Bosch JCML Center, Tsinghua University
Huazhong University of Science and Technology, "Bejjin Institute of Technology *Tsinghua Laboratory of Brain and Intelligence (THBI),
IDG/McGovern Institute for Brain Research, Tsinghua University
5 Chinese Institute for Brain Research (CIBR), Beijing 100010, China
zhang-8 g g190mails tsinghua. edu. cn, chen jn@hust edu. cn, www..ooooooottbbbttt....o.o.. edu. cn
1i Lijiani iijianmin@mmil... tawamiiimmai..sinnnnuuaa.. edu, www.ttinnhttttiiiinaaaaaa edu. cm
Code: https//gghhol.oomm///naagggggggg
Abstract
3D object detection in point clouds important for autonomous driving systems. A primary challenge in 3D object detection stems from the sparse distribution of points within the 3D scene. Existing high-performance methods typically employ 3D sparse convolutional neural networks with small kernels to extract features. To reduce computational costs, these methods resort to submanifold sparse con- volutions, which prevent the information exchange among spatially disconnected features. Some recent approaches have attempted to address this problem by in- troducing large-kernel convolutions self-attention mechanisms, but they either achieve limited accuracy improvements or incur excessive computational costs. We propose HEDNet, hierarchical encoder-decoder network for 3D object detection, which leverages encoder-decoder blocks to capture long-range dependencies among features in the spatial space, particularly for large and distant objects. We con- ducted extensive experiments on the Waymo Open and nuScenes datasets. HEDNet achieved superior detection accuracy on both datasets than previous state-of-the-art methods with competitive efficiency. The code has been released.
Introduction
Learning effective representations from sparse input data is key challenge for 3D object detection in point clouds. Existing point-based methods I
2
3 S and range-based methods 6 7
g 9 10)
either suffer from high computational costs exhibit inferior detection accuracy. Currently, voxel-
based methods [2 13 14, [5) dominate pegeooeeffrmance 3D object detection.
The voxel-based methods partition the unstructured point clouds into regular voxels and utilize sparse conventional neural network (CNNs) 12 17
[8] 19] or transformers [3]
14!
15] as backbones
for feature extraction. Most existing sparse CNNs primarily built by stacking submanifold sparse residual (SSR) blocks, each consisting of two submanifold sparse convolutions 20) with small kernels. However, submanifold sparse convolutions maintain the same sparsity between input and output features, and therefore hinder the exchange of information among spatially disconnected features. Consequently, models employing SSR blocks face challenges in effectively capturing long-range dependencies among features. One potential solution is to replace the submanifold
Corresponding Author
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

sparse convolutions in SSR block with regular sparse convolutions [2T]. However, this leads to significant decrease in feature sparsity as the network deepens, resulting in substantial computational costs. Recent research has investigated the utilization of large-kernel sparse CNNs [2 16 and transformers [14 15] to capture long-range dependencies among features. However, these approaches have either demonstrated limited improvements in detection accuracy or come with significant computational costs. Thus, the question remains: there an efficient method that enables sparse CNNs to effectively capture long-range dependencies among features?
Revisiting backbone designs in various dense prediction tasks that the mncoder-decodee structure has proven effective in capturing long-range dependencies among features. These methods typically use a high-to-lor resolution backbone an encoder to extract multi-scale features and design different decoders to recover high-resolution features that can model long-range relationships. For instance, PSPNet 24 incorporates a pyramid pooling module to capture both local and global contextual information by pooling features at multiple scales. SWFormer [3] integrates top-down pathway into its transformer backbone to capture cross- woosswwndoww correlations. However, the utilization of the encoder-decoder structure in designing sparse convolutional backbones for 3D object detection has not yet been explored, to the best of our knowledge.
26 we observe
In this work, we propose a sparse encoder-decoder (SED) block to overcome the limitations of the SSR block. The encoder extracts multi-scall features through feature wwwn-sampling facilitating information exchange among spatially disconnected regions. Meanwhile, the decoder incorporates multi-scale feature fusion to recover the lost details. A hallmark of the SED block is its ability to capture long-range dependencies while preserving the same sparsity between input and output features. Since current leading 3D detectors typically rely on object centers for detection 27 28), we further adapt the 3D SED block into 2D dense encoder-decodes (DED) block, which expands the extracted sparse features towards object centers. Leveraging the SED block and DED block, we introduce hierarchical encoder-decoder network named HEDNet for 3D object detection in point clouds. HEDNet can learn powerful representations for the detection of large and distant objects.
Extensive experiments were conducted on the challenging Waymo Open 29 and nuScenes [30] datasets demonstrate the effectiveness of the proposed HEDNet on 3D object detection. HEDNet achieved impressive performance, with a 75.0% L2 mAPH on the Waymo Open test set and 72.0% NDS on the nuScenes test set, outperforming prior methods that utilize large-kernel CNNs or transformers as backhones while exhibiting higher efficiency. For instance, HEDNet was 50% faster than DSVT, the previous sate-of-the-eee transformer-ban based method, with 1.3% L.2 mAPH gains.
2 Related work
2.1 3D object detection in point clouds
For 3D object detection in point clouds, methods can be categorized into three groups:
range-based, and voxel-based. Point-based methods 2 3 5 utilize the PointNet series [3] to directly extract geometric features from raw point clouds and make predictions. However, these methods require computationally intensive point sampling and neighbor search procedures. Range- based methods 6
9 10] convert point clouds into pseudo images, thus benefiting from the well-established designs of 2D object detectors. While computationally efficient, these methods often exhibit lower accuracy. Voxel-based approaches [1 hech-performncce 3D object detection. Most voxel-based methods employ sparse CNNs that consist of submanifold and regular sparse convolutions with small kernels to extract features. Regular sparse convolutions can capture distant contextual information but are computationally expensive. On the other hand, submanifold sparse convolutions prioritize efficiency but sacrifice the model's 's ability to capture long-range dependencies.
7
8
19 are currently the leading methods for
2.2 Capturing long-range dependencies for 3D object detection
To capture long-range dependencies for 3D object detection, recent research has explored solutions
such as large-kemell sparse convolutions 33 [6] and self-attention mechanisms [13 14 15). However,
directly applying plain large-kernel CNNs for 3D representation learning can lead to problems such overfitting and reduced efficiency. Weight-sharing strategies have been proposed to mitigate overfitting, like LargeKerne33D [12] and Link [16), however, they still suffer from low efficiency.

---

Emjty fnalure
SB conv
RS cony
Sklp
Sklp
SSR block
Skip Seature
SS conv
RS camv
Down
Expanded feature
(a) SSR block
(b) RSR block
(6) SED block (ours)
Figure 1: Comparison among SSR block (a), RSR block (b), and SED block (c). The 'Skip conn.' denotes the skip connection, and the orange dashed lines represent the convolution kernel space. Valid features have non-zero values. Expanded and empty features have zero values. In (b), convolution is applied to both valid and expanded features, i.e., the convolution kernel center traverses the regions covered by these features. The red dashed square highlights the regions from which the output feature marked by a star can receive information. In (c), we adopt 3 RS convolution with stride of 3 for feature down-sampling (Down) example. UP denotes feature up-sampling.
Other methods, such as SST [4] and DSVT [5 utilize transformers as replacements for sparse CNNs. SST employs a single-stride sparse transformer to preserve high-resolution features without using down-sampling operators that may cause loss of detail. Similarly, DSVT employs single- stride sparse transformer and performs window-basee self-attention sequentially along the X-axis and Y-axis. While both harge-kernel CNNs and transformers aim capture long-rang Long-range dependencies, they either achieve comparable performance or exhibit lower efficiency compared with sparse CNNs. In contrast, proposed HEDNet effectively captures long-range dependencies with the help of encoder-decoder blocks while achieving competitive inference speed compared with existing methods.
2.3 Encoder-decoder networks for dense prediction
The encoder-decoder structure has been extensively investigated in various dense prediction tasks.
For example, FPN-series [2 34 33| 3G] incorporates lightweight fusion modules as decoders to integrate multi-scale features extracted from image classification backbones. Decplab (25) employs an atrous spatial pyramid pooling module to combine low-level features with semantically rich high-level features. SWFormer [33
introduces top-down pathway into its transformer backbone to capture cross-wii window correlations. However, there limited exploration of the encoder-decode structure in the design of sparse CNNs. Most voxel-based approaches [17] [8 27 28 rely high-to- low resolution sparse CNNs to extract single-scale high-level features. Part-A2-Net [37] adopts the UNet (38 extract features, but performs detection using the encoder of UNet while utilizing the decoder for the auxiliary segmentation and part prediction tasks. In this study, we propose HEDNet, which primarily consists of encoder-decoder blocks to effectively capture long-range dependencies.
3 Method
3.1
Background
The sparse CNNs adopted by most voxel-based methods 18 27 28
primarily built by stacking
SSR blocks, each consisting of two submanifold sparse (SS) convolutions (20]. In addition, they usually insert regular sparse (RS) convolutions (21) into the stacked SSR blocks to reduce the resolution of feature maps progressively (similar to ResNet (391).
SS convolution and SSR block. We present the structure of single SSR block Figure I (a). Two SS convolutions are sequentially applied to the input feature map, with skip connections incorporated between the input and output feature maps the SSR block. The sparsity of feature map is defined as the ratio of the regions that are not occupied by valid (nonzero) features to the total area of the feature map. SS convolution only operates on valid features, allowing the output feature map of the SSR block to maintain the same sparsity the input feature map. However, this design hinders the

---

Sparse map
SSR block
conv (stride
Sparse Inverse conv
Dense feature
DR
Dense residual block
Down
Dense reskfusl block (stride
Dense desonv
F((1)
F.C)
Fue
Fs(1)
F-11
Fuft
F
F2U1)
SSR
Down
mn SSt
S5R
DR
Down
Down
200
10
Encodor
Encoder
Decoder
(a) SED block
(b) DED block
Figure 2: Architecture of the SED block (a) and DED block (b). As an example, we illustrate blocks of three scales. Both designs share the same structure. F //////F//////F are the names of the corresponding feature maps. The number in parentheses indicates the resolution ratio of the corresponding feature map relative the block input. The SED block is capable of processing both 2D and 3D features, depending on whether 2D or 3D sparse convolutions are used.
exchange of information among spatially disconnected features. For instance, in the top feature map, the output feature marked by a star cannot receive information from the other three feature points outside the red dashed square in the bottom feature map (marked by the red triangles). This poses challenge for the model in capturing long-range dependencies.
RS convolution and RSR block. One possible solution to problem is to replace the SS convolutions in the SSR block with RS convolutions. We call this modified structure regular sparse residual (RSR) block and illustrate its structure in [ RS convolution operates on both valid and expanded features [1 Expanded features correspond to the features that fall within the neighhorhood of the valid features. Taking 2D RS convolution with a kernel size of 3 3 an example, the neighborhood of a certain valid feature consists of the eight positions around This design leads to an output feature map with a lower sparsity compared with the input feature map. Stacking RS convolutions reduces the feature sparsity dramatically, which in turn leads to a notable decrease in model efficiency compared with using SS convolutions. This is why existing methods [8 usage of RS convolution to feature down- wwwn-sampling layers.
27
28
typically limit the
3.2 SED and DED blocks
SED block. SED block is designed to overcome the limitations of SSR block. The fundamental idea behind this design is reduce the spatial distance between distant features through feature down-sampling and recover the lost details through multi-scale feature fusion.
We illustrate two-scale SED block in Figure (c). After feature down-sampling, the spatially disconnected valid features the bottom feature map are integrated into the adjacent valid features in the middle feature map. An SSR block is subsequently applied to the middle feature map to promote interaction among valid features. Finally, the middle feature map is up-sampled to match the resolution of the input feature map. Note that the feature up-sampling layer (UP) only up-samples features to the regions covered by the valid features in the impul feature map. As a result, the proposed SED block can maintain the same sparsity between input and output feature maps. This characteristic prevents the introduction of excessive computational costs when stacking multiple SED blocks.
The architecture of three-scale SED block is presented in Figure (a). The SED block adopts an asymmetric encoder-decoder structure similar to UNet (38), with the encoder responsible for extracting multi-scale features and the decoder sequentially fusing the extracted multi-scale features with the help of skip connections. Given the input feature map X, the function of the SED block can be formulated follows:
Fi SSR""(X) (X)
F2 SSR m (Down, (F1))
F3 SSR" "M (Down2(Fz))
F, = UP2(F3) + F2
F; = UP (F4) + F
(5)

---

Sparse feature map
Dense feature map
SSR
block
SED
SED block
DED
DED block
Detection head
VFE
SSR
SED
SED
SED
To BEV
nx DED
2x
FL())
F5(2)
Point clouds
F2C
Fd(1)
F((1)
2D dense backbone
sparse backbone
Figure Architecture of the proposed HEDNet. Given the raw point clouds, we first perform voxelization to generate voxels by the VFE module, then employ the 3D sparse backbone and the 2D dense backbone extract features for the detection head. The number in the bracket denotes the resolution ratio of the corresponding feature map relative to the input. The RS convolutions for feature down-sampling that follow the feature maps F,, F2, and F3 are omitted for simplicity.
where F5, denotes the output feature map with the same resolution as the input X. The resolution ratios of the intermediate feature maps F1, F2. F3, and F4 relative to the input X are 1/2, 1/4, and 1/2, respectively. SSR" indicates consecutive SSR blocks. We adopt RS convolution as the feature down-sampling layer (Down) and sparse inverse convolution 40 as the feature up-sampling layer (UP), With an encoder-decoder structure, the SED block facilitates information exchange among spatially disconnected features, thereby enabling the model to capture long-range dependencies.
DED block. Existing peseh-eerfrmacce 3D object detectors 15
28 usually rely on object
centers for detection. However, the feature maps extracted by purely sparse CNNs may have empty holes around object centers, especially for large objects. To overcome this issue, we introduce DED block that expands sparse features towards object centers, as shown in ( The DED block shares a similar structure with the SED block but utilizes the widely used dense convolutions instead. Specifically, we replace the SSR block the SED block with dense residual (DR) block, which is similar to the SSR block but consists of two dense convolutions. Furthermore, the RS convolution employed for feature down-sampling is replaced with a DR block that has a stride For feature up-sampling, we replace the sparse inverse convolution with a dense deconvolution. These modifications enable the DED block to effectively expand sparse features towards object centers.
3.3
HEDNet
Based on the proposed SED block and DED block, we introduce HEDNet, a hierarchical encoder- decoder network designed for 3D object detection. The architecture of HEDNet illustrated in Figure 3 Given the raw point clouds, dynamic VFE module [1] is used to perform voxelization to generate grid of voxels denoted as Fg. Subsequently, sparse backbone including two SSR blocks and several SED blocks is employed to extract 3D sparse features. Before being fed into the 2D dense backbone, the sparse features are compressed into dense BEV features like 18]. The 2D dense backbone, composed of n DED blocks, is responsible for expanding the sparse features towards object centers. Finally, the output features are into the detection head for final predictions. macro level. HEDNet follows hierarchical structure similar to SECOND [18), where the resolution of feature maps progressively decreases. At a micro level, the SED and DED blocks, key components of HEDNet, employ encoder-decoder structures. This is where the name HEDNet comes from. We adopt SED and DED blocks of three scales for HEDNet by default.
Experiments
4.1 Datasets and metrics
Waymo Open contains 160k, 40k, and 30k annotated samples training, validation, and testing, respectively. The metrics for 3D object detection include mean average precision (mAP) and mAP
5

---

Results on validation data set
mPP/mAPH
Vehicle AP/APH
Pedestrian
AP/APH
Cyclist AP/APH
Method
L2
L1
L2
L1
L2
L1
L2
SECOND
18
61.0/5.22
72.3/71.7
63.9/63.3
68. 68.7/58.2
60.7/51.3
60.6/59.3
58.3/57.0
PointPillar
19
62.8/57.8
72.1/71.5
63.6/63.1
70.6/56.7
62.8/50.3
64.4/62.3
61.9/59.9
Lidar-RCNN
63.8/61.3
76.0/75.5
68.3/67.9
71.2/58.7
63.1/51.7
68.6/66.9
66.1/64.4
Part-A2-Net
37
66.9/63.8
77.1/76.5
68.5/68.0
75.2/66.9
66,2/58.6
68.6/67.4 .4
66. 1/6.9
SST [4]
6
74.2/73.8
65.5/65.1
78.7/69.6
70.0/61.7
70.7/69.6 6
68.0/66.9
CenterPoint
27
68.2/65.8
74.2/73.6
66.2/65.7
76.6/70.5
68.8/63.2
72.3/71.1
69.7/68.5
PV-RCNN 43
69.6/67.2
78.0/77.5
69.4/69.0
79.2/73.0
70.4/64.7
71.5/70.3
69.0/67.8
CenterPoint
27
69.8/67.6
76.6/76.0
68.9/68.4
79.0/73.4
71.0/65.8
72.1/71.0 -4-
69.5/68.5
SWFormer
-/-
77.8/77.3
69.2/68.8
80.9/72.7
72.5/64.9
-/-
OcTr 144
70.7/68.2
78.1/77.6
6/.8/69.3
80.8/74.4
72.5/66.5
72.6/71.5
69.9/68.9
PillarNet-34
71.0/68.5
79.1/78.6
70.9/70.5
80.6/74.0
72.3/66.2
72.3/71.2
69.7/68.7
AFDetV2 45
71.0/68.8
77.6/77.1
69.7/69.2
80.2/74.6
72.2/67.0
73.7/72.7
71.0/70.1
CenterFormer 46
71.1/68.9
75.0/74.4
69.9/69.4
78.6/73.0
73.6/68.3
72.3/71.3 -/-
69.8/68.8
LargeKernl33
33
+/-
78.1/77.6
69.8/69.4
-/-
-/-
-/-
PV-RCNN+ 47
71.7/69.5
79.3/78.8
70.6/70.2
81.3/76.3
73.2/68.0
73.7/72.7
71.2/70.2
FSD [48]
72.7/70.5
79.5/79.0
70.3/69.9
83.6/78.2
74.4/69.4
4
73.3/72.1 77.5/76.5
75.3/74.1
ASVT-Voxel 15
74.0/72.1
79.7/79.3
71.4/71.0
83.7/78.9
76.1/71.5
74.6/73.7
HEDNet (ours)
75.3/73.4
81.1/80.6
73.2/72.7
84.4/80.0
76.8/72.6
78.7/77.7
75.8/74.9
Results the test data set
mAP/mAPH
Vehicle AP/APH L1
Pedestrian AP/APH
Cyclist
AP/APH
Method
L2
L2
L1
L2
L1
L2
PV-RCNN
45
71.3/68.8
80.6/80.1
72.8/72.4
78.2/72.0
71.8/66.0
71.8/70.4
69.1/67.8
PV-RCNN+ 47
72.4/70.2
81.6/81.2
73.9/73.5
80.4/75.0
74.1/69.0
71.9/70.8
69.3/68.2
AFDetV2 45
72.2/70.3
80.5/80.0
73.0/72.6
79.8/74.3
73.7/68.6
72.4/71.2
69.8/69.7
FSD 48
74.4/72.4
82.7/82.3
74.4/74.1
82.9/77.9
75.9/71.3
75.6/74.4
72.9/718
HEDNet (ours)
76.9/75.0
84.2/83.8
77.0/76.6
84.1/79.7
78.3/74.0
78.2/77.0
75.4/74.3
Table I: Comparison with prior methods on the Waymo Open dataset (single-frame setting). Metrics: mAP/mAPH (%)f for the overall results, and AP/APH (%)t for each category. i; two-stage method.
weighted by the heading accuracy (mAPH). Both are further broken down into two difficulty levels: L1 for objects with more than five LiDAR points and L2 for objects with at least one LiDAR point.
nuScenes consists of 28k, 6k, and 6k annotated samples for training, validation, and testing, respectively. Mean average precision (mAP) and nuScenes detection score (NDS) are used as the evaluation metries. mAP is computed by averaging over the distance thresholds of 0.5m, Im, 2m, 4m across all categories. NDS is weighted average of mAP and the other five true positive metrics measuring the translation, scaling, orientation, velocity, and attribute errors.
4.2 Implementation details
We implemented our method using the open-source OpenPCDet 49 To build HEDNet, we set the hyperparameter m to 2 for all SED and DED blocks and stacked DED blocks for the 2D dense backbone by default. For 3D object detection on the Waymo Open dataset, we adopted the detection head of CenterPoint and set the voxel size to (0.08m, 0.08m, 051) We trained HEDNet for 24 epochs on the full training set (single-frame) to compare with prior methods. For ablation experiments Section[.4 we trained the models for 30 epochs on a 20% training subset. All models were trained with batch size of 16 on 8 RTX 3090 GPUs. The other training settings strictly followed DSVT [15). For 3D object detection on the nuScenes dataset, we adopted the detection head of TransFusions and set the voxel size to (0.075m, 0.075m, 0.2m). We trained HEDNet for 20 epochs with batch size of 16 on RTX 3090 GPUs The other training settings strictly followed TransFusion-L. 28
4.3 Comparison with state-of-the-art methods
Results on the Waymo Open dataset. We compared the proposed HEDNet with previous methods on the Waymo Open dataset (Table []. On the validation set, HEDNet yielded 1.3% L2 mAP and 1.3%

---

Results on the validation data
Method
NDS
mAP
Car
Truck
Bus
T.I.
C.V.
Ped.
M.T.
Bike
T.C.
B.R.
CenterPoint 27
66.5
59.2
84.9
57.4
70.7
38.1
16.9
85.1
59.0
42.0
69.8
68.3
VoxeINeXt 50
66.7
60.5
83.9
55.5
70.5
38.1
21.1
84.6
62.8
50.0
69.4
69.4
TransFusion-L.
28
70.1
65.5
86.9
60.8
73.1
43.4
25.2
87.5
72.9
57.3
77.2
70.3
HEDNet (Ours)
71.4
66.7
87.7
60.6
77.8
50.7
28.9
87.1
74.3
56.8
76.3
66.9
Results on the test data set
Method
NDS
mAP
Car
Truck
Bus
T.L.
C.V.
Ped.
M.T.
Bike
T.C.
B.R.
PointPillars
19
45.3
30.5
68.4
23.0
28.2
23.4
4.1
59.7
27.4
1.1
30.8
38.9
3DSSD
56.4
81.2
47.2
61.4
30.5
12.6
70.2
36.0
8.6
31.1
47.9
CBGS 51
63.3
52.8
81.1
48.5
54.9
42.9
10.5
80.1
51.5
22.3
70.9
65.7
CenterPoint
27
65.5
58.0
84.6
51.0
60.2
53.2
17.5
83.4
53.7
28.7
76.7
70.9
FCOS-LiDAR 9
65.7
60.2
82.2
47.7
52.9
48.8
28.8
84.5
68.0
39.0
79.2
70.7
HotSpotNet
52
66.0
59.3
83.1
50.9
56.4
53.3
23.0
81.3
63.5
36.6
73.0
71.6
CVCNET 53
66.6
58.2
82.6
49.5
59.4
51.1
16.2
83.0
61.8
38.8
69.7
69.7
AFDetV2 45
68.5
62.4
86.3
54.2
62.5
58.9
26.7
85.8
63.8
34.3
80.1
71.0
UVTR-L 54
69.7
63.9
86.3
52.2
62.8
59.7
33.7
84.5
68.8
41.1
74.7
74.9
VISTA 55
69.8
63.0
84.4
55.1
63.7
54.2
25.1
82.8
70.0
45.4
78.5
71.4
Focals Conv
156
70.0
63.8
86.7
56.3
67.7
59.5
23.8
87.5
64.5
36.3
81.4
74.1
VoxelNeXt [50
70.0
64.5
84.6
53.0
64.7
55.8
28.7
85.8
73.2
45.7
79.0
74.6
TransFusion-L.
70.2
65.5
86.2
56.7
66.3
58.8
28.2
86.1
68.3
44.2
82.0
78.2
LargeKerneI3D
12
70.6
53.8
64.4
29.7
85.9
72.7
46.8
79.9
75.5
Link [6
71.0
66.3
86.1
55.7
65.7
62.1
30.9
85.8
73.5
47.5
80.4
75.5
HEDNet (Ours)
72.0
67.7
87.1
56.5
70.4
63.5
33.6
87.9
70.4
44.8
85.1
78.1
Table 2:
Comparison with prior methods on the nuScenes dataset. Metrics: NDS (%)T and mAP (%)f for the overall results, AP (%)f for each category. "T.L.', 'C.V', 'Ped', 'M.T., 'T.C.' and 'B.R.' denote trailer, construction vehicle, pedestrian, motor, traffic cone, and barrier, respectively.
74
HIDDNet (Ours)
Overall
Detection necuracy (1.2 mAPH)
72
DSVT- Vexel
Improvement ementtss on mAPH
Vihiok
Pedestrion
FSD
Cyelist
70
LangeKemee3D
68
ConterPoint Voxel
66
RCNN
SST
64
CenterPrent-
P2-t-A2-Nett
62
Inference speed (FPS)
Distance to LiDAR sensor (m)
Figure 4: Detection accuracy (L2 mAPH) versus inference speed (FPSt) of different models on the Waymo Open validation set.
Figure 5: Improvements decomposition of HED- Net over HEDNet-single with regard to the dis- tance range of objects to the LiDAR sensor.
1.2 mAPH improvements over the prior best method DSVT- 15 HEDNet also outperformed the two-stage models PV-RCNN+ 47 and FSD 488. More importantly, our method significantly outperformed the transformer- DSVT- Voxel by 1.7% L.2 mAPH on the vehicle category, where the average scale vehicles is 10 times bigger than pedestrians and cyclists.
Results on the nuScenes dataset. We compared HEDNet with previous top performinng methods on the nuScenes dataset (Table On the nuScenes test set, HEDNet achieved impressive results with 72.0% NDS and 67.7% mAP. Compared with TransFusion-L. (which adopts the same head as HEDNet), HEDNet showcased significant improvements, with gain of 1.8% NDS and 2.2% mAP In addition, on the three categories with large objects, namely bus, trailer (T.L.), and construction vehicle (C. V.), HEDNet outperformed TransFusion-L by 4. %, 4.7%, and 5.4% mAP, respectively. These results further demonstrate the effectiveness of our method.

---

Block
Latency
LI mAPH
L2 mAPH
#Block
Latency
LI mAPH
L2 mAPH
RSR block
176 ms
74.61
68.30
0
48 ms
76.13
69.89
SSR block
43
74.42
67.93
1
54 ms
77.56
71.37
SED block
48 ms
76.13
69.89
3
63 ms
77.75
71.64
SSR blockt
49 ms
76.67
70.49
4
67 ms
78.02
71.92
SED blockt
54
77.39
71.37
5
73 ms
77.85
71.77
(a) Effectiveness of the SED block.
(b) Effectiveness of the DED block.
Sparse back.
Dense back.
[L] mAPH|L2 mAPH
#Scale
Latency
LI mAPH
L.2 mAPH
8 SSR blocks
DED block
73,91
67.80
43 ms
76.18
69.88
16 SSR blocks
DED block 73.95
67.82
ms
77.61
71.44
4 SED blocks
1 DED block 75.39
69.42
3
67 ms
78.02
71.92
SED blocks
2 DED blocks
75.62
69.67
78 ms
78.12
72.02
(c) HEDNet with 2D sparse backbone.
(d) Effectiveness of encoder-decoder design.
Table 3: Ablations the Waymo Open. I; with I DED block. In (c), 'back.' denotes backbone. In (d), the gray line denotes the HEDNet-single, and the blue line denotes the default HEDNet.
Inference speed. We further compared HEDNet with previous leading methods in terms of detection accuracy and inference speed, as depicted in Figure Remarkably, HEDNet achieved superior detection accuracy compared with LargeKerneI3D [2] and DSVT- Voxel [.5 with faster inference speed. Note that LargeKernel31 and DSVT- Voxel were developed based on large-kernel CNN and transformer, respectively. All models were evaluated on the same NVIDIA RTX 3090 GPU.
4.4 Ablation studies
To better investigate the effectiveness of HEDNet, we constructed two network variants: HEDNet- single and HEDNet-2D. For the HEDNet-single, we replaced all the SED and DED blocks in HEDNet with single-scale blocks, i.e., only keeping the first m SSR/DR blocks in each SED/DED block. For the HEDNet-2D, we replaced all 3D sparse convolutions HEDNet with 2D sparse convolutions and removed the three feature wwwn-sampling layers that follow the feature maps F1, F2, and Fs, following the settings DSVT [5]. The two SSR blocks after Fo were also removed. In HEDNet-2D. the resolution of the output feature map of the 2D dense backbone is same as that of the network input Fo. We conducted experiments on the Waymo Open dataset to analyze various design choices of HEDNet. All models were trained on a 20% training subset and evaluated on the validation set.
4.4.1 Model designs
Effectiveness the SED block.
compared the models built with RSR block, SSR block, and our proposed SED block in Table 3 (a). For the models with RSR/SSR blocks, we replaced the SED blocks in HEDNet with RSR/SSR blocks. The dense backbones the first three models removed to fully explore the potential of the three structures. The first model with RSR blocks achieved slightly better results than the second model with SSR blocks but with much higher runtime latency. The third model with SED blocks significantly outperformed the second model with SSR blocks by 1.96% L2 MAPH. Similar gains can be observed in the last two models with DED blocks.
Effectiveness of the DED block. The DED block is designed to expand sparse features towards object centers. We compared the models that include different numbers of DED blocks in Table (b). The models with DED blocks achieved large improvements over the model without DED blocks. The model with five blocks performed worse than the model with four blocks. The former may be overfitted the training data. We adopted four DED blocks for HEDNet by default.
HEDNet with 2D sparse backbone. We conducted experiments on HEDNet-2D to evaluate the effectiveness of our method with 2D inputs. the construction of 2D inputs, we set the voxel size to (0.32m, 0.32m, 6m), where the size of 6m in the Z. axis corresponds to the full size of the input point clouds. To compare our SED blocks with SSR blocks, we replaced each SED block in HEDNet-2D with SSR blocks or 4 SSR blocks, resulting in two models of different sizes (the first two models in

---

llarge object
large object
object with a few points
distant objects
Figure 6: Qualitative results on the Waymo Open. The red boxes are annotated by humans. The blue boxes and green boxes are predicted by HEDNet and the HEDNet-single, respectively. Red points correspond the points that fall inside the annotated boxes. HEDNet predicted more precise bounding boxes for the objects marked by red arrows than the single-scale variant HEDNet-single.
Table 3 (c)). From Table (c), we can make the following observations. Firstly, the model with 16 SSR blocks achieved similar performance the model with 8 SSR blocks, indicating that stacking more SSR blocks could not further boost performance. Secondly, the models incorporating SED blocks showed significant improvements over the models using SSR blocks (at least 1.6% gains on L2 mAPH). This observation demonstrates the effectiveness of our SED block. Thirdly, stacking two DED blocks achieved better performance than using a single one. These results clearly demonstrate the generality and effectiveness of our proposed SED block and DED block.
4.4.2 HEDNet versus HEDNet-single
We conducted thorough comparison between the proposed HEDNet and its single-scale variant, HEDNet-single, to explore the effectiveness of the encoder-decoder structure and investigate which objects benefit from HEDNet the most. Please note that the HEDNet is designed to capture long-range dependencies among features in the spatial space, which is the core of this work.
Firstly, compared the models built with blocks of different numbers of scales explore the effectiveness of the encoder-decoder structure. As shown in Table (d), the models with multi-scale blocks significantly outperformed the single-scale variant HEDNet-single (the line in gray color). Using more scales achieved better performance, but introduced higher runtime latency. To strike balance between accuracy and efficiency, we adopted three-scale blocks for HEDNet by default.
Secondly, we evaluated the three-scale HEDNet and the HeDNet-single in Table (d) separately for each category and analyzed the results based on the distance range of objects the LiDAR sensor. We illustrate the accuracy improvements of HEDNet over HEDNet-single at various distance ranges in Figure 5 Firstly, HEDNet showed significant improvements over HEDNet-single on the vehicle category, where the size of vehicles is 10 times larger than that of pedestrians and cyclists. This highlights the importance of capturing long-range dependencies for accurately detecting large objects. Furthermore, HEDNet achieved larger performance gains on distant objects compared with objects closer to the LiDAR sensor across all three categories. We believe this is because distant objects with fewer point clouds require more contextual information for accurate detection. Overall, these results demonstrate the effectiveness our proposed method in detecting large and distant objects.
Thirdly, we further present some visualization results of the two models in Figure 6 HEDNet-single exhibited limitations in accurately predicting boxes for large objects and the predicted boxes often

---

only covered parts of the objects (see the top row). In addition, when dealing with objects containing few points, HEDNet-single struggled to accurately estimate their orientations (see the bottom row). In contrast, HEDNNet predicted more precise bounding boxes for both scenarios, which we believe is owed the ability of HEDNet to capture long-range dependencies.
Conclusion
We propose sparse encoder-decoder structure named SED block to capture long-range dependencies among features the spatial space. Further, we propose a dense encoder-decoder structure named DED block expand sparse features towards object centers. With the SED and DED blocks, introduce a hierarchical encoder-decoder network named HEDNet for 3D object detection in point clouds. HEDNet achieved new state-of-the-ar performance on both the Waymo Open and nuScenes datasets, which demonstrates the effectiveness of our method. We hope that our work can provide some inspiration for the backbone design in 3D object detection.
Limitations HEDNet mainly focuses on 3D object detection in outdoor autonomous driving scenarios. However, the application of HEDNet other indoor applications is still an open problem.
This work was supported in part by the National Key Research and Devel- opment Program China (No. 20212D0200301) and the National Natural Science Foundation of China (Nos. U19B2034, 61836014) and THU-Bosch JCML center.
Acknowledgements
References
Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum pointnets for 3d object detection from rgb-d data. CVPR, 2018.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointremn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
Charles Qi, Or Litany, Kaiming He, and Leonidas J Guibas, Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast point r-enn. ICCV, 2019.
[6] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu. Range condi- tioned dilated convolutions for scale invariant 3d object detection. CoRI., 2020.
[7] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and ZhaoXiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In ICCV, 2021.
[8] Pei Sun, Weiyue Wang. Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchis- escu, and Dragomir Anguelov. RSN: Range sparse net for efficient, accurate lidar 3d object detection. In CVPR, 2021.
[9] Zhi Tian, Xiangxiang Chu, Xiaoming Wang, Xiaolin Wei, and Chunhua Shen. Fully convolutional one-stage 3d object detection on liDAR range images. N/PS, 2022.
[10] Danila Rukhovich, Anna Vorontsova, and Anton Konushin, Feaf3d: fully convolutional anchor-free 3d object detection. In ECCV, 2022.
[11] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillamnt: Real-time and hegh-performance pillar- based 3d object detection. In FCCVV, 2022.
[12] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels 3d cnns. In CVPR, 2023.
[13] Pei Sun, Mingxing Tan. Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV, 2022.
[14] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Embracing Single Stride 3D Object Detector with Sparse Transformer. In CVPR, 2022.
10

---

[15] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang. Di He. Bernt Schiele. and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In CVPR, 2023.
[16] Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang. Link: Linear kernel for lidar-based 3d perception. In CVPR, 2023.
[17] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
[18] Yan Yan, Yuxing Mao, and Bo Li, Second: Sparsely embedded convolutional detection. In Sensors, 2018
[19] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[20] Benjamin Graham and Laurens Van der Maaten. Submanifold sparse convolutional networks. arXiv preprint 170.0v:0006.130000 2017.
[21] Ben Graham, Sparse 3d convolutional neural networks. arXiv preprint arXiv:150..02890, 2015.
[22] Tsung-Yi Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
[23] Fisher Yu, Dequan Wang. Trevor Darrell, and Yi Zhou. Deep layer ageregation. CVPR, 2018.
[24] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[25] Liang-Chieh Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[26] Gang Zhang. Ziyi Li, Jianmin Li, and Xiaolin Hu. Cfnet: Cascade fusion network for dense prediction. arXiv preprint arXiv:2302.06052 2023.
[27] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based object detection tracking. In CVPR, 2021.
[28] Xuyang Bai, Zeyu Hu, Xinge Zhu Qingqiu Huang. Yilun Chen, Hongbo Fu, Chiew-Lan Tai. Transfu- sion: Robust iidar-camera fusion for 3d object detection with transformers. In CVPR, 2022.
[29] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[30] Holger Cacsar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Veniee Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[31] Charles Qi. Hao Su. Kaichun Mo. Leonidas Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[32] Charles Qi, Li Yi, Hao Su, Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NewrlPS, 2017.
[33] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels 31x31: Revisiting large kernel design in cnns. CVPR, 2022.
[34] Mingxing Tan. Ruoming Pang. and Quoe Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020.
[35] Golnaz Ghiasi, Tsung-Yi Yi Lin, Ruoming Pang, and Quoe Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In CVPR, 2019.
[36] Xinqi Fan, Mingjie Jiang, Raza Shahid,
Hong. Hierarchical scale convolutional neural network for facial expression recognition. In Cognitive Neurodynamicss 2022.
[37] Shaoshuai Shi, Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part- augreagggegtioon network. In TPAMI, 2019.
11

---

[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[40] Spconv Contributors. Speonv: Spatially sparse convolution library. traveller53/spcoon 2022.
https://github. com/
[41] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In CoRL. 2019.
[42] Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cnn: An efficient and universal 3d object detector. In CVPR, 2021.
[43] Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renm: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020.
[44] Chao Zhou, Yanan Zhang, Jiaxin Chen, and Di Huang. Octr: Octrec-based transformer for 3d object detection. CVPR, 2023.
[45] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. AAAI, 2022.
[46] Zixiang Zhou, Xiangehen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection, In ECCV, 2022.
[47] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Py-renn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. In IJC,, 2023.
[48] Lue Fan, Feng Wang. Naiyan Wang, and Zhaoxiang Zhang. Fully Sparse 3D Object Detection. NeurlPS, 2022.
[49] OpenPCDet Development Team. Openpedet: An open-source toolbox for 3d object detection from point clouds. https://gitthi com wtt.cccm///oc/eecccc http///peemmmmbbapeaeeeeeeee 2020.
[50] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR, 2023.
[51] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class- balsnca grouping and sampling for point cloud 3d object detection. In arXiv preprint arXiv: 0949,, 2019.
[52] Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Vuille. Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots. arXiv preprint arXiv: 1912 2020.
[53] Xiangyuan Zhu, Kchua Guo, Hui Fang, Liang Chen, Sheng Ren, and Bin Hu. Cross view capture for stereo image super-resolution. In IEEE Transactions on Multimedia, 2022.
[54] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representa- tion with transformer for 3d object detection. In NIPS, 2022.
[55] Shengheng Deng, Zhihao Liang. Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In CVPR, 2022.
[56] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In CVPR, 2022.
[57] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[58] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019.
[59] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke. Cyrill Stachniss, and Jurgen Gall. Semantickiti:: A dataset for semantic scene understanding of lidar sequences. In ICCV, 2019.
[60] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In ICCV, 2023.
12

---

Other ground
Motorcycle
Motorcyclist
Vegetation
Traffic sign
Car
Bicycle
Truck
Bus
Person
Bicyclist
Road
Parking
Sidewalk
Building
Fence
Trunk
Terrian
Pole
Method
mloU
MinkUNet34 [38]
96.8
55.0 81.4 83.2 70.2 79.5 89.8 7.8 94.8 54.6 82.8 1.5 92.0 68.3 87.9 69.4 72.8 66.1 52.4 68.3
HEDNet (Ours)
97.3 57.2 82.3 88.1 73.9 80.4 91.3 23.2 95.1 51.5 83.1 2.8 92.1 69.6 87.6 69.4
72.3 66.6 52.1 70.3
Table 4: 3D semantic segmentation results the SemanticKiTTI validation set. Metrics: mloU (%)1 for the overall results, loU (%)f for each category.
A Implementation details on 3D object detection
We implemented our method with Pytorch using the open-source OpenPCDet [49
Waymo Open dataset. We set the hyperparameter m to 2 for all SED and DED blocks and stacked 4 DED blocks for the 2D dense backbone by default. We adopted the detection head of CenterPoint 27 for HEDNet. As metioned in the main paper, we primarily followed the training and inference schemes of DSVT [1]] Specifically, the voxel size and the detection range was set to [-75.2m, 75.2m] for X and axis, and [-2m, 4m] for Z axis. We trained HEDNet for 24 epochs the entire training dataset and reported the evaluation results on the validation set to compare with previous state-of-the-ee methods. For the ablation experiments, we trained all models for 30 epochs on a 20% training subset. All models were trained with batch size of 16 on 8 RTX 3090 GPUs. We employed the Adam [57] optimizer with a one-cycle learning rate policy, and set the weight-decay to 0.05, and the max learning rate to 0.003. We also adopted the faded training strategy in the last epoch. During inference, we applied class-specific NMS with an IoU threshold of 0.75, 0.6 and 0.55 for vehicle, pedestrian, and cyclist, respectively.
(0.08m, 0.08m, 0.15m),
nuScenes dataset. We set the hyperparameter m 2 for all SED and DED blocks and stacked DED blocks for the 2D dense backbone. We adopted the detection head of TransFuuion--L. 28 for HEDNet and primarily followed the training and inference schemes of TransFusion-L (28) The voxel size was set to (0.075m. 0.075m, 0.2m), and the detection range was set to [-54m, 54m] for X and Y axis, and [-5m, 3m] for Z axis. We trained HEDNet for 20 epochs the combined training and validation sets with a batch size of 16 on 8 RTX 3090 GPUs and reported the results on the test set to compare with other methods. We employed the Adam [5 optimizer with one-cycle learning rate policy, and set the decyyt-decay to 0.1, the momentum to [0.85, 0.95], and the max learning rate to 0.001. The faded strategy was used during the last epochs. For submission to the test server, set the query number of detection head to 300 and did not use any test-time augmentation.
B Experiments on 3D semantic segmentation
We conducted experiments on the popular LiDAR semantic segmentation dataset SemanticKiTTI [59] It provides 22 sequences with 19 semantic classes, captured by a 64-beam LiDAR sensor. Following the standard practice [58 60), we report the Intersection-ors Cociiion-vvr--nnniii Union (IoU) for each category and the average score (mloU) over all categories. For the backbone network, we employed a UNet-style structure, i.e., the same designs as the first two layers of the Nee33 are first adopted to extract sparse features with a spatial down- down-sampling ratio of 4, followed by 4 SED layers to transform the resulting features, finally, two symmetrical layers are used to recover high-resolution features following MinkUNet34. The other settings strictly followed (58). Table 4 shows that the proposed model exhibited significant gains over its counterpart MinkUNet34 (i.e., 2.0% in mloU), which demonstrated the generality of our method.
C A step-wise ablation from VoxeINet to HEDNet
We conducted a step-wise ablation from the standard VoxeINet to our HEDNet on the Waymo Open dataset show the effectiveness of different components (see Table For the second model, employed the training tricks used by DSVT, including loU loss, class-specific NMS, faded strategy (disabling data augmentations in the last epoch), and weight decay of 0.05. These training tricks
13

---

No. VoxelNet Tricks" Smaller-voxel SED-block DED-block Full-data
Latency
L1 MAPH L.2 MAPH
40 ms
70.0
64.0
2
40 ms
75,4
69.1
49
76.6
70.2
v
55
77.6
71.3
ms
78.0
71.9
67 ms
79.5
73.4
Table 5: A step-wise ablation from VoxelNet to HEDNet. The first five models were trained on 20%
training subset and the last model was trained on the full training set. ": +training tricks used DSVT.
significantly boost detection accuracy. Actually, most of the training tricks have been used by previous works, such as PV-RCNN++, and FSD. The codes and training configurations of the DSVT model be found in the OpenPCDet archives. For the third model, adopt smaller input voxel size to keep more detailed information, which boosts the detection accuracy of pedestrian and cyclist. proposed SED blocks and DED blocks. The last model was trained on the full training set. Our final model (the 6th model) outperformed the previous SOTA method DSVT by 1.3% L2 mAPH while being 50% faster.
The 4th and 5th models sequentially incorporate
14


Reviews:
[{"review_id": "y7F1fraMpz", "reviewer": "Reviewer_k2sW", "summary": "The paper addresses the task of 3D object detention on real-world 3D inputs that possess challenges such as scalability to long-range dependency. The main suggestion is to incorporate encoder-decoder type blocks to capture long-range dependency, in contrast to previous works that were limited to the sparsity patterns in the input. The method is evaluated on 2 real-world 3D detection benchmarks.", "questions": "No specific quesitons", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The introduction discusses previous approaches, their limitations, and how the current suggestion can mitigate them.\n\nThe method is evaluated on real-world challenging benchmarks.\n\nI appreciate the incorporation of the qualitative evaluation and the ablation study.\n\nThe proposed solution seems to be simple and valuable.\n", "weaknesses": "Missing discussion\n\nIt is not clear what part of the design choices stemmed from the fact the architecture is aimed to tackle object detection. It would be beneficial to incorporate such as discussion. In this context, the comment on the limitation section is a bit vague.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Q1. It is not clear what part of the design choices stemmed from the fact the architecture is aimed to tackle object detection? And the comment on the limitation section is a bit vague?\nSince we only conducted experiments on 3D object detection in the paper, we didn't know whether the proposed HEDNet was suitable for other 3D tasks. This is the limitation we want to express. As reminded by you and the other two reviewers vKEU and pAFm, we realize that HEDNet can actually work on other 3D tasks. During the rebuttal period, we conducted experiments on the LiDAR-based 3D semantic segmentation task. The results show that HEDNet also performed well on semantic segmentation, effectively addressing the aforementioned limitation. For more detailed information, please refer to Q4 of the reviewer vKEU.\n"}, {"review_id": "AS43rY6QRB", "reviewer": "Reviewer_zRCr", "summary": "The proposed method considers a much more straightforward approach to incorporating larger context for 3D detection from point clouds. In contrast to existing work which consider large kernels or transformers to increase the receptive field of 3D backbones, this work focuses on developing an encoder-decoder block, taking care to maintain input and output sparsity levels. The resulting architecture achieves new state-of-the-art on LiDAR-only Waymo and nuScenes benchmarks with high efficiency.", "questions": "Please see Weaknesses above. Overall, the proposed method has solid performance with high efficiency, opting for a more straightforward backbone that demonstrates better performance than existing large-kernel or transformer structures. As such, I recommend acceptance at this stage.", "limitations": "Limitations have been included.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written, and the relevant modules are illustrated clearly.\n- The resulting pipeline achieves strong performance while being efficient and straightforward, not involving complicated modules.\n- Extensive ablations validate the importance of the SED block, as well as the # of DED blocks.\n", "weaknesses": "- While there is some analysis of what types of objects improved with the scale in the encoder-decoder block, it would be useful to see a more fine-grained analysis on nuScenes, which contains more object categories.\n- Table 3a’s row 2 appears to be similar, or same, as the standard 3D sparse convolutional backbone used in prior works. In that case, the overall model should be similar to the CenterPoint model, which on 20% data on OpenPCDet archives (70.76 + 65.49 + 67.39) / 3 = 67.88~ L1 mAPH. Why is the performance in Table 3a row 2 much better, at 74.42? Are there other differences between the two models?\n- Building on the previous point, it would be useful to see a step-by-step ablation from the standard 3D sparse convolutional backbone to the proposed backbone, to demonstrate more clearly where the improvements/latency changes are with respect to the most commonly used 3D spconv backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Q1. A more fine-grained analysis on nuScenes?\n\n|||||||||||||\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|Model|mAP|Car|Truck|Cons.veh|Bus|Trailer|Barrier|Motor.|Bicycle|Ped.|Tr.Cone|\n|TransFusion-L|65.2|86.5|60.8|25.7|74.2|41.6|70.2|71.6|56.8|87.1|77.1|\n|HEDNet (Ours)|66.9|87.8|61.4|27.7|77.6|48.0|69.9|72.6|58.9|87.2|77.6|\n|||||||||||||\n|+improvements|+1.7|+1.3|+0.6|**+2.0**|**+3.4**|**+6.4**|-0.3|+1.0|+2.0|+0.1|+0.5|\n| avg object size ($m^3$)||16.0|56.0|73.0|115.4|151.2|1.3|2.3|1.4|0.9|0.2|\n\nNote: The column mAP shows the overall results, and the other columns show AP of each category.\n\nSince our baseline model on the nuScenes dataset (TransFusion-L) uses a sparse backbone similar to that of HEDNet-single, we directly compare HEDNet with TransFusion-L. The two models only differ in their backbones.\n1) We present the detailed results on the nuScenes validation set in the above table. HEDNet significantly outperformed TransFusion-L on large objects, especially the categories Trailer, Bus, and Construction vehicle.\n2) In addition, we divided objects into three groups by object distances, *i.e.* 0-15, 15-30, and 30+ meters. HEDNet outperformed TransFusion-L by 1.6%, 2.0%, and 2.8% mAP in the three groups, respectively, indicating that HEDNet benefits the detection of distant objects more.\n\n#### Q2. A step-by-step ablation from the standard 3D sparse convolutional backbone to the proposed backbone?\n\n|||||||||||\n|-|-|-|-|-|-|-|-|-|-|\n|No.|Model|FPS|L2 mAPH|Veh/L1 APH|Veh/L2 APH|Ped/L1 APH|Ped/L2 APH|Cyc/L1 APH|Cyc/L2 APH|\n|1|CenterPoint (ResNet)|25.0 |64.0|72.2|64.4|68.0|60.3|69.8|67.3|\n|2|+DSVT tricks|25.0|69.1|76.5|68.2|75.8|67.9|73.8|71.1|\n|3|+Smaller voxel|20.4|70.2|77.1|68.7|77.3|69.6|75.2|72.4|\n|4|+SED blocks|18.1|71.3|78.4|70.2|78.3|70.8|75.9|73.1|\n|5|+DED blocks|15.0|71.9|79.5|71.4|78.2|70.8|76.4|73.6|\n|6|+100% data|15.0|73.4|80.6|72.7|80.0|72.6|77.7|74.9|\n||||||||||||\n|*|DSVT-Voxel (100%)|10.1|72.1|79.3|71.0|78.9|71.5|76.5|73.7|\n\nNote: L2 mAPH is the metric for the overall results. FPS denotes frame per second, the metric for measuring inference speed.\n\nWe conducted a step-by-step ablation on the Waymo Open dataset (20% training data for the first five models, and 100% training data for the last two models).\n1. The results of the first model are from the OpenPCDet archives.\n2. The second model employed the training tricks used by DSVT, including IoU-aware loss, multi-class NMS, faded strategy (disabling data augmentations in the last epoch), and a bigger weight decay value. These training tricks can significantly boost detection accuracy. Actually, most of the training tricks have been used by previous works, such as PV-RCNN++, and FSD. The codes and training configurations of the DSVT model can also be found in the OpenPCDet archives.\n3. For the third model, we adopt a smaller input voxel size to keep more detailed information, which boosts the detection accuracy of pedestrian and cyclist.\n4. The 4th and 5th models sequentially incorporate our proposed SED blocks and DED blocks.\n5. The last two models were trained with the full training set. Our final model (the 6th model) outperformed the previous SOTA method DSVT by 1.3% L2 mAPH while being 50% faster.\n\n#### Q3. The performance of the model in Table 3a row 2 is much better than the CenterPoint model on OpenPCDet archives?\nAll models in Table 3 were trained with the aforementioned settings (DSVT tricks and smaller input voxel size), thus significantly outperforming the models released on the OpenPCDet archives.\n"}, {"review_id": "bz0NCqGKrx", "reviewer": "Reviewer_vKEU", "summary": "This paper focuses on the long range problems in 3D object detection. Compared to the common idea of large kernel or transformers, this hierarchical design is more smart and efficient. It designs an encoder-decoder style network for 3D object detection. Comprehensive experiments on nuScenes and Ways show the effectiveness of the proposed methods. Detailed ablation studies show the efficiency compared to large kernel and transformer baselines.", "questions": "1. If the authors choose to skip the experiments on the suggestion of 3/4 in the weakness. Please at least provide a discussion on them, which would be helpful to further improve the quality of this paper.", "limitations": "It is a bit not elegant that the author implement this idea upon two separate codebases on different datasets (OpenPCDet to Waymo, mmdet3d to nuScenes). A kind suggestion is that you could unified the codebase when you released the code. For example, you could reproduce the results of nuScenes on OpenPCDet, and use the unified one to release.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written. The idea is direct and easy to follow. Figures and tables are well-organized.\n\n2. The idea of encoder-decoder is also much smart. Compared to large kernel or transformers, this design seems more relevant to the task 3D object detection itself. That is also why it is more efficient than baselines. This simple design might be motivated from semantic segmentation in 2D images.\n\n3. Experimental results on nuScenes and Waymo are state-of-the-art. These are good to show the effectiveness of the proposed methods.", "weaknesses": "1. The visualization comparison on receptive fields should be provided. As this work claimed that it achieves similar or better long-range detection capability than large kernel and transformers. This visualization is necessary to support this claim.\n\n2. A kind suggestion is that the results on nuScenes validation set should also be released. This would facilitate the following works to compare with.\n\n3. Fully sparse styles are gradually becoming popular, as FSD and VoxelNeXt. It would be curious that whether the idea will work if there are only sparse SED. Because the dense head limits the detection range, (like Argoverse2 datasets with 200m radius), as analyzed in FSD. It would be much helpful that if this work could be extended to be fully sparse.\n\n4. This idea is very related to the area of semantic segmentation. I think this framework might simultaneously work with lidar semantic segmentation,  with different head. For example, like LIdarMultiNet, training and inference together on both detection and segmentation on nuScenes and waymo. I know that this suggestion might not be validated well in the limited rebuttal period. Please just ignore this if you do not have enough time for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Q1. The visualization comparison on receptive fields should be provided. As this work claimed that it achieves similar or better long-range detection capability than large kernel and transformers.\n\nWe have presented some visualization examples in the provided PDF file (refer to `Author Rebuttal by Authors`, the global rebuttal filed above). To illustrate receptive fields, we back-propagated gradients from the output BEV feature that corresponds to each object center. In each sub-figure, the red points indicate the receptive field corresponding to the selected object center.\n\nWe conducted a comparison among the proposed HEDNet, HEDNet-single, and DSVT to underline the effectiveness of our approach. HEDNet-single is a single-scale variant of HEDNet, employing a backbone akin to the standard sparse backbone (SECOND). Further details about HEDNet-single can be found in the first paragraph of Section 4.4 of the paper. DSVT, the prior state-of-the-art method for 3D object detection on the Waymo Open dataset, utilizes a transformer-based backbone. As LargeKernel3D exhibits inferior performance compared with DSVT, we only consider DSVT here.\n\nThe visualized examples highlight that both HEDNet and DSVT can capture longer-range dependencies than HEDNet-single. Additionally, HEDNet achieves similar or slightly larger receptive fields than DSVT. To better bolster our claim, we calculated the average distance from the farthest K red points to their corresponding object center. With K set to 30, the average distance ranking across the three models was as follows: HEDNet (38.8m) > DSVT (28.2m) > HEDNet-single (22.1m). These results demonstrate that our proposed method exhibits better long-range detection capabilities than HEDNet-single and DSVT. All analyses were conducted on the Waymo validation set.\n\n#### Q2. Results on nuScenes validation set\n\n|||||||||||||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|No.|Model|NDS|mAP|Car|Truck|Bus|Trailer|Cons.|Ped.|Motor.|Bicycle|Tr.Cone|Barrier|\n|1|TransFusion-L|70.2|65.2|86.5|60.8|25.7|74.2|41.6|70.2|71.6|56.8|87.1|77.1|\n|2|HEDNet (Ours)|71.4|66.9|87.8|61.4|27.7|77.6|48.0|69.9|72.6|58.9|87.2|77.6|\n\nThe first model is our baseline TransFusion-L, which employs the standard 3D sparse backbone. The second model is the proposed HEDNet. The two models were trained with the same settings, and they only differ in their backbones.\n\n#### Q3. Experimental results on VoxelNeXt.\n\n||||||||||\n|-|-|-|-|-|-|-|-|-|\n|No.|Model|L2 mAPH|Veh/L1 APH|Veh/L2 APH|Ped/L1 APH|Ped/L2 APH|Cyc/L1 APH|Cyc/L2 APH|\n|1|VoxelNeXt|70.8|77.6|69.3|77.9|70.4|75.4|72.6|\n|2|Our model|78.2|72.1|79.8|71.6|78.3|70.8|76.7|73.9|\n||+improvements|+1.3|**+2.2**|**+2.3**|+0.4|+0.4|+1.3|+1.3|\n\nNote: L2 mAPH is the metric for the overall results.\n\nWe agree with your opinion about the fully sparse detectors and conducted experiments with VoxelNeXt on the 100% Waymo Open dataset. We replaced the 3D sparse backbone of VoxelNeXt with our proposed sparse backbone. Our model (the second model) outperformed the standard VoxelNeXt by 1.3% L2 mAPH, demonstrating that our method also performed well on fully sparse detectors.\n\n#### Q4. Experimental results on LiDAR-based 3D semantic segmentation.\n\n||||||||||||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|No.|Model|mIoU|Motorcyclist|Truck|Bicycle|Motorcycle|Bicyclist|Trunk|Person|Sidewalk|Traffic sign|Building|\n|1|MinkUNet34|68.3|9.9|80.2|52.8|80.4|89.8|68.3|79.3|83.1|52.3|92.3|\n|2|Our model|70.3|23.2|88.1|57.2|82.3|91.3|69.4|80.4|83.1|52.1|92.1|\n||+improvements|+2.0|+13.3|+7.9|+4.4|+1.9|+1.5|+1.1|+1.1|+0.0|-0.2|-0.2|\n\nNote: mIoU is the metric for the overall results.\n\nDue to the limited rebuttal period, we only conducted LiDAR-based 3D semantic segmentation experiments on the SemanticKITTI dataset, which has a smaller size than the nuScenes and Waymo Open datasets. We adopted the well-known model MinkUNet implemented in MMDetection3d as our baseline (the first model). For the second model, we replaced the sparse UNet of the first model with three SED blocks. Both models were trained with the same settings and only differ in their backbones. Our model outperformed the plain MinkUNet by 2.0% mIoU. This validates the effectiveness of our design on LiDAR-based 3D semantic segmentation.\n\n#### Q5. Unify the codebase?\nThanks for your suggestion. We will unify the codebase to release our codes."}, {"review_id": "OABOEImEsd", "reviewer": "Reviewer_Nfby", "summary": "The paper proposed a Hierarchical Encoder-Decoder Network, which faciliated sparse encoder-decoder block to enhance the network to model long-range dependencies among sparse voxel features while maintaining high sparsity. The method consistently improve the detection performance with a higher detection rate.", "questions": "It is nice to witness a simple idea being effectively work within3D object detection. My major concern revolve around the technical soundness of this encoder-decoder design and its ability to truly preserve sparsity and save computation.\n\nIt would be better to anonymously publish the results in the NuScenes and Waymo leaderboard.", "limitations": "The author addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. The idea is simple and efficient.\n2. Clearly identify the issue among current detection model w.r.t the submanifold sparse conv and regular sparse conv.\n", "weaknesses": "1. Since the authors adopt a 3×3 RS convolution with a stride of 3 for feature down-sampling (Down), this will definitely introduce more non-zero elements in the resulted downsampled feature. In relation to the original convolutional block in SECOND, which involves performing SSxN + RS(down), I am curious about how the SED block, encompassing SS + RS(down) + SS + RS(down) + SS + UP + UP, can effectively reduce the computation and maintain the sparsity. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Q1. How the SED block can effectively reduce the computation and maintain the sparsity?\n1. As you said, each SED block encompasses SS + RS_1(down) + SS + RS_2(down) + SS + UP_2 + UP_1. To distinguish different Down and Up layers, we add subscripts here. We adopt the sparse inverse conv implemented in the open source toolboxes [spconv](https://github.com/traveller59/spconv) as the UP layer, and it is the key for SED block to maintain feature sparsity. Each UP layer is bound with the Down layer which has the same subscript, and it only upsamples features to the same non-zero positions as that of the features before the bound Down layer. As a result, the input and output features of each SED block have the same feature sparsity.\n\n2. The sparse backbone of SECOND includes several convolutional blocks, where each block involves performing SSxN + RS(down). Our proposed SED block is designed to replace the N consecutive SS layers before the RS layer. Compared with the sparse backbone in SECOND, our design has a slightly higher computation cost (only 5ms, see Table 3 (a) in the paper, row 2 *vs.* row 3). But our final HEDNet is still more efficient than previous methods that incorporate large kernel convolutions or self-attention mechanisms, such as LargeKernel3D and DSVT, while achieving higher detection accuracy. (see Figure 4 in the paper)\n\n#### Q2. It would be better to anonymously publish the results in the NuScenes and Waymo leaderboards.\nWe have anonymously published the results in the NuScenes and Waymo leaderboards and sent the links to AC following the rules of the rebuttal. The results on the nuScenes leaderboard were the same as that presented in Table 2 of the paper. The results on the Waymo leaderboard were inferred from the test set by the model presented in Table 1 of the paper (only used the training set for training, based on single-frame data). As we are not sure whether you can see the link, we present the results on the Waymo leaderboard as follows:\n\n||||||||\n|-|-|-|-|-|-|-|\n|L2 mAPH|Veh/L1 APH|Veh/L2 APH|Ped/L1 APH|Ped/L2 APH|Cyc/L1 APH|Cyc/L2 APH|\n|73.8|83.4|76.0|79.0|73.3|74.8|72.1|\n\nNote: L2 mAPH is the metric for the overall results."}, {"review_id": "ZRL4Cf5pMT", "reviewer": "Reviewer_pAfm", "summary": "This paper focuses on 3D object detection using outdoor LiDAR data. The authors proposed a multi-scale network that is built upon several encoder-decoder blocks, e.g., SED and DED. The motivation behind it is to use sub-manifold sparse feature down-sampling and up-sampling steps to capture long-range voxel relationships without greatly sacrificing computation efficiency. The authors benchmark their model, HEDNet, on both Waymo Open Dataset and nuScenes and demonstrate competitive detection quality.", "questions": "[Technical Clarity]\n\n1. L-146 to L-147: “we adopt RS convolution as the feature down-sampling layer (DOWN)”. However, Figure 2 (a) says that the down-sampling layer is SS conv, instead of RS conv as explained in the text. They are conflicting with each other. Which one is correct?\n\n2. Table-1: are the presented results all based on single frame or multiple frames? This is not mentioned in the main paper.", "limitations": "N/A.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "[Strong Benchmark Performance]\n\n1. This paper demonstrates strong detection quality on standard benchmarks. For example, HEDNet outperforms the prior SOTA DSVT while being 50% faster.\n\n[Method Novelty]\n\n2. Essentially, the network architecture plotted in Figure 3 can be thought as some type of “hierarchical stack-hourglass” models. Each block of SED and DED modules is a tiny \"hourglass\" with skip connections.  In that sense, HEDNet differentiates itself from other 3D object detectors which mostly follow FPN structures (e.g., SWFormer) or single-stride designs (e.g., SST). I agree that HEDNet brings some fresh air into this field.", "weaknesses": "[Lack of Insights or Theoretical Contributions]\n\n1. While acknowledging the method novelties of HEDNet as mentioned above, I also think this work might lack insights or theoretical contributions. For example, I’m not sure the findings/designs are something that is sufficient for a NeurIPS publication or is generalizable to other 3D deep learning tasks. This paper is a good fit for other conferences like CVPR or ICCV.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Q1. Might lack of insights or theoretical contributions? Not sure the findings/designs are generalizable to other 3D deep learning tasks?\n\nTo the best of our knowledge, not all works published at NeurIPS prioritize theoretical contributions. In fact, NeurIPS has been a platform for the publication of numerous works that offer valuable insights. For instance, within the domain of image object detection and segmentation, notable publications include Faster R-CNN (NeurIPS 2016), RFCN (NeurIPS 2016), SOLOv2 (NeurIPS 2020), and MaskFormer (NeurIPS 2021). When it comes to 3D object detection, recently published methods such as Object DGCNN (NeurIPS 2021), FSD (NeurIPS 2022), DeepInteraction (NeurIPS 2022), and BEVFusion (NeurIPS 2022) stand out.\n\nThe essence of our paper lies in the introduction of the encoder-decoder block, which offers a simple yet effective approach to capturing long-range dependencies for LiDAR-based 3D vision tasks. We sincerely appreciate the insightful feedback from you and the other two reviewers, k2sW and vKEU. As a result, we've integrated the proposed encoder-decoder design into the LiDAR-based 3D semantic segmentation task, yielding a model that achieved excellent performance. For more detailed information, please refer to Q4 of the reviewer vKEU.\n\n#### Q2. Figure 2 (a) says that the down-sampling layer is SS conv?\nThe down-sampling layer is RS conv and the description in L146-147 is correct. We made a mistake in Figure 2(a) and we apologize for the caused confusion. We will update the paper to fix the error.\n\n#### Q3. Table-1: are the presented results all based on a single frame or multiple frames?\nSingle frame. We will update the paper to clarify this.\n"}]

Author rebuttal:
To All Reviewers: 
Thank you so much for your affirmations and insightful feedback on our work. We are very glad to receive your constructive suggestions. Here, we give more discussions and explanations on those concerns mentioned by each reviewer respectively.

To Reviewer vKEU: 
The visualization comparison on receptive fields is presented in the attached PDF file.


Meta-review: The paper is motivated to address the limitation of submanifold sparse conv. and proposes a hierarchical encoder-decoder network for 3D object detection, which is able to capture long-range dependencies among features in the spatial space. Experiments on the Waymo and nuScenes datasets show the efficacy of the proposed method. 

Reviewers are initially concerned about some paper presentations and visualizations, about the lack of insights such as analysis on how and where the proposed method improves 3D detection,  and about the applicability of the method on scenarios other than self-driving cars. The authors provide responses that address these concerns. Congratulations on acceptance of the paper!
Decision: Accept (poster)