SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results on the test data set
Method	NDS	mAP	Car	Truck	Bus	T.L.	C.V.	Ped.	M.T.	Bike	T.C.	B.R.
PointPillars [19]	45.3	30.5	68.4	23.0	28.2	23.4	4.1	59.7	27.4	1.1	30.8	38.9
3DSSD [3]	56.4	42.6	81.2	47.2	61.4	30.5	12.6	70.2	36.0	8.6	31.1	47.9
CBGS [51]	63.3	52.8	81.1	48.5	54.9	42.9	10.5	80.1	51.5	22.3	70.9	65.7
CenterPoint [27]	65.5	58.0	84.6	51.0	60.2	53.2	17.5	83.4	53.7	28.7	76.7	70.9
FCOS-LiDAR [9]	65.7	60.2	82.2	47.7	52.9	48.8	28.8	84.5	68.0	39.0	79.2	70.7
HotSpotNet [52]	66.0	59.3	83.1	50.9	56.4	53.3	23.0	81.3	63.5	36.6	73.0	71.6
CVCNET [53]	66.6	58.2	82.6	49.5	59.4	51.1	16.2	83.0	61.8	38.8	69.7	69.7
AFDetV2 [45]	68.5	62.4	86.3	54.2	62.5	58.9	26.7	85.8	63.8	34.3	80.1	71.0
UVTR-L [54]	69.7	63.9	86.3	52.2	62.8	59.7	33.7	84.5	68.8	41.1	74.7	74.9
VISTA [55]	69.8	63.0	84.4	55.1	63.7	54.2	25.1	82.8	70.0	45.4	78.5	71.4
Focals Conv [56]	70.0	63.8	86.7	56.3	67.7	59.5	23.8	87.5	64.5	36.3	81.4	74.1
VoxelNeXt [50]	70.0	64.5	84.6	53.0	64.7	55.8	28.7	85.8	73.2	45.7	79.0	74.6
TransFusion-L [28]	70.2	65.5	86.2	56.7	66.3	58.8	28.2	86.1	68.3	44.2	82.0	78.2
LargeKernel3D [12]	70.6	65.4	85.5	53.8	64.4	59.5	29.7	85.9	72.7	46.8	79.9	75.5
LinK [16]	71.0	66.3	86.1	55.7	65.7	62.1	30.9	85.8	73.5	47.5	80.4	75.5
HEDNet (Ours)	72.0	67.7	87.1	56.5	70.4	63.5	33.6	87.9	70.4	44.8	85.1	78.1
Table 2: Comparison with prior methods on the nuScenes dataset. Metrics: NDS  $(\%)\uparrow$  and mAP  $(\%)\uparrow$  for the overall results, AP  $(\%)\uparrow$  for each category. 'T.L.', 'C.V.', 'Ped.', 'M.T.', 'T.C.', and 'B.R.' denote trailer, construction vehicle, pedestrian, motor, traffic cone, and barrier, respectively.
Figure 4: Detection accuracy (L2 mAPH) versus inference speed (FPS↑) of different models on the Waymo Open validation set.
Figure 5: Improvements decomposition of HEDNet over HEDNet-single with regard to the distance range of objects to the LiDAR sensor.
L2 mAPH improvements over the prior best method DSVT-Voxel [15]. HEDNet also outperformed the two-stage models PV-RCNN++ [47] and FSD [48]. More importantly, our method significantly outperformed the transformer-based DSVT-Voxel by  $1.7\%$  L2 mAPH on the vehicle category, where the average scale of vehicles is 10 times bigger than pedestrians and cyclists.
Results on the nuScenes dataset. We compared HEDNet with previous top-performing methods on the nuScenes dataset (Table 2). On the nuScenes test set, HEDNet achieved impressive results with  $72.0\%$  NDS and  $67.7\%$  mAP. Compared with TransFusion-L (which adopts the same head as HEDNet), HEDNet showcased significant improvements, with a gain of  $1.8\%$  NDS and  $2.2\%$  mAP. In addition, on the three categories with large objects, namely bus, trailer (T.L.), and construction vehicle (C.V.), HEDNet outperformed TransFusion-L by  $4.1\%$ ,  $4.7\%$ , and  $5.4\%$  mAP, respectively. These results further demonstrate the effectiveness of our method.
Block	Latency	L1 mAPH	L2 mAPH
RSR block	176 ms	74.61	68.30
SSR block	43 ms	74.42	67.93
SED block	48 ms	76.13	69.89
SSR \( block^† \)	49 ms	76.67	70.49
SED \( block^† \)	54 ms	77.39	71.37
(a) Effectiveness of the SED block.
#Block	Latency	L1 mAPH	L2 mAPH
0	48 ms	76.13	69.89
1	54 ms	77.56	71.37
3	63 ms	77.75	71.64
4	67 ms	78.02	71.92
5	73 ms	77.85	71.77
(b) Effectiveness of the DED block.
Sparse back.	Dense back.	L1 mAPH	L2 mAPH
8 SSR blocks	1 DED block	73.91	67.80
16 SSR blocks	1 DED block	73.95	67.82
4 SED blocks	1 DED block	75.39	69.42
4 SED blocks	2 DED blocks	75.62	69.67
(c) HEDNet with 2D sparse backbone.
#Scale	Latency	L1 mAPH	L2 mAPH
1	43 ms	76.18	69.88
2	59 ms	77.61	71.44
3	67 ms	78.02	71.92
4	78 ms	78.12	72.02
(d) Effectiveness of encoder-decoder design.
Table 3: Ablations on the Waymo Open. †: with 1 DED block. In (c), 'back.' denotes backbone. In (d), the gray line denotes the HEDNet-single, and the blue line denotes the default HEDNet.
Inference speed. We further compared HEDNet with previous leading methods in terms of detection accuracy and inference speed, as depicted in Figure 4. Remarkably, HEDNet achieved superior detection accuracy compared with LargeKernel3D [12] and DSVT-Voxel [15] with faster inference speed. Note that LargeKernel3D and DSVT-Voxel were developed based on large-kernel CNN and transformer, respectively. All models were evaluated on the same NVIDIA RTX 3090 GPU.
To better investigate the effectiveness of HEDNet, we constructed two network variants: HEDNet-single and HEDNet-2D. For the HEDNet-single, we replaced all the SED and DED blocks in HEDNet with single-scale blocks, i.e., only keeping the first  $m$  SSR/DR blocks in each SED/DED block. For the HEDNet-2D, we replaced all 3D sparse convolutions in HEDNet with 2D sparse convolutions and removed the three feature down-sampling layers that follow the feature maps  $\mathbf{F}_1$ ,  $\mathbf{F}_2$ , and  $\mathbf{F}_3$ , following the settings in DSVT [15]. The two SSR blocks after  $\mathbf{F}_0$  were also removed. In HEDNet-2D, the resolution of the output feature map of the 2D dense backbone is same as that of the network input  $F_0$ . We conducted experiments on the Waymo Open dataset to analyze various design choices of HEDNet. All models were trained on a 20% training subset and evaluated on the validation set.
4.4.1 Model designs
Effectiveness of the SED block. We compared the models built with RSR block, SSR block, and our proposed SED block in Table 3 (a). For the models with RSR/SSR blocks, we replaced the SED blocks in HEDNet with RSR/SSR blocks. The 2D dense backbones in the first three models were removed to fully explore the potential of the three structures. The first model with RSR blocks achieved slightly better results than the second model with SSR blocks but with much higher runtime latency. The third model with SED blocks significantly outperformed the second model with SSR blocks by  $1.96\%$  L2 mAPH. Similar gains can be observed in the last two models with DED blocks.
Effectiveness of the DED block. The DED block is designed to expand sparse features towards object centers. We compared the models that include different numbers of DED blocks in Table 3(b). The models with DED blocks achieved large improvements over the model without DED blocks. The model with five blocks performed worse than the model with four blocks. The former may be overfitted to the training data. We adopted four DED blocks for HEDNet by default.
HEDNet with 2D sparse backbone. We conducted experiments on HEDNet-2D to evaluate the effectiveness of our method with 2D inputs. For the construction of 2D inputs, we set the voxel size to (0.32m, 0.32m, 6m), where the size of 6m in the Z axis corresponds to the full size of the input point clouds. To compare our SED blocks with SSR blocks, we replaced each SED block in HEDNet-2D with 2 SSR blocks or 4 SSR blocks, resulting in two models of different sizes (the first two models in
Figure 6: Qualitative results on the Waymo Open. The red boxes are annotated by humans. The blue boxes and green boxes are predicted by HEDNet and the HEDNet-single, respectively. Red points correspond to the points that fall inside the human-annotated boxes. HEDNet predicted more precise bounding boxes for the objects marked by red arrows than the single-scale variant HEDNet-single.
Table 3 (c)). From Table 3 (c), we can make the following observations. Firstly, the model with 16 SSR blocks achieved similar performance to the model with 8 SSR blocks, indicating that stacking more SSR blocks could not further boost performance. Secondly, the models incorporating SED blocks showed significant improvements over the models using SSR blocks (at least  $1.6\%$  gains on L2 mAPH). This observation demonstrates the effectiveness of our SED block. Thirdly, stacking two DED blocks achieved better performance than using a single one. These results clearly demonstrate the generality and effectiveness of our proposed SED block and DED block.
4.4.2 HEDNet versus HEDNet-single
We conducted a thorough comparison between the proposed HEDNet and its single-scale variant, HEDNet-single, to explore the effectiveness of the encoder-decoder structure and investigate which objects benefit from HEDNet the most. Please note that the HEDNet is designed to capture long-range dependencies among features in the spatial space, which is the core of this work.
Firstly, we compared the models built with blocks of different numbers of scales to explore the effectiveness of the encoder-decoder structure. As shown in Table 3 (d), the models with multi-scale blocks significantly outperformed the single-scale variant HEDNet-single (the line in gray color). Using more scales achieved better performance, but introduced higher runtime latency. To strike a balance between accuracy and efficiency, we adopted three-scale blocks for HEDNet by default.
Secondly, we evaluated the three-scale HEDNet and the HEDNet-single in Table 3 (d) separately for each category and analyzed the results based on the distance range of objects to the LiDAR sensor. We illustrate the accuracy improvements of HEDNet over HEDNet-single at various distance ranges in Figure 5. Firstly, HEDNet showed significant improvements over HEDNet-single on the vehicle category, where the size of vehicles is 10 times larger than that of pedestrians and cyclists. This highlights the importance of capturing long-range dependencies for accurately detecting large objects. Furthermore, HEDNet achieved larger performance gains on distant objects compared with objects closer to the LiDAR sensor across all three categories. We believe this is because distant objects with fewer point clouds require more contextual information for accurate detection. Overall, these results demonstrate the effectiveness of our proposed method in detecting large and distant objects.
Thirdly, we further present some visualization results of the two models in Figure 6. HEDNet-single exhibited limitations in accurately predicting boxes for large objects and the predicted boxes often
only covered parts of the objects (see the top row). In addition, when dealing with objects containing a few points, HEDNet-single struggled to accurately estimate their orientations (see the bottom row). In contrast, HEDNet predicted more precise bounding boxes for both scenarios, which we believe is owed to the ability of HEDNet to capture long-range dependencies.
We propose a sparse encoder-decoder structure named SED block to capture long-range dependencies among features in the spatial space. Further, we propose a dense encoder-decoder structure named DED block to expand sparse features towards object centers. With the SED and DED blocks, we introduce a hierarchical encoder-decoder network named HEDNet for 3D object detection in point clouds. HEDNet achieved a new state-of-the-art performance on both the Waymo Open and nuScenes datasets, which demonstrates the effectiveness of our method. We hope that our work can provide some inspiration for the backbone design in 3D object detection.
Limitations HEDNet mainly focuses on 3D object detection in outdoor autonomous driving scenarios. However, the application of HEDNet in other indoor applications is still an open problem.
Acknowledgements This work was supported in part by the National Key Research and Development Program of China (No. 2021ZD0200301) and the National Natural Science Foundation of China (Nos. U19B2034, 61836014) and THU-Bosch JCML center.
[1] Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018.
[2] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointcnn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[3] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
[4] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
[5] Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast point r-cnn. In ICCV, 2019.
[6] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu. Range conditioned dilated convolutions for scale invariant 3d object detection. In CoRL, 2020.
[7] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and ZhaoXiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In ICCV, 2021.
[8] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov. RSN: Range sparse net for efficient, accurate lidar 3d object detection. In CVPR, 2021.
[9] Zhi Tian, Xiangxiang Chu, Xiaoming Wang, Xiaolin Wei, and Chunhua Shen. Fully convolutional one-stage 3d object detection on LiDAR range images. In NIPS, 2022.
[10] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Fcaf3d: fully convolutional anchor-free 3d object detection. In ECCV, 2022.
[11] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillar-based 3d object detection. In ECCV, 2022.
[12] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. In CVPR, 2023.
[13] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV, 2022.
[14] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing Single Stride 3D Object Detector with Sparse Transformer. In CVPR, 2022.
Method	Car	Bicycle	Motorcycle	Truck	Bus	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalk	Other ground	Building	Fence	Vegetation	Trunk	Terrier	Pole	Traffic sign	mIoU
MinkUNet34 [58]	96.8	55.0	81.4	83.2	70.2	79.5	89.8	7.8	94.8	54.6	82.8	1.5	92.0	68.3	87.9	69.4	72.8	66.1	52.4	68.3
HEDNet (Ours)	97.3	57.2	82.3	88.1	73.9	80.4	91.3	23.2	95.1	51.5	83.1	2.8	92.1	69.6	87.6	69.4	72.3	66.6	52.1	70.3
Table 4: 3D semantic segmentation results on the SemanticKiTTI validation set. Metrics: mIoU (\%)↑ for the overall results, IoU (\%)↑ for each category.
A Implementation details on 3D object detection
We implemented our method with Pytorch using the open-source OpenPCDet [49].
Waymo Open dataset. We set the hyperparameter  $m$  to 2 for all SED and DED blocks and stacked 4 DED blocks for the 2D dense backbone by default. We adopted the detection head of CenterPoint [27] for HEDNet. As metioned in the main paper, we primarily followed the training and inference schemes of DSVT [15]. Specifically, the voxel size was set to (0.08m, 0.08m, 0.15m), and the detection range was set to [-75.2m, 75.2m] for X and Y axis, and [-2m, 4m] for Z axis. We trained HEDNet for 24 epochs on the entire training dataset and reported the evaluation results on the validation set to compare with previous state-of-the-art methods. For the ablation experiments, we trained all models for 30 epochs on a 20% training subset. All models were trained with a batch size of 16 on 8 RTX 3090 GPUs. We employed the Adam [57] optimizer with a one-cycle learning rate policy, and set the weight-decay to 0.05, and the max learning rate to 0.003. We also adopted the faded training strategy in the last epoch. During inference, we applied class-specific NMS with an IoU threshold of 0.75, 0.6 and 0.55 for vehicle, pedestrian, and cyclist, respectively.
nuScenes dataset. We set the hyperparameter  $m$  to 2 for all SED and DED blocks and stacked 5 DED blocks for the 2D dense backbone. We adopted the detection head of TransFusion-L [28] for HEDNet and primarily followed the training and inference schemes of TransFusion-L [28]. The voxel size was set to (0.075m, 0.075m, 0.2m), and the detection range was set to [-54m, 54m] for X and Y axis, and [-5m, 3m] for Z axis. We trained HEDNet for 20 epochs on the combined training and validation sets with a batch size of 16 on 8 RTX 3090 GPUs and reported the results on the test set to compare with other methods. We employed the Adam [57] optimizer with a one-cycle learning rate policy, and set the weight-decay to 0.1, the momentum to [0.85, 0.95], and the max learning rate to 0.001. The faded strategy was used during the last 5 epochs. For submission to the test server, we set the query number of detection head to 300 and did not use any test-time augmentation.
B Experiments on 3D semantic segmentation
We conducted experiments on the popular LiDAR semantic segmentation dataset SemanticKiTTI [59], It provides 22 sequences with 19 semantic classes, captured by a 64-beam LiDAR sensor. Following the standard practice [58, 60], we report the Intersection-over-Union (IoU) for each category and the average score (mIoU) over all categories. For the backbone network, we employed a UNet-style structure, i.e., the same designs as the first two layers of the MinkUNet34 are first adopted to extract sparse features with a spatial down-sampling ratio of 4, followed by 4 SED layers to transform the resulting features, finally, two symmetrical layers are used to recover high-resolution features following MinkUNet34. The other settings strictly followed [58]. Table 4 shows that the proposed model exhibited significant gains over its counterpart MinkUNet34 (i.e.,  $2.0\%$  in mIoU), which demonstrated the generality of our method.
C A step-wise ablation from VoxelNet to HEDNet
We conducted a step-wise ablation from the standard VoxelNet to our HEDNet on the Waymo Open dataset to show the effectiveness of different components (see Table 5). For the second model, we employed the training tricks used by DSVT, including IoU loss, class-specific NMS, faded strategy (disabling data augmentations in the last epoch), and a weight decay of 0.05. These training tricks
No.	VoxelNet	Tricks*	Smaller-voxel	SED-block	DED-block	Full-data	Latency	L1 mAPH	L2 mAPH
1	✓	40 ms	70.0	64.0
2	✓	✓	40 ms	75.4	69.1
3	✓	✓	✓	49 ms	76.6	70.2
4	✓	✓	✓	✓	55 ms	77.6	71.3
5	✓	✓	✓	✓	✓	67 ms	78.0	71.9
6	✓	✓	✓	✓	✓	✓	67 ms	79.5	73.4
Table 5: A step-wise ablation from VoxelNet to HEDNet. The first five models were trained on a  $20\%$  training subset and the last model was trained on the full training set. \*: training tricks used in DSVT.
can significantly boost detection accuracy. Actually, most of the training tricks have been used by previous works, such as PV-RCNN++, and FSD. The codes and training configurations of the DSVT model can be found in the OpenPCDet archives. For the third model, we adopt a smaller input voxel size to keep more detailed information, which boosts the detection accuracy of pedestrian and cyclist. The 4th and 5th models sequentially incorporate our proposed SED blocks and DED blocks. The last model was trained on the full training set. Our final model (the 6th model) outperformed the previous SOTA method DSVT by  $1.3\%$  L2 mAPH while being  $50\%$  faster.