SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gang Zhang $^{1}$ , Junnan Chen $^{2}$ , Guohuan Gao $^{3}$ , Jianmin Li $^{1}$ , Xiaolin Hu $^{1,4,5*}$
$^{1}$ Department of Computer Science and Technology, Institute for AI, BNRist, THU-Bosch JCML Center, Tsinghua University
$^{2}$ Huazhong University of Science and Technology,  $^{3}$ Beijing Institute of Technology
4Tsinghua Laboratory of Brain and Intelligence (THBI),
IDG/McGovern Institute for Brain Research, Tsinghua University
$^{5}$  Chinese Institute for Brain Research (CIBR), Beijing 100010, China
zhang-g19@mails.tsinghua.edu.cn,chen_jn@hust.edu.cn,gaoguohuan@bit.edu.cn
lijianmin@mail.tsinghua.edu.cn,xlhu@tsinghua.edu.cn
Code: https://github.com/zhanggang001/HEDNet
3D object detection in point clouds is important for autonomous driving systems. A primary challenge in 3D object detection stems from the sparse distribution of points within the 3D scene. Existing high-performance methods typically employ 3D sparse convolutional neural networks with small kernels to extract features. To reduce computational costs, these methods resort to submanifold sparse convolutions, which prevent the information exchange among spatially disconnected features. Some recent approaches have attempted to address this problem by introducing large-kernel convolutions or self-attention mechanisms, but they either achieve limited accuracy improvements or incur excessive computational costs. We propose HEDNet, a hierarchical encoder-decoder network for 3D object detection, which leverages encoder-decoder blocks to capture long-range dependencies among features in the spatial space, particularly for large and distant objects. We conducted extensive experiments on the Waymo Open and nuScenes datasets. HEDNet achieved superior detection accuracy on both datasets than previous state-of-the-art methods with competitive efficiency. The code has been released.
Learning effective representations from sparse input data is a key challenge for 3D object detection in point clouds. Existing point-based methods [1, 2, 3, 4, 5] and range-based methods [6, 7, 8, 9, 10] either suffer from high computational costs or exhibit inferior detection accuracy. Currently, voxel-based methods [11, 12, 13, 14, 15] dominate high-performance 3D object detection.
The voxel-based methods partition the unstructured point clouds into regular voxels and utilize sparse conventional neural network (CNNs) [11, 12, 16, 17, 18, 19] or transformers [13, 14, 15] as backbones for feature extraction. Most existing sparse CNNs are primarily built by stacking submanifold sparse residual (SSR) blocks, each consisting of two submanifold sparse convolutions [20] with small kernels. However, submanifold sparse convolutions maintain the same sparsity between input and output features, and therefore hinder the exchange of information among spatially disconnected features. Consequently, models employing SSR blocks face challenges in effectively capturing long-range dependencies among features. One potential solution is to replace the submanifold
sparse convolutions in SSR block with regular sparse convolutions [21]. However, this leads to a significant decrease in feature sparsity as the network deepens, resulting in substantial computational costs. Recent research has investigated the utilization of large-kernel sparse CNNs [12, 16] and transformers [14, 15] to capture long-range dependencies among features. However, these approaches have either demonstrated limited improvements in detection accuracy or come with significant computational costs. Thus, the question remains: is there an efficient method that enables sparse CNNs to effectively capture long-range dependencies among features?
Revisiting backbone designs in various dense prediction tasks [13, 22, 23, 24, 25, 26], we observe that the encoder-decoder structure has proven effective in capturing long-range dependencies among features. These methods typically use a high-to-low resolution backbone as an encoder to extract multi-scale features and design different decoders to recover high-resolution features that can model long-range relationships. For instance, PSPNet [24] incorporates a pyramid pooling module to capture both local and global contextual information by pooling features at multiple scales. SWFormer [13] integrates a top-down pathway into its transformer backbone to capture cross-window correlations. However, the utilization of the encoder-decoder structure in designing sparse convolutional backbones for 3D object detection has not yet been explored, to the best of our knowledge.
In this work, we propose a sparse encoder-decoder (SED) block to overcome the limitations of the SSR block. The encoder extracts multi-scale features through feature down-sampling, facilitating information exchange among spatially disconnected regions. Meanwhile, the decoder incorporates multi-scale feature fusion to recover the lost details. A hallmark of the SED block is its ability to capture long-range dependencies while preserving the same sparsity between input and output features. Since current leading 3D detectors typically rely on object centers for detection [27, 28], we further adapt the 3D SED block into a 2D dense encoder-decoder (DED) block, which expands the extracted sparse features towards object centers. Leveraging the SED block and DED block, we introduce a hierarchical encoder-decoder network named HEDNet for 3D object detection in point clouds. HEDNet can learn powerful representations for the detection of large and distant objects.
Extensive experiments were conducted on the challenging Waymo Open [29] and nuScenes [30] datasets to demonstrate the effectiveness of the proposed HEDNet on 3D object detection. HEDNet achieved impressive performance, with a  $75.0\%$  L2 mAPH on the Waymo Open test set and a  $72.0\%$  NDS on the nuScenes test set, outperforming prior methods that utilize large-kernel CNNs or transformers as backbones while exhibiting higher efficiency. For instance, HEDNet was  $50\%$  faster than DSVT, the previous state-of-the-art transformer-based method, with  $1.3\%$  L2 mAPH gains.
2.1 3D object detection in point clouds
For 3D object detection in point clouds, methods can be categorized into three groups: point-based, range-based, and voxel-based. Point-based methods [1, 2, 3, 4, 5] utilize the PointNet series [31, 32] to directly extract geometric features from raw point clouds and make predictions. However, these methods require computationally intensive point sampling and neighbor search procedures. Range-based methods [6, 7, 8, 9, 10] convert point clouds into pseudo images, thus benefiting from the well-established designs of 2D object detectors. While computationally efficient, these methods often exhibit lower accuracy. Voxel-based approaches [11, 17, 18, 19] are currently the leading methods for high-performance 3D object detection. Most voxel-based methods employ sparse CNNs that consist of submanifold and regular sparse convolutions with small kernels to extract features. Regular sparse convolutions can capture distant contextual information but are computationally expensive. On the other hand, submanifold sparse convolutions prioritize efficiency but sacrifice the model's ability to capture long-range dependencies.
2.2 Capturing long-range dependencies for 3D object detection
To capture long-range dependencies for 3D object detection, recent research has explored solutions such as large-kernel sparse convolutions [33, 16] and self-attention mechanisms [13, 14, 15]. However, directly applying plain large-kernel CNNs for 3D representation learning can lead to problems such as overfitting and reduced efficiency. Weight-sharing strategies have been proposed to mitigate overfitting, like LargeKernel3D [12] and Link [16], however, they still suffer from low efficiency.
(a) SSR block
(b) RSR block
(c) SED block (ours)
Figure 1: Comparison among SSR block (a), RSR block (b), and our SED block (c). The 'Skip conn.' denotes the skip connection, and the orange dashed lines represent the convolution kernel space. Valid features have non-zero values. Expanded and empty features have zero values. In (b), convolution is applied to both valid and expanded features, i.e., the convolution kernel center traverses the regions covered by these features. The red dashed square highlights the regions from which the output feature marked by a star can receive information. In (c), we adopt a  $3 \times 3$  RS convolution with a stride of 3 for feature down-sampling (Down) as an example. UP denotes feature up-sampling.
Other methods, such as SST [14] and DSVT [15], utilize transformers as replacements for sparse CNNs. SST employs a single-stride sparse transformer to preserve high-resolution features without using down-sampling operators that may cause a loss of detail. Similarly, DSVT employs a single-stride sparse transformer and performs window-based self-attention sequentially along the X-axis and Y-axis. While both large-kernel CNNs and transformers aim to capture long-range dependencies, they either achieve comparable performance or exhibit lower efficiency compared with sparse CNNs. In contrast, our proposed HEDNet effectively captures long-range dependencies with the help of encoder-decoder blocks while achieving competitive inference speed compared with existing methods.
2.3 Encoder-decoder networks for dense prediction
The encoder-decoder structure has been extensively investigated in various dense prediction tasks. For example, the FPN-series [22, 34, 35, 36] incorporates lightweight fusion modules as decoders to integrate multi-scale features extracted from image classification backbones. DeeplabV3+ [25] employs an atrous spatial pyramid pooling module to combine low-level features with semantically rich high-level features. SWFormer [13] introduces a top-down pathway into its transformer backbone to capture cross-window correlations. However, there is limited exploration of the encoder-decoder structure in the design of sparse CNNs. Most voxel-based approaches [17, 18, 27, 28] rely on high-to-low resolution sparse CNNs to extract single-scale high-level features. Part-A2-Net [37] adopts the UNet [38] to extract features, but it performs detection using the encoder of UNet while utilizing the decoder for the auxiliary segmentation and part prediction tasks. In this study, we propose HEDNet, which primarily consists of encoder-decoder blocks to effectively capture long-range dependencies.
The sparse CNNs adopted by most voxel-based methods [18, 27, 28] are primarily built by stacking SSR blocks, each consisting of two submanifold sparse (SS) convolutions [20]. In addition, they usually insert regular sparse (RS) convolutions [21] into the stacked SSR blocks to reduce the resolution of feature maps progressively (similar to ResNet [39]).
SS convolution and SSR block. We present the structure of a single SSR block in Figure 1 (a). Two SS convolutions are sequentially applied to the input feature map, with skip connections incorporated between the input and output feature maps of the SSR block. The sparsity of feature map is defined as the ratio of the regions that are not occupied by valid (nonzero) features to the total area of the feature map. SS convolution only operates on valid features, allowing the output feature map of the SSR block to maintain the same sparsity as the input feature map. However, this design hinders the
Figure 2: Architecture of the SED block (a) and DED block (b). As an example, we illustrate blocks of three scales. Both designs share the same structure.  $\mathrm{F}_1 / \mathrm{F}_2 / \mathrm{F}_3 / \mathrm{F}_4 / \mathrm{F}_5$  are the names of the corresponding feature maps. The number in parentheses indicates the resolution ratio of the corresponding feature map relative to the block input. The SED block is capable of processing both 2D and 3D features, depending on whether 2D or 3D sparse convolutions are used.
exchange of information among spatially disconnected features. For instance, in the top feature map, the output feature marked by a star cannot receive information from the other three feature points outside the red dashed square in the bottom feature map (marked by the red triangles). This poses a challenge for the model in capturing long-range dependencies.
RS convolution and RSR block. One possible solution to problem is to replace the SS convolutions in the SSR block with RS convolutions. We call this modified structure regular sparse residual (RSR) block and illustrate its structure in Figure 1 (b). RS convolution operates on both valid and expanded features [21]. Expanded features correspond to the features that fall within the neighborhood of the valid features. Taking a 2D RS convolution with a kernel size of  $3 \times 3$  as an example, the neighborhood of a certain valid feature consists of the eight positions around it. This design leads to an output feature map with a lower sparsity compared with the input feature map. Stacking RS convolutions reduces the feature sparsity dramatically, which in turn leads to a notable decrease in model efficiency compared with using SS convolutions. This is why existing methods [18, 27, 28] typically limit the usage of RS convolution to feature down-sampling layers.
3.2 SED and DED blocks
SED block. SED block is designed to overcome the limitations of SSR block. The fundamental idea behind this design is to reduce the spatial distance between distant features through feature down-sampling and recover the lost details through multi-scale feature fusion.
We illustrate a two-scale SED block in Figure 1 (c). After feature down-sampling, the spatially disconnected valid features in the bottom feature map are integrated into the adjacent valid features in the middle feature map. An SSR block is subsequently applied to the middle feature map to promote interaction among valid features. Finally, the middle feature map is up-sampled to match the resolution of the input feature map. Note that the feature up-sampling layer (UP) only up-samples features to the regions covered by the valid features in the input feature map. As a result, the proposed SED block can maintain the same sparsity between input and output feature maps. This characteristic prevents the introduction of excessive computational costs when stacking multiple SED blocks.
The architecture of a three-scale SED block is presented in Figure 2 (a). The SED block adopts an asymmetric encoder-decoder structure similar to UNet [38], with the encoder responsible for extracting multi-scale features and the decoder sequentially fusing the extracted multi-scale features with the help of skip connections. Given the input feature map X, the function of the SED block can be formulated as follows:
$$
F _ {1} = \operatorname {S S R} ^ {m} (X) \tag {1}
$$
$$
\mathrm {F} _ {2} = \operatorname {S S R} ^ {m} \left(\operatorname {D o w n} _ {1} \left(\mathrm {F} _ {1}\right)\right) \tag {2}
$$
$$
\mathrm {F} _ {3} = \operatorname {S S R} ^ {m} \left(\operatorname {D o w n} _ {2} \left(\mathrm {F} _ {2}\right)\right) \tag {3}
$$
$$
\mathrm {F} _ {4} = \mathrm {U P} _ {2} \left(\mathrm {F} _ {3}\right) + \mathrm {F} _ {2} \tag {4}
$$
$$
\mathrm {F} _ {5} = \mathrm {U P} _ {1} \left(\mathrm {F} _ {4}\right) + \mathrm {F} _ {1} \tag {5}
$$
Figure 3: Architecture of the proposed HEDNet. Given the raw point clouds, we first perform voxelization to generate voxels by the VFE module, then employ the 3D sparse backbone and the 2D dense backbone to extract features for the detection head. The number in the bracket denotes the resolution ratio of the corresponding feature map relative to the input. The RS convolutions for feature down-sampling that follow the feature maps  $F_{1}$ ,  $F_{2}$ , and  $F_{3}$  are omitted for simplicity.
where  $\mathrm{F}_5$  denotes the output feature map with the same resolution as the input X. The resolution ratios of the intermediate feature maps  $\mathrm{F}_1, \mathrm{F}_2, \mathrm{F}_3$ , and  $\mathrm{F}_4$  relative to the input X are 1, 1/2, 1/4, and 1/2, respectively.  $\mathrm{SSR}^m$  indicates  $m$  consecutive SSR blocks. We adopt RS convolution as the feature down-sampling layer (Down) and sparse inverse convolution [40] as the feature up-sampling layer (UP). With an encoder-decoder structure, the SED block facilitates information exchange among spatially disconnected features, thereby enabling the model to capture long-range dependencies.
DED block. Existing high-performance 3D object detectors [15, 27, 28] usually rely on object centers for detection. However, the feature maps extracted by purely sparse CNNs may have empty holes around object centers, especially for large objects. To overcome this issue, we introduce a DED block that expands sparse features towards object centers, as shown in Figure 2 (b). The DED block shares a similar structure with the SED block but utilizes the widely used dense convolutions instead. Specifically, we replace the SSR block in the SED block with a dense residual (DR) block, which is similar to the SSR block but consists of two dense convolutions. Furthermore, the RS convolution employed for feature down-sampling is replaced with a DR block that has a stride of 2. For feature up-sampling, we replace the sparse inverse convolution with a dense deconvolution. These modifications enable the DED block to effectively expand sparse features towards object centers.
3.3 HEDNet
Based on the proposed SED block and DED block, we introduce HEDNet, a hierarchical encoder-decoder network designed for 3D object detection. The architecture of HEDNet is illustrated in Figure 3. Given the raw point clouds, a dynamic VFE module [41] is used to perform voxelization to generate a grid of voxels denoted as  $\mathbf{F}_0$ . Subsequently, a sparse backbone including two SSR blocks and several SED blocks is employed to extract 3D sparse features. Before being fed into the 2D dense backbone, the sparse features are compressed into dense BEV features like in [18]. The 2D dense backbone, composed of  $n$  DED blocks, is responsible for expanding the sparse features towards object centers. Finally, the output features are fed into the detection head for final predictions. At a macro level, HEDNet follows a hierarchical structure similar to SECOND [18], where the resolution of feature maps progressively decreases. At a micro level, the SED and DED blocks, key components of HEDNet, employ encoder-decoder structures. This is where the name HEDNet comes from. We adopt SED and DED blocks of three scales for HEDNet by default.
4.1 Datasets and metrics
Waymo Open contains 160k, 40k, and 30k annotated samples for training, validation, and testing, respectively. The metrics for 3D object detection include mean average precision (mAP) and mAP
Results on the validation data set
Method	mAP/mAPH L2	Vehicle AP/APH L1	L2	Pedestrian AP/APH L1	L2	Cyclist AP/APH L1	L2
SECOND [18]	61.0/57.2	72.3/71.7	63.9/63.3	68.7/58.2	60.7/51.3	60.6/59.3	58.3/57.0
PointPillar [19]	62.8/57.8	72.1/71.5	63.6/63.1	70.6/56.7	62.8/50.3	64.4/62.3	61.9/59.9
Lidar-RCNN [42]†	65.8/61.3	76.0/75.5	68.3/67.9	71.2/58.7	63.1/51.7	68.6/66.9	66.1/64.4
Part-A2-Net [37]†	66.9/63.8	77.1/76.5	68.5/68.0	75.2/66.9	66.2/58.6	68.6/67.4	66.1/64.9
SST [14]	67.8/64.6	74.2/73.8	65.5/65.1	78.7/69.6	70.0/61.7	70.7/69.6	68.0/66.9
CenterPoint [27]	68.2/65.8	74.2/73.6	66.2/65.7	76.6/70.5	68.8/63.2	72.3/71.1	69.7/68.5
PV-RCNN [43]†	69.6/67.2	78.0/77.5	69.4/69.0	79.2/73.0	70.4/64.7	71.5/70.3	69.0/67.8
CenterPoint [27]†	69.8/67.6	76.6/76.0	68.9/68.4	79.0/73.4	71.0/65.8	72.1/71.0	69.5/68.5
SWFormer [13]	-/-	77.8/77.3	69.2/68.8	80.9/72.7	72.5/64.9	-/-	-/-
OcTr [44]	70.7/68.2	78.1/77.6	69.8/69.3	80.8/74.4	72.5/66.5	72.6/71.5	69.9/68.9
PillarNet-34 [11]	71.0/68.5	79.1/78.6	70.9/70.5	80.6/74.0	72.3/66.2	72.3/71.2	69.7/68.7
AFDetV2 [45]	71.0/68.8	77.6/77.1	69.7/69.2	80.2/74.6	72.2/67.0	73.7/72.7	71.0/70.1
CenterFormer [46]	71.1/68.9	75.0/74.4	69.9/69.4	78.6/73.0	73.6/68.3	72.3/71.3	69.8/68.8
LargeKernel3D[33]	-/-	78.1/77.6	69.8/69.4	-/-	-/-	-/-	-/-
PV-RCNN++ [47]†	71.7/69.5	79.3/78.8	70.6/70.2	81.3/76.3	73.2/68.0	73.7/72.7	71.2/70.2
FSD [48]†	72.7/70.5	79.5/79.0	70.3/69.9	83.6/78.2	74.4/69.4	75.3/74.1	73.3/72.1
DSVT-Voxel [15]	74.0/72.1	79.7/79.3	71.4/71.0	83.7/78.9	76.1/71.5	77.5/76.5	74.6/73.7
HEDNet (ours)	75.3/73.4	81.1/80.6	73.2/72.7	84.4/80.0	76.8/72.6	78.7/77.7	75.8/74.9
Results on the test data set
Method	mAP/mAPH L2	Vehicle AP/APH L1	L2	Pedestrian AP/APH L1	L2	Cyclist AP/APH L1	L2
PV-RCNN [43]	71.3/68.8	80.6/80.1	72.8/72.4	78.2/72.0	71.8/66.0	71.8/70.4	69.1/67.8
PV-RCNN++ [47]	72.4/70.2	81.6/81.2	73.9/73.5	80.4/75.0	74.1/69.0	71.9/70.8	69.3/68.2
AFDetV2 [45]	72.2/70.3	80.5/80.0	73.0/72.6	79.8/74.3	73.7/68.6	72.4/71.2	69.8/69.7
FSD [48]	74.4/72.4	82.7/82.3	74.4/74.1	82.9/77.9	75.9/71.3	75.6/74.4	72.9/71.8
HEDNet (ours)	76.9/75.0	84.2/83.8	77.0/76.6	84.1/79.7	78.3/74.0	78.2/77.0	75.4/74.3
Table 1: Comparison with prior methods on the Waymo Open dataset (single-frame setting). Metrics: mAP/mAPH  $(\%)\uparrow$  for the overall results, and AP/APH  $(\%)\uparrow$  for each category.  $\dagger$  : two-stage method.
weighted by the heading accuracy (mAPH). Both are further broken down into two difficulty levels: L1 for objects with more than five LiDAR points and L2 for objects with at least one LiDAR point.
nuScenes consists of 28k, 6k, and 6k annotated samples for training, validation, and testing, respectively. Mean average precision (mAP) and nuScenes detection score (NDS) are used as the evaluation metrics. mAP is computed by averaging over the distance thresholds of  $0.5\mathrm{m}$ ,  $1\mathrm{m}$ ,  $2\mathrm{m}$ ,  $4\mathrm{m}$  across all categories. NDS is a weighted average of mAP and the other five true positive metrics measuring the translation, scaling, orientation, velocity, and attribute errors.
We implemented our method using the open-source OpenPCDet [49]. To build HEDNet, we set the hyperparameter  $m$  to 2 for all SED and DED blocks and stacked 4 DED blocks for the 2D dense backbone by default. For 3D object detection on the Waymo Open dataset, we adopted the detection head of CenterPoint and set the voxel size to (0.08m, 0.08m, 0.15m). We trained HEDNet for 24 epochs on the full training set (single-frame) to compare with prior methods. For ablation experiments in Section 4.4, we trained the models for 30 epochs on a  $20\%$  training subset. All models were trained with a batch size of 16 on 8 RTX 3090 GPUs. The other training settings strictly followed DSVT [15]. For 3D object detection on the nuScenes dataset, we adopted the detection head of TransFusion-L and set the voxel size to (0.075m, 0.075m, 0.2m). We trained HEDNet for 20 epochs with a batch size of 16 on 8 RTX 3090 GPUs. The other training settings strictly followed TransFusion-L [28].
4.3 Comparison with state-of-the-art methods
Results on the Waymo Open dataset. We compared the proposed HEDNet with previous methods on the Waymo Open dataset (Table 1). On the validation set, HEDNet yielded  $1.3\%$  L2 mAP and  $1.3\%$
Results on the validation data set
Method	NDS	mAP	Car	Truck	Bus	T.L.	C.V.	Ped.	M.T.	Bike	T.C.
CenterPoint [27]	66.5	59.2	84.9	57.4	70.7	38.1	16.9	85.1	59.0	42.0	69.8
VoxelNeXt [50]	66.7	60.5	83.9	55.5	70.5	38.1	21.1	84.6	62.8	50.0	69.4
TransFusion-L [28]	70.1	65.5	86.9	60.8	73.1	43.4	25.2	87.5	72.9	57.3	77.2
HEDNet (Ours)	71.4	66.7	87.7	60.6	77.8	50.7	28.9	87.1	74.3	56.8	76.3