SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yingjie Wang $^{1,2*}$ , Jiajun Deng $^{3\dagger}$ , Yuenan Hou $^{2}$ , Yao Li $^{1}$ , Yu Zhang $^{1}$ , Jianmin Ji $^{1}$ , Wanli Ouyang $^{2}$ , Yanyong Zhang $^{1\dagger}$
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>Shanghai AI Laboratory, <sup>3</sup>The University of Adelaide, AIML
Currently, LiDAR-based 3D detectors are broadly categorized into two groups, namely, BEV-based detectors and cluster-based detectors. BEV-based detectors capture the contextual information from the Bird's Eye View (BEV) and fill their center voxels via feature diffusion with a stack of convolution layers, which, however, weakens the capability of presenting an object with the center point. On the other hand, cluster-based detectors exploit the voting mechanism and aggregate the foreground points into object-centric clusters for further prediction. In this paper, we explore how to effectively combine these two complementary representations into a unified framework. Specifically, we propose a new 3D object detection framework, referred to as CluB, which incorporates an auxiliary cluster-based branch into the BEV-based detector by enriching the object representation at both feature and query levels. Technically, CluB is comprised of two steps. First, we construct a cluster feature diffusion module to establish the association between cluster features and BEV features in a subtle and adaptive fashion. Based on that, an imitation loss is introduced to distill object-centric knowledge from the cluster features to the BEV features. Second, we design a cluster query generation module to leverage the voting centers directly from the cluster branch, thus enriching the diversity of object queries. Meanwhile, a direction loss is employed to encourage a more accurate voting center for each cluster. Extensive experiments are conducted on Waymo and nuScenes datasets, and our CluB achieves state-of-the-art performance on both benchmarks.
3D perception on point clouds has attracted much attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics [29, 17, 11, 28, 32]. LiDAR-based 3D object detection is one of the vital tasks, which takes sparse point clouds as input to estimate the 3D positions and categories of objects [20, 7, 30, 40].
Currently, the mainstream 3D object detectors [35, 21, 18, 38, 22, 1, 39] convert unstructured point clouds into regular grids in the Bird's Eye View (BEV) for feature representation (Figure 1 (a)), named as BEV-based 3D object detectors. The regular grids in BEV are beneficial to feature extraction and contextual information capturing. As the key point to make a prediction, i.e., the center point of an object, can be empty, the features of the center points are often diffused from the surrounding points via a stack of convolution layers. This practice significantly weakens the capability of presenting an object with the center point.
* This work was done during Yingjie's internship at Shanghai Artificial Intelligence Laboratory.
† Corresponding Authors: Jiajun Deng and Yanyong Zhang.
Figure 1: Illustration of (a) context-aware BEV-based representation[10] and (b) object-centric cluster-based representation. The BEV representation can capture the rich contextual information of an object. However, the missing feature of the center point (red square), which is also the key to making the prediction, is diffused from occupied voxels. The cluster-based representation is generated using the voting mechanism by aggregating the foreground points into the object-specific feature for the voting center (red dot). This representation mainly focuses on the foreground points and fails to utilize the contextual information.
Another stream of detectors [10, 19, 13, 36] resorts to the cluster-based feature representation (Figure 1 (b)). This line of methods exploits the voting mechanism and groups the foreground points into object-centric clusters for further prediction. The cluster features largely preserve the 3D structure details of each object, while avoiding detection based on center points that are absent in the raw data. However, the clustering operation mainly focuses on foreground points, and fails to fully utilize the contextual information. It heavily hinges on foreground segmentation accuracy. Based on the potential synergy between these two paradigms, in this paper, we focus on exploring the combination of context-aware BEV representation and object-centric cluster representation to improve the accuracy of 3D object detection, which remains under-explored in the literature.
Towards this goal, an intuitive solution is to scatter all cluster features to the BEV space based on the locations of the predicted voting centers<sup>1</sup> and directly concatenate the obtained vote BEV features (vote BEV in short) to the dense BEV features (dense BEV in short). However, the established hard association<sup>2</sup> between cluster features and BEV features heavily relies on the quality of vote clusters, not to mention that the representation of these two streams is not well aligned, which may affect the overall stability of representation learning [5]. This potential issue makes the intuitive strategy fail to fully explore the feature intertwining of both representations.
Therefore, by consolidating the idea of leveraging the object-centric property of cluster representation and contextual information of BEV representation, we propose a novel 3D object detection framework, i.e., CluB, which integrates an auxiliary Cluster-based branch to the mainstream BEV-based detector by enriching the object representation at both feature and query levels. Specifically, CluB begins with projecting the two features into the BEV space (vote BEV and dense BEV) for unifying the object representation from corresponding branches. Then, we propose a two-step approach to enhance the representation capability from the two feature and query aspects.
Firstly, to address the issue of the hard association between cluster features and BEV features, we propose a Cluster Feature Diffusion (CFD) module that adaptively diffuses the valid votes on the vote BEV to their neighboring regions according to the predicted class of each vote, which repositions the association from hard to soft and adaptive. Based on that, an imitation loss is introduced to transfer object-centric knowledge to the BEV branch and encourage the stability of overall representation learning. In this way, the object representation can be jointly enhanced by fusing features from both branches.
Secondly, since most of the BEV detectors detect objects as center points, it is non-trivial to enrich the diversity of object queries using the voting centers directly from the cluster branch. We thus design a Cluster Query Generation (CQG) module, which activates and then selects the top-ranked candidates based on the vote BEV. The top-selected cluster queries are then appended to the original object queries together for better prediction. On the one hand, we also utilize direction supervision to alleviate the overlap problem for close clusters, thus producing more accurate cluster queries.
<sup>1</sup>The expression of the vote center is referred to as VoteNet [19]. VoteNet processes existing points to generate votes, which serve as the cluster centers.
2In fact, the vote centers indicate the potential locations of the centers of objects within a certain area.
The design of CluB strengthens the object representation through feature integration and query expansion, eventually bringing significant performance gains. In summary, we make the following contributions:
- We develop the CluB framework to improve the accuracy of 3D object detection by taking advantage of both BEV-based and cluster-based paradigms.
- We present an elegant solution to integrating the context-aware BEV representation and object-centric cluster representation at both the feature level and the query level, which is a non-trivial problem.
- Our approach outperforms state-of-the-art methods by a remarkable margin on two prevalent datasets, i.e., Waymo Open Dataset and nuScenes, demonstrating the effectiveness and generality of our proposed method.
BEV-based LiDAR Detectors. There are mainly two ways to convert the point cloud into BEV representation. One is to extract BEV features in 2D space and design efficient networks to produce predictions based on the BEV features [15, 21, 9]. PointPillars [15] introduces the pillar representation (a particular form of the voxel) and process it with 2D convolutions. PillarNet [21] resorts to the powerful encoder network for feature extraction and designs the neck network to achieve spatial-semantic feature fusion, thus greatly improving the detection performance of pillar-based models. The other way is to extract point cloud features in 3D space and scatter them to the BEV plane, yielding more accurate detection results [37, 33, 35]. With the BEV representation, CenterPoint [35] designs an anchor-free one-stage detector, which extracts BEV features from voxelized point clouds to estimate object centers and all object properties such as 3D size and orientation. Later, the adoption of the popular transformer architecture as the detection head has substantially improved the accuracy and robustness of BEV-based methods [38, 1, 27, 39]. SWFormer [27] applies the window-based transformer on LiDAR BEV features. In particular, SWFormer solves the center feature missing problem for the 3D sparse features on the BEV planes with the voxel diffusion strategy, while not considering the semantic consistency. Transfusion [1] designs an effective and robust transformer-based LiDAR-camera fusion framework, which can also serve as a strong LiDAR-based 3D detector featured by the BEV-based paradigm.
Cluster-based LiDAR Detectors. The cluster-based methods focus on extracting discriminative features directly from clustered point clouds and have shown promising results in various benchmarks [10, 19, 13, 36]. VoteNet [19] is a pioneering work in the field of cluster-based LiDAR detectors. It employs the voting mechanism to predict object centers from the input point cloud. Motivated by that, FSD [10] designs the fully sparse network in the outdoor scene using point clustering and group correction. PSA-Det3D [13] proposes a pillar set abstraction method (PSA) to learn representative features for sparse point clouds of small objects, which mainly benefits from point-wise feature aggregation. Current benchmarks [26, 2, 31] are dominated by the aforementioned BEV-based and cluster-based 3D detectors. Notably, a coherent work VoxelNext [4] predicts objects directly based on sparse voxel features, without relying on cluster proxies or the dense BEV representation.
In summary, cluster-based detectors are specialized at capturing fine-grained features for individual objects consisting of foreground points, while BEV-based detectors are well-suited for capturing contextual information of point clouds but may have weaker center point representation capabilities. Therefore, by combining these approaches, we can leverage their respective advantages to enhance the detection performance. Our method aims to integrate a BEV-based detector with an auxiliary cluster-based branch, thereby harnessing the strengths of both paradigms to enhance the overall detection performance.
In this section, we introduce the proposed CluB in detail, as shown in the Figure 2.
Overview. Our CluB is built from scratch using the following three steps. First, we utilize a sparse voxel encoder to extract voxel features and aggregate the vote cluster features, which is considered
Figure 2: The overall framework of CluB. The cluster-based side branch (a) is integrated into the BEV-based main branch (b) through a two-step scheme, i.e., feature-level integration (c) and query-level expansion (d). Specifically, the raw point clouds are first voxelized and then fed into the sparse U-Net to extract both 3D voxel features and sparse voxel features. The former is used to obtain the dense BEV feature. The latter is processed in the cluster branch to generate a set of vote cluster features  $\mathbf{F}_{clu}$  and corresponding voting centers  $\mathbf{C}_{clu}$ . Based on the two outputs, i.e.,  $\mathbf{F}_{clu}$  and  $\mathbf{C}_{clu}$ , the Cluster Feature Diffusion (CFD) module obtains a diffused vote BEV feature  $\mathbf{F}_{voteB}$  to enhance the dense BEV feature  $\mathbf{F}_{denseB}$  supervised by an imitation loss, and the Cluster Query Generation (CQG) module outputs a set of top-ranking cluster queries  $\mathbf{Q}_{clu}$ , which are concatenated to the BEV queries  $\mathbf{Q}_{BEV}$  obtained from the heatmap. The decoder takes hybrid object queries as input and makes the final predictions based on the concatenated BEV features.
as the cluster-based auxiliary branch (Section 3.2). Next, a BEV-based main branch (Section 3.1) is served as the detection backbone. Finally, the cluster-based side branch is incorporated into the BEV-based main branch through a two-level enhancement scheme, i.e., feature-level integration (Section 3.3) and query-level expansion (Section 3.4).
3.1 BEV-based Main Branch
The main BEV branch takes the voxelized 3D point clouds as input and then leverages a sparse encoder consisting of sparse convolution layers to produce 3D voxel features [24]. Then we compress the 3D voxel features into 2D feature maps in the BEV space. The 2D BEV feature map is fed to the 2D convolution-based backbone to generate a dense BEV feature map  $\mathbf{F}_{denseB} \in \mathbb{R}^{C \times H \times W}$  (short for dense BEV), where  $(H, W)$  indicate the height and width of the feature map. Compared to the anchor/center-based detection head [35, 37], the transformer decoder demonstrates great potential for feature fusion besides the notable performance improvement [34, 8]. We thus utilize a transformer-based detection head [1] to extract class-level feature representations from dense BEV for object prediction. The decoder layer follows the design of DETR [3]. We generate object candidates from the class-specific heatmap and select the top-ranked candidates for all the categories as our initial BEV queries  $\mathbf{Q}_{BEV}$ . The fused BEV features are also used as key-value sequences for the transformer decoder.
3.2 Cluster-based Auxiliary Branch
The cluster side branch takes the same voxelized 3D point clouds as input to extract sparse voxel features via a sparse convolution based encoder-decoder. These sparse voxel features are fed into two heads for foreground classification and object center voting. Based on the voting results, we group the foreground points into clusters using the grouping strategy (e.g., ball query), and then use a vote cluster module to aggregate each cluster feature. The vote cluster module could be implemented with simple vote aggregation following [19]. In order to extract more powerful vote cluster features, we utilize a highly efficient sparse instance recognition module for further extraction [10]. The final
Figure 3: Illustration of Cluster Feature Diffusion (CFD) module. We first transform the vote cluster features to the BEV space based on the position of each vote, forming a sparse representation termed vanilla vote BEV features. Next, the non-empty grids of vanilla vote BEV are adaptively expanded to their neighboring regions according to the predicted class of the vote through a class-aware BEV diffusion block. Finally, we use a fully connected layer to project the class-aware vote BEV to the same channel dimension as the vanilla vote BEV, thus obtaining a diffused vote BEV feature.
output of the vote cluster module consists of two parts, i.e., object-specific cluster features  $\mathbf{F}_{clu}$  and their corresponding cluster centers  $\mathbf{C}_{clu}$ .
3.3 Feature-level Integration
In this subsection, we introduce the Cluster Feature Diffusion (CFD) module to generate diffused vote features in the unified BEV space, which are fused with the dense BEV features for representation enhancement. The unified BEV features are supervised by an imitation loss.
Cluster Feature Diffusion. To establish a soft and adaptive association between cluster and BEV features for better fusing the two representations at the feature level, we propose the CFD module depicted in Figure 3, which involves two steps, i.e., first converting the cluster features to the BEV space, and then using a class-aware BEV diffusion block to generate diffused cluster features. In the first step, given a series of vote centers  $\mathbf{C}_{clu} = \{c_1, c_2, \dots, c_N\} \in \mathbb{R}^{N \times 3}$  and vote cluster features  $\mathbf{F}_{clu} = \{f_1, f_2, \dots, f_N\} \in \mathbb{R}^{N \times C}$  from the cluster branch, we first map the voting centers to the BEV plane using the following Equation 1.
$$
p _ {x} = \frac {c _ {i} ^ {x} - B _ {x}}{v \cdot d}, p _ {y} = \frac {c _ {i} ^ {y} - B _ {y}}{v \cdot d}, i = 1, 2, \dots , N, \tag {1}
$$
where  $c_{i}^{x}$  is the coordinate of  $x$ -axis for the  $i$ -th vote center,  $B_{x}$  is the boundary of  $x$ -axis for the given scene,  $v$  is the voxel size of the BEV plane, and  $d$  is the downsampling factor of the BEV feature map. Therefore, each cluster feature corresponds to one non-empty BEV pixel located at  $p = (p_{x}, p_{y})$ , thus generating the vanilla vote BEV features (short for vote BEV). However, the established hard association between cluster and BEV features heavily relies on the quality of vote clusters. In fact, the cluster centers represent the potential locations of the centers of objects within a certain area.
To address this issue, in the second step, we propose the class-aware BEV diffusion block. Specifically, we first match the classification results from the cluster branch to the corresponding non-empty grids in the vanilla vote BEV. Secondly, we expand non-empty grids by diffusing their features into neighboring locations with simple max pooling operations on the BEV plane, where the diffusion factor varies in different vote classes to control the expansion magnitude. Thirdly, the generated class-aware vote BEV are together fed into a fully connected layer, which generates the required diffused vote  $\mathbf{BEV}_{voteB} \in \mathbb{R}^{C \times H \times W}$  as the output. A fully connected layer is used for channel-wise projection. Eventually, we can successfully convert the cluster features to the unified BEV plane in a soft and adaptive fashion, thus benefiting joint feature learning.
Imitation Loss. Considering that the center feature of diffused vote BEV  $\mathbf{F}_{voteB}$  and dense BEV  $\mathbf{F}_{denseB}$  is not well aligned, the overall stability of representation learning may be hampered [5]. Therefore, to better distill object-centric knowledge from the cluster branch to the BEV branch and meanwhile avoid the imitation of areas of non-interest (i.e., edge information of the object), we apply an imitation loss to the two features in the BEV space using Equations 2 and 3.
$$
\mathcal {L} _ {i m i} = \operatorname {M a s k} \left(\mathbf {F} _ {\text {v o t e} B}\right) \cdot \| \mathbf {F} _ {\text {v o t e} B} - \mathbf {F} _ {\text {d e n s e} B} \| _ {2}, \tag {2}
$$
$$
\operatorname {M a s k} \left(\mathbf {F} _ {\text {v o t e B}} (p)\right) = \left\{ \begin{array}{l} 0, \text {i f} \mathbf {F} _ {\text {v o t e B}} (p) \text {i s v a i l d}, \\ 1, \text {e l s e}, \end{array} \right. \tag {3}
$$
where  $\mathbf{F}_{\text{vote}B}$  and  $\mathbf{F}_{\text{dense}B}$  denote the diffused vote BEV features from the CFD module and the dense BEV features from the BEV-based main branch. The  $\operatorname{Mask}(\cdot)$  operation only focuses on the certain center-oriented regions highlighted on the diffused vote BEV features, which indicates the potential areas of object centers.
3.4 Query-level Expansion
In this subsection, we introduce the Cluster Query Generation (CQG) module by expanding selected cluster queries which are implicitly supervised by a direction loss, thus enriching the object representation at the query level.
Cluster Query Generation. Most BEV detectors detect objects as center points. It is of significance to directly enrich the diversity of object queries by using the voting centers from the cluster branch. Therefore, we adopt a  $3 \times 3$  convolution with sigmoid on the vanilla vote BEV, to generate a vote activation map. The location of the top  $K$  activation scores will be taken out as cluster queries  $\mathbf{Q}_{clu}$ . The cluster queries are then directly concatenated with BEV queries  $\mathbf{Q}_{BEV}$  (top  $M$  candidates generated from BEV features). The positions and features of the hybrid queries  $\{\mathbf{Q}_{BEV}; \mathbf{Q}_{clu}\}$  are used to initialize the query positions and query features of initial objects. The query embeddings are then fed into a transformer decoder to predict objects.
Direction loss. The quality of cluster queries is implicitly determined by the voting center of all the clusters. To ensure a more accurate voting center for each cluster, we employ a direction loss to constrain the direction of predicted offset vectors, which encourages all foreground points to point to their voting centers precisely. The direction loss [14] is defined based on the cosine similarities using the Equation 4.
$$
\mathcal {L} _ {d i r} = - \frac {1}{\sum_ {i} m _ {i}} \cdot \frac {o _ {i}}{\| o _ {i} \| _ {2}} \cdot \frac {b _ {i} - p _ {i}}{\| b _ {i} - p _ {i} \| _ {2}} \cdot m _ {i}, \tag {4}
$$
where  $m_{i}$  is the foreground mask from the segmentation head,  $o_{i}$  is the offset vector,  $p_{i}$  is the point coordinates and  $b_{i}$  corresponds to the center of the ground-truth box in which the  $p_{i}$  is located.
In this section, we first make comparisons with the state-of-the-art methods on Waymo and nuScenes datasets. Then, we conduct ablation studies to examine the effect of each component of the proposed CluB.
Waymo Open Dataset. In Waymo Open Dataset [26] (WOD), 798, 202 and 150 sequences are used for training, validation and testing, respectively. We adopt the official evaluation metrics, i.e., mean average precision (mAP) and mAPH (mAP weighted by heading) for Vehicle (Veh.), Pedestrian (Ped.), and Cyclist (Cyc.). The metrics are further split into two difficulty levels according to the number of points in ground truth boxes: LEVEL_1 (>5) and LEVEL_2 (≥1).
NuScenes Dataset. NuScenes [2] dataset has 1,000 driving scenes, where 700, 150, and 150 scenes are chosen for training, validation and testing, respectively. The point cloud of nuScenes is collected by a 32-beam LiDAR. There are 10 classes in total. The evaluation metrics used by nuScenes are mAP and nuScenes detection score (NDS). The mAP is defined by the BEV center distance instead of the 3D IoU, and the final mAP is computed by averaging over distance thresholds of  $0.5\mathrm{m}$ ,  $1\mathrm{m}$ ,  $2\mathrm{m}$ ,  $4\mathrm{m}$  across ten classes. NDS is a consolidated metric of mAP and other attribute metrics.
Training. For Waymo, we follow previous voxel-based methods [22, 35, 23, 21] to use point cloud range of  $[-75.2m, 75.2m] \times [-75.2m, 75.2m] \times [-2.0m, 4.0m]$  with voxel size  $[0.1m, 0.1m, 0.15m]$  in  $x$ ,  $y$ , and  $z$ -axes respectively. For nuScenes, we use point cloud range