Title: CluB: Cluster Meets BEV for LiDAR-Based 3D Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D object detection ; Point clouds
Abstract: Currently, LiDAR-based 3D detectors are broadly categorized into two groups, namely, BEV-based detectors and cluster-based detectors.
BEV-based detectors capture the contextual information from the Bird's Eye View (BEV) and fill their center voxels via feature diffusion with a stack of convolution layers, which, however, weakens the capability of presenting an object with the center point.
On the other hand, cluster-based detectors exploit the voting mechanism and aggregate the foreground points into object-centric clusters for further prediction.
In this paper, we explore how to effectively combine these two complementary representations into a unified framework.
Specifically, we propose a new 3D object detection framework, referred to as CluB, which incorporates an auxiliary cluster-based branch into the BEV-based detector by enriching the object representation at both feature and query levels.
Technically, CluB is comprised of two steps.
First, we construct a cluster feature diffusion module to establish the association between cluster features and BEV features in a subtle and adaptive fashion. 
Based on that, an imitation loss is introduced to distill object-centric knowledge from the cluster features to the BEV features.
Second, we design a cluster query generation module to leverage the voting centers directly from the cluster branch, thus enriching the diversity of object queries.
Meanwhile, a direction loss is employed to encourage a more accurate voting center for each cluster.
Extensive experiments are conducted on Waymo and nuScenes datasets, and our CluB achieves state-of-the-art performance on both benchmarks.

CluB: Cluster Meets BEV for LiDAR-Based 3D Object Detection

Abstract

Currently, LiDAR-based 3D detectors are broadly categorized into two groups, namely, BEV-based detectors and cluster-based detectors. BEV-based detectors capture the contextual information from the Bird's Eye View (BEV) and fill their center voxels via feature diffusion with a stack of convolution layers, which, however, weakens the capability of presenting an object with the center point. On the other hand, cluster-based detectors exploit the voting mechanism and aggregate the foreground points into object-centric clusters for further prediction. In this paper, we explore how to effectively combine these two complementary representations into a unified framework. Specifically, we propose a new 3D object detection framework, referred to as CluB, which incorporates an auxiliary cluster-based branch into the BEV-based detector by enriching the object representation at both feature and query levels. Technically, CluB is comprised of two steps. First, we construct a cluster feature diffusion module to establish the association between cluster features and BEV features in a subtle and adaptive fashion. Based on that, an imitation loss is introduced to distill object-centric knowledge from the cluster features to the BEV features. Second, we design a cluster query generation module to leverage the voting centers directly from the cluster branch, thus enriching the diversity of object queries. Meanwhile, a direction loss is employed to encourage a more accurate voting center for each cluster. Extensive experiments are conducted on Waymo and nuScenes datasets, and our CluB achieves state-of-the-art performance on both benchmarks.

1 Introduction

3D perception on point clouds has attracted much attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics [29, 17, 11, 28, 32]. LiDAR-based 3D object detection is one of the vital tasks, which takes sparse point clouds as input to estimate the 3D positions and categories of objects [20, 7, 30, 40].

Currently, the mainstream 3D object detectors [35, 21, 18, 38, 22, 1, 39] convert unstructured point clouds into regular grids in the Bird's Eye View (BEV) for feature representation (Figure 1 (a)), named as BEV-based 3D object detectors. The regular grids in BEV are beneficial to feature extraction and contextual information capturing. As the key point to make a prediction, i.e. , the center point of an object, can be empty, the features of the center points are often diffused from the surrounding points via a stack of convolution layers. This practice significantly weakens the capability of presenting an object with the center point.

(a) BEV-based Representation

(b) Cluster-based Representation

Voting

Aggregation

Diffusion

Extracting

Another stream of detectors [10, 19, 13, 36] resorts to the cluster-based feature representation (Figure 1 (b)). This line of methods exploits the voting mechanism and groups the foreground points into object-centric clusters for further prediction. The cluster features largely preserve the 3D structure details of each object, while avoiding detection based on center points that are absent in the raw data. However, the clustering operation mainly focuses on foreground points, and fails to fully utilize the contextual information. It heavily hinges on foreground segmentation accuracy. Based on the potential synergy between these two paradigms, in this paper, we focus on exploring the combination of context-aware BEV representation and object-centric cluster representation to improve the accuracy of 3D object detection, which remains under-explored in the literature.

Towards this goal, an intuitive solution is to scatter all cluster features to the BEV space based on the locations of the predicted voting centers 1 and directly concatenate the obtained vote BEV features (vote BEV in short) to the dense BEV features (dense BEV in short). However, the established hard association 2 between cluster features and BEV features heavily relies on the quality of vote clusters, not to mention that the representation of these two streams is not well aligned, which may affect the overall stability of representation learning [5]. This potential issue makes the intuitive strategy fail to fully explore the feature intertwining of both representations.

Therefore, by consolidating the idea of leveraging the object-centric property of cluster representation and contextual information of BEV representation, we propose a novel 3D object detection framework, i.e. , CluB , which integrates an auxiliary Clu ster-based branch to the mainstream B EV-based detector by enriching the object representation at both feature and query levels. Specifically, CluB begins with projecting the two features into the BEV space (vote BEV and dense BEV) for unifying the object representation from corresponding branches. Then, we propose a two-step approach to enhance the representation capability from the two feature and query aspects.

Firstly, to address the issue of the hard association between cluster features and BEV features, we propose a Cluster Feature Diffusion (CFD) module that adaptively diffuses the valid votes on the vote BEV to their neighboring regions according to the predicted class of each vote, which repositions the association from hard to soft and adaptive. Based on that, an imitation loss is introduced to transfer object-centric knowledge to the BEV branch and encourage the stability of overall representation learning. In this way, the object representation can be jointly enhanced by fusing features from both branches.

Secondly, since most of the BEV detectors detect objects as center points, it is non-trivial to enrich the diversity of object queries using the voting centers directly from the cluster branch. We thus design a Cluster Query Generation (CQG) module, which activates and then selects the top-ranked candidates based on the vote BEV. The top-selected cluster queries are then appended to the original object queries together for better prediction. On the one hand, we also utilize direction supervision to alleviate the overlap problem for close clusters, thus producing more accurate cluster queries.

The design of CluB strengthens the object representation through feature integration and query expansion, eventually bringing significant performance gains. In summary, we make the following contributions:

Wedevelop the CluB framework to improve the accuracy of 3D object detection by taking advantage of both BEV-based and cluster-based paradigms.

We present an elegant solution to integrating the context-aware BEV representation and objectcentric cluster representation at both the feature level and the query level, which is a non-trivial problem.

Our approach outperforms state-of-the-art methods by a remarkable margin on two prevalent datasets, i.e. , Waymo Open Dataset and nuScenes, demonstrating the effectiveness and generality of our proposed method.

2 Related Work

BEV-based LiDAR Detectors. There are mainly two ways to convert the point cloud into BEV representation. One is to extract BEV features in 2D space and design efficient networks to produce predictions based on the BEV features [15, 21, 9]. PointPillars [15] introduces the pillar representation (a particular form of the voxel) and process it with 2D convolutions. PillarNet [21] resorts to the powerful encoder network for feature extraction and designs the neck network to achieve spatialsemantic feature fusion, thus greatly improving the detection performance of pillar-based models. The other way is to extract point cloud features in 3D space and scatter them to the BEV plane, yielding more accurate detection results [37, 33, 35]. With the BEV representation, CenterPoint [35] designs an anchor-free one-stage detector, which extracts BEV features from voxelized point clouds to estimate object centers and all object properties such as 3D size and orientation. Later, the adoption of the popular transformer architecture as the detection head has substantially improved the accuracy and robustness of BEV-based methods [38, 1, 27, 39]. SWFormer [27] applies the window-based transformer on LiDAR BEV features. In particular, SWFormer solves the center feature missing problem for the 3D sparse features on the BEV planes with the voxel diffusion strategy, while not considering the semantic consistency. Transfusion [1] designs an effective and robust transformerbased LiDAR-camera fusion framework. which can also serve as a strong LiDAR-based 3D detector featured by the BEV-based paradigm.

Cluster-based LiDAR Detectors. The cluster-based methods focus on extracting discriminative features directly from clustered point clouds and have shown promising results in various benchmarks [10, 19, 13, 36]. VoteNet [19] is a pioneering work in the field of cluster-based LiDAR detectors. It employs the voting mechanism to predict object centers from the input point cloud. Motivated by that, FSD [10] designs the fully sparse network in the outdoor scene using point clustering and group correction. PSA-Det3D [13] proposes a pillar set abstraction method (PSA) to learn representative features for sparse point clouds of small objects, which mainly benefits from point-wise feature aggregation. Current benchmarks [26, 2, 31] are dominated by the aforementioned BEV-based and cluster-based 3D detectors. Notably, a coherent work VoxelNext [4] predicts objects directly based on sparse voxel features, without relying on cluster proxies or the dense BEV representation.

In summary, cluster-based detectors are specialized at capturing fine-grained features for individual objects consisting of foreground points, while BEV-based detectors are well-suited for capturing contextual information of point clouds but may have weaker center point representation capabilities. Therefore, by combining these approaches, we can leverage their respective advantages to enhance the detection performance. Our method aims to integrate a BEV-based detector with an auxiliary cluster-based branch, thereby harnessing the strengths of both paradigms to enhance the overall detection performance.

3 Methodology

In this section, we introduce the proposed CluB in detail, as shown in the Figure 2.

Overview. Our CluB is built from scratch using the following three steps. First, we utilize a sparse voxel encoder to extract voxel features and aggregate the vote cluster features, which is considered

Transformer

Decoder

Imitation

Loss

(a) Cluster-based Side Branch

Prediction Head

Cluster Queries

𝑸

𝒄𝒍𝒖

Cluster Feature

Diffusion (CFD)

Diffused Vote BEV

𝑭

𝒗𝒐𝒕𝒆𝑩

(b) BEV-based Main Branch

BEV Queries

𝑸

𝑩𝑬𝑽

SparseConv

Encoder

SparseConv

Decoder

Dense BEV

𝑭

𝒅𝒆𝒏𝒔𝒆𝑩

Heatmap

Q

K,V

To BEV

3D Voxel Features

Sparse Voxel Features

Vote Cluster

Module

Voting

Classification

2D Backbone

Generation (CQG)

Vote

Vote

(c)

(d)

…

𝑭

𝒄𝒍𝒖

,

𝑪

𝒄𝒍𝒖

C

：

Concatenate

C

as the cluster-based auxiliary branch (Section 3.2). Next, a BEV-based main branch (Section 3.1) is served as the detection backbone. Finally, the cluster-based side branch is incorporated into the BEV-based main branch through a two-level enhancement scheme, i.e. , feature-level integration (Section 3.3) and query-level expansion (Section 3.4).

3.1 BEV-based Main Branch

The main BEV branch takes the voxelized 3D point clouds as input and then leverages a sparse encoder consisting of sparse convolution layers to produce 3D voxel features [24]. Then we compress the 3D voxel features into 2D feature maps in the BEV space. The 2D BEV feature map is fed to the 2D convolution-based backbone to generate a dense BEV feature map F denseB ∈ R C × H × W (short for dense BEV), where ( H,W ) indicate the height and width of the feature map. Compared to the anchor/center-based detection head [35, 37], the transformer decoder demonstrates great potential for feature fusion besides the notable performance improvement [34, 8]. We thus utilize a transformer-based detection head [1] to extract class-level feature representations from dense BEV for object prediction. The decoder layer follows the design of DETR [3]. We generate object candidates from the class-specific heatmap and select the top-ranked candidates for all the categories as our initial BEV queries Q BEV . The fused BEV features are also used as key-value sequences for the transformer decoder.

3.2 Cluster-based Auxiliary Branch

The cluster side branch takes the same voxelized 3D point clouds as input to extract sparse voxel features via a sparse convolution based encoder-decoder. These sparse voxel features are fed into two heads for foreground classification and object center voting. Based on the voting results, we group the foreground points into clusters using the grouping strategy ( e.g. , ball query), and then use a vote cluster module to aggregate each cluster feature. The vote cluster module could be implemented with simple vote aggregation following [19]. In order to extract more powerful vote cluster features, we utilize a highly efficient sparse instance recognition module for further extraction [10]. The final

N×

3

…

N×

C

N×

N×

3

C

一对多：

diffusion

…

Class-aware

BEV Diffusion

C

×

H

×

W

To BEV

Class-aware Vote BEV

Diffused Vote BEV

FC layer

Vanilla Vote BEV

3C

×

H

×

W

C

×

H

×

W

Vote

Vote

Vote

Vote

Vote Cluster

Features

Predicted Vote Classes

output of the vote cluster module consists of two parts, i.e. , object-specific cluster features F clu and their corresponding cluster centers C clu .

3.3 Feature-level Integration

In this subsection, we introduce the Cluster Feature Diffusion (CFD) module to generate diffused vote features in the unified BEV space, which are fused with the dense BEV features for representation enhancement. The unified BEV features are supervised by an imitation loss.

Cluster Feature Diffusion. To establish a soft and adaptive association between cluster and BEV features for better fusing the two representations at the feature level, we propose the CFD module depicted in Figure 3, which involves two steps, i.e. , first converting the cluster features to the BEV space, and then using a class-aware BEV diffusion block to generate diffused cluster features. In the first step, given a series of vote centers C clu = { c 1 , c 2 , · · · , c N } ∈ R N × 3 and vote cluster features F clu = { f 1 , f 2 , · · · , f N } ∈ R N × C from the cluster branch, we first map the voting centers to the BEV plane using the following Equation 1.

where c x i is the coordinate of x -axis for the i -th vote center, B x is the boundary of x -axis for the given scene, v is the voxel size of the BEV plane, and d is the downsampling factor of the BEV feature map. Therefore, each cluster feature corresponds to one non-empty BEV pixel located at p = ( p x , p y ) , thus generating the vanilla vote BEV features (short for vote BEV). However, the established hard association between cluster and BEV features heavily relies on the quality of vote clusters. In fact, the cluster centers represent the potential locations of the centers of objects within a certain area.

To address this issue, in the second step, we propose the class-aware BEV diffusion block. Specifically, we first match the classification results from the cluster branch to the corresponding non-empty grids in the vanilla vote BEV. Secondly, we expand non-empty grids by diffusing their features into neighboring locations with simple max pooling operations on the BEV plane, where the diffusion factor varies in different vote classes to control the expansion magnitude. Thirdly, the generated class-aware vote BEV are together fed into a fully connected layer, which generates the required diffused vote BEV F voteB ∈ R C × H × W as the output. A fully connected layer is used for channelwise projection. Eventually, we can successfully convert the cluster features to the unified BEV plane in a soft and adaptive fashion, thus benefiting joint feature learning.

Imitation Loss. Considering that the center feature of diffused vote BEV F voteB and dense BEV F denseB is not well aligned, the overall stability of representation learning may be hampered [5]. Therefore, to better distill object-centric knowledge from the cluster branch to the BEV branch and meanwhile avoid the imitation of areas of non-interest ( i.e. , edge information of the object), we apply an imitation loss to the two features in the BEV space using Equations 2 and 3.

cluster queries

where F voteB and F denseB denote the diffused vote BEV features from the CFD module and the dense BEV features from the BEV-based main branch. The Mask( · ) operation only focuses on the certain center-oriented regions highlighted on the diffused vote BEV features, which indicates the potential areas of object centers.

3.4 Query-level Expansion

In this subsection, we introduce the Cluster Query Generation (CQG) module by expanding selected cluster queries which are implicitly supervised by a direction loss, thus enriching the object representation at the query level.

Cluster Query Generation. Most BEV detectors detect objects as center points. It is of significance to directly enrich the diversity of object queries by using the voting centers from the cluster branch. Therefore, we adopt a 3 × 3 convolution with sigmoid on the vanilla vote BEV, to generate a vote activation map. The location of the top K activation scores will be taken out as cluster queries Q clu . The cluster queries are then directly concatenated with BEV queries Q BEV (top M candidates generated from BEV features). The positions and features of the hybrid queries { Q BEV ; Q clu } are used to initialize the query positions and query features of initial objects. The query embeddings are then fed into a transformer decoder to predict objects.

Direction loss. The quality of cluster queries is implicitly determined by the voting center of all the clusters. To ensure a more accurate voting center for each cluster, we employ a direction loss to constrain the direction of predicted offset vectors, which encourages all foreground points to point to their voting centers precisely. The direction loss [14] is defined based on the cosine similarities using the Equation 4.

where m i is the foreground mask from the segmentation head, o i is the offset vector, p i is the point coordinates and b i corresponds to the center of the ground-truth box in which the p i is located.

4 Experiments

In this section, we first make comparisons with the state-of-the-art methods on Waymo and nuScenes datasets. Then, we conduct ablation studies to examine the effect of each component of the proposed CluB.

Waymo Open Dataset. In Waymo Open Dataset [26] (WOD), 798, 202 and 150 sequences are used for training, validation and testing, respectively. We adopt the official evaluation metrics, i.e. , mean average precision (mAP) and mAPH (mAP weighted by heading) for Vehicle (Veh.), Pedestrian (Ped.), and Cyclist (Cyc.). The metrics are further split into two difficulty levels according to the number of points in ground truth boxes: LEVEL_1 ( > 5) and LEVEL_2 ( ≥ 1).

NuScenes Dataset. NuScenes [2] dataset has 1,000 driving scenes, where 700, 150, and 150 scenes are chosen for training, validation and testing, respectively. The point cloud of nuScenes is collected by a 32-beam LiDAR. There are 10 classes in total. The evaluation metrics used by nuScenes are mAP and nuScenes detection score (NDS). The mAP is defined by the BEV center distance instead of the 3D IoU, and the final mAP is computed by averaging over distance thresholds of 0.5m, 1m, 2m, 4m across ten classes. NDS is a consolidated metric of mAP and other attribute metrics.

4.1 Implementation Details

Training. For Waymo, we follow previous voxel-based methods [22, 35, 23, 21] to use point cloud range of [ -75 . 2 m, 75 . 2 m ] × [ -75 . 2 m, 75 . 2 m ] × [ -2 . 0 m, 4 . 0 m ] with voxel size [0 . 1 m, 0 . 1 m, 0 . 15 m ] in x , y , and z -axes respectively. For nuScenes, we use point cloud range

of [ -51 . 2 m, 51 . 2 m ] × [ -51 . 2 m, 51 . 2 m ] × [ -5 . 0 m, 3 . 0 m ] with voxel size [0 . 1 m, 0 . 1 m, 0 . 2 m ] in x , y , and z -axes respectively. We adopt the same data augmentation setting as [35], including random flipping, global scaling, global rotation, and groundtruth (GT) sampling [33] for Waymo dataset. These settings are similar to those of nuScenes dataset following [1]. We use the one-cycle [25] learning rate schedule and AdamW [16] optimizer with the maximal learning rate 0.001. In addition to the 12-epoch schedule for ablation studies, we adopt a longer schedule (36 epochs) to obtain the best performance on the validation and test set. During the evaluation, we use the NMS IoU threshold of [0.7, 0.25, 0.25]. Our code is built on MMdetection3D [6].

Network. The 3D sparse encoder and decoder are built upon sparse U-Net in PartA2 [24]. For the BEV backbone, we use the same architecture as SECOND [33] and obtain multi-scale BEV features with FPN structure. The top 300 BEV queries are selected from the predicted heatmap following [1] The vote cluster module generates cluster features with 128 dimensions via simple aggregation [19] or further instance-wise feature extraction [10]. For the CFD module, we set the diffusion factors according to the size of different classes. For example, we set 1, 3, and 5 for pedestrians, cyclists, and vehicles in Waymo. For the CQG module. we generate top-ranking cluster queries from the activation map. Since the maximum number of objects in one frame is 185 and 142 for Waymo and nuScenes datasets [1], the number of cluster queries is set as 200 and 150, respectively. We set the number of transformer layers and heads to 3 and 8, and the category labels are simply encoded as one-hot embeddings following [1].

4.2 Main Results

Waymo Results. We summarize the performance of CluB and state-of-the-art (SOTA) 3D detection methods on WOD validation and test sets in Table 1 and Table 2. As shown in Table 1, our CluB achieves competitive single-frame performance and sets new records on Cyclist and Pedestrian of the WOD validation set. CluB achieves comparable mAPH/L2 performance on the validation set compared with the previous best transformer-based model ConQueR [39].

Meanwhile, CluB achieves 0.6% mAPH/L2 performance gain than the powerful detector FSD [10] featured by cluster-based paradigm, even FSD takes longer-range points as input. CluB surpasses the BEV-based method TransFusion-L, which is also our BEV-based baseline, by 6.5% mAPH/L2. It is noteworthy that CluB achieves 1.3% mAPH/L2 performance improvement compared to VoxelNext [4] which utilizes a larger model size for model enhancement( e.g., 5 × kernel size for 3D sparse encoder). Furthermore, with the help of further cluster-wise feature extraction, CluB improves the detection performance of the light version CluB light on all classes. Notably, CluB light consistently demonstrates competitive performance on the Cyclist class , compared with all previous single-frame methods in terms of both APH/L1 and APH/L2.

In addition to offline results, we also report the detection performance on Waymo test set to explore the potential of CluB. As shown in Table 2, our CluB outperforms the previous single-frame LiDARonly methods on the test set, e.g. , 2.0% / 0.2% mAPH/L2 improvement compared with PV-RCNN++ / ConQueR, respectively.

Here, we also give our analysis of why CluB performs better on the cyclist and pedestrian classes. Actually, the cluster-based branch works better when the point clouds are more compact, which means the improvements are more significant for cyclists than that for vehicles. Due to the fixed voxelization and over-downsampling, it is challenging to capture sufficient details of small objects with the BEVbased branch. On the contrary, the cluster branch becomes highly beneficial to extracting fine-grained features from sparser points with smaller cluster voxel sizes, which leads to notable performance improvement.

NuScenes Results. We summarize the performance of CluB and different baselines on the nuScenes test set in Table 3. Notably, compared with the concurrent fully sparse detector VoxelNext [4], CluB achieves 1.2 %/ 1.5 %improvement on NDS / mAP (70.0 → 71.2 , 64.5 → 66.0). Compared with the strong BEV-based baseline Transfusion-L, CluB also leads to 1.0 %and 0.5 %improvement on NDS and mAP (70.2 → 71.2, 65.5 → 66.0 ). The experimental results on Waymo and nuScenes benchmarks validate the effectiveness of CluB by integrating the cluster representation to the BEV branch via the two-level enhancement scheme.

4.3 Ablation Study

To validate the effect of each component in CluB, we conduct ablation studies on the WOD. The models compared in this section are trained with 20% training samples of WOD, and evaluated on the whole validation set.

Effect of Components in CluB. To understand how each module contributed to the final performance in CluB, we test each component independently and report its performance in Table 4. Our baseline detector (a) employs the mainstream BEV paradigm, which starts from 60.5% mAPH/L2. When the CFD module is applied, the mAPH/L2 of Method (b) is raised by 1.4%, which indicates that adaptively integrating the cluster feature in a soft manner is effective to improve 3D detection. Method (c) brings a 2.0% mAPH/L2 improvement by introducing an imitation loss, which validates that explicitly transferring knowledge (object-specific features) to the BEV stream is necessary. On the other hand, Method (d) adds the CQG module based on the Method (a) and brings a total of 0.5% mAPH/L2 enhancement by taking advantage of querying expansion. Method (e) further introduces a direction loss to ensure the quality of cluster queries with 1.1 improvements. The combination of all the components in Method (f) achieves 63.9% mAPH/L2 (3.4 % absolute improvement), validating the effectiveness of CluB. This indicates that enhancing the object representation from two branches at the feature and query levels is of great use to improve detection accuracy.

Effect of Diffusion Strategies in the CFD Module. We compare our proposed class-aware BEV diffusion namely Strategy (d) with other strategies, which are shown in Table 5. Intuitively, we can directly concatenate the cluster feature with the BEV feature based on the location of the voting centers, which is a sub-optimal strategy discussed in Section 1. We consider the naive Strategy (a) relying on hard association as the baseline strategy in Table 5. Strategy (b) uses a 2D convolution layer to diffuse valid features to the neighboring, which not only introduces extra parameters but degrades the overall detection performance. In contrast, Strategy (c) utilizes the max pooling operation, an elegant solution to avoid some unnecessary computation for a more effective representation learning procession. Compared with the class-agnostic manner (c), our diffusion scheme (d) leads to better performance, e.g. , 61.0 → 62.9, 63.4 → 64.4 APH/L2 on vehicle and pedestrian by considering semantic consistency. Strategy (d) establishes the soft and adaptive association between cluster and BEV features, thus successfully enhancing object representation at the feature level.

Effect of Design Choices of Cluster Module. We examine the effects of different design choices of cluster modules in Table 6. The base competitor Method (a) is a BEV-based 3D detector we implemented. Compared with CluB, it abandons the cluster branch. As evidenced in the Method (a) and the Method (b) , by integrating the cluster branch via a simple aggregation strategy, i.e. , K-Nearest Neighbor (KNN), Method (b) performs better, e.g. , 61.0 → 61.4, 60.4 → 64.3 on vehicle and pedestrian. Combining more local features for each cluster (8 neighbors of KNN), our Method (c) achieves a significant improvement on vehicle class, i.e. , 61.0 → 63.2. However, for the pedestrian class, the performance degrades from 60.4 to 59.9 because points that do not belong to pedestrians are aggregated. That indicates that a well-designed clustering algorithm can improve the overall performance. Motivated by that, Method (d) replaces the KNN algorithm with the ball query algorithm, and the detection performance steadily improves among the three classes. Method (e) utilize the elaborate cluster module following [10], which groups points into instance-wise clusters, achieving the satisfactory detection performance shown in the last row in Table 6.

5 Conclusion

In this paper, we introduce CluB, a unified 3D object detection framework that takes advantage of both BEV-based and cluster-based paradigms for improving the accuracy of 3D object detection. To effectively combine the context-aware BEV representation and object-centric cluster representation,

we propose to integrate an auxiliary cluster-based branch to the mainstream BEV-based detector by strengthening the object representation at both feature and query levels. On the one hand, we first establish the association between cluster and BEV features in a soft adaptive way using the CFD module. Based on the association, we transfer the object-specific knowledge from the cluster to the BEV branch supervised by an imitation loss. On the other hand, we design a CQG module to enrich the diversity of object queries using the voting centers directly from the cluster branch. Simultaneously, a direction loss is employed to encourage a more accurate voting center for each cluster. The two-level enhancement ( i.e. , feature integration, and query expansion) design of CluB greatly enriches the object representation ability. With our proposed framework, CluB outperforms the previous single-frame models for detecting 3D objects on both Waymo and nuScenes datasets.

Limitations. Remarkably, CluB presents an elegant view of how to combine the context-aware BEV representation with object-centric cluster representation in a unified detection framework and achieves promising performance via two-level enhancement (feature, query) for object representation. Yet, the computational and spatial complexity on BEV feature maps is quadratic to the perception range, which makes it hard to adopt our CluB for long-range detection. We leave the exploration of computational efficient version of our method as the future work.

References

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR , pages 1080-1089. IEEE, 2022.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , pages 11618-11628. Computer Vision Foundation / IEEE, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV (1) , volume 12346 of Lecture Notes in Computer Science , pages 213-229. Springer, 2020.

Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR , pages 21674-21683. IEEE, 2023.

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal BEV distillation for multi-view 3d object detection. CoRR , abs/2211.09386, 2022.

MMDetection3D Contributors. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https://github.com/open-mmlab/mmdetection3d , 2020.

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: towards high performance voxel-based 3d object detection. In AAAI , pages 1201-1209. AAAI Press, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR . OpenReview.net, 2021.

Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR , pages 8448-8458. IEEE, 2022.

Lue Fan, Feng Wang, Naiyan Wang, and Zhao-Xiang Zhang. Fully sparse 3d object detection. In NeurIPS , 2022.

Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 43(12):43384364, 2021.

Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. In AAAI , pages 969-979. AAAI Press, 2022.

Zhicong Huang, Zhijie Zheng, Jingwen Zhao, Haifeng Hu, Zixin Wang, and Dihu Chen. Psadet3d: Pillar set abstraction for 3d object detection. Pattern Recognit. Lett. , 168:138-145, 2023.

Jean Lahoud, Bernard Ghanem, Martin R. Oswald, and Marc Pollefeys. 3d instance segmentation via multi-task metric learning. In ICCV , pages 9255-9265. IEEE, 2019.

Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR , pages 12697-12705. Computer Vision Foundation / IEEE, 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster) . OpenReview.net, 2019.

Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang, Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, Dinesh Manocha, and Xinge Zhu. Vision-centric BEV perception: A survey. CoRR , abs/2208.02797, 2022.

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV , pages 3144-3153. IEEE, 2021.

Charles R. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV , pages 9276-9285. IEEE, 2019.

Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , pages 77-85. IEEE Computer Society, 2017.

Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillarbased 3d object detection. In ECCV (10) , volume 13670 of Lecture Notes in Computer Science , pages 35-52. Springer, 2022.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: point-voxel feature set abstraction for 3d object detection. In CVPR , pages 10526-10535. Computer Vision Foundation / IEEE, 2020.

Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN++: point-voxel feature set abstraction with local vector representation for 3d object detection. Int. J. Comput. Vis. , 131(2):531-551, 2023.

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE Trans. Pattern Anal. Mach. Intell. , 43(8):2647-2664, 2021.

Leslie N. Smith and Nicholay Topin. Super-convergence: very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications , volume 11006, page 1100612. SPIE, 2019.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR , pages 2443-2451. Computer Vision Foundation / IEEE, 2020.

Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV (10) , volume 13670 of Lecture Notes in Computer Science , pages 426-442. Springer, 2022.

Luequan Wang, Hongbin Xu, and Wenxiong Kang. Mvcontrast: Unsupervised pretraining for multi-view 3d object recognition. Machine Intelligence Research , pages 1-12, 2023.

Yingjie Wang, Qiuyu Mao, Hanqi Zhu, Yu Zhang, Jianmin Ji, and Yanyong Zhang. Multi-modal 3d object detection in autonomous driving: a survey. CoRR , abs/2106.12735, 2021.

Yue Wang and Justin M. Solomon. Object DGCNN: 3d object detection using dynamic graphs. In NeurIPS , pages 20745-20758, 2021.

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS Datasets and Benchmarks , 2021.

Dong Wu, Man-Wen Liao, Wei-Tian Zhang, Xing-Gang Wang, Xiang Bai, Wen-Qing Cheng, and Wen-Yu Liu. Yolop: You only look once for panoptic driving perception. Machine Intelligence Research , 19(6):550-562, 2022.

Yan Yan, Yuxing Mao, and Bo Li. SECOND: sparsely embedded convolutional detection. Sensors , 18(10):3337, 2018.

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, Jie Zhou, and Jifeng Dai. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. CoRR , abs/2211.10439, 2022.

Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. In CVPR , pages 11784-11793. Computer Vision Foundation / IEEE, 2021.

Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In CVPR , pages 18931-18940. IEEE, 2022.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR , pages 4490-4499. Computer Vision Foundation / IEEE Computer Society, 2018.

Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV (38) , volume 13698 of Lecture Notes in Computer Science , pages 496-513. Springer, 2022.

Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. In CVPR , pages 9296-9305. IEEE, 2023.

Xinge Zhu, Yuexin Ma, Tai Wang, Yan Xu, Jianping Shi, and Dahua Lin. SSN: shape signature networks for multi-class object detection from point clouds. In ECCV (25) , volume 12370 of Lecture Notes in Computer Science , pages 581-597. Springer, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper combines two streams of LiDAR-based 3D object detectors into a single model: conventional BEV-based detectors and emerging cluster-based detectors. The main contribution is that It is the first one to combine the two streams of methods, leveraging the high-recall of BEV representation and fine-grained structure of cluster representation. It reveals the potential of such a combination, providing a possibility for a new sub-direction.\n\nSTRENGTHS:\n- This paper is well written with clear representation and concise illustration. For example, Figure 2 is very informative and concise, and I could clearly get the overall idea at first glance.\n\n- To my best knowledge, it is the first one to explicitly combine the two streams of methods, leveraging the high-recall of BEV representation and fine-grained structure of cluster representation. Although straightforward, the idea is well-motivated and makes sense to me.\n\n- Every module is well ablated. Overall performance on Waymo Open Dataset and nuScenes dataset is good. Especially, on nuScenes, it achieves state-of-the-art performance.\n\nWEAKNESSES:\n- Although the overall writing is clear, the authors are not very careful with some details. For example, Figure 1 (a) is very similar to a figure in the FSD paper, and it would be appropriate to add a citation. In L108, the citations should be placed after \"benchmarks\" instead of detectors. In Eqn (4), $Ldir$ should be $L_{dir}$.  There are many errors in Table 1: the best and second-best entries of L2 Vehicle should be PillarNet and PVRCNN++. The performance of FSD seems inaccurate because it has a similar vehicle performance to PVRCNN++ if I recall correctly. Please check. The second-best Pedestrian L2 APH is not underlined. And this table size is out of the range on the right.\n\n- Although experiments thoroughly verify the effectiveness of proposed modules, they are straightforward ablation without insight. I encourage authors to add more detailed performance analysis to reveal the inner workings of CluB. For example, how do the two kinds of queries affect the final performance?  Since BEV-based representation has higher recall and cluster-based representation preserves fine-grained structure, authors should design experiments to demonstrate such properties, which cannot be demonstrated by simple numbers of the final performance. Table 1 in supplementary materials should be moved to the main paper, and the current 9-page space is not fully utilized.\n\n- I encourage authors to conduct runtime evaluations and report the latency.\n\nQUESTIONS:\nI like the idea of combining the two kinds of representation. However, the authors should carefully address my concerns.\nThe manuscript should be updated in the OpenReview system if possible to address the first weakness. Additional experiments could be simply posted on this page for now.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work combines the BEV-based and cluster-based representations into a unified framework named CluB. At the feature level, the Cluster Feature Diffusion module and the imitation loss are proposed to fuse the features obtained from the BEV branch and cluster branch. At the query level, the Cluster Query Generation module and the direction loss are proposed to provide more accurate object queries from the cluster branch. CluB achieves state-of-the-art performance on the Waymo and nuScenes datasets.\n\nSTRENGTHS:\nThis work mentions that the features of the object centers in BEV-based detectors are diffused from their neighbors and are not in high quality, which is an interesting hypothesis. Based on that, the cluster branch is applied to provide clustered features of the object centers. The whole structure of the work is compact and unified, and the promotion on precision is clearly shown.\n\nWEAKNESSES:\n1. The argumentation of the core hypothesis is insufficient. I think that the max pooling in BEV-based detectors is able to reduce the feature shape and aggregate the feature of the object to the center, just like the cluster-based methods. Additional quantitative analysis is preferred to show the difference.\n2. The captions of figures and tables are too long and mainly repeat the text of the main body.\n\nQUESTIONS:\n1. Some results in the ablation study do not match with each other. For example, the mAPH of Table 4 (b) should be the same as (d), but in fact they are not. Are there differences between the two configurations?\n2. I am wondering why CluB performs better on the cyclist class of Waymo, because the point clouds of cyclists are more compact than the vehicles and should benefit less from the cluster branch.\n3. How efficient is CluB compared to the baseline? The time cost could be significantly increased by applying U-Net in the cluster branch.\n\nLIMITATIONS:\nIt would be nice to apply different BEV-based detectors as the baseline to test the universality of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors explore and analyze the existing LiDAR-based 3D object detection framework, and propose to adopt BEV-based and cluster-based methods to aggregate those features from LiDAR input. The experimental results on Waymo and nuScenes are better compared to the existing methods.

STRENGTHS:
1. The task of 3D object detection is very important in the 3D community. Interestingly, the authors propose to extract and combine BEV and cluster-based features.

2. The paper is easy to follow.

3. The authors conduct the experiments on two widely-used datasets, including nuScenes and Waymo.

WEAKNESSES:
1. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D detection methods. The time consumption might be large since the proposed method includes transforming the vote cluster to BEV matching and aggregation.

2. Most 3D object detection SOTA is multi-frame based. It would be interesting to see how the multiple frames fitted into the proposed architecture.

QUESTIONS:
Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.

LIMITATIONS:
Yes. The authors presented one of the limitations in terms of computational tradeoffs.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new 3D object detection framework called CluB that combines the strengths of BEV-based and cluster-based detectors. CluB effectively integrates the context-aware BEV representation and object-centric cluster representation at both the feature level and the query level.\nThe proposed method outperforms state-of-the-art methods by a remarkable margin on two prevalent datasets, i.e., Waymo Open Dataset and nuScenes, demonstrating the effectiveness and generality of the proposed method.\n\nSTRENGTHS:\n1. The paper introduces a novel framework that combines the strengths of BEV-based and cluster-based detectors at both the feature level and the query level. The proposed combination method is of a certain degree of novelty.\n2. The paper is well-written and easy to follow. The authors provide clear explanations of the proposed method and the experimental results. \n3. The paper presents a well-designed and comprehensive experimental evaluation of the proposed CluB framework. The authors provide detailed analysis and comparison with state-of-the-art methods on prevalent datasets.\n\nWEAKNESSES:\n1. Limited novelty: While the CluB framework is unique and original in its approach to integrating BEV-based and cluster-based detectors, the paper does not introduce any fundamentally new concepts or techniques. The proposed method is built on existing methods and combines them in a novel way. The novelty is limited for NeurIPS.\n\n2. Limited analysis of real-time performance: The paper does not provide a detailed analysis of the real-time performance of the proposed CluB framework. To address this weakness, the authors could provide a more detailed analysis of the real-time performance of the proposed method and explore ways to make the method more suitable for real-time scenarios.\n\n3. Limited analysis of the combination result of two level: According to the ablation study in table 4. The comprehensive improvement effect of the two levels (3.4%) is greater than the sum of their individual improvement effects (2% + 1.1%). I think this may be the essential reason why the proposed framework is effective, but this issue lacks in-depth analysis.\n\nQUESTIONS:\n1. According to the expression in the paper, Diffused Vote BEV features and Dense BEV features represent the center point semantic information and edge information respectively. But in the design of the Club, Imitation Loss is used to make the two similar. This seems to be contradictory to the previous statement. Can you give some further explanation?\n2. As stated in Weakness #3, can you provide a more detailed analysis of the the combination result of two level? \n2. Can you provide a more detailed analysis of the computational complexity?\n3. Can you provide a more detailed analysis of the impact of the proposed CluB framework on real-time performance?\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a CluB framework to improve the accuracy of 3D object detection by taking advantage of both BEV-based and cluster-based paradigms. The motivation is that the cluster features in voting-based cluster method can largely preserve the 3D structure details of each object, thus supplements the weakened center point features in BEV-based convolutional methods. To achieve the above goals, the authors propose a Cluster Feature Diffusion (CFD) module that adaptively diffuses the valid votes on a vote BEV and fuse it with dense BEV features. An imitation loss is also introduced to transfer object-centric knowledge to the BEV branch and encourage the stability of overall representation learning. Meanwhile, a Cluster Query Generation (CQG) module is proposed to enrich the diversity of object queries by using the voting centers from the cluster branch. Extensive experiments are conducted on the Waymo and NuScenes dataset. Both the ablation studies and the comparison with the SOTA methods demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\nOverall the paper is well-written and well-structured.\nThe experiment results are convincing.\nThe motivation of combining the BEV-based and voting-based method is somewhat novel.\nFrom the technical aspect, although the design of CFD and CQG modules is vulgaris, however it is also effective.\n\nWEAKNESSES:\nThe demonstration of the class-aware BEV diffusion is not clear. How to leverage the classification results to control the expansion magnitude? I think there should be a formal formula. Figure 3 is also confusing, how to get Class-aware Vote BEV?\nThere is a lack of discussion of Model Efficiency. It seems that the cluster branch is time-consuming.\n\nQUESTIONS:\nIntuitively, it seems like that the convolution operation performed on feature maps is also like an implicit clustering method that aggregates the surrounding features. Is it necessary to make such big effort to introduce a new cluster branch? If you want to alleviate the phenomenon that a stack of convolution layers will weaken the capability of presenting an object with the center point and reduce structure information, how about using deformable convolution?\nOther question, please refer the weakness part above and give your demonstration.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}