CluB: Cluster Meets BEV for LiDAR-Based 3D Object Detection
Authors: Yingjie Wang
Jiajun Deng
Yuenan Hou
Yao Li
Yu Zhang
Jianmin Ji
Wanli Ouyang
Yanyong Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D object detection ; Point clouds

Abstract: Currently, LiDAR-based 3D detectors are broadly categorized into two groups, namely, BEV-based detectors and cluster-based detectors.
BEV-based detectors capture the contextual information from the Bird's Eye View (BEV) and fill their center voxels via feature diffusion with a stack of convolution layers, which, however, weakens the capability of presenting an object with the center point.
On the other hand, cluster-based detectors exploit the voting mechanism and aggregate the foreground points into object-centric clusters for further prediction.
In this paper, we explore how to effectively combine these two complementary representations into a unified framework.
Specifically, we propose a new 3D object detection framework, referred to as CluB, which incorporates an auxiliary cluster-based branch into the BEV-based detector by enriching the object representation at both feature and query levels.
Technically, CluB is comprised of two steps.
First, we construct a cluster feature diffusion module to establish the association between cluster features and BEV features in a subtle and adaptive fashion. 
Based on that, an imitation loss is introduced to distill object-centric knowledge from the cluster features to the BEV features.
Second, we design a cluster query generation module to leverage the voting centers directly from the cluster branch, thus enriching the diversity of object queries.
Meanwhile, a direction loss is employed to encourage a more accurate voting center for each cluster.
Extensive experiments are conducted on Waymo and nuScenes datasets, and our CluB achieves state-of-the-art performance on both benchmarks.

CluB: Cluster Meets BEV for LiDAR-Based 3D Object Detection
Yingjie Wangl.2 .Jijun Deng", Yuenan Hou", Yao Li', Yu Zhang', Jianmin Ji', Wanli Ouyang', Yanyong Zhang"! University of Science and Technology of China, Shanghai Al Laboratory, 3The University of Adelaide, AIML
Abstract
Currently, LiDAR-basee 3D detectors are broadly categorized into two groups, namely, BEV-based detectors and cluster-based detectors. BEV-based detectors capture the contextual information from the Bird's Eye View (BEV) and fill their center voxels via feature diffusion with a stack of convolution layers, which, how- ever, weakens the capability of presenting object with the center point. On the other hand, cluster-based detectors exploit the voting mechanism and aggregate the foreground points into object-centric clusters for further prediction. In this paper, we explore how to effectively combine these two complementary representations into a unified framework. Specifically, we propose new 3D object detection framework, referred to as CluB, which incorporates an auxiliary cluster-based branch into the BEV-based detector by enriching the object representation at both feature and query levels. Technically, CluB is comprised of two steps. First, we construct cluster feature diffusion module to establish the association between cluster features and BEV features in subtle and adaptive fashion. Based on that, an imitation loss is introduced to distill object-centric knowledge from the cluster features to the BEV features. Second, we design cluster query generation module to leverage the voting centers directly from the cluster branch, thus enriching the diversity of object queries. Meanwhile, a direction loss is employed to encourage a more accurate voting center for each cluster. Extensive experiments are con- ducted on Waymo and nuScenes datasets, and our CluB achieves state-of-the-an performance on both benchmarks.
Introduction
3D perception on point clouds has attracted much attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics 20 [7 [T [28 32). LiDAR-based 3D object detection is one of the vital tasks, which takes sparse point clouds as input to estimate the 3D positions and categories of objects 20
40)
convert unstructured point clouds into regular grids in the Bird's Eye View (BEV) for feature representation (Figure (a)), named as BEV-based 3D object detectors. The regular grids in BEV are beneficial feature extraction and contextual information capturing. As the key point to make a prediction, i.e., the center point of an object, can be empty, the features of the center points are often diffused from the surrounding points via stack of convolution layers. This practice significantly weakens the capability of presenting an object with the center point.
Currently, the mainstream 3D object detectors [35 21
38
This work was done during Yingjie's 's internship at Shanghai Artificial Intelligence Laboratory. Corresponding Authors: Jiajun Deng Yanyong Zhang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Extracting
Diffusion
Voting
Aggregation
(a) BEV-based Representation
(b) Cluster-based Representation
Figure I: Illustration of (a) context-aware BEV-based representation [0] and (b) object-centric cluster-based representation. The BEV representation can capture the rich contextual information of object. However, the missing feature of the center point (red square), which is also the key to making the prediction, diffused from occupied voxels. The cluster- based representation is generated using the voting mechanism by aggregating the foreground points into the object-specific feature for the voting center (red dot). This representation mainly focuses on the foreground points and fails utilize the contextual information.
Another stream of detectors [0 19
36 resorts to the cluster-based feature representation (Figure [1 (b)). This line of methods exploits the voting mechanism and groups the foreground points into object-centric clusters for further prediction. The cluster features largely preserve the 3D structure details of each object, while avoiding detection based on center points that are absent in the raw data. However, the clustering operation mainly focuses on foreground points, and fails to fully utilize the contextual information. heavily hinges on foreground segmentation accuracy. Based on the potential synergy between these two paradigms, in this paper, we focus exploring the combination of context-aware BEV representation and object-centric cluster representation to improve the accuracy of 3D object detection, which remains under-explored in the literature.
13
Towards this goal, an intuitive solution is scatter all cluster features to the BEV space based on the locations of the predicted voting centers" and directly concatenate the obtained vote BEV features (vote BEV in short) to the dense BEV features (dense BEV in short). However, the established hard association betweee between cluster features and BEV features heavily relies on the quality of vote clusters, not mention that the representation of these two streams is not well aligned, which may affect the overall stability of representation learning (5). This potential issue makes the intuitive strategy fail to fully explore the feature intertwining of both representations.
Therefore, by consolidating the idea of leveraging the object-centric property of cluster representation and contextual information of BEV representation, we propose novel 3D object detection framework. i.e., CluB, which integrates an auxiliary Cluster-based branch to the mainstream BEV-based detector by enriching the object representation at both feature and query levels. Specifically, CluB begins with projecting the two features into the BEV space (vote BEV and dense BEV) for unifying the object representation from corresponding branches. Then, we propose two-step approach to enhance the representation capability from the two feature and query aspects.
Firstly, address the issue of the hard association between cluster features and BEV features, propose a Cluster Feature Diffusion (CFD) module that adaptively diffuses the valid votes on the vote BEV to their neighboring regions according to the predicted class of each vote, which repositions the association from hard to soft and adaptive. Based on that, an imitation loss is introduced to transfer object-centric knowledge to the BEV branch and encourage the stability of overall representation learning. In this way, the object representation can be jointly enhanced by fusing features from both branches.
Secondly, since most of the BEV detectors detect objects as center points, it is non-trivial to enrich the diversity of object queries using the voting centers directly from the cluster branch. We thus design Cluster Query Generation (CQG) module, which activates and then selects the top-ranked candidates based the vote BEV. The top-selected cluster queries then appended to the original object queries together for better prediction. On the one hand, we also utilize direction supervision to alleviate the overlap problem for close clusters, thus producing more accurate cluster queries.
'The expression of the vote center is referred to as VoteNet 19 generate votes, which serve as the cluster centers. In fact, the vote centers indicate the potential locations of the centers of objects within certain
VoteNet processes existing points to

---

The design of CluB strengthens the object representation through feature integration and query expansion, eventually bringing sigmificant performance gains. In summary. we make the following contributions:
We develop the CluB framework to improve the accuracy of 3D object detection by taking advantage of both BEV-based and cluster-based paradigms.
We present an elegant solution to integrating the context-aware BEV representation and object- centric cluster representation at both the feature level and the query level, which is a non-trivial problem.
Our approach outperforms state-of-the-art methods by a remarkable margin two prevalent datasets, i.e., Waymo Open Dataset and nuScenes, demonstrating the effectiveness and generality of our proposed method.
2 Related Work
BEV-based LiDAR Detectors. There are mainly two ways to convert the point cloud into BEV representation. One to extract BEV features 2D space and design efficient networks to produce predictions based on the BEV features [15 21 9 PointPillars [[5] introduces the pillar representation (a particular form of the voxel) and process it with 2D convolutions. PillarNet (21) resorts to the powerful encoder network for feature extraction and designs the neck network to achieve spatial- semantic feature fusion, thus greatly improving the detection performance of pillar- basee models. The other way is to extract point cloud features in 3D space and scatter them to the BEV plane, yielding more accurate detection results [37 33 (35] With the BEV representation, CenterPoint 335) designs anchor-free one-stage detector, which extracts BEV features from voxelized point clouds to estimate object centers and all object properties such as 3D size and orientation. Later, the adoption of the popular transformer architecture as the detection head has substantially improved the accuracy and robustness of BEV-based methods [38 M 27 39]. SWFormer [27] applies the hindow-based transformer LiDAR BEV features. In particular, SWFormer solves the center feature missing problem for the 3D sparse features on the BEV planes with the voxel diffusion strategy, while not considering the semantic consistency. Transfusion [1] designs effective and robust transformer- based LiDAR --ammrr fusion framework. which can also serve as a strong LiDAR-based 3D detector featured by the BEV-based paradigm.
Cluster-based LiDAR Detectors. The cluster-based methods focus on extracting discriminative features directly from clustered point clouds and have shown promising results in various bench- marks 110 19.
13 36). VoteNet [19] is a pioncering work in the field of cluster-based LiDAR detectors. It employs the voting mechanism to predict object centers from the input point cloud. Motivated by that, FSD [0] designs the fully sparse network the outdoor scene using point cluster- ing and group correction. PSA-Det3D [3 proposes pillar set abstraction method (PSA) to learn representative features for sparse point clouds of small objects, which mainly benefits from point-wise dominated by the aforementioned BEV-based and cluster-based 3D detectors. Notably, a coherent work VoxeINext [4) predicts objects directly based on sparse voxel features, without relying on cluster proxies the dense BEV representation.
feature aggregation. Current benchmarks 33
In summary, cluster-based detectors are specialized at capturing fine-grained features for individual objects consisting of foreground points, while BEV-based detectors are well-suited for capturing contextual information of point clouds but may have weaker center point representation capabilities. Therefore, by combining these approaches, we can leverage their respective advantages to enhance the detection performance. Our method aims to integrate a BEV-based detector with an auxiliary cluster-based branch, thereby harnessing the strengths of both paradigms to enhance the overall detection performance.
3 Methodology
In this section, we introduce the proposed CluB in detail, as shown in the Figure 2
Overview. Our CluB built from scratch using the following three steps. First, we utilize a sparse voxel encoder to extract voxel features and aggregate the vote cluster features, which is considered
3

---

(a) Cluster-based Side Branch
Custer Generation (COG)
Classification cation
Vote Cluster Medule
Fetue
Votte
Cluster Feature
(d)
Votting
(c)
f f Sparse Vowal Features
Querles Qri
SpanseCorv Decoder
Imitation
Diffiveed Vote BEV Freees
BEV Qiveries Quer
pesg u00 mg
20 Backkborr
Transformer Decodor
LIDAR Points
SonrseConv
Dense BEV
Heatmap
k.
Encoder
(b) BEV-based Main Branch
Conaeeeate
Figure 2: The overall framework of CluB. The cluster-based side branch (a) is integrated into the BEV-based main branch (b) through two-step scheme, i.e., feature-level integration (c) and query- level expansion (d), Specifically, the raw point clouds are first voxelized and then fed into the sparse U-Net extract both 3D voxel features and sparse voxel features. The former used obtain the dense BEV feature. The latter is processed in the cluster branch to generate set of vote cluster features ctu and corresponding voting centers Ceiu- Based on the two outputs, i.e., Fetu and Celu. the Cluster Feature Diffusion (CFD) module obtains diffused vote BEV feature F Fatte to enhance the dense BEV feature denseBB supervised by an imitation loss, and the Cluster Query Generation (CQG) module outputs set of top-ranking cluster queries Qetu- which are concatenated to the BEV queries QBEV obtained from the heatmap. The decoder takes hybrid object queries as input and makes the final predictions based the concatenated BEV features.
as the cluster-based auxiliary branch (Section 3.2) Next, a BEV-based main branch (Section 3.1 is served as the detection backbone. Finally, the cluster-based side branch is incorporated into the BEV-based main branch through a two-level enhancement scheme, i.e., feature-level integration 3.4
(Section 3.3) and query-level expansion (Section 3.4)
3.1 BEV-based Main Branch
The main BEV branch takes the voxelized 3D point clouds as input and then leverages sparse encoder consisting of sparse convolution layers to produce 3D voxel features (24). Then compress the 3D voxel features into 2D feature maps in the BEV space. The 2D BEV feature map is fed to the 2D convolution-basee backbone to generate dense BEV feature map Fdensen Cx (short for dense BEV), where HH, W)) indicate the height and width of the feature map. Compared the anchor/cemter-basee detection head 35 37). the transformer decoder demonstrates great potential for feature fusion besides the notable performance improvement [34] 8
We thus utilize transformer-bee bassed detection head [ extract class-level feature representations from dense BEV for object prediction. The decoder layer follows the design of DETR [3]. We generate object candidates from the class-specific heatmap and select the top-ranked candidates for all the categories as our initial BEV queries QBEV. The fused BEV features are also used as key-value sequences for the transformer decoder.
3.2 Cluster-based Auxiliary Branch
The cluster side branch takes the same voxelized 3D point clouds as input to extract sparse voxel features via sparse convolution based encoder-decoder. These sparse voxel features fed into two heads for foreground classification and object center voting. Based on the voting results, we group foreground points into clusters using the grouping strategy (e.g., ball query), and then use vote cluster module to aggregate each cluster feature. The vote cluster module could be implemented with simple vote aggregation following [19]. In order to extract more powerful vote cluster features, we utilize highly efficient sparse instance recognition module for further extraction [10]. The final

---

Vote
CHHKW
3C
C
Vote
To BEV
Class-aware
Fayye
BEV Diffusion
Vote
Vote Cluster
Vanilla Vate BEV
Class-aware Vote BEV
Diffused Vote BEV
Features
Predicted Vote Classes
Figure 3: Illustration of Cluster Feature Diffusion (CFD) module. We first transform the vote cluster features the BEV space based the position of each vote, forming sparse representation termed vanilla vote BEV features. Next, the non-empty grids of vanilla vote BEV are adaptively expanded their neighboring regions according to the predicted class of the vote through class-aware BEV diffusion block. Finally, we use fully connected layer to project the class-aware vote BEV to the same channel dimension as the vanilla BEV, thus obtaining diffused vote BEV feature.
output of the vote cluster module consists of two parts, i.e., object-specific cluster features F Fetu and their corresponding cluster centers C.t..
3.3 Feature-level Integration
In this subsection, we introduce the Cluster Feature Diffusion (CFD) module generate diffused vote features in the unified BEV space, which are fused with the dense BEV features for representation enhancement. The unified BEV features are supervised by an imitation loss.
Cluster Feature Diffusion. To establish soft and adaptive association between cluster and BEV features for better fusing the two representations at the feature level, we propose the CFD module depicted in Figure 3] which involves two steps, i.e, first converting the cluster features to the BEV space, and then using class-aware BEV diffusion block to generate diffused cluster features. the first step, given series of vote centers Catu = {ei, 62, 2..... ,N } E R-N and vote cluster features Feu- = {f1, f2.... ... fN E NxC from the cluster branch, we first map the voting centers to the BEV plane using the following Equation
B2 U v.d Pa
C
By v.d
i = =,2, 1.2,...,,N.
where of is the coordinate of x-axis for the i-th vote center, B2 is the boundary of r-axis for the given scene, v the voxel size of the BEV plane, and d is the downsampling factor of the BEV feature map. Therefore, each cluster feature corresponds to one non-empty BEV pixel located at p (Pa. Py), thus generating the vanilla vote BEV features (short for BEV). However, the established hard association between cluster and BEV features heavily relies on the quality of vote clusters. In fact, the cluster centers represent the potential locations of the centers of objects within certain area.
To address this issue, in the second step, we propose the awase BEV diffiusion block. Specifically, first match the classification results from the cluster branch to the corresponding non-empty grids in the vanilla vote BEV. Secondly, we expand non-empty grids by diffusing their features into neighboring locations with simple max pooling operations the BEV plane, where the diffusion factor varies in different vote classes to control the expansion magnitude. Thirdly, the generated class-aware BEV are together fed into a fully connected layer, which generates the required diffused vote BEV F Fvote E CC x as the output. A fully connected layer is used for channel- wise projection. Eventually, can successfully convert the cluster features to the unified BEV plane in soft and adaptive fashion, thus benefiting joint feature learning.
Imitation Loss. Considering that the center feature of diffused vote BEV F Fote and dense BEV F B is not well aligned, the overall stability of representation learning may be hampered (5). Therefore, to better distill object-centric knowledge from the cluster branch to the BEV branch and
meanwhile avoid the imitation of areas of non- con-interest (i.e., edge information of the object), we apply an imitation loss to the two features in the BEV space using Equations
and
Limi Mask (F B) || Foote F FaeneeB
(2)
5

---

0, if F (p) is vaild, Mask (F voteB p))
1, else,
where F FooteB and F denseB denote the diffused vote BEV features from the CFD module and the dense BEV features from the BEV-based main branch. The Mask(-) operation only focuses on the certain center-oriented regions highlighted on the diffused vote BEV features, which indicates the potential areas of object centers.
3.4 Query-level Expansion
In this subsection, we introduce the Cluster Query Generation (CQG) module by expanding se- lected cluster queries which are implicitly supervised by direction loss, thus enriching the object representation at the query level.
Cluster Query Generation. Most BEV detectors detect objects as center points. It is of significance to directly enrich the diversity object queries by using the voting centers from the cluster branch. Therefore, we adopt 3 convolution with sigmoid the vanilla vote BEV, to generate vote activation map. The location of the top K activation scores will be taken out as cluster queries Qatu- The cluster queries then directly concatenated with BEV queries QBEV (top M candidates generated from BEV features). The positions and features of the hybrid queries {QBEV: Qutu } are used to initialize the query positions and query features of initial objects. The query embeddings then fed into a transformer decoder to predict objects.
Direction loss. The quality of cluster queries is implicitly determined by the voting center of all the clusters. To ensure a more accurate voting center for each cluster, we employ a direction loss to constrain the direction of predicted offset vectors, which encourages all foreground points point to their voting centers precisely. The direction loss [14] is defined based on the cosine similarities using the Equation
O2
b: Ps
Ldir Σ, m,
m.;
T0:11 1b: Pill2
where m, is the foreground mask from the segmentation head, o; is the offset vector, ps is the point coordinates and b, corresponds the center of the ground-truth box in which the p, is located.
4 Experiments
In this section, we first make comparisons with the state-of-the-art methods on Waymo and nuScenes datasets. Then, we conduct ablation studies examine the effect of each component of the proposed CluB.
Waymo Open Dataset. Waymo Open Dataset 226 (WOD), 798, 202 and 150 sequences are used for training, validation and testing, respectively. We adopt the official evaluation metrics, i.e., mean average precision (mAP) and mAPH (mAP weighted by heading) for Vehicle (Veh.), Pedestrian (Ped.), and Cyclist (Cyc.). The metrics are further split into two difficulty levels according to the number of points ground truth boxes: LEVEL_ (>5) and LEVEL (21).
NuScenes Dataset. NuScenes 2 dataset has 1,000 driving scenes, where 700, 150, and 150 scenes are chosen for training, validation and testing, respectively. The point cloud of nuScenes is collected by 32-beam LiDAR. There are 10 classes in total. The evaluation metrics used by nuScenes are mAP and nuScenes detection score (NDS). The mAP is defined by the BEV center distance instead of the 3D IoU, and the final mAP is computed by averaging over distance thresholds of 0.5m, Im, 2m, 4m across ten classes. NDS is a consolidated metric of mAP and other attribute metrics.
4.1 Implementation Details
Training. For Waymo, we follow previous voxel-based methods 22 point cloud range of [-75.2m, 75.2m] x [-75.2m, 75.2m] x 1-2.0m 4.0m) with voxel size 10.1m, 0. Om, 05ml] in æ, y, and z-axes respectively. For nuScenes, point cloud range
35 X
21
to use

---

mAP/mAPH
Vehicle 3D
APIAPH
Pedestrian 3D
APIAPH
Cyclist 3D APIAPH
Methods
L2
L2
L1
L2
L1
L2
LI
IA-SSDA 36
62.3/58.1
61.6/6100
70.5/69.1
60.3/50.7
69.4/58.5
65.0/62.7
67. 67.7/65.3
FSDAf 10
72.9/70.8
70.5/70.1
79.2/78.8
73.9/ 69.1
82.6/77.3
74.4/73.3
77.1/76.0
CenterPoint
35
-/67.4
-/67.9
-/-
-/65.6
-/-
-/68.6
PV-RCNN
66.8/63.3
69.0/68.4
77.5/76.9
66.0/57.6
75.0/65.6
65.4/64.0
67 8/66/
AFDetV2 12
71.0/68.8
69.7/69.2
77.6/77.1
72.2/67.0
80.2/74.6
71.0/70.1 -/-
73.7/72.7
SST_TS, [
-/-
68.0/67.6
76.2/75.8
72.8/65.9
81.4/74.1
PillarNet-34
71.0/68.5
70.9/70.5
79.1/78.6
72.3/66.2
80.6/74.0
69.7/68.7
72.3/71.2
PV-RCNN+
71.7/69.5
70.6/70.2
79.3/78.8
73.2/68.0
81.3/76.3
71.2/70.2
73.7/12.7
TransFusion-L
./64.9
./65.1
-/63.7
-165.9
CenterFormer
38
71.2/69.0
70.2/69.7
75.2/74.7
73.6/68.3
78.6/73.0
69.8/68.8
72.3/71.3
ConQueR 39
70.3/67.7
68. 68.7/68.2
76.1/75.6
70.9/64.7
79.0/72.3
71.4/70.1
73.9/72.5
ConQueR+
39
74.0/71.6
71.0/70.5
78.4/77.9
75.8/70.1
82.4/76.6
75.2/74.1
77.5/76.4
VoxelNext
70.9/68.2
69.7/69.2
77.9/77.5
72.2/65.9
80,2/73.5
7
73.3/72.2
VoxeINext-K33
72.2/70.1
69.9/69.4
78.2/77.7
73.5/68.6
81.5/76.3
73.3/72.2
76.1/74.9
CluBrighe (ours)
71.2/69.3
68.4/68.0
76.5/76.0
71.1/66.1
79.4/73.9
74.3/73.0
76.7/75.3
CluB (ours)
73.4/71.4
70.4/69.9
79.1/78.6
74.3/69.5
83.1/77.9
75.6/74.9
78.2/77.5
Table I: Performances on the WOD validation split. All models take single- frame input, no pre- training ensembling is required. light denotes light version that involves simple feature aggrega- tion without further feature extraction for cluster features [19]. denotes using a longer input range for the point cloud. denotes using larger model size (e.g., 5x kernel size for 3D sparse encoder) for enhancement. * denotes an enhancement version (i.e., wider backbone network and additional NMS for Pedestrian and Cyclist). A denotes that belonging the 3D detector featured by cluster-based paradigm. We highlight the top-2 entries with bold font.
of [-51.2m, 51.2m) [-51.2m. 51.22mm.11mm. 51.2m) [-5.0m, 3.0m) with voxel size [0.1m, 0.1m, 0.2m) in a,
and z-axes respectively. We adopt the same data augmentation setting as [35 including random
flipping, global scaling, global rotation, and groundtruth (GT) sampling [33] for Waymo dataset.
These settings are similar to those of nuScenes dataset following []. We use the one-cycle (25) learning rate schedule and AdamW [6 optimizer with the maximal learning rate 0.001. In addition to the 12-epoch schedule for ablation studies, we adopt longer schedule (36 epochs) to obtain the
best performance on the validation and test set. During the evaluation, we use the NMS IoU threshold of [0.7, 0.25, 0.25]. Our code is built on MMdetection3D 6
Network. The 3D sparse encoder and decoder are built upon sparse U-Net in PartA2 [24]. For the BEV backbone, use the same architecture as SECOND (33] and obtain multi-scale BEV features with FPN structure. The top 300 BEV queries are selected from the predicted heatmap following M The vote cluster module generates cluster features with 128 dimensions via simple aggregation [19] or further instance-wite feature extraction [0]] For the CFD module, we set the diffusion factors according the size of different classes. For example, we set 1, 3, and 5 for pedestrians, cyclists, and vehicles in Waymo. For the CQG module. we generate top-ranking cluster queries from the activation map. Since the maximum number of objects in frame is 185 and 142 for Waymo and nuScenes datasets [1]. the number of cluster queries is set as 200 and 150, respectively. We set the number of transformer layers and heads to 3 and 8, and the category labels are simply encoded as one-hot embeddings following M
4.2 Main Results
Methods
All
Veh.
Ped.
Cyc.
Waymo Results. We summarize the perfor- mance of CluB and state-of-the-an (SOTA) 3D detection methods on WOD validation and test and sets in Table and Table As shown in Tab
CenterPoint [35
69.0
71.9
67.0
68.2
PV-RCNN+ 23
70,2
73.5
69.0
68.2
AFDetv2 [12
70.0
72.6
68.6
68.7
our CluB achieves competitive fanll--rraee per- formance and sets new records on Cyclist and Pedestrian of the WOD validation set. CluB achieves comparable mAPH/L.2 performance on the validation set compared with the previous best transformer-bass model ConQueR [39
PillarNet-34 [1]
69.6
74.7
68.5
65.5
ConQueR (39
72.0
73.3
70.9
71.9
CluB (Ours)
72.2
73.4
70.8
72.2
Table 2: Performance comparison of different single-fram framm models on the WOD test set. APH/L2 results reported.
7

---

Methods
NDS
mAP
Car
Truck
Bus
Trailer
C.V.
Ped.
Motor.
Bicye.
T.C.
Barrier
AFDetV2
12
68.5
62.4
86.3
54.2
62.5
58.9
26.7
85.8
63.8
34.3
80.1
71.0
CenterPoint
35
67.3
60.3
85.2
53.5
63.6
56.0
20.0
84.6
59.5
30.7
78.4
71.1
TransFusion-L.
0
70.2
65.5
86.2
56.7
66.3
58.8
28.2
86.1
68.3
44.2
82.0
78.2
VoxeINext [
70.0
64.5
84.6
53.0
64.7
55.8
28.7
85.8
73.2
45.7
79.0
74.6
CluB (Ours)
71.2
66.0
87.2
57.0
66.4
59.0
28.8
87.2
69.0
45,4
82.1
78.2
Table 3: Performance comparison of different models on the nuScenes dataset.
Meanwhile, CluB achieves 0.6% mAPH/L22 performance gain than the powerful detector FSD [0] featured by cluster- based paradigm, even FSD takes longer-range points input. CluB surpasses the BEV-based method TransFusion-L, which is also our BEV-based baseline, by 6.5% mAPH/L2. It is noteworthy that CluB achieves 1.3% mAPH/L2 performance improvement compared to Voxel- Next [4] which utilizes a larger model size for model enhancement(e.g: 5x kernel size for 3D sparse encoder). Furthermore, with the help of further cluster-wist feature extraction, CluB improves the detection performance of the light version CluBright on all classes. Notably, CluBright consistently demonstrates competitive performance on the Cyclist class, compared with all previous single-frame methods in terms of both APH/LI and APH/L2.
In addition to offline results, we also report the detection performance on Waymo test set to explore the potential of CluB. As shown in Table? 2 our CluB outperforms the previous single-frame LiDAR- only methods on the test set, e.g., 2.0% 0.2% mAPH/L2 improvement compared with PV-RCNN+ / ConQueR, respectively.
Here, we also give our analysis of why CluB performs better on the cyclist and pedestrian classes. Actually, the cluster-hased branch works better when the point clouds are more compact, which means the improvements are more significant for cyclists than that for vehicles. Due to the fixed voxelization and over-downsampling
challenging capture sufficient details of small objects with the BEV- based branch. On the contrary, the cluster branch becomes highly beneficial to extracting fine-grained features from sparser points with smaller cluster voxel sizes, which leads to notable performance improvement.
NuScenes Results. We summarize the performance of CluB and different baselines on the nuScenes test in Table Notably, compared with the concurrent fully sparse detector VoxeINext 44, CluB achieves 1.2% % 1.5% improvement on NDS/ / mAP (70.0 + 71.2, 64.5 + 66.0). Compared with the strong BEV-based baseline Transfusion-L, CluB also leads to 1.0% and 0.5% improvement on NDS and mAP (70.2 + 71.2. 65.5 - 66.0) ). The experimental results on Waymo and nuScenes benchmarks validate the effectiveness of CluB by integrating the cluster representation to the BEV branch via the two-level enhancement scheme.
4.3 Ablation Study
To validate the effect of each component in CluB, we conduct ablation studies on the WOD. The models compared this section are trained with 20% training samples of WOD, and evaluated on the whole validation set.
Effect of Components in CluB. To understand how each module contributed to the final perfor- mance in CluB, test each component independently and report its performance in Table 4 Our baseline detector (a) employs the mainstream BEV paradigm, which starts from 60.5% mA PH/1.2 When the CFD module applied, the mAPH/L2 of Method (b) is raised by 1.4%, which indicates that adaptively integrating the cluster feature in soft manner is effective to improve 3D detection. Method (c) brings a 2.0% mAPH/L2 improvement by introducing an imitation loss, which validates that explicitly transferring knowledge the other hand, Method (d) adds the CQG module based on the Method (a) and brings a total of 0.5% mAPH/L2 enhancement by taking ad advantage of querying expansion. Method (e) further introduces direction loss to ensure the quality cluster queries with 1.1 improvements. The combination of all the components in Method (f) achieves 63.9 63.9% MAPH/L22 (3.4 % absolute improvement), validating the effectiveness of CluB. This indicates that enhancing the object representation from two branches at the feature and query levels is of great use to improve detection accuracy.
cbbject-specific features) to the BEV stream is necessary. On
8

---

Feature-level
Query-level
Method
mAPH/L2
CFD Module
Imitation Loss
CQG Module
Direction Loss
60.5
(b)
61.9 f 1.4
62.5
2.0
61.0 0.5
61.6 1.1
63.9
Table 4: Effect of each component in CluB, This experiment reveals that the combined object representation is indeed strengthened by elaborately fusing the context-aware BEV representation and object-centric cluster representation at both the feature level and the query level.
Using diffusion
Using
APH/L2
Strategy
Operation
class infor.
Veh.
Ped.
Cyc.
(a)
Direct concatenation
61.9
63.0
63.9
(h) 2D convolution
61.4., 0.5
62.5. 0.5
63.7. 0.4
Max pooling
61.0,. 0.9
63.41
0.4
64.11 0.2
(d)
Max pooling
62.9 1.0
64.41 1.4
64.5t 0.6
Table 5: Effect of different strategies for the CFD module. This experiment reveals the importance of establishing the association between cluster and BEV features soft and adaptive way when enhancing object representation at the feature level.
Effect of Diffussoon Strategies in the CFD Module. We compare our proposed class-aware BEV diffusion namely Strategy (d) with other strategies, which are shown in Table Intuitively, we can directly concatenate the cluster feature with the BEV feature based on the location of the voting centers, which is a sub-optimal strategy discussed Section T We consider the naive Strategy (a) relying on hard association as the baseline strategy in Table [ Strategy (b) uses 2D convolution layer diffuse valid features the neighboring, which not only introduces extra parameters but degrades the overall detection performance. In contrast, Strategy (c) utilizes the max pooling operation, an elegant solution avoid some unnecessary computation for more effective representation learning procession. Compared with the class-agnostic manner (c), our diffusion scheme (d) leads to better performance, e.g., 61.0 - 62.9, 63.4 64.4 APH/L2 on vehicle and pedestrian by considering semantic consistency. Strategy (d) establishes the soft and adaptive association between cluster and BEV features, thus successfully enhancing object representation at the feature level.
Effeet of Design Choices of Cluster Module. We examine the effects of different design choices of cluster modules in Table 6 The base competitor Method (a) is BEV-based 3D detector we implemented. Compared with CluB, abandons the cluster branch. As evidenced in the Method (a) and the Method (b), by integrating the cluster branch via simple aggregation strategy, i.e., K-Nearest Neighbor (KNN), Method (b) performs better, e.g., 61.0 61.4, 60.4 + 64.3 on vehicle and pedestrian. Combining more local features for each cluster (8 neighbors of KNN), our Method (c) achieves a significant improvement on vehicle class, i.e., 61.0 - 63.2. However, class, the performance degrades from 60.4 to 59.9 because points that do not belong to pedestrians are aggregated. That indicates that a well-designed clustering algorithm can improve the overall performance. Motivated by that, Method (d) replaces the KNN algorithu with the ball query algorithm, and the detection performance steadily improves among the three classes. Method (e) utilize the elaborate cluster module following [10], which groups points into aastance-wise clusters, achieving the satisfactory detection performance shown in the last row in Table
the pedestrian
5 Conclusion
In this paper, we introduce CluB, unified 3D object detection framework that takes advantage of both BEV-based and cluster-based paradigms for improving the accuracy of 3D object detection. To effectively combine the context-aware BEV representation and object-centric cluster representation,

---

APH/1.2
Methods
Cluster Branch
Cluster Module Design
Veh.
Ped.
Cyc.
61.0
60.4
60.1
KNN-4 KNN-8
61.4
64.3
64.1
63.2
59.9
64.4
Ball query SIR
62.7
63.8
64.4
62.9
64.4
64.5
Table 6: Effects on different design choices of vote cluster modules. The experimental result reveals that integrating the cluster branch is beneficial for detection performance, and more elaborately designed grouping strategy may bring more gains.
we propose to integrate an auxiliary cluster- based branch to the mainstream BEV-based detector by strengthening the object representation at both feature and query levels. On the one hand, we first establish the association between cluster and BEV features in soft adaptive way using the CFD module. Based on the association, we transfer the object-specific knowledge from the cluster to the BEV branch supervised by imitation loss. On the other hand, we design a CQG module to enrich the diversity of object queries using the voting centers directly from cluster branch. Simultaneously, a direction loss is employed to encourage more accurate voting center for each cluster. The two-level enhancement (i.e., feature integration, and query expansion) design CluB greatly enriches the object representation ability. With proposed framework, CluB outperforms the previous single-franm frame models for detecting 3D objects on both Waymo and nuScenes datasets.
Limitations. Remarkably, CluB presents an elegant view of how to combine the context-aware BEV representation with object-centric cluster representation in unified detection framework and achieves promising performance via two-level enhancement (feature, query) for object representation. Yet, the computational and spatial complexity on BEV feature maps quadratic the perception range, which makes it hard to adopt our CluB for long-range detection. We leave the exploration of computational efficient version of our method the future work.
6 Acknowledgement
We hereby thank Pengfei Luo for his contribution. This work was supported by the Chinese Academy of Sciences Frontier Science Key Research Project ZDBS-LY-JSC001, the National Key R&D Pro- gram of China NN..20277D01600000 and in part by Shanghai Committee of Science and Technology (N0.21DZ1100100)
References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan
Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, pages 1080-1089. IEEE, 2022.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal
dataset for autonomous driving. In CVPR. pages 1618--16223 Computer Vision Foundation /
IEEE, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV (1), volume 12346
of Lecture Notes in Computer Science, pages 213-229. Springer, 2020.
[4] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR, pages 21674-21683, IEEE, 2023.
[5] Zehui Chen, Zhenyu Lii, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal BEV distillation for multi-view 3d object detection, CoRR. abs/2211.00 1.09386. 2022. [6] MMDetection33 Contributors. MMDetection3::: Mpen next-generation platform for general 3D object detection. https: http:///pgttubb.mmmmppee.... wwtp:///ooo///a///oaaaeaaeeeeaae www.w/ htt.a//mmmmmmmmmttttiioo..... 2020.
10

---

[7] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: towards high performance voxel-based 3d object detection. In AAAI, pages 1201-1209. AAAI Press, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image
recognition at scale. In ICLR. OpenReview.ner 2021.
[9] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang. and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, pages 8448 8458. IEEE, 2022.
[10] Lue Fan, Feng Wang. Naiyan Wang, and Zhao-Xiang Zhang. Fully sparse 3d object detection.
In NeurlPS, 2022.
[11] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 43(12):4338
4364, 2021.
[12] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang. Kun Li, and Qiang Liu. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. In
AAAI, pages 969-979. AAAI Press, 2022.
[13]
Zhicong Huang, Zhijie Zheng. Jingwen Zhao, Haifeng Hu, Zixin Wang, and Dihu Chen. Psa- det3d: Pillar set abstraction for 3d object detection. Pattern Recognit. Letr., 108:138-144 2023.
[14] Jean Lahoud, Bernard Ghanem, Martin R. Oswald, and Mare Pollefeys. 3d instance segmenta- tion via multi-task metric learning. In ICCV, pages
IEEE, 2019.
[15] Alex H. Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, pages 12697-12705. Computer Vision Foundation IEFE, 2019.
[16] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster).
OpenReview.net, 2019.
[17] Yuexin Ma. Tai Wang, Xuyang Bai, Huitong Yang. Yuenan Hou, Yaming Wang, Yu Qiao, Ruigang Yang, Dinesh Manocha, and Xinge Zhu. Vision-centric BEV perception: A survey, CoRR, abs/2208... 02797, 2022.
[18] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng. Xiaodan Liang. Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV, pages 3 3144-3153, IEEE, 2021.
[19] Charles R. Qi, Or Litany, Kaiming Hc, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, pages 9276-9285 IEEF, 2019. Charles Ruizhongtai Qi, Hao Su, Kaichun Mo. and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 77-85. IEEE Computer Society, 2017.
[20]
[21] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillar- based 3d object detection. In ECCV (I0), volume 13670 of Lecture Notes in Computer Science, pages 35-52 Springer, 2022.
[22] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng
Li. PV-RCNN: poxnt-voxel feature set abstraction for 3d object detection. In CVPR, pages
10526-10535 Computer Vision Foundation / IEEE, 2020.
[23] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang. and Hongsheng Li. PV-RCNN++: point-voxel feature set abstraction with local vector represen- tation for 3d object detection. Int. J. Comput. Vis., 13 131(2)533155)
2023.
[24]
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and aggeeeggrggttion network. IEEE
Trans. Pattern Anal. Mach. Intell., 43(8):2647-2666 2021.
[25] Leslie N. Smith and Nicholay Topin. Super-convergence very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for
Operations Applications, volume 1 1006, page 1106112. SPIE, 2019.
[26] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pages Foundation/ / IEEE, 2020.
Computer Vision
11

---

[27] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV (10), volume 13670 of Lecture Notes in Computer Science, pages 426-442. Springer, 2022.
[28]
Luequan Wang. Hongbin Xu, and Wenxiong Kang. Mvcontrast: Unsupervised pretraining for multi-view 3d object recognition. Machine Intelligence Research, pages 1-12, 2023.
[29]
Yingji Wang, Qiuyu Mao, Hanqi Zhu, Yu Zhang. Jianmin Ji, and Yanyong Zhang. Multi-modal
3d object detection in autonomous driving: survey, CoRR, abs/2106. 12735, 2021.
[30]
Yue Wang and Justin M. Solomon. Object DGCNN: 3d object detection using dynamic graphs. In NeurlPS, pages 20745-20758 2021.
[31] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert. Jagject Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurlPS Datasets and Benchmarks, 2021.
[32] Dong Wu, Man- Liao, Wei-Tian Zhang, Xing-Gang Wang, Xiang Bai, Wen-Qing Cheng, and Wen-Yu Yu Liu. Yolop: You only look once for panoptic driving perception. Machine Intelligence Research, 19(6):550-50 16000-66.2200 2022.
[33] Yan Yan, Yuxing Mao, and Bo Li. SECOND: sparsely embedded convolutional detection. Sensors, 18(10):333,, 2018
[34] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, Jie Zhou, and Jifeng Dai. Bevformer v2: Adapting modern image backbones to bird's e-ee-viwww recognition via perspective supervision. CoRR, abs/2211.10433 2022.
[35] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. In CVPR, pages 11784-1193.. Computer Vision Foundation / IEEE, 2021.
[36] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In CVPR, pages 18931-18940. IEEE, 2022.
[37] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, pages 4490-4499. Computer Vision Foundation/ / IEEE Computer Society, 2018.
[38] Zixiang Zhou, Xiangchen Zhao, Yu Wang. Panqu Wang. and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV (38), volume 13698 of Lecture Notes in Computer Science, pages 496-513. Springer, 2022.
Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. CVPR, pages 9296-9305 IEEE, 2023. [40] Xinge Zhu, Yuexin Ma, Tai Wang. Yan Xu, Jianping Shi, and Dahua Lin. SSN: shape signature networks for multi-class object detection from point clouds. In ECCV (25), volume 12370 of Lecture Notes in Computer Science, pages 581 181-597. Springer, 2020.
[39]
12


Reviews:
[{"review_id": "GyDBbs3UA4", "reviewer": "Reviewer_Z2j1", "summary": "This paper proposes a CluB framework to improve the accuracy of 3D object detection by taking advantage of both BEV-based and cluster-based paradigms. The motivation is that the cluster features in voting-based cluster method can largely preserve the 3D structure details of each object, thus supplements the weakened center point features in BEV-based convolutional methods. To achieve the above goals, the authors propose a Cluster Feature Diffusion (CFD) module that adaptively diffuses the valid votes on a vote BEV and fuse it with dense BEV features. An imitation loss is also introduced to transfer object-centric knowledge to the BEV branch and encourage the stability of overall representation learning. Meanwhile, a Cluster Query Generation (CQG) module is proposed to enrich the diversity of object queries by using the voting centers from the cluster branch. Extensive experiments are conducted on the Waymo and NuScenes dataset. Both the ablation studies and the comparison with the SOTA methods demonstrate the effectiveness of the proposed method.", "questions": "Intuitively, it seems like that the convolution operation performed on feature maps is also like an implicit clustering method that aggregates the surrounding features. Is it necessary to make such big effort to introduce a new cluster branch? If you want to alleviate the phenomenon that a stack of convolution layers will weaken the capability of presenting an object with the center point and reduce structure information, how about using deformable convolution?\nOther question, please refer the weakness part above and give your demonstration.", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall the paper is well-written and well-structured.\nThe experiment results are convincing.\nThe motivation of combining the BEV-based and voting-based method is somewhat novel.\nFrom the technical aspect, although the design of CFD and CQG modules is vulgaris, however it is also effective.", "weaknesses": "The demonstration of the class-aware BEV diffusion is not clear. How to leverage the classification results to control the expansion magnitude? I think there should be a formal formula. Figure 3 is also confusing, how to get Class-aware Vote BEV?\nThere is a lack of discussion of Model Efficiency. It seems that the cluster branch is time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We respond to your comments below.\n\n> *The demonstration of the class-aware BEV diffusion is not clear. How to leverage the classification results to control the expansion magnitude? Figure 3 is also confusing, how to get Class-aware Vote BEV?*\n\nThanks for your valuable comments. The expansion magnitute is adapted according to the predictions by setting different max pooling kernel sizes. Specifically, on Waymo Open Dataset, we set the kernel size to 1, 3 and 5 if a cluster is with maximum likelihood to be pedestrian, cyclist, and vehicle, respectively. Here we adopt the different expansion magnitute since they are in different scales. These details are included in Section 4.1 of the manuscript. We will further emphasize them.\nBesides, we have revised Figure 3 (shown in **Figure 1 of the PDF**) to make it clearer, and elaborate the process of getting Class-aware Vote BEV as follows: \n\n1. We convert the cluster features to the BEV space and generate vanilla vote BEV features F.\n2. Since the feature maps are class-agnostic, we merge the classification results (assuming c categories) from the cluster branch into masks { $W_1,W_2, …,W_{c}$ }. The vote BEV feature $F_i$ for each class is then computed using the following equation:\n$F_i=W_i \\cdot F$.\n3. As objects in different classes vary in size, we choose **different kernel sizes according to the classification results of each cluster** (i.e., 1 for pedestrian, 3 for cyclist and 5 for vehicle). \n4. Based on the defined kernel sizes, we perform max pooling on each per-class vote BEV feature simultaneously, which generates the diffused vote BEV feature of each class $D_i$.\n$D_i=\\text{maxpooling}(F_i,\\text{kernel size})$\n5. Finally, **the generated class-aware vote BEV** { $D_1,D_2, …,D_c$ } are together fed into a fully connected layer, which generates the required class-aware vote BEV.\n\nIn the final version, we will accordingly revise Section 3.3 to provide a more detailed and intuitive explanation.\n\n\n> *lack of discussion of Model Efficiency.*\n\nThank you for the suggestion. We present the comparison of model efficiency between our CluB and the BEV-only baseline in terms of latency and FLOPs accordingly. The results are obtained on Waymo Open Dataset with one NVIDIA 3090 GPU.\nThe computational overhead of CluB is 1.2 / 1.3  times that of the BEV-only baseline in terms of latency / FLOPs. The cost is affordable since the auxiliary cluster branch is built with fully sparse operations[1]. Detailed comparison is shown in the following table. We will add the experimental results and analysis in the future revision. Thanks again for your advice.\n\n|             | Baseline | CluB   |\n|--------------|-------------------|--------|\n| Latency (ms) | 144               | 167    |\n| FLOPs (G)    | 95.71             | 112.35 |\n\n> *It seems like that the convolution operation performed on feature maps is also like an implicit clustering method that aggregates the surrounding features. Is it necessary to make such big effort to introduce a new cluster branch?*\n\nWe agree that the convolution operation can aggregate the surrounding features. However, the filters of convolution have fixed kernel sizes, which means the receptive field might not adapt optimally to objects with different scales and aspect ratios. By contrast, our cluster-based branch groups points (voxels) into clusters with a flexible receptive field. Besides, the center voting operation is leveraged in the cluster branch, which further benefits introducing the object-centric feature.\n\n> *If you want to alleviate the phenomenon that a stack of convolution layers will weaken the capability of presenting an object with the center point and reduce structure information, how about using deformable convolution?*\n\nThanks for your valuable comments. We agree that it is feasible to exploit deformable convolution for extracting center features, but the performance by leveraging it is unknown to the point cloud domain. To verify it, we conducted an additional experiment. Specifically, we replace all the 2D convolution layers in the BEV backbone with deformable convolution layers. Compared with the baseline (60.5%), this approach leads to inferior detection performance (20.6%). Accordingly, although deformable convolution has been proven to be effective for learning the deformable shape and scale, it barely works when the object center is empty. We will add the experiment and analysis in the revision.\n\n[1] Lue Fan et al., Fully Sparse 3D Object Detection, NeurIPS 2022.\n\n"}, {"review_id": "GLxjxIQeKc", "reviewer": "Reviewer_oemX", "summary": "The paper introduces a new 3D object detection framework called CluB that combines the strengths of BEV-based and cluster-based detectors. CluB effectively integrates the context-aware BEV representation and object-centric cluster representation at both the feature level and the query level.\nThe proposed method outperforms state-of-the-art methods by a remarkable margin on two prevalent datasets, i.e., Waymo Open Dataset and nuScenes, demonstrating the effectiveness and generality of the proposed method.", "questions": "1. According to the expression in the paper, Diffused Vote BEV features and Dense BEV features represent the center point semantic information and edge information respectively. But in the design of the Club, Imitation Loss is used to make the two similar. This seems to be contradictory to the previous statement. Can you give some further explanation?\n2. As stated in Weakness #3, can you provide a more detailed analysis of the the combination result of two level? \n2. Can you provide a more detailed analysis of the computational complexity?\n3. Can you provide a more detailed analysis of the impact of the proposed CluB framework on real-time performance?", "limitations": "Authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel framework that combines the strengths of BEV-based and cluster-based detectors at both the feature level and the query level. The proposed combination method is of a certain degree of novelty.\n2. The paper is well-written and easy to follow. The authors provide clear explanations of the proposed method and the experimental results. \n3. The paper presents a well-designed and comprehensive experimental evaluation of the proposed CluB framework. The authors provide detailed analysis and comparison with state-of-the-art methods on prevalent datasets.", "weaknesses": "1. Limited novelty: While the CluB framework is unique and original in its approach to integrating BEV-based and cluster-based detectors, the paper does not introduce any fundamentally new concepts or techniques. The proposed method is built on existing methods and combines them in a novel way. The novelty is limited for NeurIPS.\n\n2. Limited analysis of real-time performance: The paper does not provide a detailed analysis of the real-time performance of the proposed CluB framework. To address this weakness, the authors could provide a more detailed analysis of the real-time performance of the proposed method and explore ways to make the method more suitable for real-time scenarios.\n\n3. Limited analysis of the combination result of two level: According to the ablation study in table 4. The comprehensive improvement effect of the two levels (3.4%) is greater than the sum of their individual improvement effects (2% + 1.1%). I think this may be the essential reason why the proposed framework is effective, but this issue lacks in-depth analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and detailed comments. We respond to your comments below.\n\n> *According to the expression in the paper, Diffused Vote BEV features and Dense BEV features represent the center point semantic information and edge information respectively. But in the design of the Club, Imitation Loss is used to make the two similar. This seems to be contradictory to the previous statement. Can you give some further explanation?*\n> \nThanks for your valuable comment. The imitation loss is leveraged to align the center feature between the cluster branch and the BEV branch in the `align-and-fusion` manner, instead of erasing the different information of these two branches. Particularly, as presented in Equation (2) and Equation (3), the valid pixels with imitation loss are within the center region, while no constraint has been added for the edge. In the revision, we will include this explanation in Section 3.3.\n\n\n> *As stated in Weakness #3, can you provide a more detailed analysis of the combination result of two level?*\n> \nThank you for the thought-provoking comment. It is possible that the overall performance improvement is greater than the summation of their individual improvement. When the two-level enhancements are applied together, the enriched queries are in fact utilized by the enhanced feature from the  feature-level. We provide a more detailed analysis as follows.\n\nAccordingly, we provide results for each category in the following table (Table 1 in the PDF). Compared with BEV-based baseline method (a), the performance improvements are similar over three classes when applying feature-level enhancement. As the features are enhanced in the first step, the improvements for adding queries furthermore are more pronounced, which indicates these two-level enhancements could work synergistically. We observe the same phenomenon you mentioned is mainly on small objects, e.g., 4.0% > 2.3% + 1.4% for pedestrians, which shows that the CluB framework is more effective in capturing fine-grained targets.\nWe will add the experimental results and analysis in the future revision.\n\n| Method | Feature-level | Query-level | Vehicle  | Pedestrian | Cyclist  | mAPH/L2  |\n|--------|---------------|-------------|----------|------------|----------|----------|\n| (a)    |               |             | 61.0     | 60.4       | 60.1     | 60.5     |\n| (b)    | √             |             | 62.6 ↑1.6 | 62.7 ↑2.3   | 62.0 ↑1.9 | 62.5 ↑2.0 |\n| (c)    |               | √           | 61.7 ↑0.7 | 61.8 ↑1.4   | 61.4 ↑1.3 | 61.6 ↑1.1 |\n| (d)    | √             | √           | 62.9 ↑1.9 | 64.4 ↑4.0   | 64.5 ↑4.4 | 63.9 ↑3.4 |\n\n\n> *Can you provide a more detailed analysis of the computational complexity?*\n> \nThank you for the suggestion. The computational overhead of CluB is 1.3 times that of the BEV-only baseline in terms of FLOPs. The computation cost is affordable since the auxiliary cluster branch is built with fully sparse operation, such as sparse instance recognition (SIR) module [1]. Please refer to the table below:\n\n|           | Baseline | CluB   |\n|-----------|----------|--------|\n| FLOPs (G) | 95.71    | 112.35 |\n\n \n>*Can you provide a more detailed analysis of the impact of the proposed CluB framework on real-time performance?*\n\nThank you for the suggestion. We evaluate the runtime on one NVIDIA GEFORCE RTX 3090. The computational overhead of CluB is 1.2 times that of the BEV-only baseline in terms of latency.  Please see the table below:\n\n|              | Baseline | CluB |\n|--------------|----------|------|\n| Latency (ms) | 144      | 167  |\n\n[1] Lue Fan et al., Fully Sparse 3D Object Detection, NeurIPS 2022.\n\n"}, {"review_id": "bQgc4woA5h", "reviewer": "Reviewer_BdkH", "summary": "The authors explore and analyze the existing LiDAR-based 3D object detection framework, and propose to adopt BEV-based and cluster-based methods to aggregate those features from LiDAR input. The experimental results on Waymo and nuScenes are better compared to the existing methods.", "questions": "Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.", "limitations": "Yes. The authors presented one of the limitations in terms of computational tradeoffs. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The task of 3D object detection is very important in the 3D community. Interestingly, the authors propose to extract and combine BEV and cluster-based features.\n\n2. The paper is easy to follow.\n\n3. The authors conduct the experiments on two widely-used datasets, including nuScenes and Waymo.", "weaknesses": "1. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D detection methods. The time consumption might be large since the proposed method includes transforming the vote cluster to BEV matching and aggregation.\n\n2. Most 3D object detection SOTA is multi-frame based. It would be interesting to see how the multiple frames fitted into the proposed architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We respond to your comments below\n\n> *Computation/memory footprint comparison.*\n\nThank you for the suggestion. We have computed the statistics for computation and memory usage of our CluB model and the widely used BEV-based baseline detector Transfusion-L [1] on a single NVIDIA GEFORCE RTX 3090 GPU. \n\nAs shown in the table, the computational overhead of CluB is 1.2 times that of the BEV-only baseline in terms of latency. Meanwhile, the FLOPs and memory cost do not go up significantly as the cluster branch primarily involves fully sparse operations on the given point set [2].\n \n|                 | Baseline   | CluB       |\n|-----------------|------------|------------|\n| FLOPs (G)       | 95.71      | 112.35     |\n| Latency (ms)    | 144        | 167        |\n| GPU Memory (MB) | $1.11 × 10^4$ | $1.38 × 10^4$ |\n\n \n \n> *It would be interesting to see how the multiple frames fitted into the proposed architecture.*\n\nThanks for your constructive feedback. It points out an insightful and crucial direction for our future research. We think it is possible to fit multiple frames into the CluB framework.  We kindly refer you to the PDF (**Figure 2**) for the overview of the framework. Next, we give a detailed illustration of multi-frame-based CluB architecture.\n\nSince our CluB is a query-based 3D detector, we could conveniently model the temporal interaction by utilizing object queries. This is inspired by the object-query-centric temporal modeling architecture of StreamPETR [3]. As illustrated in Figure 2 of the PDF, we first build a memory queue to store the historical object queries. The current queries from our CluB framework are fed into the propagation transformer to interact with historical queries and current BEV features, obtaining temporal and spatial information. The output queries are further used to generate detection results and the top-K queries are pushed into the memory queue. Through the recurrent update of the memory queue, the long-term temporal information is propagated frame by frame. Note that the memory queue follows the first-in, first-out (FIFO) rule.\n\nWe would take this constructive suggestion and extend our CluB framework to a multi-frame version in the future.\n\n[1] Xuyang bai et al., TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers, CVPR 2022.\n\n[2] Lue Fan et al., Fully Sparse 3D Object Detection, NeurIPS 2022.\n\n[3] Shihao Wang et al., Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection, arXiv 2023.\n\n"}, {"review_id": "mvCUvUkWgx", "reviewer": "Reviewer_KK49", "summary": "This work combines the BEV-based and cluster-based representations into a unified framework named CluB. At the feature level, the Cluster Feature Diffusion module and the imitation loss are proposed to fuse the features obtained from the BEV branch and cluster branch. At the query level, the Cluster Query Generation module and the direction loss are proposed to provide more accurate object queries from the cluster branch. CluB achieves state-of-the-art performance on the Waymo and nuScenes datasets.", "questions": "1. Some results in the ablation study do not match with each other. For example, the mAPH of Table 4 (b) should be the same as (d), but in fact they are not. Are there differences between the two configurations?\n2. I am wondering why CluB performs better on the cyclist class of Waymo, because the point clouds of cyclists are more compact than the vehicles and should benefit less from the cluster branch.\n3. How efficient is CluB compared to the baseline? The time cost could be significantly increased by applying U-Net in the cluster branch.", "limitations": "It would be nice to apply different BEV-based detectors as the baseline to test the universality of the method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work mentions that the features of the object centers in BEV-based detectors are diffused from their neighbors and are not in high quality, which is an interesting hypothesis. Based on that, the cluster branch is applied to provide clustered features of the object centers. The whole structure of the work is compact and unified, and the promotion on precision is clearly shown.", "weaknesses": "1. The argumentation of the core hypothesis is insufficient. I think that the max pooling in BEV-based detectors is able to reduce the feature shape and aggregate the feature of the object to the center, just like the cluster-based methods. Additional quantitative analysis is preferred to show the difference.\n2. The captions of figures and tables are too long and mainly repeat the text of the main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and detailed comments.\n\n> *The argumentation of the core hypothesis is insufficient. I think that the max pooling in BEV-based detectors is able to reduce the feature shape and aggregate the feature of the object to the center, just like the cluster-based methods. Additional quantitative analysis is preferred to show the difference.*\n\nThanks for your thought-provoking comment.  Accordingly, we conducted an additional experiment that quantitatively verifies max pooling in BEV-based detectors is not able to work like the cluster-based method. The results are summarized as the following table (Table 2 in the PDF).\n| Method           | Vehicle | Pedestrian | Cyclist | mAPH/L2 |\n|------------------|---------|------------|---------|---------|\n| Baseline         | 61.0     | 60.4       | 60.1    | 60.5    |\n| Baseline_maxpool | 60.5    | 59.9       | 59.1    | 60.0      |\n| Baseline_CFD     | **61.9**    | **61.2**       | **62.5**    | **61.9**    |\n\n\nSpecifically, Baseline_maxpool takes sparse voxels directly from the sparse U-Net and transforms them into a dense BEV feature. Subsequently, several pre-defined max pooling layers are applied to amplify the object's feature towards its center. In contrast, Baseline_CFD replaces the multiple max-pooling layers with our cluster feature diffusion (CFD) module. This module dynamically merges cluster features with BEV features to enhance the center's feature.\n\nAs demonstrated in the table, the mAPH/L2 is dropped from 60.5% to 60.0% for Baseline_maxpool, which demonstrates that max pooling operations on BEV features cannot effectively aggregate features to the center of the target to enhance the representation of target features. In comparison, the cluster representation with its adaptable receptive fields demonstrates superior effectiveness in achieving this enhancement. We will include the experimental results and analysis in the future revision. Thanks again for your advice.\n \n> *The captions of figures and tables are too long and mainly repeat the text of the main body.*\n\nThanks for your kind advice. We will shorten the captions of figures and tables in the revised manuscript  as suggested.\n \n> *The mAPH of Table 4 (b) should be the same as (d), but in fact they are not. Are there differences between the two configurations?*\n\nThanks for the comment. The configurations of methods (b) and (d) in Table 4 are in fact different. Compared to the baseline, Method (b) adds the CFD module (feature-level) while Method (d) introduces the CQG module (query-level). \n\nSpecifically, Method (b) adds the Cluster Feature Diffusion (CFD) module compared to the baseline BEV-based detector, which adaptively generates diffused vote features in the unified BEV space for the feature representation enhancement. Method (d) only introduces the Cluster Query Generation (CQG) module compared to the baseline, which enriches the diversity of object queries by using the positions of voting centers from the cluster branch.\n \n> *I am wondering why CluB performs better on the cyclist class of Waymo, because the point clouds of cyclists are more compact than the vehicles and should benefit less from the cluster branch.*\n\nActually, the cluster-based branch works better when the point clouds are more compact, which means the improvements are more significant for cyclists than that for vehicles. Due to the fixed voxelization and over-downsampling, it is challenging to capture sufficient details of small objects with the BEV-based branch. On the contrary, the cluster branch becomes highly beneficial to extract fine-grained features from sparser points with smaller cluster voxel size, which leads to notably performance improvement.\n \n> *How efficient is CluB compared to the baseline? The time cost could be significantly increased by applying U-Net in the cluster branch.*\n\nThank you for the suggestion. The computational overhead of CluB is 1.2 / 1.3  times that of the BEV-only baseline in terms of latency / FLOPs respectively. We present the efficiency-related statistics of our CluB and baseline (w.o. cluster branch) on Waymo dataset below:\n|              | Baseline | CluB   |\n|--------------|----------|--------|\n| Latency (ms) | 144      | 167    |\n| FLOPs (G)    | 95.71    | 112.35 |\n\nThe time cost is not significantly increased for the following reasons. First, the decoder of the U-Net is shared between both the BEV-based baseline and our approach to extract 3D voxel features. Besides, the cluster branch primarily involves fully sparse operations [1] on the given point set. The latency is evaluated using one NVIDIA GEFORCE RTX 3090 GPU. \n \n> *It would be nice to apply different BEV-based detectors as the baseline to test the universality of the method.*\n\nThanks for the valuable suggestion. We choose CenterPoint [2], which is also a prevalent BEV baseline in the community, to test the universality of the method. We provide the results in the following table(Table 3 in the PDF). \n\nCenterPoint is an anchor-free one-stage detector, which extracts BEV features from voxelized point clouds to find object centers and regress to 3D bounding boxes. \nTo adapt the CluB framework for this comparison, we excluded the query-level enhancement, since CenterPoint does not employ a transformer architecture. Remarkably, our method still achieves higher accuracy (from 67.4% to 68.2%) compared to CenterPoint, **even when utilizing only feature-level enhancement**. We have added these results to the appendix of the revised version to show the universality of the method.\n\n\n| Method      | Vehicle | Pedestrian | Cyclist | mAPH/L2 |\n|-------------|---------|------------|---------|---------|\n| CenterPoint | 67.9    | 65.6       | 68.6    | 67.4    |\n| +CluB       | 68.4    | 66.5       | 69.6    | 68.2    |\n\n\n\n\n[1] Lue Fan et al., Fully Sparse 3D Object Detection, NeurIPS 2022.\n\n[2] Tianwei Yin et al., Center-Based 3D Object Detection and Tracking, CVPR 2021.\n\n"}, {"review_id": "EGWvN22ujw", "reviewer": "Reviewer_UKd9", "summary": "This paper combines two streams of LiDAR-based 3D object detectors into a single model: conventional BEV-based detectors and emerging cluster-based detectors. The main contribution is that It is the first one to combine the two streams of methods, leveraging the high-recall of BEV representation and fine-grained structure of cluster representation. It reveals the potential of such a combination, providing a possibility for a new sub-direction. \n", "questions": "I like the idea of combining the two kinds of representation. However, the authors should carefully address my concerns.\nThe manuscript should be updated in the OpenReview system if possible to address the first weakness. Additional experiments could be simply posted on this page for now.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written with clear representation and concise illustration. For example, Figure 2 is very informative and concise, and I could clearly get the overall idea at first glance.\n\n- To my best knowledge, it is the first one to explicitly combine the two streams of methods, leveraging the high-recall of BEV representation and fine-grained structure of cluster representation. Although straightforward, the idea is well-motivated and makes sense to me.\n\n- Every module is well ablated. Overall performance on Waymo Open Dataset and nuScenes dataset is good. Especially, on nuScenes, it achieves state-of-the-art performance.", "weaknesses": "- Although the overall writing is clear, the authors are not very careful with some details. For example, Figure 1 (a) is very similar to a figure in the FSD paper, and it would be appropriate to add a citation. In L108, the citations should be placed after \"benchmarks\" instead of detectors. In Eqn (4), $Ldir$ should be $L_{dir}$.  There are many errors in Table 1: the best and second-best entries of L2 Vehicle should be PillarNet and PVRCNN++. The performance of FSD seems inaccurate because it has a similar vehicle performance to PVRCNN++ if I recall correctly. Please check. The second-best Pedestrian L2 APH is not underlined. And this table size is out of the range on the right.\n\n- Although experiments thoroughly verify the effectiveness of proposed modules, they are straightforward ablation without insight. I encourage authors to add more detailed performance analysis to reveal the inner workings of CluB. For example, how do the two kinds of queries affect the final performance?  Since BEV-based representation has higher recall and cluster-based representation preserves fine-grained structure, authors should design experiments to demonstrate such properties, which cannot be demonstrated by simple numbers of the final performance. Table 1 in supplementary materials should be moved to the main paper, and the current 9-page space is not fully utilized.\n\n- I encourage authors to conduct runtime evaluations and report the latency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and detailed comments. We respond to your comments below.\n\n> *detail errors and format correct*\n\nThanks for pointing out these typos. We have addressed all the errors you mentioned, and we will proofread them very carefully. Since the manuscript can not be updated in the OpenReview system at present, we will make sure the fore-mentioned errors are corrected in the final manuscript.\n\n> *Errors of Table 1.*\n\nThanks for pointing it out. We make the corresponding corrections to address the problems.\n- We checked the performance of FSD and modified the numbers in the revised Table 1. The best vehicle  performance in the FSD (70.1%) paper is indeed similar to PVRCNN++ (70.2%) in terms of APH/L2.\n- As FSD adopts a longer point cloud range, we apply the same input range on our CluB to further enhance the performance (71.4% mAPH/L2). The enhanced CluB is trained on 8 NVIDIA RTX A6000 GPUs with larger batch size of 4. The latest result is also reported on the revised Table 1. \n- We have carefully proofread the label of the best and second-best entries and revised the table size as pointed out.\n\nSince the manuscript can not be updated in the OpenReview system at present, we present **part of the revised Table 1** (Table 5 in the PDF). Thank you again for your detailed thorough review. \n\n\n> *Although experiments thoroughly verify the effectiveness of proposed modules, they are straightforward ablation without insight. I encourage authors to add more detailed performance analysis to reveal the inner workings of CluB. For example, how do the two kinds of queries affect the final performance? Since BEV-based representation has higher recall and cluster-based representation preserves fine-grained structure, authors should design experiments to demonstrate such properties, which cannot be demonstrated by simple numbers of the final performance.*\n\nThanks for your insightful comment. Based on your suggestion, we conduct both quantitative and qualitative experiments to show how the two cluster queries and BEV queries affect the final performance. We first provide the quantitative results on the Waymo dataset in the table below (Table 4 in the PDF).\n\nCompared to CluB (the first row), if we remove the cluster queries (the second row), we observe a significant drop in the performance of cyclist and pedestrian classes. This decline suggests that cluster queries play a beneficial role in enhancing the diversity of object queries, particularly for fine-grained targets.  By removing the BEV queries alone (the last entry),  overall performance drops over 6.3% mAPH/L2. We conclude that BEV queries directly indicate the possible positions of objects on the BEV space, making it easier for the decoder to detect target objects. Therefore, BEV queries and cluster queries have their respective advantages in query initialization for the decoder, thus enhancing detection performance.\n\nTo intuitively demonstrate the contribution of these two queries, we also visualize the two kinds of queries on the same BEV features using Waymo dataset, which can be seen in the Figure 3 of the PDF. The yellow square represents the BEV query activated by the BEV heatmap, while the pink cross signifies the cluster query derived from vote center positions in 3D space. The red circle denotes a potential object initialization position overlooked by the BEV query. This comparison reveals that BEV queries provide a more comprehensive coverage of potential object locations, whereas cluster queries enrich object diversity with finer granularity.\n\nWe will add the experimental results and analysis in the future revision. Thanks again for your advice.\n\n\n| BEV queries | Cluster queries | Vehicle | Pedestrian | Cyclist | mAPH/L2 |\n|-------------|-----------------|---------|------------|---------|---------|\n| √           | √               | 62.9    | 64.4       | 64.5    | 63.9    |\n| √           |                 | 62.6 ↓0.3    | 62.7 ↓1.7      | 62.0 ↓2.5     | 62.5 ↓1.4   |\n|             | √               | 56.9 ↓6.0   | 57.7 ↓ 6.7      | 58.1 ↓6.4   | 57.6 ↓6.3   |\n\t \n> *I encourage authors to conduct runtime evaluations and report the latency.*\n\nThank you for the suggestion. We evaluate the runtime on single NVIDIA GEFORCE RTX 3090. The computational overhead of CluB is 1.2 times that of the BEV-only baseline in terms of latency. Please see the table below:\n \n|              | Baseline | CluB |\n|--------------|----------|------|\n| Latency (ms) | 144      | 167  |\n\n\n\n \n"}]

Author rebuttal:
We sincerely thank the reviewers for their detailed and insightful comments, as well as the favorable recommendations. We also thank the area chair for your time and efforts in handling our paper. We appreciate the positive comments, e.g., "well-written" and "novel motivation" from Reviewer Z2j1, "well-designed and comprehensive evaluation" from Reviewer oemX, "easy to follow"  from Reviewer BdkH, "interesting hypothesis" and "compact structure" from Reviewer KK49, "well-motivated","concise illustration" and "well-ablated" from Reviewer UKd9.

Following the valuable comments and suggestions, we carefully revised the manuscript and updated the suggested experiments on the supplementary material. We hereby refer the reviewers to the detailed responses to each comment. We also upload a PDF with tables and figures due to the limited characters of each response. We provide a summary of changes presented in the attached PDF.

- Tables
  - Table 1: As suggested by Reviewer oemX,  we provide a more detailed comparison on the effect of the two-level enhancement in CluB on individual classes, showing that the two-level enhancements could work synergistically.
  - Table 2: As suggested by Reviewer KK49, we conduct an ablation study on different ways to aggregate the features to the center of the object, showing the effectiveness of cluster representation with its adaptable receptive.
  - Table 3: As suggested by Reviewer KK49, we show the experimental result of applying CluB framework on the different BEV-based detector,  which demonstrates the universality of our method.
  - Table 4: As suggested by Reviewer UKd9, we conduct an ablation study on the effect of the two kinds of object queries for the transformer decoder, showing the two have respective advantages in query initialization for the decoder.
  - Table 5: As suggested by Reviewer UKd9, we present part of the revised Table 1 of the manuscript.

- Figures 
  - Figure 1: As suggested by Reviewer Z2j1, we provide a detailed illustration of the Cluster Feature Diffusion (CFD) module, which is carefully revised based on the Figure 3 of the manuscript.
  - Figure 2: As suggested by Reviewer BdkH, we give an illustration on the possible multi-frame version of our proposed CluB, which is inspired by the recent work named StreamPETR.
 
We look forward to discussing with you over the next few days.


Meta-review: The paper studies how to combine the strengths of BEV-based and cluster-based representations for 3D object detection. Reviewers generally found the problem and insight interesting and novel, the proposed approach effective, and the experiments comprehensive and solid. Meanwhile, reviewers raised several technical questions.

After the rebuttal, many of the reviewers' questions are addressed. Several reviewers increased the ratings. The paper now has a positive rating: average 5.4 (5, 5, 5, 6, 6). The AC thus recommends an acceptance. The AC suggests that the authors incorporate the detailed rebuttal into the final version.
Decision: Accept (poster)