SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Methods	mAP/mAPH L2	Vehicle 3D AP/APH L2	L1	Pedestrian 3D AP/APH L2	L1	Cyclist 3D AP/APH L2	L1
IA-SSD△ [36]	62.3/58.1	61.6/61.0	70.5/69.1	60.3/50.7	69.4/58.5	65.0/62.7	67.7/65.3
FSD△† [10]	72.9/70.8	70.5/70.1	79.2/78.8	73.9/69.1	82.6/77.3	74.4/73.3	77.1/76.0
CenterPoint [35]	-/67.4	-/67.9	-/-	-/65.6	-/-	-/68.6	-/-
PV-RCNN [22]	66.8/63.3	69.0/68.4	77.5/76.9	66.0/57.6	75.0/65.6	65.4/64.0	67.8/66.4
AFDetV2 [12]	71.0/68.8	69.7/69.2	77.6/77.1	72.2/67.0	80.2/74.6	71.0/70.1	73.7/72.7
SST_TS [9]	-/-	68.0/67.6	76.2/75.8	72.8/65.9	81.4/74.1	-/-	-/-
PillarNet-34 [21]	71.0/68.5	70.9/70.5	79.1/78.6	72.3/66.2	80.6/74.0	69.7/68.7	72.3/71.2
PV-RCNN++ [23]	71.7/69.5	70.6/70.2	79.3/78.8	73.2/68.0	81.3/76.3	71.2/70.2	73.7/72.7
TransFusion-L [1]	-/64.9	-/65.1	-/-	-/63.7	-/-	-/65.9	-/-
CenterFormer [38]	71.2/69.0	70.2/69.7	75.2/74.7	73.6/68.3	78.6/73.0	69.8/68.8	72.3/71.3
ConQueR [39]	70.3/67.7	68.7/68.2	76.1/75.6	70.9/64.7	79.0/72.3	71.4/70.1	73.9/72.5
ConQueR★ [39]	74.0/71.6	71.0/70.5	78.4/77.9	75.8/70.1	82.4/76.6	75.2/74.1	77.5/76.4
VoxelNext [4]	70.9/68.2	69.7/69.2	77.9/77.5	72.2/65.9	80.2/73.5	70.7/69.6	73.3/72.2
VoxelNext-K3‡ [4]	72.2/70.1	69.9/69.4	78.2/77.7	73.5/68.6	81.5/76.3	73.3/72.2	76.1/74.9
CluBlight (ours)	71.2/69.3	68.4/68.0	76.5/76.0	71.1/66.1	79.4/73.9	74.3/73.0	76.7/75.3
CluB (ours)	73.4/71.4	70.4/69.9	79.1/78.6	74.3/69.5	83.1/77.9	75.6/74.9	78.2/77.5
Table 1: Performances on the WOD validation split. All models take single-frame input, no pretraining or assembling is required.  $\text{light}$  denotes a light version that involves simple feature aggregation without further feature extraction for cluster features [19].  $\dagger$  denotes using a longer input range for the point cloud.  $\ddagger$  denotes using larger model size (e.g.,  $5 \times$  kernel size for 3D sparse encoder) for enhancement.  $\star$  denotes an enhancement version (i.e., wider backbone network and additional NMS for Pedestrian and Cyclist).  $\triangle$  denotes that belonging to the 3D detector featured by cluster-based paradigm. We highlight the top-2 entries with bold font.
of  $[-51.2m, 51.2m] \times [-51.2m, 51.2m] \times [-5.0m, 3.0m]$  with voxel size  $[0.1m, 0.1m, 0.2m]$  in  $x$ ,  $y$ , and  $z$ -axes respectively. We adopt the same data augmentation setting as [35], including random flipping, global scaling, global rotation, and groundtruth (GT) sampling [33] for Waymo dataset. These settings are similar to those of nuScenes dataset following [1]. We use the one-cycle [25] learning rate schedule and AdamW [16] optimizer with the maximal learning rate 0.001. In addition to the 12-epoch schedule for ablation studies, we adopt a longer schedule (36 epochs) to obtain the best performance on the validation and test set. During the evaluation, we use the NMS IoU threshold of [0.7, 0.25, 0.25]. Our code is built on MMdetection3D [6].
Network. The 3D sparse encoder and decoder are built upon sparse U-Net in PartA2 [24]. For the BEV backbone, we use the same architecture as SECOND [33] and obtain multi-scale BEV features with FPN structure. The top 300 BEV queries are selected from the predicted heatmap following [1] The vote cluster module generates cluster features with 128 dimensions via simple aggregation [19] or further instance-wise feature extraction [10]. For the CFD module, we set the diffusion factors according to the size of different classes. For example, we set 1, 3, and 5 for pedestrians, cyclists, and vehicles in Waymo. For the CQG module, we generate top-ranking cluster queries from the activation map. Since the maximum number of objects in one frame is 185 and 142 for Waymo and nuScenes datasets [1], the number of cluster queries is set as 200 and 150, respectively. We set the number of transformer layers and heads to 3 and 8, and the category labels are simply encoded as one-hot embeddings following [1].
Waymo Results. We summarize the performance of CluB and state-of-the-art (SOTA) 3D detection methods on WOD validation and test sets in Table 1 and Table 2. As shown in Table 1, our CluB achieves competitive single-frame performance and sets new records on Cyclist and Pedestrian of the WOD validation set. CluB achieves comparable mAPH/L2 performance on the validation set compared with the previous best transformer-based model ConQueR [39].
Methods	All	Veh.	Ped.	Cyc.
CenterPoint [35]	69.0	71.9	67.0	68.2
PV-RCNN++ [23]	70.2	73.5	69.0	68.2
AFDetv2 [12]	70.0	72.6	68.6	68.7
PillarNet-34 [21]	69.6	74.7	68.5	65.5
ConQueR [39]	72.0	73.3	70.9	71.9
CluB (Ours)	72.2	73.4	70.8	72.2
Table 2: Performance comparison of different single-frame models on the WOD test set. APH/L2 results are reported.
Methods	NDS	mAP	Car	Truck	Bus	Trailer	C.V.	Ped.	Motor.	Bicyc.	T.C.	Barrier
AFDetV2 [12]	68.5	62.4	86.3	54.2	62.5	58.9	26.7	85.8	63.8	34.3	80.1	71.0
CenterPoint [35]	67.3	60.3	85.2	53.5	63.6	56.0	20.0	84.6	59.5	30.7	78.4	71.1
TransFusion-L [1]	70.2	65.5	86.2	56.7	66.3	58.8	28.2	86.1	68.3	44.2	82.0	78.2
VoxelNext [4]	70.0	64.5	84.6	53.0	64.7	55.8	28.7	85.8	73.2	45.7	79.0	74.6
CluB (Ours)	71.2	66.0	87.2	57.0	66.4	59.0	28.8	87.2	69.0	45.4	82.1	78.2
Table 3: Performance comparison of different models on the nuScenes dataset.
Meanwhile, CluB achieves  $0.6\%$  mAPH/L2 performance gain than the powerful detector FSD [10] featured by cluster-based paradigm, even FSD takes longer-range points as input. CluB surpasses the BEV-based method TransFusion-L, which is also our BEV-based baseline, by  $6.5\%$  mAPH/L2. It is noteworthy that CluB achieves  $1.3\%$  mAPH/L2 performance improvement compared to VoxelNext [4] which utilizes a larger model size for model enhancement(e.g.,  $5\times$  kernel size for 3D sparse encoder). Furthermore, with the help of further cluster-wise feature extraction, CluB improves the detection performance of the light version  $\mathrm{CluB}_{light}$  on all classes. Notably,  $\mathrm{CluB}_{light}$  consistently demonstrates competitive performance on the Cyclist class, compared with all previous single-frame methods in terms of both APH/L1 and APH/L2.
In addition to offline results, we also report the detection performance on Waymo test set to explore the potential of CluB. As shown in Table 2, our CluB outperforms the previous single-frame LiDAR-only methods on the test set, e.g.,  $2.0\% / 0.2\%$  mAPH/L2 improvement compared with PV-RCNN++ / ConQueR, respectively.
Here, we also give our analysis of why CluB performs better on the cyclist and pedestrian classes. Actually, the cluster-based branch works better when the point clouds are more compact, which means the improvements are more significant for cyclists than that for vehicles. Due to the fixed voxelization and over-downsampling, it is challenging to capture sufficient details of small objects with the BEV-based branch. On the contrary, the cluster branch becomes highly beneficial to extracting fine-grained features from sparser points with smaller cluster voxel sizes, which leads to notable performance improvement.
NuScenes Results. We summarize the performance of CluB and different baselines on the nuScenes test set in Table 3. Notably, compared with the concurrent fully sparse detector VoxelNext [4], CluB achieves  $1.2\% / 1.5\%$  improvement on NDS/mAP  $(70.0 \rightarrow 71.2, 64.5 \rightarrow 66.0)$ . Compared with the strong BEV-based baseline Transfusion-L, CluB also leads to  $1.0\%$  and  $0.5\%$  improvement on NDS and mAP  $(70.2 \rightarrow 71.2, 65.5 \rightarrow 66.0)$ . The experimental results on Waymo and nuScenes benchmarks validate the effectiveness of CluB by integrating the cluster representation to the BEV branch via the two-level enhancement scheme.
To validate the effect of each component in CluB, we conduct ablation studies on the WOD. The models compared in this section are trained with  $20\%$  training samples of WOD, and evaluated on the whole validation set.
Effect of Components in CluB. To understand how each module contributed to the final performance in CluB, we test each component independently and report its performance in Table 4. Our baseline detector (a) employs the mainstream BEV paradigm, which starts from  $60.5\%$  mAPH/L2. When the CFD module is applied, the mAPH/L2 of Method (b) is raised by  $1.4\%$ , which indicates that adaptively integrating the cluster feature in a soft manner is effective to improve 3D detection. Method (c) brings a  $2.0\%$  mAPH/L2 improvement by introducing an imitation loss, which validates that explicitly transferring knowledge (object-specific features) to the BEV stream is necessary. On the other hand, Method (d) adds the CQG module based on the Method (a) and brings a total of  $0.5\%$  mAPH/L2 enhancement by taking advantage of querying expansion. Method (e) further introduces a direction loss to ensure the quality of cluster queries with 1.1 improvements. The combination of all the components in Method (f) achieves  $63.9\%$  mAPH/L2 ( $3.4\%$  absolute improvement), validating the effectiveness of CluB. This indicates that enhancing the object representation from two branches at the feature and query levels is of great use to improve detection accuracy.
Method	Feature-level	Query-level	mAPH/L2
CFD Module	Imitation Loss	CQG Module	Direction Loss
(a)	60.5
(b)	✓	61.9 ↑ 1.4
(c)	✓	✓	62.5 ↑ 2.0
(d)	✓	61.0 ↑ 0.5
(e)	✓	✓	61.6 ↑ 1.1
(f)	✓	✓	✓	✓	63.9 ↑ 3.4
Table 4: Effect of each component in CluB. This experiment reveals that the combined object representation is indeed strengthened by elaborately fusing the context-aware BEV representation and object-centric cluster representation at both the feature level and the query level.
Strategy	Operation	Using diffusion	Using class infor.	Veh.	APH/L2 Ped.	Cyc.
(a)	Direct concatenation	61.9	63.0	63.9
(b)	2D convolution	✓	61.4↓0.5	62.5↓0.5	63.7↓0.4
(c)	Max pooling	✓	61.0↓0.9	63.4↑0.4	64.1↑0.2
(d)	Max pooling	✓	✓	62.9↑1.0	64.4↑1.4	64.5↑0.6
Table 5: Effect of different strategies for the CFD module. This experiment reveals the importance of establishing the association between cluster and BEV features in a soft and adaptive way when enhancing object representation at the feature level.
Effect of Diffusion Strategies in the CFD Module. We compare our proposed class-aware BEV diffusion namely Strategy  $(d)$  with other strategies, which are shown in Table 5. Intuitively, we can directly concatenate the cluster feature with the BEV feature based on the location of the voting centers, which is a sub-optimal strategy discussed in Section 1. We consider the naive Strategy  $(a)$  relying on hard association as the baseline strategy in Table 5. Strategy  $(b)$  uses a 2D convolution layer to diffuse valid features to the neighboring, which not only introduces extra parameters but degrades the overall detection performance. In contrast, Strategy  $(c)$  utilizes the max pooling operation, an elegant solution to avoid some unnecessary computation for a more effective representation learning procession. Compared with the class-agnostic manner (c), our diffusion scheme (d) leads to better performance, e.g.,  $61.0\rightarrow 62.9$ $63.4\rightarrow 64.4$  APH/L2 on vehicle and pedestrian by considering semantic consistency. Strategy  $(d)$  establishes the soft and adaptive association between cluster and BEV features, thus successfully enhancing object representation at the feature level.
Effect of Design Choices of Cluster Module. We examine the effects of different design choices of cluster modules in Table 6. The base competitor Method (a) is a BEV-based 3D detector we implemented. Compared with CluB, it abandons the cluster branch. As evidenced in the Method (a) and the Method (b), by integrating the cluster branch via a simple aggregation strategy, i.e., K-Nearest Neighbor (KNN), Method (b) performs better, e.g.,  $61.0 \rightarrow 61.4$ ,  $60.4 \rightarrow 64.3$  on vehicle and pedestrian. Combining more local features for each cluster (8 neighbors of KNN), our Method (c) achieves a significant improvement on vehicle class, i.e.,  $61.0 \rightarrow 63.2$ . However, for the pedestrian class, the performance degrades from 60.4 to 59.9 because points that do not belong to pedestrians are aggregated. That indicates that a well-designed clustering algorithm can improve the overall performance. Motivated by that, Method (d) replaces the KNN algorithm with the ball query algorithm, and the detection performance steadily improves among the three classes. Method (e) utilize the elaborate cluster module following [10], which groups points into instance-wise clusters, achieving the satisfactory detection performance shown in the last row in Table 6.
In this paper, we introduce CluB, a unified 3D object detection framework that takes advantage of both BEV-based and cluster-based paradigms for improving the accuracy of 3D object detection. To effectively combine the context-aware BEV representation and object-centric cluster representation,
Methods	Cluster Branch	Cluster Module Design	Veh.	APH/L2 Ped.	Cyc.
(a)	61.0	60.4	60.1
(b)	✓	KNN-4	61.4	64.3	64.1
(c)	✓	KNN-8	63.2	59.9	64.4
(d)	✓	Ball query	62.7	63.8	64.4
(e)	✓	SIR	62.9	64.4	64.5
Table 6: Effects on different design choices of vote cluster modules. The experimental result reveals that integrating the cluster branch is beneficial for detection performance, and a more elaborately designed grouping strategy may bring more gains.
we propose to integrate an auxiliary cluster-based branch to the mainstream BEV-based detector by strengthening the object representation at both feature and query levels. On the one hand, we first establish the association between cluster and BEV features in a soft adaptive way using the CFD module. Based on the association, we transfer the object-specific knowledge from the cluster to the BEV branch supervised by an imitation loss. On the other hand, we design a CQG module to enrich the diversity of object queries using the voting centers directly from the cluster branch. Simultaneously, a direction loss is employed to encourage a more accurate voting center for each cluster. The two-level enhancement (i.e., feature integration, and query expansion) design of CluB greatly enriches the object representation ability. With our proposed framework, CluB outperforms the previous single-frame models for detecting 3D objects on both Waymo and nuScenes datasets.
Limitations. Remarkably, CluB presents an elegant view of how to combine the context-aware BEV representation with object-centric cluster representation in a unified detection framework and achieves promising performance via two-level enhancement (feature, query) for object representation. Yet, the computational and spatial complexity on BEV feature maps is quadratic to the perception range, which makes it hard to adopt our CluB for long-range detection. We leave the exploration of computational efficient version of our method as the future work.
We hereby thank Pengfei Luo for his contribution. This work was supported by the Chinese Academy of Sciences Frontier Science Key Research Project ZDBS-LY-JSC001, the National Key R&D Program of China (No.2022ZD0160100), and in part by Shanghai Committee of Science and Technology (No.21DZ1100100).
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, pages 1080-1089. IEEE, 2022.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11618-11628. Computer Vision Foundation / IEEE, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV (1), volume 12346 of Lecture Notes in Computer Science, pages 213-229. Springer, 2020.
[4] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnet: Fully sparse voxelnet for 3d object detection and tracking. In CVPR, pages 21674-21683. IEEE, 2023.
[5] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal BEV distillation for multi-view 3d object detection. CoRR, abs/2211.09386, 2022.
[6] MMDetection3D Contributors. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https://github.com/open-mmlab/mmdetection3d, 2020.
[27] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV (10), volume 13670 of Lecture Notes in Computer Science, pages 426-442. Springer, 2022.
[28] Luequan Wang, Hongbin Xu, and Wenxiong Kang. Mvcontrast: Unsupervised pretraining for multi-view 3d object recognition. Machine Intelligence Research, pages 1-12, 2023.
[29] Yingjie Wang, Qiuyu Mao, Hanqi Zhu, Yu Zhang, Jianmin Ji, and Yanyong Zhang. Multi-modal 3d object detection in autonomous driving: a survey. CoRR, abs/2106.12735, 2021.
[30] Yue Wang and Justin M. Solomon. Object DGCNN: 3d object detection using dynamic graphs. In NeurIPS, pages 20745-20758, 2021.
[31] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS Datasets and Benchmarks, 2021.
[32] Dong Wu, Man-Wen Liao, Wei-Tian Zhang, Xing-Gang Wang, Xiang Bai, Wen-Qing Cheng, and Wen-Yu Liu. Yolop: You only look once for panoptic driving perception. Machine Intelligence Research, 19(6):550-562, 2022.
[33] Yan Yan, Yuxing Mao, and Bo Li. SECOND: sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
[34] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, Jie Zhou, and Jifeng Dai. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. CoRR, abs/2211.10439, 2022.
[35] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, pages 11784-11793. Computer Vision Foundation / IEEE, 2021.
[36] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In CVPR, pages 18931-18940. IEEE, 2022.
[37] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, pages 4490-4499. Computer Vision Foundation / IEEE Computer Society, 2018.
[38] Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV (38), volume 13698 of Lecture Notes in Computer Science, pages 496-513. Springer, 2022.
[39] Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. In CVPR, pages 9296-9305. IEEE, 2023.
[40] Xinge Zhu, Yuexin Ma, Tai Wang, Yan Xu, Jianping Shi, and Dahua Lin. SSN: shape signature networks for multi-class object detection from point clouds. In ECCV (25), volume 12370 of Lecture Notes in Computer Science, pages 581-597. Springer, 2020.