SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first train and evaluate Uni3DETR on indoor 3D detection datasets and list the comparison with existing state-of-the-art indoor 3D detectors in Tab. 1. Here we omit some grouping methods like [58], which relies on mask annotations for better grouping and clustering. Our method obtains the  $67.0\%$ $\mathrm{AP}_{25}$  and  $50.3\%$ $\mathrm{AP}_{50}$  on SUN RGB-D, which surpasses FCAF3D, the state-of-the-art indoor detector based on the CNN architecture, by almost  $3\%$ . On the ScanNet dataset, Uni3DETR surpasses FCAF3D by  $1\%$  on  $\mathrm{AP}_{50}$ . It is also noticeable that with only point clouds participating in training, our method even obtains better performance than existing multi-modal approaches that require both point clouds and RGB images. On the SUN RGB-D dataset, our model is  $2.1\%$  higher on  $\mathrm{AP}_{25}$  than TokenFusion. This strongly demonstrates the effectiveness of Uni3DETR. The visualized results of Uni3DETR on SUN RGB-D can be seen in the left two of Fig. 5.
Our method also significantly outperforms existing transformer-based indoor detectors, 3DETR and GroupFree. The superiority of our method is more significant, especially in localization: Uni3DETR
Table 2: The performance of Uni3DETR for outdoor 3D object detection on the KITTI validation set with 11 recall positions.  $M$ :  $\checkmark$  means training on three classes and the blank means training only on the car class.  $*$ : AP on the moderate car is the most important metric.
Method	M	Car-3D (IoU=0.7)	Ped.-3D (IoU=0.5)	Cyc.-3D (IoU=0.5)
Easy	Mod.*	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
SECOND [64]	✓	88.61	78.62	77.22	56.55	52.98	47.73	80.59	67.16	63.11
PointPillar [24]	✓	86.46	77.28	74.65	57.75	52.29	47.91	80.06	62.69	59.71
PointRCNN [51]	✓	89.06	78.74	78.09	67.69	60.74	55.83	86.16	71.16	67.92
Part-A2[52]	✓	89.56	79.41	78.84	65.69	60.05	55.45	85.50	69.90	65.49
PV-RCNN [49]	✓	89.35	83.69	78.70	63.12	54.84	51.78	86.06	69.48	64.50
CT3D [46]	✓	89.11	85.04	78.76	64.23	59.84	55.76	85.04	71.71	68.05
RDIoU [47]	✓	89.16	85.24	78.41	63.26	57.47	52.53	83.32	68.39	63.63
Uni3DETR (ours)	✓	89.61	86.57	78.96	70.18	62.49	58.32	87.18	72.90	68.86
3DSSD [65]	88.82	78.58	77.47	-	-	-	-	-	-
STD [66]	89.70	79.80	79.30	-	-	-	-	-	-
Voxel-RCNN [13]	89.41	84.52	78.93	-	-	-	-	-	-
VoTr-TSD [33]	89.04	84.04	78.68	-	-	-	-	-	-
CT3D [46]	89.54	86.06	78.99	-	-	-	-	-	-
BtcDet [63]	-	86.57	-	-	-	-	-	-	-
RDIoU [47]	89.76	86.62	79.04	-	-	-	-	-	-
Uni3DETR (ours)	90.23	86.74	79.31	-	-	-	-	-	-
Table 3: The performance of Uni3DETR for outdoor 3D object detection on the nuScenes validation set. We compare with previous methods without test-time augmentation. \*: the implementation from OpenPCDet [55].
Method	NDS(%)	mAP(%)	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
PointPillar [24]	49.1	34.3	0.424	0.284	0.529	0.377	0.194
CBGS [75]	61.5	51.9	-	-	-	-	-
CenterPoint [67]	64.9	56.6	0.291	0.252	0.324	0.284	0.189
VoxelNeXt* [8]	66.7	60.5	0.301	0.252	0.406	0.217	0.186
PillarNet [48]	67.4	59.8	0.277	0.252	0.289	0.247	0.191
UVTR [27]	67.7	60.9	0.334	0.257	0.300	0.204	0.182
Uni3DETR (ours)	68.5	61.7	0.288	0.249	0.303	0.216	0.181
outperforms them by  $5.1\%$  on SUN RGB-D and  $5.5\%$  on ScanNet in  $\mathrm{AP}_{50}$ . Such results validate that compared with existing transformers in 3D detection, our detection transformer on voxel features with the mixture of query points is more appropriate for 3D detection.
3.2 Outdoor 3D Object Detection
KITTI. We then conduct experiments on the outdoor KITTI dataset. We report the detection results on the KITTI validation set in three difficulty levels - easy, moderate, and hard in Tab. 2. We notice that Uni3DETR also achieves the satisfying performance with the same structure as that for indoor scenes. For the most important KITTI metric, AP on the moderate level of car, we obtain the  $86.57\%$  AP, which is more than  $1.5\%$  higher than CT3D and  $1.3\%$  higher than RDIoU. With only the car class in training, the car moderate AP is  $86.74\%$ , which is also higher than existing methods like BtcDet. Its ability in outdoor scenes is thus demonstrated. The superiority of Uni3DETR is also consistent for the pedestrian and cyclist class. This illustrates that our model can also distinguish small and scarce objects well, which is one main aspect that hinders existing indoor detectors in the outdoor environments. The visualized results are shown in the right two of Fig. 5.
nuScenes. We further conduct experiments on the nuScenes dataset. Compared to the KITTI dataset, the range of scenes in nuScenes is larger, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). These make the nuScenes more challenging and even some existing outdoor detectors fail to address the detection problem on nuScenes. We list the comparative results in Tab. 3. We obtain the  $68.5\%$  NDS, which surpasses recent methods like PillarNet, UVTR, VoxelNeXt. Compared to the most recent method VoxelNeXt, Uni3DETR is  $1.8\%$  higher in NDS and  $1.4\%$  higher in mAP. Besides the detection metric mAP, Uni3DETR also achieves promising results for predicting other attributes of boxes. The ability of our model in the outdoor scenes is further validated.
Figure 5: The visualized results of Uni3DETR on the indoor SUN RGB-D dataset (the left two) and the outdoor KITTI dataset (the right two). Better zoom in with colors.
Table 4: The 3D performance of our detection transformer on the SUN RGB-D dataset compared with some existing transformer decoder structures.
transformer	AP25	AP50
3DETR [35]	59.1	32.7
Deform [77]	61.3	45.4
DAB-Deform [29]	62.0	44.3
DN-Deform [25]	62.0	44.6
ours ( {P_l})	62.6	46.4
ours (mixed)	66.4	49.6
Table 5: Effect of the mixture of query points on 3D indoor and outdoor detection. We compare with different combinations of queries. \*: AP on the moderate car is the most important metric for KITTI.
query	SUN RGB-D	KITTI car
AP25	AP50	Easy	Mod.*	Hard
{Pl}	62.6	46.4	90.20	85.59	78.89
{Pnl}	54.0	39.9	89.42	79.24	78.29
{Pnlv}	57.4	43.1	89.70	79.34	78.14
{Pl, Pnl}	65.1	46.9	90.06	85.94	78.64
{Pl, Pnlv}	64.5	46.9	90.04	85.86	78.75
{Pl, Pnl, Pnlv}	66.4	49.6	90.12	86.26	79.01
{Pl, Pnl, Pnlv, Prd}	67.0	50.3	90.23	86.74	79.31
3D detection transformer. We first compare the detection AP of our detection transformer with existing transformer structures and list the results in Tab. 4. As 3DETR is built on point-based features, its performance is restricted by the less effectiveness of point representations. We then implement other methods based on voxel features. As anchors are negatively affected by the center point missing in 3D point clouds, formulating queries as 3D anchors like DAB- or DN-DETR is less effective than our query points. In comparison, our detection transformer better adapts the 3D points, thus achieves the highest performance among existing methods. Besides, it accommodates the mixture of query points, which brings further detection improvement.
Mixture of query points. Tab. 5 analyzes the effect of the mixture of query points on 3D object detection. For indoor detection on SUN RGB-D, the simple usage of the learnable query points  $\{P_l\}$  obtains the  $62.6\%$ $\mathrm{AP}_{25}$ . Substituting the learnable query with the non-learnable query hurts the performance. This is because non-learnable query points are fixed during training and cannot adaptively attend to information around objects. After we introduce  $\{P_l, P_{nl}\}$ , we find such mixed usage notably boosts  $\mathrm{AP}_{25}$ , from  $62.6\%$  to  $65.1\%$ : the global information in  $P_{nl}$  overcomes the lack of local information in  $P_l$  and contributes to more comprehensive detection results. As a result, the object-related metric is enhanced greatly. Utilizing  $\{P_l, P_{nlv}\}$  similarly improves  $\mathrm{AP}_{25}$ , but less than the effect of  $P_{nl}$ . This is because voxelization may remove some points, making the global information not that exhaustive. With  $\{P_l, P_{nl}, P_{nlv}\}$ , the three different query points further enhance the performance.  $P_{nlv}$  here provides more location-accurate global information, thus  $AP_{50}$  is improved by almost  $3\%$ . Random points at the test time further brings a  $0.6\%$  improvement.
For outdoor detection on KITTI, we notice that the effect of non-learnable query points is less significant than that in indoor scenes. Using  $\{P_l, P_{nl}\}$  is just  $0.35\%$  higher than  $\{P_l\}$ . Since outdoor point clouds are large-range and sparse, the effect of global information in non-learnable query will be counteracted by excessive background information. Therefore, local information matters more in the outdoor task. The collaboration of the two-level information in the mixture of query makes it suitable for 3D detection in various scenes.
We further combine multiple groups of learnable query points and list the performance in Tab. 6 to compare with the mixture of query points. Such a combination is actually analogous to Group DETR [5, 6] in 2D detection. We observe that unlike 2D detection, multiple groups of learnable queries do not bring an improvement. Even though the number of groups has been raised to 7, the AP on SUN RGB-D still keeps almost the same. The reason is that multiple groups in 2D mainly speed up the convergence, which does not matter much in 3D. In this situation, by providing multi-level information about the scene, the mixture of query points works better for 3D detection.
Table 6: Comparison with multiple groups of learnable query points on the SUN RGB-D dataset.
query	AP25	AP50
{Pl}	62.6	46.4
{Pl} × 2	62.2	46.2
{Pl} × 3	62.7	46.6
{Pl} × 5	62.3	46.7
{Pl} × 7	62.7	46.6
{Pl, Pnl}	65.1	46.9
{Pl, Pnl, Pnlv}	66.4	49.6
Table 7: Effect of the decoupled IoU on 3D indoor and outdoor detection.  $\mathrm{IoU}_{aa-3D}$  denotes the 3D IoU in axis-aligned way,  $\mathrm{IoU}_{xy}$  is the 2D IoU in the  $xy$  space,  $\mathrm{IoU}_z$  is the 1D IoU in the  $z$  space.
IoU	SUN RGB-D	KITTI car
AP25	AP50	Easy	Mod.*	Hard
w/o IoU	20.7	3.7	69.81	61.97	64.85
IoU3D	48.3	29.4	88.70	78.52	77.60
IoUaa-3D	50.4	31.6	89.57	79.44	76.81
RD-IoU [47]	29.1	14.0	82.08	73.50	73.98
IoUxy	65.8	49.0	89.91	79.58	77.93
IoUz	64.7	44.1	90.09	79.74	78.57
IoUde	67.0	50.3	90.23	86.74	79.31
Table 8: Comparison of performance and computational complexity against existing methods on the indoor SUN RGB-D and the outdoor nuScenes dataset. The metrics are  $\mathrm{AP}_{25}$  for SUN RGB-D and mAP for nuScenes. The computational cost is measured on a single RTX 3090 GPU.
Method	performance	efficiency
avg.	indoor	outdoor	latency	params	FLOPS
CenterPoint [67]	37.75	18.9	56.6	0.32 s	9.17 M	121.10 G
VoxelNeXt [8]	39.30	18.1	60.5	0.29 s	7.12 M	42.57 G
PillarNet [48]	44.00	28.2	59.8	0.31 s	12.55 M	100.10 G
UVTR [27]	55.55	50.2	60.9	0.51 s	26.12 M	451.12 G
Uni3DETR (ours)	64.35	67.0	61.7	0.52 s	26.71 M	458.74 G
Decoupled IoU. In Tab. 7, we compare our decoupled IoU with different types of IoU in the 3D space. On the SUN RGB-D dataset, optimizing with no IoU loss, only classification and L1 regression loss, just obtains the  $20.7\% \mathrm{AP}_{25}$ , indicating that 3D IoU is necessary to address the scale problem of L1 loss. Introducing the normal  $\mathrm{IoU}_{3D}$  alleviates the limitation of L1 loss to some extent, boosting the  $\mathrm{AP}_{25}$  to  $48.3\%$ . However, it is restricted by the negative coupling effect, thus is even inferior to axis-aligned 3D IoU, which does not take the rotation angle into consideration. In comparison, since this problem does not exist in 2D IoU, even the  $\mathrm{IoU}_{xy}$ , without considering the  $z$  axis, can be  $17.5\%$  higher than  $\mathrm{IoU}_{3D}$ . Our decoupled IoU, which can be viewed as  $(\mathrm{IoU}_{xy} + \mathrm{IoU}_z) / 2$ , takes all directions into account, thus further improves the  $\mathrm{AP}_{25}$  to  $67.0\%$ . On the outdoor KITTI dataset, the decoupled IoU is equally critical,  $24.77\%$  higher than w/o IoU and  $8.22\%$  higher than  $\mathrm{IoU}_{3D}$ . Its necessity for transformer-based 3D detection is further validated.
3.4 Comparison of Computational Cost
Here we further list the comparison of both performance and efficiency in the Tab. 8. We can observe that the computational budget compared with these methods is not significant: the inference time (latency) is almost the same as UVTR and the FLOPS is only  $1.16\%$  more than UVTR. In addition, we obtain significantly better detection performance on both indoor and outdoor datasets. Compared with VoxelNeXt, one model that mainly focuses on reducing the FLOPS of 3D detectors, we achieve more than  $40\%$  indoor AP and more than  $25\%$  average AP improvement. In this paper, we mainly target a unified structure. To ensure that the detector can accommodate both indoor and outdoor detection, we have, to a certain extent, made sacrifices in terms of efficiency, in order to prioritize its unified ability. A more efficient and unified structure can be left as one of the future works.
A unified voxel size. The word "unified" in our paper specifically refers to the architecture aspect. Since point clouds are collected with different sensors, their ranges and distributions vary significantly (about  $3\mathrm{m}$  for indoor but more than  $70\mathrm{m}$  for outdoor datasets). For the experiments above, we adopt different values for the voxel size, one data-related parameter. We further conduct the experiment with the same KITTI voxel size  $(0.05\mathrm{m}, 0.05\mathrm{m}, 0.1\mathrm{m})$  and list the results in Tab. 9. Compared with other outdoor detectors, our superiority is still obvious. Therefore, even if standardizing such a data-related parameter, our model can still obtain a higher AP.
Table 9: Comparison with existing methods on the indoor SUN RGB-D dataset and outdoor KITTI car dataset with the same voxel size. \*: results from training with different voxel sizes.
Method	indoor	outdoor
3DSSD [65]	9.5	78.6
CenterPoint [67]	18.9	74.4
UVTR [27]	35.9	72.0
Uni3DETR (ours)	47.3	86.7
Uni3DETR (ours) *	67.0	86.7
Table 10: The cross-dataset performance on the indoor SUN RGB-D dataset compared with existing methods. We compare with the RGB image based Cude RNN with its metric  $\mathrm{AP}_{3D}$ .
Method	Trained on	AP3D
Cube RCNN [2]	SUN RGB-D	34.7
Cube RCNN	OMNI3DIN	35.4
Uni3DETR (ours)	SUN RGB-D	64.3
Uni3DETR (ours)	ScanNet	50.9
Cross-dataset experiments. We further conduct the cross-dataset evaluation with the Omni3D metric [2]. From the results in Tab. 10, it is worth noticing that Uni3DETR has a good cross-dataset generalization ability. The cross-dataset AP (ScanNet to SUN RGB-D) is  $16.2\%$  higher than Cube RCNN trained on the SUN RGB-D dataset. The reason is that our Uni3DETR takes point clouds as input for 3D detection, while Cube RCNN takes RGB images for detection. By introducing 3D space information from point clouds, the superiority of a unified architecture for point clouds over Cube RCNN can be demonstrated. We further emphasize that cross-dataset evaluation is a more difficult problem for point cloud based 3D object detection, as the dataset-interference issue is more serious. We believe our Uni3DETR can become the basic platform and facilitate related research.
Limitation. One limitation is that we still require separate models for different datasets. Currently, some methods [2, 68] have tried one single model for multiple datasets for 3D object detection. We hope our Uni3DETR can become the foundation for indoor and outdoor 3D dataset joint training.
Existing research on 3D Object Detection has been separated into indoor and outdoor categories. Indoor 3D detectors [38, 70, 9, 58] usually cluster points first based on extracted point-wise features, then conduct classification and detection. Recently, 3D sparse convolution based detectors [43, 45] also adopt the anchor-free manner [56] in 2D for 3D indoor detection. In comparison, outdoor 3D detectors [64, 52, 49, 13, 46, 7, 50] usually convert 3D features into the 2D BEV space and adopt 2D convolutions for predicting 3D boxes. However, because of the significant difference in point clouds between indoor and outdoor scenes, a unified 3D detector in various environments is still absent. Some recent methods [20, 31, 60, 44] have performed experiments on both indoor and outdoor datasets. However, they rely on RGB images to bridge the difference of point clouds.
Transformer-based object detectors have been widely used in 2D object detection [4, 77, 34, 29, 25, 69], predicting object instances in the set-to-set manner. Recently, the transformer structure has been introduced in 3D object detection. Methods like [33, 18, 15, 54, 14] introduce transformers into the backbone for 3D feature extraction, while still adopting CNNs for predicting boxes. In comparison, approaches like [35, 61, 36, 74, 76] leverage transformers in the detection head for box prediction. However, these methods still rely on point-wise features or BEV features for either indoor or outdoor 3D detection, thus are restricted by the singular scene.
In this paper, we propose Uni3DETR, a unified transformer-based 3D detection framework that addresses indoor and outdoor 3D object detection within the same network structure. By feeding the mixture of query points into the detection transformer for point-voxel interaction and supervising the transformer decoder with the decoupled IoU, our Uni3DETR fills the gap of existing research in unified 3D detection under various scenes. Extensive experiments demonstrate that our model can address both indoor and outdoor 3D detection with a unified structure. We believe our work will stimulate following research along the unified and universal 3D object detection direction, and Uni3DETR will become a basic tool and platform.
Acknowledgment. This work is supported by the National Key Research and Development Program of China in the 14th Five-Year with Nos. 2021YFF0602103 and 2021YFF0602102, National Natural Science Foundation of China (No. 62201484), HKU Startup Fund and Seed Fund for Basic Research.
[63] Qiangeng Xu, Yiqi Zhong, and Ulrich Neumann. Behind the curtain: Learning occluded shapes for 3d object detection. In AAAI, 2022.
[64] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[65] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
[66] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In ICCV, 2019.
[67] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021.
[68] Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi-dataset 3d object detection. In CVPR, 2023.
[69] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023.
[70] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In ECCV, 2020.
[71] Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning a scene-conditioned 3d object detector. In CVPR, 2022.
[72] Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. Iou loss for 2d/3d object detection. In 3DV, 2019.
[73] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In CoRL, 2020.
[74] Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV, 2022.
[75] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv:1908.09492, 2019.
[76] Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. In CVPR, 2023.
[77] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. *ICLR*, 2021.
A Datasets and Implementation Details
SUN RGB-D [53]. When training on the SUN RGB-D dataset, the input point clouds are filtered in the range  $[-3.2\mathrm{m}, 3.2\mathrm{m}]$  for the  $x$  axis,  $[-0.2\mathrm{m}, 6.2\mathrm{m}]$  for the  $y$  axis and  $[-2\mathrm{m}, 0.56\mathrm{m}]$  for the  $z$  axis. The grid size is set to  $0.02\mathrm{m}$ . We randomly flip the input data along the  $x$  axis and randomly sample 20,000 points for data augmentation. The common global translation, rotation and scaling strategies are also adopted here. We train Uni3DETR with the initial learning rate of 1.67e-4 and the batch size of 32 for 90 epochs, and the learning rate is decayed by 10x on the 70th and 80th epoch. The ratio of the classification score and the predicted IoU is 0.8:0.2.
ScanNet [12]. For the ScanNet dataset, here we adopt the range of  $[-6.4\mathrm{m}, 6.4\mathrm{m}]$  for the  $x$  and  $y$  axis and  $[-0.1\mathrm{m}, 2.46\mathrm{m}]$  for the  $z$  axis after global alignment, with the  $0.02\mathrm{m}$  grid size. Here we do not adopt the RGB information of the dataset for training. The input data are randomly flipped along both the  $x$  and  $y$  axis. We utilize dynamic voxelization [73] on the ScanNet dataset. The initial learning rate is set to 7.5e-5, with the batch size of 24. We train Uni3DETR for 240 epochs, where the learning rate is decayed on the 192nd and the 228th epoch. Other hyper-parameters and operations are the same as the SUN RGB-D dataset.
S3DIS [1]. As the S3DIS dataset point clouds are distributed only on the positive planes, which is adverse to the random flipping augmentation, we first translate the input data to make it centered at the original point. Besides the coordinate information, we also utilize the RGB information of the point clouds. We train the Uni3DETR for 520 epochs, with the learning rate decaying at the 416th and 494th epoch. The batch size is set to 8, with the initial learning rate of 3.33e-5. Other hyper-parameters and operations are the same as the ScanNet dataset.
KITTI [16]. For the KITTI dataset, the data augmentation operations are basically the same as previous methods like [13]. For the ground-truth sampling augmentation, we sample at most 20 cars, 10 pedestrians and 10 cyclists from the database. 18000 points are randomly sampled at the training time. During training, we also adopt van class objects as car objects. The ground-truth sampling augmentation and the object-level noise strategy are removed at the last 2 epochs. We train Uni3DETR for 70 epochs, with the learning rate decaying at the 64th epoch. The initial learning rate is set to 9e-5, with the batch size of 8. As the KITTI dataset suffers from the sparse objects seriously, we follow [21] to repeat the ground truth labels 5 times during training. The predicted car objects are filtered at the threshold of 0.5 after inference. The ratio of the classification score and the predicted IoU is 0.2:0.8 on the KITTI dataset.
nuScenes [3]. Compared to the KITTI dataset, the nuScenes dataset covers a larger range, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). Besides, the nuScenes dataset contains 10 classes, with the severe long-tail problem. The initial learning rate is set to 1e-5, with the batch size of 16 and the cyclic schedule. We train the Uni3DETR for 20 epochs.
B Test Set Results