SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We further conduct the experiment and evaluate our method on the test set of KITTI and nuScenes dataset. The comparison is listed in the Tab. 11 and Tab. 12 separately. For the most important KITTI metric, AP on the moderate level of car, we obtain the  $82.26\%$  AP, which is 0.83 points higher than PV-RCNN, 0.49 points higher than CT3D, and 0.38 points higher than PV-RCNN++. On the test set of the nuScenes dataset, we obtain the  $65.2\%$  mAP and  $70.8\%$  NDS. The consistent superiority further demonstrates the ability of Uni3DETR on outdoor 3D detection.
C Visualized Results
Comparison results about the mixture of query points. We first provide comparative visualized results in Fig. 6 to illustrate the effectiveness of the mixture of query points. For the first case, it can be seen that training with only the learnable query points concentrates on the right region of the bed and the nightstand, and ignores the left sofa. This similarly applies to the rest two cases. For the second case, the right nightstand is not detected and for the third case, three chairs are ignored. The common point among these three cases is that these ignored objects are partly occluded, thus with insufficient points. The limited quantities of point clouds therefore restrict the performance of the 3D
Table 11: The performance of Uni3DETR for outdoor 3D object detection on the KITTI test set with 40 recall positions. We train the models on the car category only. *: AP on the moderate car is the most important metric.
Method	Easy	Mod.*	Hard
SECOND [64]	88.61	78.62	77.22
PointPillar [24]	82.58	74.31	68.99
Part- \( A^2 \) [52]	87.81	78.49	73.51
PV-RCNN [49]	90.25	81.43	76.82
CT3D [46]	87.83	81.77	77.16
PV-RCNN++ [50]	-	81.88	-
Uni3DETR (ours)	91.14	82.26	77.58
Table 12: The performance of Uni3DETR for outdoor 3D object detection on the nuScenes test set. We compare with previous methods with double-flip testing.
Method	mAP(%)	NDS(%)
PointPillar [24]	30.5	45.3
CBGS [75]	52.8	63.3
CenterPoint [67]	58.0	65.5
Focals Conv [7]	63.8	70.0
UVTR [27]	63.9	69.7
PillarNet [48]	65.0	70.8
Uni3DETR (ours)	65.2	70.8
Figure 6: The comparative visualized results of our mixture of query points, compared to the result obtained from learnable query points only. The examples are from the SUN RGB-D dataset.
detector. After we introduce the mixture of query points, global information is better considered with the help of non-learnable query points. As a result, even under the circumstance of insufficient point cloud information, our Uni3DETR can still recognize these objects and detect them out relying on the knowledge about the whole scene. More visualized examples are also provided in Fig. 7.
Comparison results about the decoupled IoU. We then compare the visualized results of Uni3DETR with the normal 3D IoU and plot the results in Fig. 8. It can be seen that when supervising the detector with the normal 3D IoU, although the 3D detector has the ability to detect the foreground objects out, the localization precision remains significantly low. For example, for the second case, two chairs are indeed detected, but the overlaps between the detected instances with the corresponding objects are minimal. Furthermore, the low degree of the overlapped area also results in many duplicated boxes, especially for the above one. The same thing also occurs at the third case. For the first case, besides the localization error, only one chair is detected out. This is because 3D IoU is hard to optimize
Figure 7: More comparative visualized results of our mixture of query points, compared to the result obtained from learnable query points only. The left two examples are from the ScanNet dataset and the right is from the KITTI dataset.
thus fails to alleviate the scale problem of L1 loss. In comparison, our decoupled IoU addresses the coupling problem of 3D IoU, thus contributing to better localization accuracy.
More visualized results obtained by Uni3DETR. We further provide more visualized results obtained by our Uni3DETR on different datasets. Uni3DETR obtains satisfying detection results on all five datasets, which further demonstrate its effectiveness and universality.
D Per-category Results
Table 13: Per-category  $\mathrm{AP}_{25}$  for the 10 classes on the SUN RGB-D dataset.
bathtub	bed	bookshelf	chair	desk	dresser	nightstand	sofa	table	toilet	mAP
H3DNet [70]	73.8	85.6	31.0	76.7	29.6	33.4	65.5	66.5	50.8	88.2	60.1
BRNet [9]	76.2	86.9	29.7	77.4	29.6	35.9	65.9	66.4	51.8	91.3	61.1
GroupFree [30]	80.0	87.8	32.5	79.4	32.6	36.0	66.7	70.0	53.8	91.1	63.0
FCAF3D [43]	79.0	88.3	33.0	81.1	34.0	40.1	71.9	69.7	53.0	91.3	64.2
Uni3DETR (ours)	80.7	89.1	30.7	85.6	38.6	42.7	74.7	75.1	59.2	93.9	67.0
Table 14: Per-category  $\mathrm{AP}_{50}$  for the 10 classes on the SUN RGB-D dataset.
bathtub	bed	bookshelf	chair	desk	dresser	nightstand	sofa	table	toilet	mAP
H3DNet [70]	47.6	52.9	8.6	60.1	8.4	20.6	45.6	50.4	27.1	69.1	39.0
BRNet [9]	55.5	63.8	9.3	61.6	10.0	27.3	53.2	56.7	28.6	70.9	43.7
GroupFree [30]	64.0	67.1	12.4	62.6	14.5	21.9	49.8	58.2	29.2	72.2	45.2
FCAF3D [43]	66.2	69.8	11.6	68.8	14.8	30.1	59.8	58.2	35.5	74.5	48.9
Uni3DETR (ours)	67.4	66.2	10.7	71.7	14.8	33.5	60.3	63.0	36.7	78.6	50.3
We first list the per-category results for the 10 classes on the SUN RGB-D dataset in Tab. 13 and Tab. 14. For the  $\mathrm{AP}_{25}$  metric, Uni3DETR achieves the best for 9 classes out of the total 10 classes. The
ground-truth
3D IoU
decoupled IoU
Figure 8: The comparative visualized results of our decoupled IoU, compared to the result obtained from the normal 3D IoU.
ground-truth
Figure 9: The visualized results of Uni3DETR on the SUN RGB-D dataset.
Uni3DETR ground-truth
Figure 10: The visualized results of Uni3DETR on the ScanNet dataset.
Figure 11: The visualized results of Uni3DETR on the S3DIS dataset.
Table 15: Per-category  $\mathrm{AP}_{25}$  for the 18 classes on the ScanNet dataset.
cab	bed	chair	sofa	tabl	door	wind	bkshf	pic	cntr	desk	curt	fridg	showr	toil	sink	bath	ofurn	mAP
VoteNet [38]	36.3	87.9	88.7	89.6	58.8	47.3	38.1	44.6	7.8	56.1	71.7	47.2	45.4	57.1	94.9	54.7	92.1	37.2	58.7
GSDN [17]	41.6	82.5	92.1	87.0	61.1	42.4	40.7	51.5	10.2	64.2	71.1	54.9	40.0	70.5	100	75.5	93.2	53.1	62.8
H3DNet [70]	49.4	88.6	91.8	90.2	64.9	61.0	51.9	54.9	18.6	62.0	75.9	57.3	57.2	75.3	97.9	67.4	92.5	53.6	67.2
GroupFree[30]	52.1	92.9	93.6	88.0	70.7	60.7	53.7	62.4	16.1	58.5	80.9	67.9	47.0	76.3	99.6	72.0	95.3	56.4	69.1
FCAF3D [43]	57.2	87.0	95.0	92.3	70.3	61.1	60.2	64.5	29.9	64.3	71.5	60.1	52.4	83.9	99.9	84.7	86.6	65.4	71.5
Uni3DETR (ours)	58.1	87.0	94.9	91.2	71.7	66.9	58.5	59.6	34.6	73.2	81.0	55.6	52.7	81.2	99.6	78.2	83.5	63.7	71.7
Table 16: Per-category  $\mathrm{AP}_{50}$  for the 18 classes on the ScanNet dataset.
cab	bed	chair	sofa	tabl	door	wind	bkshf	pic	cntr	desk	curt	fridg	showr	toil	sink	bath	ofurn	mAP
VoteNet [38]	8.1	76.1	67.2	68.8	42.4	15.3	6.4	28.0	1.3	9.5	37.5	11.6	27.8	10.0	86.5	16.8	78.9	11.7	33.5
GSDN [17]	13.2	74.9	75.8	60.3	39.5	8.5	11.6	27.6	1.5	3.2	37.5	14.1	25.9	1.4	87.0	37.5	76.9	30.5	34.8
H3DNet [70]	20.5	79.7	80.1	79.6	56.2	29.0	21.3	45.5	4.2	33.5	50.6	37.3	41.4	37.0	89.1	35.1	90.2	35.4	48.1
GroupFree[30]	26.0	81.3	82.9	70.7	62.2	41.7	26.5	55.8	7.8	34.7	67.2	43.9	44.3	44.1	92.8	37.4	89.7	40.6	52.8
FCAF3D [43]	35.8	81.5	89.8	85.0	62.0	44.1	30.7	58.4	17.9	31.3	53.4	44.2	46.8	64.2	91.6	52.6	84.5	57.1	57.3
Uni3DETR (ours)	39.5	82.5	90.4	83.1	63.8	50.5	31.9	56.4	23.5	38.6	62.8	38.4	42.2	61.6	97.8	50.9	80.2	56.3	58.3
most significant improvement comes from the sofa and table class,  $5.4\%$  and  $6.2\%$  respectively. For the AP50 metric, UniDETR also achieves the best for 8 classes. For the sofa class, the improvement is up to  $4.8\%$ . The effectiveness of UniDETR is thus further demonstrated.
ground-truth
Uni3DETR
Figure 12: The visualized results of Uni3DETR on the KITTI dataset.
Uni3DETR
Figure 13: The visualized results of Uni3DETR on the nuScenes dataset.
Table 17: Per-category  $\mathrm{AP}_{25}$  for the 5 classes on the S3DIS dataset.
table	chair	sofa	bkcase	board	mAP
GSDN [17]	73.7	98.1	20.8	33.4	12.9	47.8
FCAF3D [43]	69.7	97.4	92.4	36.7	37.3	66.7
Uni3DETR (ours)	74.4	98.7	77.4	47.7	52.2	70.1
Table 18: Per-category  $\mathrm{AP}_{50}$  for the 5 classes on the S3DIS dataset.
table	chair	sofa	bkcase	board	mAP
GSDN [17]	36.6	75.3	6.1	6.5	1.2	25.1
FCAF3D [43]	45.4	88.3	70.1	19.5	5.6	45.9
Uni3DETR (ours)	45.4	87.9	64.1	19.0	23.7	48.0
Table 19: Per-category AP for the 10 classes on the nuScenes dataset.
NDS	mAP	Car	Truck	Bus	Trailer	C.V.	Ped	Mot	Byc	T.C.	Bar
UVTR [27]	67.7	60.9	85.3	53.0	69.1	41.4	24.0	82.6	70.4	52.9	67.1	63.4
Uni3DETR (ours)	68.5	61.7	87.0	59.0	70.8	41.7	23.9	86.1	66.4	46.0	67.8	68.0
The per-category results for the 18 classes on the ScanNet dataset are listed in Tab. 15 and Tab. 16. We achieve the best for 7 classes for the  $\mathrm{AP}_{25}$  metric and for 9 classes for the  $\mathrm{AP}_{50}$  metric. For the S3DIS dataset, the per-category results are listed in Tab. 17 and Tab. 18. For the  $\mathrm{AP}_{25}$  metric, we achieve the best for 4 classes out of the total 5 ones. For the  $\mathrm{AP}_{50}$  metric, we are  $18.1\%$  higher than FCAF3D on the board class. These per-category results further demonstrate the ability of Uni3DETR on indoor scenes.
We also list the per-category AP on the nuScenes dataset in Tab. 19. Our Uni3DETR achieves the best for 7 out of 10 classes. The ability of Uni3DETR is further demonstrated on outdoor scenes.