SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rope3D. Rope3D [53] is a comprehensive real-world benchmark for infrastructure-side 3D detection, featuring over 50,000 images and more than 1.5 million 3D objects in diverse scenes. The benchmark comprises various settings, such as different cameras with ambiguous mounting positions, diverse camera specifications, as well as different environmental conditions. For the evaluation metrics, we follow the official settings using  $AP_{3D|R40}$  and  $Ropescore$  [53], which is a combined metric of the 3D AP and other similarities, such as Average Ground Center Similarity (ACS). We follow the proposed homologous setting to utilize  $70\%$  of the images as training, and the remaining as validating. All images are randomly sampled.
DAIR-V2X-I. DAIR-V2X-I [55] is a subset of DAIR-V2X, which is a large-scale multimodal vehicle-infrastructure collaborative perception dataset. It contains 10,000 infrastructure-side images and 493,000 3D bounding boxes with 10 classes. We follow the official train/val division for evaluation, and use  $AP_{3D|R40}$  as the main metric, with an IOU threshold of 0.5.
KITTI. KITTI [10] is a widely used benchmark consisting of 7481 training images and 7518 testing images for vehicle-side monocular 3D detection. The dataset contains three classes: Car, Pedestrian, and Cyclist, each with three difficulty levels: Easy, Moderate, and Hard. Moderate difficulty is considered the official rank level of the KITTI leaderboard. To ensure a fair comparison with previous methods, results are submitted to an official server for evaluation on the test set.  $AP_{3D|R40}$  is used as the main metric with IOU thresholds of 0.7 for cars, and 0.5 for Pedestrians and Cyclists respectively.
Method	AP3D
Pedestrian	Cyclist
Easy	Mod.	Hard	Easy	Mod.	Hard
DFR-Net [61]	6.09	3.62	3.39	5.69	3.58	3.10
CaDDN [39]	12.87	8.14	6.76	7.00	3.41	3.30
MonoCon [48]	13.10	8.41	6.94	2.80	1.92	1.55
GUPNet [27]	14.95	9.76	8.41	5.58	3.21	2.66
MonoDTR [12]	15.33	10.18	8.61	5.05	3.27	3.19
MonoUNI	15.78	10.34	8.74	7.34	4.28	3.78
Table 3: Monocular 3D detection performance of Pedestrian and Cyclist category on KITTI test set.
Table 4: Monocular 3D detection performance of Big Vehicle category on Rope3D val set.
Method	Big Vehicle
AP3D	Rscore
M3D-RPN [1]	32.19	40.52
MonoDLE [30]	44.71	52.83
SMOKE [25]	52.04	59.23
GUPNet [27]	45.27	52.19
BEVFormer [24]	34.58	45.16
BEVDepth [22]	45.02	54.64
BEVHeight [49]	48.93	57.70
MonoUNI	76.30	79.20
Table 5: Ablation Study on different components of our overall framework on Rope3D and KITTI val set for Car category.
Experiments	Normlized Depth	Cube	Rope3D	KITTI
Focal	Pitch	AP3D	Rscore	Easy	Mod.	Hard
(a)	79.37	81.38	21.71	14.93	12.10
(b)	✓	83.42	84.91	22.35	15.29	12.34
(c)	✓	81.55	83.53	21.68	14.85	12.12
(d)	✓	✓	86.69	87.99	22.19	14.94	12.45
(e)	✓	87.07	88.16	24.66	17.07	14.06
(f)	✓	✓	✓	92.45	92.63	24.51	17.18	14.01
Waymo. Waymo [42] assesses objects using a dual-tiered categorization: Level 1 and Level 2. The assessment is performed across three distance intervals: [0, 30), [30, 50), and  $[50,\infty)$  meters. Waymo employs the  $APH_{3D}$  percentage metric, which integrates heading data into the  $AP_{3D}$ , as a reference benchmark for evaluation.
nuScenes. nuScenes [3] comprises 28,130 training and 6,019 validation images captured from the front camera. We use validation split for cross-dataset evaluation.
Our proposed MonoUNI is trained on 4 Tesla V100 GPUs with a batch size of 16 for 150 epochs. We use Adam [17] as our optimizer with an initial learning rate  $1.25 \times e - 3$ . Images are all resized to the same size of  $960 \times 512$  for infrastructure side and  $1280 \times 384$  for vehicle side. Following [37], the ROI-Align size  $d \times d$  is set to  $7 \times 7$ . Inspired by [33], we adopt the multi-bin strategy for heading angle and depth prediction in our baseline. Random crop and expand along principal points are used to achieve more physical data augmentation.
Results of Car Category. Since there was no previous method designed for supporting both the vehicle and infrastructure sides at the same time, the results of some methods were reproduced by us or other published papers (BEVHeight [49], Rope3D [53] and DAIR-V2X [55]). As shown in Table 2, our proposed MonoUNI achieves superior performance than previous methods on infrastructure-side benchmarks, even those with extra data. Specifically, compared with BEVHeight which is the recent top1-ranked image-only method for infrastructure side, MonoUNI gains significant improvement of  $17.85\% / 13.91\%$  in  $AP_{3D}$  and  $R_{score}$  on Rope3D benchmark. According to DAIR-V2X, our method achieves over  $21\%$  improvement compared with BEVHeight on Moderate difficulty. For vehicle-side benchmark, although our method did not rank first, it reached a competitive result on Easy difficulty. On the Moderate and Hard difficulties, MonoUNI is slightly inferior to methods such as MonoATT [60] and MoGDE [59], probably because the gain of the 3D cube depth is weakened in the case of far-distance or severe truncation.
Results of Other Categories. In Table 3, we present the results of pedestrians and cyclists on the test set of KITTI. MonoUNI outperforms all methods by a large margin. This may be because the
Table 6: Analysis of MonoUNI for different focal lengths on Rope3D with new train/val division.
Method	Train	AP3D(IOU=0.5|R40)
val_2100	val_2700	val_all
MonoUNI	2100	25.78	21.30	24.47
MonoUNI	2700	5.77	23.42	9.25
MonoUNI	all	26.63	38.10	28.91
Table 7: Cross-dataset evaluation of the KITTI val model on KITTI val and nuScenes val cars with depth MAE.
Method	KITTI Val	nuScenes frontal Val
0-20	20-40	40-∞	All	0-20	20-40	40-∞	All
M3D-RPN [1]	0.56	1.33	2.73	1.26	0.94	3.06	10.36	2.67
MonoRCNN [41]	0.46	1.27	2.59	1.14	0.94	2.84	8.65	2.39
GUPNet [27]	0.45	1.10	1.85	0.89	0.82	1.70	6.20	1.45
DEVIANT [20]	0.40	1.09	1.80	0.87	0.76	1.60	4.50	1.26
MonoUNI	0.38	0.92	1.79	0.865	0.72	1.79	4.98	1.43
Table 8: Monocular 3D detection performance of Vehicle category on Waymo val set.
IOU3D	Difficulty	Method	Extra	AP3D	APH3D
All	0-30	30-50	50-∞	All	0-30	30-50	50-∞
0.7	Level_1	CaDDN [39]	LIDAR	5.03	15.54	1.47	0.10	4.99	14.43	1.45	0.10
PatchNet [28] in [43]	Depth	0.39	1.67	0.13	0.03	0.39	1.63	0.12	0.03
PCT [43]	Depth	0.89	3.18	0.27	0.07	0.88	3.15	0.27	0.07
M3D-RPN [1] in [39]	None	0.35	1.12	0.18	0.02	0.34	1.10	0.18	0.02
GUPNet [27] in [20]	None	2.28	6.15	0.81	0.03	2.27	6.11	0.80	0.03
DEVIANT [20]	None	2.69	6.95	0.99	0.02	2.67	6.90	0.98	0.02
MonoUNI (Ours)	None	3.20	8.61	0.87	0.13	3.16	8.50	0.86	0.12
0.7	Level_2	CaDDN [39]	LIDAR	4.49	14.50	1.42	0.09	4.45	14.38	1.41	0.09
PatchNet [28] in [43]	Depth	0.38	1.67	0.13	0.03	0.36	1.63	0.11	0.03
PCT [43]	Depth	0.66	3.18	0.27	0.07	0.66	3.15	0.26	0.07
M3D-RPN [1] in [39]	None	0.35	1.12	0.18	0.02	0.33	1.10	0.17	0.02
GUPNet [27] in [20]	None	2.14	6.13	0.78	0.02	2.12	6.08	0.77	0.02
DEVIANT [20]	None	2.52	6.93	0.95	0.02	2.50	6.87	0.94	0.02
MonoUNI (Ours)	None	3.04	8.59	0.85	0.12	3.00	8.48	0.84	0.12
0.5	Level_1	CaDDN [39]	LIDAR	17.54	45.00	9.24	0.64	17.31	44.46	9.11	0.62
PatchNet [28] in [43]	Depth	2.92	10.03	1.09	0.23	2.74	9.75	0.96	0.18
PCT [43]	Depth	4.20	14.70	1.78	0.39	4.15	14.54	1.75	0.39
M3D-RPN [1] in [39]	None	3.79	11.14	2.16	0.26	3.63	10.70	2.09	0.21
GUPNet [27] in [20]	None	10.02	24.78	4.84	0.22	9.94	24.59	4.78	0.22
DEVIANT [20]	None	10.98	26.85	5.13	0.18	10.89	26.64	5.08	0.18
MonoUNI (Ours)	None	10.98	26.63	4.04	0.57	10.73	26.30	3.98	0.55
0.5	Level_2	CaDDN [39]	LIDAR	16.51	44.87	8.99	0.58	16.28	44.33	8.86	0.55
PatchNet [28] in [43]	Depth	2.42	10.01	1.07	0.22	2.28	9.73	0.97	0.16
PCT [43]	Depth	4.03	14.67	1.74	0.36	4.15	14.51	1.71	0.35
M3D-RPN [1] in [39]	None	3.61	11.12	2.12	0.24	3.46	10.67	2.04	0.20
GUPNet [27] in [20]	None	9.39	24.69	4.67	0.19	9.31	24.50	4.62	0.19
DEVIANT [20]	None	10.29	26.75	4.95	0.16	10.20	26.54	4.90	0.16
MonoUNI (Ours)	None	10.38	26.57	3.95	0.53	10.24	26.24	3.89	0.51
introduction of depth clues promotes pedestrians and cyclists better learn spatial features. Table 4 shows the results of Big Vehicle on the Rope3D benchmark.
Effectiveness of Different Components in MonoUNI on Rope3D and KITTI val set for Car category. We evaluate the effectiveness of the normalized depth and the 3D cube depth through ablations on two datasets. As shown in Table 5, the normalized depth, particularly the focal length, has a significant improvement on the infrastructure side, but only a minor effect on the vehicle side. This is due to the fact that the focal length of the camera is similar across the KITTI data, and the pitch angle is close to 0. On the other hand, the cube depth improves results for both the vehicle and infrastructure sides, providing evidence of its efficacy.
Analysis for different focal lengths. Table 6 presents the performance of MonoUNI at different focal lengths. Compared with the results in Table 1, our method has obvious benefits in solving the problem of focal length diversity. In the case of using all training data, our method can improve  $19.53\%$  and  $20.43\%$  AP respectively compared with GUPNet and SMOKE.
Figure 5: Qualitative visualization on the Rope3D and KITTI val set. The 3D green boxes are produced by MonoUNI and the red boxes are the ground truths.
Cross-dataset evaluation. Tabel 7 shows the result of our KITTI val model on the KITTI val and nuScenes [3] frontal val images, using mean absolute error (MAE) of the depth [41]. MonoUNI is better than GUPNet [27] and achieves similar competitive performance to DEVIANT [20]. This is because DEVIANT is equivariant to the depth translations and is more robust to data distribution changes.
Performance of cube depth on large dataset Waymo. In order to fully verify the effectiveness of cube depth, we verified MonoUNI on Waymo, which is a large-scale vehicle-side 3D detection dataset, as shown in Table 8.
We visualize the detection results of the MonoUNI on the both vehicle and infrastructure sides in Fig. 5. As can be seen, the MonoUNI can accurately estimate the 3D position of objects, even those not labeled by the annotators due to severe occlusion in the first row of KITTI. However, for far-distance and severely truncated obstacles, our model suffers from missed detections.
In this paper, we propose a new optimization target named normalized depth to unify monocular 3D detection for both vehicle and infrastructure sides, addressing the problem of focal length and pitch angle diversity. Furthermore, we introduce 3D Cube Depth as an additional supervision clue to improve the 3D detection performance. Experiments on five datasets (Rope3D, DAIR-V2X-I, KITTI, Waymo and nuScenes) fully demonstrate the effectiveness of our method.
Although the normalized depth unifies the optimization target of the vehicle and infrastructure sides, separate training for the two sides is still necessary instead of direct hybrid training. In the future, we aim to enable one model file to support 3D detection for both sides which would hold significant value for industrial applications, such as the domain gap problem in the vehicle-infrastructure collaborative perception and fusion. In addition, global depth clues such as depth similarity (i.e., contrastive learning) or the global topological relationship can be further developed.