SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinrang Jia*
jiajinrang@baidu.com
Zhenjia Li*
lizhenjia@baidu.com
Yifeng Shi†
shiyifeng@baidu.com
Monocular 3D detection of vehicle and infrastructure sides are two important topics in autonomous driving. Due to diverse sensor installations and focal lengths, researchers are faced with the challenge of constructing algorithms for the two topics based on different prior knowledge. In this paper, by taking into account the diversity of pitch angles and focal lengths, we propose a unified optimization target named normalized depth, which realizes the unification of 3D detection problems for the two sides. Furthermore, to enhance the accuracy of monocular 3D detection, 3D normalized cube depth of obstacle is developed to promote the learning of depth information. We posit that the richness of depth clues is a pivotal factor impacting the detection performance on both the vehicle and infrastructure sides. A richer set of depth clues facilitates the model to learn better spatial knowledge, and the 3D normalized cube depth offers sufficient depth clues. Extensive experiments demonstrate the effectiveness of our approach. Without introducing any extra information, our method, named MonoUNI, achieves state-of-the-art performance on five widely used monocular 3D detection benchmarks, including Rope3D and DAIR-V2X-I for the infrastructure side, KITTI and Waymo for the vehicle side, and nuScenes for the cross-dataset evaluation.
Accurate 3D detection [14, 21, 25] is crucial for autonomous driving. Although LIDAR sensors [8, 18, 47, 51] provide high precision, camera sensors [7, 39, 52] are cost-effective and have a wider range of perception. Typically, autonomous driving systems employ frontal-view cameras mounted on the vehicle for 3D detection. As intelligent transportation continues to advance, there is increasing interest in using infrastructure-side cameras for 3D detection [13, 49, 53].
Due to the different installations and focal lengths between the vehicle and infrastructure-side cameras, researchers usually design algorithms to solve these two problems separately, which adds an additional application limitation. Fig. 1 illustrates the imaging process and the corresponding visual feature under different installations and focal lengths. With respect to the different installations, most vehicle-side cameras are installed on the top of the vehicle with a near-zero pitch angle, leading to a prior assumption that the optical axis is parallel to the ground. Most vehicle-side methods [4, 19, 27] are designed based on this assumption. In contrast, the infrastructure-side cameras, which are mounted on poles, typically have large pitch angles, rendering most of the existing vehicle-side
Image captured by camera 1
Image captured by camera 2
Image captured by camera 3
Image captured by camera 4
Figure 1: Schematic diagram of the camera imaging process and visual feature under different focal lengths and pitch angles. We assume that there is a scene where four cameras in different positions capture the same car (the white car highlighted in the four corresponding images). Camera 1,2,3 are infrastructure-side cameras, and camera 4 is the vehicle-side camera. The pitch angles of camera 1 and camera 2 are similar. Camera 2 is closer to the vehicle, and the focal length is relatively small. Due to the substantial variation in focal length between the two cameras, although the distance between the two cameras and the white car is significantly different, the visual features of the vehicle in the two images are nearly identical, which creates ambiguity in the depth estimation. The focal lengths of camera 2 and camera 3 are comparable. However, due to the different pitch angles of the two cameras, the same car at a similar depth appears with different visual features, resulting in increased difficulty for accurate depth regression. Camera 4 is on the vehicle side with a near-zero pitch angle, leading to a prior assumption that the optical axis is parallel to the ground which is not satisfied on the infrastructure side.
methods unsuitable for direct application, and the diversity of pitch angles also further increases the difficulty of object detection. According to the various focal lengths, the type of vehicle-side cameras are relatively uniform, and the focal lengths are thus similar, while the infrastructure-side cameras have a wide range of focal lengths. This results in a new challenge of 1-to-N ambiguity for depth estimation in monocular 3D detection, which has already been an ill-posed problem.
In this work, to address the above problems, we developed the obstacle projection models of the vehicle and infrastructure sides, and found that the former can actually be regarded as a special case of the latter, where the pitch angle is approximately 0 and the focal length is fixed. Consequently, we propose a unified optimization target: normalized depth, which is independent of focal length and pitch angle, so that the prediction of the obstacle depth on the two sides is no longer disturbed by the diversity of focal length and pitch angle. Furthermore, to enhance the performance of 3D detection, we draw inspiration from methods like AutoShape [26] and DID-M3D [37], which introduce new depth clues. For instance, AutoShape uses dense key points on the vehicle surface to establish geometric constraints, while DID-M3D employs depth maps to enable the network to predict object surface depth. They both add spatially correlated depth clues to the model in different ways with auxiliary data. We suggest that incorporating rich depth clues has a greater impact on depth estimation accuracy. Thus, we use the geometric relationship to create a 3D normalized cube depth for the obstacle, allowing the model to predict the normalized depth of the obstacle's 3D bounding box without introducing any extra data. By predicting the bias depth from each point on the 3D bounding box to the obstacle center, we can obtain the final depth conveniently. The approach of predicting obstacle 3D normalized cube depth maximizes depth clues and improves detection performance without additional data.
In summary, incorporating the above techniques, our method MonoUNI first achieves state-of-the-art (SOTA) performance on both vehicle and infrastructure-side monocular 3D detection. The main contributions of this work are as follows: 1) We proposed a unified optimization target, namely normalized depth, to address the differences between vehicle and infrastructure-side 3D detection caused by the diversity of pitch angle and focal length. This optimization target can be considered as
the standard solution for future monocular 3D detection of both sides. 2) We posit that rich depth clues are crucial in 3D detection, and thus propose the use of a 3D normalized cube depth to facilitate the learning of depth information. 3) Without using any additional information, MonoUNI ranks 1st in both the Rope3D [53] and DAIR-V2X-I [55] infrastructure-side benchmarks. Moreover, we apply our method to the vehicle-side benchmarks, such as KITTI [10] and Waymo [42], and also achieve competitive results. Cross-dataset evaluation of the KITTI val model on the nuScenes [3] val set demonstrates the generalizability of our method. The code is available at https://github.com/TrafficX/MonoUNI.
Vehicle-side Monocular 3D Object Detection. Vehicle-side monocular 3D detection refers to the process of analyzing a single image captured by a camera mounted on a vehicle to predict the 3D locations, dimensions, and orientations of the interest obstacles. These methods can be mainly categorized into two groups based on whether additional data is utilized. The first type of method uses additional data, such as depth maps [9, 28, 38], CAD models [5, 26, 35, 34] or LIDAR [6, 39] to enhance the detection accuracy. Pseudo-LIDAR methods [29, 46] utilize the depth map predicted by the additional network to assist the monocular image to generate a pseudo point cloud and then adopt existing LIDAR-based 3D object detection pipeline. DID-M3D [37] uses a dense depth map to generate visual depth for powerful data augmentation. CaDDN [39] uses LIDAR points to supervise additional monocular network estimates dense depth map and converts the feature to BEV perspective for prediction. AutoShape [26] utilizes CAD models to generate dense key points to alleviate the sparse constraints. NeurOCS [32] attempts to introduce the Neural Radiance Field (NeRF) [31] to solve the problem of lack of supervision information. Mix-Teaching [50] utilizes unlabeled data to achieve semi-supervised learning. Although utilizing additional information allows these methods to achieve improved performance, it inevitably leads to increased labeling and computational costs. Moreover, obtaining such data is challenging in various scenarios, especially when it comes to the infrastructure side.
The second kind of method only uses a single image as input without any extra information. Some methods [23, 27, 45, 56] use geometric projection assumptions to improve the accuracy of 3D detection. GUPNet [27] uses the 2D and 3D heights of the object to construct similar triangles to assist in regression depth. MonoFlex [56] and MonoDDE [23] further extend this similar triangle relationship using the position of key corner points to assist regression depth. PGD [45] constructs geometric relation graphs across predicted objects and uses the graph to facilitate depth estimation. Due to the installed pitch angle of infrastructure-side cameras, these geometric projections cannot be applied to infrastructure-side monocular 3D detection. Other methods [25, 30, 44] directly predict the dimensions, orientations, and locations of obstacles without the aid of geometric projections. However, these methods are designed for the vehicle side, ignoring the pitch angle diversity and the ambiguity introduced by the focal length gap between different cameras on the infrastructure side.
Infrastructure-side Monocular 3D Object Detection. Compared to vehicle-side monocular 3D object detection, which is typically limited to short-range perception, infrastructure-side 3D detection can overcome the problem of frequent occlusion in vehicle-based detection by increasing the sensor installation height, thereby providing long-range perception capabilities. Recently, DAIR-V2X-I [55] and Rope3D [53] are proposed to promote the development of 3D perception in infrastructure-side scenarios. Nonetheless, the challenges mentioned above, including the focal length difference and the variable pitch angle of the cameras, make it arduous to transfer the vehicle-side 3D detection method to the infrastructure-side setting, which results in relatively sluggish progress in infrastructure-side monocular 3D detection. BEVHeight [49] mitigates the issues arising from variations in camera pose parameters by directly predicting object height instead of object depth and achieves competitive performance in Rope3D and DAIR-V2X-I.
Monocular 3D object detection extracts features from a single RGB image, predicting the category and 3D bounding box for each object in the image. The 3D bounding box can be further divided
Figure 2: Overview. The left side of the picture describes the network architecture of MonoUNI, which uses a 2D detector to obtain object features, and then adopts different heads to estimate the cube depth, bias depth, and corresponding uncertainty, as well as other 3D properties. The right side depicts how we use the 3D label to generate the normalized cube depth and bias depth.
into 3D center location  $(x, y, z)$ , dimension  $(h, w, l)$  and orientation (yaw angle)  $\theta$ . Among these properties, dimension, and orientation are easily learned by the network due to their strong correlation with visual features [15], but the 3D location is challenging because depth estimation is ill-posed.
The main idea of MonoUNI is to unify 3D detection targets for both vehicle and infrastructure sides, and further achieve accurate 3D location. The overall framework is depicted in Fig. 2. We use CenterNet [57] as the base model to generate discriminative representation, with DLA34 [54] serving as the backbone for feature extraction from images. Several network heads are established in MonoUNI to predict object properties, such as categorical heatmap, 2D bounding box, 3D offset, dimension, orientation, 3D normalized cube depth, bias depth, and depth uncertainty items [16, 27]. Depth uncertainty is widely used in 3D detection, which can enhance the loss's robustness against noisy inputs. The loss function is roughly similar to DID-M3D [37]. In the inference process, by leveraging the predicted 3D normalized cube depth and bias depth, and utilizing the known pitch angle and focal length information, our method can obtain the final obstacle depth.
3.2 Unified Optimization Target
Problem Analysis. In order to construct a unified optimization target for the vehicle and infrastructure sides, we analyze the significant difference in monocular 3D detection between them. First, the camera on the infrastructure side is typically installed at an elevated position with a specific pitch angle (the angle between the optical axis of the camera and the ground) to capture a wider view and detect more potential obstacles, which invalidates the commonly used assumption on the vehicle side that the optical axis of the camera is parallel to the ground. Not only that, due to different scenarios and installation methods, the pitch angle of the installed camera is varied. For instance, in the Rope3D dataset, the pitch angle of the camera ranges from 5 to 20 degrees. Second, to cope with different situations, the internal parameters of different infrastructure-side cameras usually have a relatively large difference. For example, the camera focal length of Rope3D ranges from 2100 to 2800 pixels, while the camera focal lengths of the KITTI dataset are all between 715 pixels and 721 pixels. The huge gap in focal length can lead to an ambiguity that obstacles of similar visual features in two images captured by cameras with different focal lengths can have different depths. To explore the influence of focal length diversity, we conducted experiments on Rope3D using both the popular single-stage and two-stage monocular object detection methods. Since the focal length of most images is centered around 2100 and 2700 pixels, we further split the dataset into train_2100, train_2700, val_2100, and val_2700. The experimental results are shown in Table 1.
As a consequence of the aforementioned ambiguity, utilizing two train sets for mixed training will decrease the accuracy of both validation sets when compared to training the network solely with images in a single focal length range. In particular, the average precision (AP) of GUPNet experiences a significant reduction of  $70\%$  on the val_2700. This decrease in accuracy can be attributed to the dominance of images with a focal length of around 2100 pixels during the training process.
Normalized Depth. In this subsection, we first build the simple projection models of the vehicle (Fig. 3 (a)) and infrastructure sides (Fig. 3 (b)). Taking the infrastructure-side projection model as an example,  $O$  in the figure is the optical center of the camera, the ray  $OZ$  denotes the optical axis (z-axis), point  $C$  represents the center point of the obstacle,  $P$  represents the intersection point
Table 1: Analysis for different focal lengths on Rope3D dataset with new train/val division.
Method	Train_set	AP3D(IOU=0.5|R40)
val_2100	val_2700	val_all
GUPNet [27]	train_2100	13.20	0.03	7.42
GUPNet [27]	train_2700	0.17	21.65	3.07
GUPNet [27]	train_all	10.82	5.85	9.38
SMOKE [25]	train_2100	9.77	0.13	6.19
SMOKE [25]	train_2700	0.04	23.20	3.64
SMOKE [25]	train_all	6.04	18.01	8.48
(a) Vehicle-side projection model
(b) Infrastructure-side projection model
Figure 3: Simple projection models of the vehicle and infrastructure-side.
between the vertical line from the center point  $C$  to the ground and the ground plane,  $H$  denotes the 3D distance of  $PC$ , and  $h$  is the pixel distance from the center point  $C$  to the ground in the imaging plane,  $\theta$  is the pitch angle of the camera,  $z$  denotes the depth of center point  $C$ ,  $f$  represents the focal length and  $\delta$  is the included angle between the line connecting the point  $P$  to the optical center  $O$  and  $OZ$ . Extend a line from the obstacle's center point  $C$  along the camera's imaging plane direction, intersecting line  $OP$  at point  $P'$ . The distance  $CP'$  is denoted as  $H'$ . According to the parallel relationship, the following can be easily deduced:
$$
\frac {H ^ {\prime}}{h} = \frac {z}{f} \tag {1}
$$
After a simple geometric calculation, the following equation can be derived<sup>3</sup>:
$$
H ^ {\prime} = H * (\cos \theta - \sin \theta * \tan \delta) \tag {2}
$$
Substitute it into equation (1) and we get:
$$
z = \frac {H}{h} * (\cos \theta - \sin \theta * \tan \delta) * f \tag {3}
$$
Among them,  $H$  and  $h$  are easy to learn based on visual features, while  $\delta$ ,  $\theta$ , and  $f$  are challenging to directly learn from the image. Therefore, learning  $z$  directly requires the model to have a strong ability to infer focal length and angle information. However, from the above analysis, it is difficult or even ambiguous. Typically, the focal length  $f$  can be obtained as prior knowledge, the pitch angle  $\theta$  can be calculated as  $\theta = \arctan(\gamma/\beta)$  from the ground equation  $\alpha x + \beta y + \gamma z + d = 0$ , and  $\delta$  can be calculated through simple calculation  $\delta = \arctan((v_p - c_y)/f)$ , where  $v_p$  is the y-axis pixel coordinates of Point  $P$  and  $c_y$  is the principal point in y-axis. In particular,  $v_p$  can be simply and roughly replaced by  $v_c$  which may introduce an average relative error of  $2.9\%$  (statistics based on the
See the Supplementary Material for more proof details.
Figure 4: The generation process of the 3D cube depth.
Rope3D dataset), or be predicted by the model. Therefore, a simple transformation of the formula (3) yields:
$$
N o r m a l i z e d \_ d e p t h = \frac {z}{(\cos \theta - \sin \theta * \tan \delta) * f} \tag {4}
$$
Formula (4) represents the new unified optimization target: normalized depth, which makes depth prediction independent of pitch angle and focal length, simplifying the task difficulty. When  $\theta$  is equal to 0, the normalized depth degenerates into a situation where only the focal length is used for normalization  $(\frac{z}{f})$ , that is, the vehicle-side projection model. In fact, simplifying the normalized depth with either the focal length  $(\frac{z}{f})$  or pitch angle  $(\frac{z}{\cos\theta - \sin\theta*\tan\delta})$  can also partially reduces the learning difficulty and improve detection accuracy, which we will analyze in detail in the ablation experiments in section 4.4. During inference, the model outputs normalized depth, and we multiply it by the denominator in the formula (4) to get the final instance depth.
3.3 3D Normalized Cube Depth
To enrich depth clues and further improve detection performance, we utilize original labels to generate 3D cube depth for each obstacle. Based on the analysis in section 3.2, our 3D cube depth need to be normalized and become 3D normalized cube depth finally. The generation process of the 3D cube depth is shown in Fig. 4. Firstly, we use the location, orientation, and dimension information from the label to obtain the positions of the 8 corner points in the 3D bounding box in the camera coordinate system. These points are then marked in a specific sequence (1-8). Secondly, we calculate the plane equation  $\alpha_{i}x + \beta_{i}y + \gamma_{i}z + d_{i} = 0$  for each surface of the obstacle. To determine the equation, we select any 3 of the 4 vertices of the surface since a plane can be uniquely defined by 3 non-collinear points. Thirdly, we generate the 3D cube depth for the obstacle and project it onto the imaging plane. Specifically, each pixel  $(u,v)$  in the 2D bounding box is traversed to determine whether it belongs to any surface in the 2D image. Then, utilizing the internal parameters of the camera, the intersection point  $(x,y,z)$  of the pixel's ray with the corresponding plane can be computed and the depth  $z$  is thus achieved. The mathematical expression for this computation is as follows:
$$
\left\{ \begin{array}{l} \alpha_ {i} x + \beta_ {i} y + \gamma_ {i} z + d _ {i} = 0, f o r i = 1, \dots , 6 \\ f _ {x} \frac {x}{z} + c _ {x} = u \\ f _ {y} \frac {\tilde {y}}{z} + c _ {y} = v \end{array} \right. \tag {5}
$$
where  $f_{x}$  and  $f_{y}$  are the focal length of the camera, and  $c_{x}$  and  $c_{y}$  are the principal points.  $i$  denotes the index of the surface on the 3D bounding box. The depth  $z$  can be calculated as:
$$
z = \frac {- d _ {i}}{\frac {\alpha_ {i} (u - c _ {x})}{f _ {x}} + \frac {\beta_ {i} (v - c _ {y})}{f _ {y}} + \gamma_ {i}} \tag {6}
$$
In cases where a pixel belongs to multiple faces in the 2D image, we select the smallest depth value as the final depth. This is because, during the camera imaging process, points with larger depth values are obstructed by ones closer to the camera. The same strategy is also applied when a pixel simultaneously belongs to multiple obstacles. Once the 3D cube depth is obtained, we apply Equation 3 to each depth value in the cube, resulting in a 3D normalized cube depth that can be used for
Table 2: Monocular 3D detection performance of Car category on Rope3D val, DAIR-V2X-I val and KITTI test sets. We highlight the best results in bold and the second ones in underlined. For the extra data, the first column means whether using depth maps as additional input on the infrastructure side and the second column means on the vehicle side. '-' means that no official results or reasonable reproduction results in the new dataset.
Method	Extra Data	Rope3D	DAIR	KITTI
AP3D	Rscore	Easy	Mod.	Hard	Easy	Mod.	Hard
M3D-RPN [1]	Depth | None	67.17	73.14	-	-	-	14.76	9.71	7.42
MonoDLE [30]	Depth | None	77.50	80.84	-	-	-	7.23	12.26	10.29
MonoFlex [56]	Depth | None	59.78	66.66	-	-	-	19.94	13.89	12.07
DID-M3D [37]	None | Depth	-	-	-	-	-	24.40	16.29	13.75
CMKD [11]	None | Depth	-	-	-	-	-	25.09	16.99	15.30
LPCG+MonoFlex [36]	None | Depth	-	-	-	-	-	25.56	17.80	15.38
MonoEF [58]	None | None	-	-	-	-	-	21.29	13.87	11.71
DEVIANT [20]	None | None	-	-	-	-	-	21.88	14.46	11.89
MonoCon [48]	None | None	-	-	-	-	-	22.50	16.46	13.95
MonoATT [60]	None | None	-	-	-	-	-	24.72	17.37	15.00
MoGDE [59]	None | None	-	-	-	-	-	27.07	17.88	15.66
Kinematic3D [2]	None | None	50.57	58.86	-	-	-	19.07	12.72	9.17
SMOKE [25]	None | None	72.13	76.26	66.03	62.24	60.71	14.03	9.76	7.84
GUPNet [27]	None | None	66.52	70.14	62.22	55.94	55.90	22.26	15.02	13.12
Imvoxelnet [40]	None | None	-	-	44.78	37.58	37.55	17.15	10.97	9.15
BEVFormer [24]	None | None	50.62	58.78	61.37	50.73	50.73	-	-	-
BEVDepth [22]	None | None	69.63	74.70	75.50	63.58	63.67	-	-	-
BEVHeight [49]	None | None	74.60	78.72	77.78	65.77	65.85	-	-	-
MonoUNI(Ours)	None | None	92.45	92.63	90.92	87.24	87.20	24.75	16.73	13.49
supervision. In order to get the center point depth, we also supervise the bias depth from each point on the cube depth to the center point depth. The utilization of cube depth can maximize the enrichment of depth clues based on existing labels without additional information such as depth maps, CAD, or LIDAR points. Compared with only supervising the depth of the center point and corner points, the 3D cube depth is a sufficient way to utilize the depth information.
4.1 Dataset and Evaluation Metrics