MonoUNI: A Unified Vehicle and Infrastructure-side Monocular 3D Object Detection Network with Sufficient Depth Clues
Authors: Jinrang Jia
Zhenjia Li
Yifeng Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D detection
deep learning
autonomous driving
TL;DR: A new unified method of monocular 3D detection for the vehicle and infrastructure-side, based on the sufficient depth clues

Abstract: Monocular 3D detection of vehicle and infrastructure sides are two important topics in autonomous driving. Due to diverse sensor installations and focal lengths, researchers are faced with the challenge of constructing algorithms for the two topics based on different prior knowledge. In this paper, by taking into account the diversity of pitch angles and focal lengths, we propose a unified optimization target named normalized depth, which realizes the unification of 3D detection problems for the two sides. Furthermore, to enhance the accuracy of monocular 3D detection, 3D normalized cube depth of obstacle is developed to promote the learning of depth information. We posit that the richness of depth clues is a pivotal factor impacting the detection performance on both the vehicle and infrastructure sides. A richer set of depth clues facilitates the model to learn better spatial knowledge, and the 3D normalized cube depth offers sufficient depth clues. Extensive experiments demonstrate the effectiveness of our approach. Without introducing any extra information, our method, named MonoUNI, achieves state-of-the-art performance on five widely used monocular 3D detection benchmarks, including Rope3D and DAIR-V2X-I for the infrastructure side, KITTI and Waymo for the vehicle side, and nuScenes for the cross-dataset evaluation.

MonoUNI: A Unified Vehicle and Infrastructure-side Monocular 3D Object Detection Network with Sufficient Depth Clues
Jinrang Jia" Baidu Inc.
Zhenjia Li Baidu Inc.
Yifeng Shi! Baidu Inc.
Beijing, China jiajinrangobaidu.co com
Beijing, China lizhenj ia@baidu com
Beijing, China
shiylfenggbaidu.com
Abstract
Monocular 3D detection of vehicle and infrastructure sides are two important topics in autonomous driving. Due to diverse sensor installations and focal lengths, researchers are faced with the challenge topics based on different prior knowledge. In this paper, by taking into account the diversity of pitch angles and focal lengths, we propose a unified optimization target named normalized depth, which realizes the unification of 3D detection problems for the two sides. Furthermore, to enhance the accuracy of monocular 3D detection, 3D normalized cube depth of obstacle is developed to promote the learning of depth information. We posit that the richness of depth clues pivotal factor impacting the detection performance richer set of depth clues facilitates the model to learn better spatial knowledge, and the 3D normalized cube depth offers sufficient depth clues. Extensive experiments demonstrate the effectiveness of our approach. Without introducing any extra information, our method, named MonoUNI, achieves state-of-the-arl performance on five widely used monocular 3D detection benchmarks, including Rope3D and DAIR V2X-I for the infrastructure side, KITTI and Waymo for the vehicle side, and nuScenes for the cross-dataset evaluation.
constructing algorithms for the two
both the vehicle and infrastructure sides. A
Introduction
Accurate 3D detection 14 21 25 is crucial for autonomous driving. Although LIDAR sensors are cost-effective and have a wider range of perception. Typically, autonomous driving systems employ frontal-view cameras mounted on the vehicle for 3D detection. As intelligent transportation continues to advance, there increasing interest in using infrastructure-side cameras for 3D detection
88 51 provide high precision, camera sensors 7
39
52
[3 49 53)
Due to the different installations and focal lengths between vehicle and infrastructure-side cameras, researchers usually design algorithms solve these two problems separately, which adds an additional application limitation. Fig. [ illlsstraee the imaging process and the corresponding visual feature under different installations and focal lengths. With respect to the different installations, most vehiele-side cameras are installed on the top the vehicle with a near-zer pitch angle, leading to prior assumption that the optical axis is parallel to the ground. Most vehicle-side methods [ [9] 27 are designed based on this assumption. In contrast, the infrastructure-sinn cameras, which mounted on poles, typically have large pitch angles, rendering most of the existing vehicle-side
Eqqual contribution. Cooreespondigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image cuptured by camera
Image cuptured by cumera 2
Image cuptured by camera
Image cuptured by camera 4
Camera
0g
Planey
0, 0, 00
1,2 12
fa fechunturr
2, 2 18
Plane
Camara
Camera
Planez
Plana
Figure 1: Schematic diagram of the camera imaging process and visual feature under different focal lengths and pitch angles. We assume that there is a scene where four cameras in different positions capture the same car (the white car highlighted in the four corresponding images). Camera 1.2,3 are infrastructure-side cameras, and camera 4 is the vehicle-side camera. The pitch angles of camera and camera 2 are similar. Camera 2 is closer to the vehicle, and the focal length is relatively small. Due the substantial variation in focal length between the cameras, although the distance between the two cameras and the white car is significantly different, the visual features of the vehicle in the two images are nearly identical, which creates ambiguity in the depth estimation. The focal lengths of camera 2 and camera 3 are comparable. However, due to the different pitch angles of the two cameras, the same car a similar depth appears with different visual features, resulting in increased difficulty for accurate depth regression. Camera 4 is on the vehicle side with near-zero pitch angle, leading to a prior assumption that the optical axis is parallel to the ground which is not satisfied on the infrastructure side.
methods unsuitable for direct application, and the diversity of pitch angles also further increases the difficulty of object detection. According the various focal lengths, the type of vehicle-side cameras are relatively uniform, and the focal lengths are thus similar, while the infrastructure-stt have a wide range of focal lengths. This results in a new challenge of 1-to-N ambiguity for depth estimation in monocular 3D detection, which has already been an ill-posed problem.
cameras
In this work, address the above problems, we developed the obstacle projection models of the vehicle and infrastructure sides, and found that the former can actually be regarded as a special case of the latter, where the pitch angle approximately and the focal length is fixed. Consequently, we propose unified optimization target: normalized depth, which is independent of focal length and pitch angle, so that the prediction of the obstacle depth on the two sides is no longer disturbed by the diversity of focal length and pitch angle. Furthermore, to enhance the performance of 3D detection, we draw inspiration from methods like AutoShape (26 and DID-M3D 37,, which introduce new depth clues. For instance, AutoShape uses dense key points on the vehicle surface to establish geometric constraints, while DID-M3D employs depth maps to enable the network to predict object surface depth. They both add spatially correlated depth clues to the model in different ways with auxiliary data. We suggest that incorporating rich depth clues has greater impact on depth estimation accuracy. Thus, we use the geometric relationship to create 3D normalized cube depth for the obstacle, allowing the model to predict the normalized depth of the obstacle's 's 3D bounding box without introducing any extra data. By predicting the bias depth from each point the 3D bounding box to the obstacle center, we can obtain the final depth conveniently. The approach of predicting obstacle 3D normalized cube depth maximizes depth clues and improves detection performance without additional data.
In summary, incorporating the above techniques, our method MonoUNI first achieves state-of-the-an (SOTA) performance on both vehicle and infrastructure-sidee monocular 3D detection. The main contributions of this work are as follows: 1) We proposed a unified optimization target, namely normalized depth, to address the differences between vehicle and infrastructure--sdde 3D detection caused by the diversity of pitch angle and focal length. This optimization target can be considered as

---

the standard solution for future monocular 3D detection of both sides. 2) We posit that rich depth clues are crucial in 3D detection, and thus propose the use of 3D normalized cube depth to facilitate the learning of depth information. 3) Without using any additional information, MonoUNI ranks 1st benchmarks. Moreover, we apply our method to the vehicle-side benchmarks, such as KITTI [TO] and Waymo 42 and also achieve competitive results. Cross-dataset evaluation of the KITTI val model on the nuScenes [3] val set demonstrates the generalizability of our method. The code is available at https://gtthub.com/ crafffffic-----
in both the Rope3l) [53] and DAIR-V2X- (55) infrastructure-Sfi
X/MonoUNI.
2 Related Work
Vehicle-side Monocular 3D Object Detection. Vehicle-side monocular 3D detection refers to the process of analyzing a single image captured by camera mounted on vehicle predict the 3D locations, dimensions, and orientations of the interest obstacles. These methods can be mainly categorized into two groups based on whether additional data is utilized. The first type of method
uses additional data, such as depth maps 9 28 38 enhance the detection accuracy. Pseudo-LIDAR methods 29 46 utilize the depth map predicted by the additional network to assist the monocular image to generate a pseudo point cloud and then adopt existing LIDAR- based 3D object detection pipeline. DID-M3D [37 uses dense depth map to generate visual depth for powerful data augmentation. CaDDN 33 uses LIDAR points to supervise additional monocular network estimates dense depth map and converts the feature to BEV perspective for prediction. AutoShape [26] utilizes CAD models to generate dense key points to alleviate the sparse constraints. NeurOCS 32 attempts introduce the Neural Radiance Field (NeRF) 31] to
CAD models (5) 26 35 34 or LIDAR [6] 39
solve the problem of lack of supervision information. Mix- Mix-Teaching (50) utilizes unlabeled data to achieve -upi-sueee learning. Although utilizing additional information allows these methods to achieve improved performance, it inevitably leads to increased labeling and computational costs. Moreover, obtaining such data is challenging in various scenarios, especially when it comes to the infrastructure side.
The second kind of method only uses single image as input without any extra information. Some methods [23] 27 45 (56] use geometric projection assumptions to improve the accuracy of 3D detection. GUPNet (27] uses the 2D and 3D heights of the object construct similar triangles to assist in regression depth. MonoFlex [56] and MonoDDE (23 further extend this similar triangle
relationship using the position of key corner points to assist regression depth. PGD 445 constructs geometric relation graphs across predicted objects and uses the graph to facilitate depth estimation. Due to the installed pitch angle of infrastructure-side cameras, these geometric projections cannot be applied to infrastructure-side monocular 3D detection. Other methods 25 30 the dimensions, orientations, and locations of obstacles without the aid of geometric projections. However, these methods designed for the vehicle side, ignoring the pitch angle diversity and the ambiguity introduced by the focal length gap between different cameras on the infrastructure side.
directly predict
Infrastructure-side Monocular 3D Object Detection. Compared to vehicle-side monocular 3D object detection, which is typically limited to short-range perception, infrastructure-side 3D detection can overcome the problem of frequent occlusion in vehicle-based detection by increasing the sensor installation height, thereby providing long-range perception capabilities. Recently, DAIR-V2X. (55) and Rope3D [53] are proposed to promote the development of 3D perception in infrastructure-Sae scenarios. Nonetheless, the challenges mentioned above, including the focal length difference and the variable pitch angle of the cameras, make it arduous to transfer the vehicle-side 3D detection method to the infrastructure-siide setting, which results in relatively sluggish progress in infrastructure-side monocular 3D detection. BEVHeight [49 mitigates the issues arising from variations in camera pose parameters by directly predicting object height instead of object depth and achieves competitive
performance in Rope3D and DAIR-V2X.
3 Method
3.1 Overview
Monocular 3D object detection extracts features from single RGB image, predicting the category and 3D bounding box for each object in the image. The 3D bounding box can be further divided
3

---

MonoUNI
Normalized Cube Depth Generation
Object label
Bias Depth
Vebicle-side
Bits depth
Calculste the biss
Location
Depth
Dimension
2D Detector
Cube Depth uncersirry
Orientation
Norm cube depth
Noematae
Lyaas
Cute depth
hor
Deyth
Object Festure
Camera parameters
Infrastmucture-sii side
properties
Pitch angle
length
Figure 2: Overview. The left side of the picture describes the network architecture of MonoUNI, which uses a 2D detector to obtain object features, and then adopts different heads to estimate the cube depth, bias depth, and corresponding uncertainty, as well as other 3D properties. The right side depicts how we use the 3D label to generate the normalized cube depth and bias depth.
into 3D center location (xr, y, z), dimension (h, w, 1) and orientation (yaw angle) Among these properties, dimension, and orientation are easily learned by the network due to their strong correlation with visual features [5]. but the 3D location is challenging because depth estimation is ill ill-posed.
The main idea of MonoUNI is to unify 3D detection targets for both vehicle and infrastructure sides. and further achieve accurate 3D location. The overall framework depicted in Fig. 2 We use CenterNet [57] as the base model to generate discriminative representation, with DLA34 [54] serving as the backbone for feature extraction from images. Several network heads are established in MonoUNI to predict object properties, such categorical heatmap, 2D bounding box. 3D offset, dimension, orientation, 3D normalized cube depth, bias depth, and depth uncertainty items [T6] 27].
Depth uncertainty is widely used in 3D detection, which can enhance the loss's ' robustness against noisy inputs. The loss function roughly similar to DID-M3D (37. the inference process, by leveraging the predicted 3D normalized cube depth and bias depth, and utilizing the known pitch angle and focal length information, our method can obtain the final obstacle depth.
3.2 Unified Optimization Target
Problem Analysis. In order to construct unified optimization target for the vehicle and infras- tructure sides, we analyze the significant difference in monocular 3D detection between them. First, the camera the infrastructure side is typically installed at an elevated position with specific pitch angle (the angle between the optical axis of the camera and the ground) to capture a wider view and detect more potential obstacles, which invalidates the commonly used assumption vehicle side that the optical axis of the camera is parallel to the ground. Not only that, due to different scenarios and installation methods, the pitch angle of the installed camera is varied. For instance, in the Rope3D dataset, the pitch angle of the camera ranges from 5 to 20 degrees. Second, to cope with different situations, the internal parameters of different infrastructure-side cameras usually have a relatively large difference. For example, the camera focal length of Rope3D ranges from 2100 2800 pixels, while the camera focal lengths of the KITTI dataset are all between 715 pixels and 721 pixels. The huge gap in focal length can lead to an ambiguity that obstacles of similar visual features in two images captured by cameras with different focal lengths can have different depths. To explore the influence of focal length diversity, we conducted experiments on Rope3D using both the popular single-stage and two-stage monocular object detection methods. Since the focal length of most images is centered around 2100 and 2700 pixels, we further split the dataset into train _2000 train 2700, val 2100, and val 2700. The experimental results are shown in Table [
the
As consequence of the aforementioned ambiguity, utilizing two train sets for mixed training will decrease the accuracy of both validation sets when compared to training the network solely with images in a single focal length range. In particular, the average precision (AP) of GUPNet experiences significant reduction of 70% on the val 2700. This decrease in accuracy can be attributed the dominance of images with a focal length of around 2100 pixels during the training process.
Normalized Depth. this subsection, we first build the simple projection models of the vehicle (Fig. 3 (a)) and infrastructure sides (Fig. 3 (b)). Taking the infrastructurr-sside projection model as an example, O the figure the optical center of the camera, the ray OZ denotes the optical axis (z-axis), point C represents the center point of the obstacle, P represents the intersection point

---

Table 1: Analysis for different focal lengths on Rope3D dataset with new train/val division.
APad [IOU = Rao)
Method
Train set
val_ 2T00
val val_2700
val_al all
GUPNet 27
train 2100
13.20
0.03
7.42
GUPNet
27
train_2700 2700 train
0.17
21.65
3.07
GUPNet
27
10.82
5.85
9.38
SMOKE
25
train_2100
9.77
0.13
6.19
SMOKE
25
train 2700
0.04
23,20
3.64
SMOKE
25
train all
6.04
18.01
8.48
Imaging Plane
Imaging Plane
Z
Y
P
y+ +
+ By
H H
11
(a) Vehicle-side projection model
(h)
Infrastructur-sidde projection model
Figure 3: Simple projection models of the vehicle and infrastructure-side.
between the vertical line from the center point to the ground and the ground plane, denotes 3D distance of PC, and h is the pixel distance from the center point C to the ground in the imaging plane, the pitch angle of the camera, z denotes the depth of center point C, f represents the focal length and 5 is the included angle between the line connecting the point P to the optical center O and OZ. Extend a line from the obstacle's s center point C along the camera's 's imaging plane direction, intersecting line OP at point P'. The distance CP' is denoted H'. According to the parallel relationship, the following can be easily deduced:
H' h
After a simple geometric calculation, the following equation can be derived
H' = H (cos O - sin 0 tan 6)
(2)
Substitute it into equation 11 and we get:
H 2 h
(cos sin 0 tan d)
Among them, and are easy to learn based on visual features, while ó, 9, and f are challenging to directly learn from the image. Therefore, learning directly requires the model to have strong ability to infer focal length and angle information. However, from the above analysis, is difficult or ambiguous. Typically, the focal length can be calculated as 0 = arctan(7/3) from the ground equation orr + By + 72 be calculated through simple calculation 8 arctan cy)/f), f), where up is the y-axis pixel coordinates of Point and c, is the principal point in y-axis. In particular, tp can be simply and roughly replaced by Ue which may introduce an average relative error of 2.9% (statistics based on the
be obtained prior knowledge, the pitch angle
O, and
"See the Supplementary Material for proof details.

---

Original Label
Curner Points Generate
Planes Firting
Depth Filling
Cube Depth
Pima
Plansy
Location
Zarj
Orientation
Ze
Ze
X
Xonj
Dimension
Yao)
Oc
02
YY
Plane; 135
Plane;
2D Box
Figure The generation process of the 3D cube depth.
Rope3D dataset), or be predicted by the model. Therefore, a simple transformation of the formula (3) yields:
formaliizedddeppe (cos 0 sin 0 tan
Formula 4 represents the unified optimization target: normalized depth, which makes depth prediction independent of pitch angle and focal length, simplifying the task difficulty. When is equal 0, the normalized depth degenerates into situation where only the focal length is used for normalization (f), that is, the vehicle-side projection model. In fact, simplifying the normalized depth with either the focal length (j) or pitch angle (cce 8-n eetan&) can also partially reduces the learning difficulty and improve detection accuracy, which we will analyze in detail in the ablation experiments in section 4.4 During inference, the model outputs normalized depth, and we multiply by the denominator in the formula (4) to get the final instance depth.
3.3 3D Normalized Cube Depth
To enrich depth clues and further improve detection performance, we utilize original labels to generate 3D cube depth for each obstacle. Based the analysis in section [3.]] our 3D cube depth need to be normalized and become 3D normalized cube depth finally. The generation process the 3D cube depth is shown in Fig. 4 Firstly, we the location, orientation, and dimension information from the label to obtain the positions of the 8 corner points in the 3D bounding box in the camera coordinate system. These points equation o,a + Bay + N2 + d, = 0 for each surface of the obstacle. To determine the equation, we select any 3 of the vertices of the surface since a plane can be uniquely defined by non-collinear points. Thirdly, we generate the 3D cube depth for the obstacle and project it onto the imaging plane. Specifically, each pixel (u, v) in the 2D bounding box is traversed to determine whether belongs to any surface in the 2D image. Then, utilizing the internal parameters of the camera, the intersection point (x., y, 2) of the pixel's 's ray with the corresponding plane can be computed and the depth is thus achieved. The mathematical expression for this computation is as follows:
then marked in specific sequence (1-8). Secondly, we calculate the plane
a, a, +Byt n2 d: f. i 1.....
+ =u f +c=
where fa and fy are the focal length of the camera, and er and cy are the principal points. denotes the index of the surface on the 3D hounding box. The depth can be calculated as:
d
(6)
0:fu
744
In cases where a pixel belongs to multiple faces in the 2D image, we select the smallest depth value as the final depth. This because, during the camera imaging process, points with larger depth values are obstructed by ones closer the camera. The same strategy is also applied when pixel simultaneously belongs to multiple obstacles. Once the 3D cube depth obtained, we apply Equation 3 to each depth value in the cube, resulting in 3D normalized cube depth that can be used for

---

Table 2: Monocular 3D detection performance of Car category Rope3D val, DAIR-V2X- val and KITTI test sets. We highlight the best results in bold and the second ones in underlined. For the extra data, the first column means whether using depth maps additional input the infrastructure side and the second column means on the vehicle side. '-' means that no official results or reasonable reproduction results in the dataset.
Rope3D
DAIR
KITTI
Method
Extra Data
APaD
Racore
Easy
Mod.
Hard
Easy
Mod.
Hard
M3D-RPN
Depth I None
67.17
73.14
14.76
9.71
7.42
MonoDLE
30
Depth None
77.50
80.84
7.23
12.26
10.29
MonoFlex
56
Depth None
59.78
66.66
19.94
13.89
12.07
DID-M3D
None Depth
24.40
16.29
13.75
CMKD [1]
None Depth
25.09
16.99
15.30
LPCG+MMonolle 36
None Depth
25.56
17.80
15.38
MonoET 58
None None
21.29
13.87
11.71
DEVIANT 20
None None
21.88
14.46
11.89
MonoCon 48
None None
22.50
16.46
13.95
MоnоATT 60
None None
24.72
17.37
15,00
MoGDE 59
None None
27.07
17.88
15.66
Kinematic3D
None None 50.57
58.86
19.07
12.72
9.17
SMOKE 25
None None 72.13
76.26
66.03
62.24
24
60.71
14.03
9.76
7.84
GUPNet 27
None None
66.52
70.14
62.22
55.94
55.90
22.26
15.02
13.12
Imvoxelnet 40
None None
44.78
37.58
37.55
17.15
10.97
9.15
BEVFormer 24
None None 50.62
58.78
61.37
50.73
50.73
BEVDepth 22
None None 69.63
74.70
75.50
63.58
63.67
BEVHeight
None None 74.60
78.72
77.78
65.77
65.85
(OurS))(Ours)
None None 92.45
92.63
90.92
87.24
87.20
24.75
16.73
13.49
supervision. In order to get the center point depth, also supervise the bias depth from each point the cube depth to the center point depth. The utilization of cube depth can maximize the enrichment of depth clues based on existing labels without additional information such as depth maps, CAD, LIDAR points. Compared with only supervising the depth the center point and corner points, the 3D cube depth is sufficient way utilize the depth information.
Experiments
4.1 Dataset and Evaluation Metrics
Rope3D. Rope3D 53 is comprehensive real-world benchmark for infrastructure-side 3D de- tection, featuring over 50,000 images and more than 1.5 million 3D objects in diverse scenes. The benchmark comprises various settings, such as different cameras with ambiguous mounting positions, diverse camera specifications, as well as different environmental conditions. For the evaluation metrics, we follow the official settings using AP AL and Rope [53]. which is a combined metric of the 3D AP and other similarities, such as Average Ground Center Similarity (ACS). We follow the proposed homologous setting to utilize 70% of the images as training, and the remaining as validating. All images are randomly sampled.
DAIR-V2X-L. DAIR-V2X-I 55 is subset of DAIR- -2X,, which is large-scale multimodal cehicle-ifrrastruetuuee collaborative perception dataset. It contains 10,000 infrastructure-side images and 493,000 3D bounding boxes with 10 classes. We follow the official train/val division for evaluation, and use A APadIR1o as the main metric, with an IOU threshold of 0.5.
a widely used benchmark consisting of 7481 training images and 7518 testing images for vehicle-side monocular 3D detection. The dataset contains three classes: Car, Pedestrian, and Cyclist, each with three difficulty levels: Easy, Moderate, and Hard. Moderate difficulty considered the official rank level of the KITTI leaderboard. To ensure fair comparison with previous methods, results are submitted to an official server for evaluation on the test set. A AP3D)RA R4o used as the main metric with IOU thresholds of 0.7 for cars. and 0.5 for Pedestrians and Cyclists respectively.
KITTI. KITTI
7

---

Table 3: Monocular 3D detection performance Table 4: Monocular 3D detection performance of Big Vehicle category on Rope3D val set.
of Pedestrian and Cyclist category on KITTI test set.
Big Vehicle
Method
Al'n
Al'D
Nscore
Method
Pedestrian
Cyclist
M3D M3D-RPN
32.19
40.52
Easy
Mod.
Hard
Easy
Mod.
Hard
MonoDLE
30
44.71
52.83
DFR-Net [61
6.09
3.62
3.39
5.69
3.58
3.10
SMOKE
25
52.04
59.23
CaDDN 39
12.87
8.14
6.76
7.00
3.41
3.30
GUPNet
45.27
52.19
MonoCon [48
13.10
8,41
6.94
2.80
1.92
1.55
BEVFormer
24
34.58
45.16
GUPNet 27
14.95
9.76
8.41
5.58
3.21
2.66
BEVDepth
22
45.02
54.64
MonoDTR 12
15,33
10.18
5.05
3.19
BEVHeight
49
48.93
57.70
UNnoUNT
15.78
10.34
8.74
7.34
4.28
3.78
MonoUNI
76.30
79.20
Table 5: Ablation Study on different components of our overall framework on Rope3D and KITTI val set for Car category.
Normlized Depth
Rope3D
KITTI
Experiments
Cube
Focal
Pitch
AP5n
Racore
Easy
Mod.
Hard
(a)
79.37
81.38
21.71
14.93
12.10
83.42
84.91
22.35
15.29
12.34
(c)
81.55
83.53
21.68
14.85
12.12
86.69
87.99
22.19
14.94
12.45
(c)
87.07
88.16
24.66
17.07
14.06
92.45
92.63
24.51
17.18
14.01
assesses objects using a dual-tiered categorization: Level and Level The assessment is performed across three distance intervals: [0, 30), [30, 50), and [50, x)) meters. Waymo employs the APH3D percentage metric, which integrates heading data into the AP3.. as reference benchmark for evaluation.
Waymo. Waymo 42
nuScenes. nuScenes 3 comprises 28,130 training and 6,019 validation images captured from front camera. We use validation split for cross-dataset evaluation.
4.2 Implementation Details
Our proposed MonoUNI trained on Tesla V100 GPUs with batch size of 16 for 150 epochs. We
use Adam [17] as our optimizer with initial learning rate 1.25 x - 3. Images are all resized to the same size of 960 x 512 for infrastructure side and 1280 384 for vehicle side. Following (37], the ROI-Align size d x is set to 7 x 7. Inspired by [33,, we adopt the multi-bin strategy for heading angle and depth prediction our baseline. Random crop and expand along principal points are used to achieve more physical data augmentation.
4.3 Main Results
Results of Car Category. Since there was no previous method designed for supporting both the
vehicle and infrastructure sides at the same time, the results of some methods were reproduced by us
(33 and DAIR- V2X (55)) As shown in Table 2 our proposed MonoUNI achieves superior performance than previous methods on infrastructure-sie benchmarks, even those with extra data. Specifically, compared with BEVHeight which is the recent topl-ranked image-only method for infrastructure side, MonoUNI gains significant improvement of 17.85 %/1.991 % A PnD and Rscore on Rope3I) benchmark. According to DAIR-V'S V2X, our method achieves over 21 % improvement compared with BEVHeight on Moderate difficulty. For vehicle-side benchmark, although our method did rank first, reached competitive result on Easy difficulty. On the Moderate and Hard difficulties, MonoUNI is slightly inferior to methods such Мо (60) and MoGDE (59], probably because the gain the 3D cube depth is weakened in the case of far-distance severe truncation.
or other published papers (BEVHeight 449 Rope3D
Results of Other Categories. In Table we present the results of pedestrians and cyclists on the test set of KITTL MonoUNI outperforms all methods by large margin. This may be because the
8

---

Table 6: Analysis of UNI for different focal lengths on Rope3D with new train/val division.
Table 7: Cross-dataset evaluation of the KITTI val model on KITTI val and nuScenes val cars with depth MAE.
KITTI Val
nuScenes frontal Val
Method
(P's = 0.5| R40
0-20
20
40-oc
All
0-20
40
40-00
All
Method
Train
val 2100val
val all
M3D-RPP 11
0.36
1.33
2.73
1.26
0.94
3.06
10.36
2.67
MonoRCNN
41
0.46
1.27
2.59
1.14
0.94
2.84
8.65
2.39
MonoUN12100
25.78
21.30
24.47
GUPNet 27
0.45
1.10
1.85
0.89
0.82
1.70
6.20
1.45
MonoUNI 2700
5.77
23.42
9.25
DEVIANT 200
0.40
1.09
1.80
0.87
0.76
1.60
4.50
1.26
MonoUNI all
26.63
38.10
28.91
MonoUNI
0.38
0.92
1.79
0.865
0.72
1.79
4.98
1.43
Table 8: Monocular 3D detection performance of Vehicle category on Waymo val set.
AP3n
APHap
10Usp
Difficulty
Method
Extra
All
0-30
30-50
50-c0
All
0-30
30-50
50-00
CaDDN
39
LIDAR
5.03
15.54
1.47
0.10
4.99
14.43
1.45
0.10
PatchNet
28
in
48
Depth
0.39
1.67
0.13
0.03
0.39
1.63
0.12
0.03
PCT 43
Depth
0.89
3.18
0.27
0.07
0.88
3.15
0.27
0.07
0.7
Level_
M3D-RPN
39
None
0.35
1.12
0.18
0.02
0.34
1.10
0.18
0.02
GUPNet 27
20
None
2.28
6.15
0.81
0.03
2.27
6.11
0.80
0.03
DEVIANT 20
None
2.69
6.95
0.99
0.02
2.67
6.90
0.02
MonoUNI (Ours)
None
3.20
8.61
0.87
0.13
3.16
8.50
0.12
CaDDN 39
LIDAR
4.49
14.50
1.42
0.09
4.45
14.38
1.41
0.09
PatchNet 28
in
Depth
0.38
1.67
0.13
0.03
0.36
1.63
0.11
0.03
PCT 43
Depth
0.66
3.18
0.27
0.07
0.66
3.15
0.26
0.07
0.7
Level_2
M3D-RPN
in
39
None
0.35
1.12
0.18
0.02
0,33
1.10
0.17
0.02
GUPNet 27
20
None
2.14
6.13
0.78
0.02
2.12
6.08
0.77
0.02
DEVIANT 20
None
2.52
6.93
0.95
0.02
2.50
6.87
0.94
0.02
MonoUNT (Ours)
None
3.04
8.59
0.85
0.12
3,00
8.48
0.84
0.12
CaDDN 39
LIDAR
17.54
45.00
9.24
0.64
17.31
44.46
9.11
0.62
PatchNet 28
in
43
Depth
2.92
10.03
1.09
0.23
2.74
9.75
0.96
0.18
PCT 43
Depth
4.20
14.70
1.78
0.39
4.15
14.54
1.75
0.39
0.5
Level_
M3D-RPN
39
None
3.79
11.14
2.16
0.26
3.63
10.70
2.09
0.21
GUPNet 27
20
None
10.02
24.78
4.84
0.22
9.94
24.59
4.78
0.22
DEVIANT 20
None
10.98
26.85
5.13
0.18
10.89
26.64
0.18
MonoUNI (Ours)
None
10.98
26.63
4.04
0.57
10.73
26.30
3.98
0.55
CaDDN 39
LIDAR
16.51
44.87
8.99
0.58
16.28
44.33
8,86
0.55
PatchNet 28
in
43
Depth
2.42
10.01
1.07
0.22
2.28
9.73
0.97
0.16
PCT 43
Depth
4.03
14.67
1.74
0.36
4.15
14.51
1.71
0.35
0.5
Level 2
M3D-RPN
39
None
3.61
11.12
2.12
0.24
3.46
10.67
2.04
0.20
GUPNet 27
20
None
9.39
24.69
4.67
0.19
9.31
24.50
4.62
0.19
DEVIANT 20
None
10.29
26.75
4.95
0.16
10.20
26.54
4.90
0.16
MonoUNT (Ours)
None
10.38
26.57
3.95
0.53
10.24
26.24
3.89
0.51
introduction of depth clues promotes pedestrians and cyclists better learn spatial features. Table shows the results Big Vehicle on the Rope3D benchmark.
4.4 Ablation Study
Effectiveness of Different Components in MonoUNI Rope3D and KITTI val set for Car category. We evaluate the effectiveness of the normalized depth and the 3D cube depth through ablations two datasets. As shown in Table the normalized depth, particularly the focal length, has a significant improvement on the infrastructure side, but only a minor effect on the vehicle side. This is due to the fact that the focal length of the camera is similar across the KITTI data, and the pitch angle close to O. On the other hand, the cube depth improves results for both the vehicle and infrastructure sides, providing evidence its efficacy.
Analysis for different focal lengths. Table presents the performance MonoUNI at different focal lengths. Compared with the results in Table [ our method has obvious benefits in solving the problem of focal length diversity. the case of using all training data, our method can improve 19.53% and 20.43% AP respectively compared with GUPNet and SMOKE.

---

Rope3D
KITTI
D0
180
Figure 5: Qualitative visualization on the Rope3D and KITTI val set. The 3D green boxes are produced by MonoUNI and the red boxes are the ground truths.
Cross-dataset evaluation. nuScenes 3 frontal val images, using mean absolute error (MAR) of the depth [41]. MonoUNT is
the result of our KITTI val model on the KITTI val and
better than GUPNet (27) and achieves similar competitive performance to DEVIANT (20. This because DEVIANT is equivariant to the depth translations and is more robust to data distribution
changes.
Performance of cube depth on large dataset Waymo. In order to fully verify the effectiveness of cube depth, we verified MonoUNI Waymo, which is large-scale vehicle-side 3D detection dataset, as shown in Table
4.5 Qualitative Results
We visualize the detection results of the MonoUNI on the both vehicle and infrastructure sides in Fig. [J As can be seen, the MonoUNI can accurately estimate the 3D position of objects, even those labeled by the annotators due to severe occlusion in the first row of KITTI. However, for far-distance and severely truncated obstacles, our model suffers from missed detections.
5
Conclusion
In this paper, we propose new optimization target named normalized depth to unify monocular 3D detection for both vehicle and infrastructure sides, addressing the problem of focal length and pitch angle diversity. Furthermore, we introduce 3D Cube Depth as an additional supervision clue to improve the 3D detection performance. Experiments five datasets (Rope3D, DAIR-V2X-I, KITTI, Waymo and nuScenes) fully demonstrate the effectiveness our method.
6 Limitations and Future Work
Although the normalized depth unifies the optimization target the vehicle and infrastructure sides. separate training for the two sides is still necessary instead of direct hybrid training. In the future, aim to enable one model file to support 3D detection for both sides which would hold significant value for industrial applications, such the domain gap problem in the vehicle-infrastructure collaborative perception and fusion. addition, global depth clues such as depth similarity (i.e., contrastive learning) or the global topological relationship be further developed.
10

---

References
[1] Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019,
[2] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection in monocular vidco. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August
23-28, 2020, Proceedings, Part XXIII 16, pages 135-152, 2020.
[3] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11621-1 1631,
2020.
[4] Yingjie Cai, Buyu Li, Zeyu Jiao. Hongsheng Li, Xingyu Zeng, and Xiaogang Wang. Monocular 3d object detection with decoupled structured polygon estimation and height-guided depth estimation. 2021.
[5] Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Celine Teuliere, and Thierry Chatcau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
[6] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C (CVPR), pages 10379-10388, June 2021.
[7] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. In IEEE/CVF Conference on Computer Vixion and Pottern Recognition (CVPR), June 2020.
[8] Ziming Chen, Yifeng Shi, and Jinrang Jia. Transiff: An instance-level feature fusion framework for vehicle- infrastructure cooperative 3d detection with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 18205-18214 14 October 2023.
[9] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guides convolutions for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361.
IEEE, 2012.
[11] Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection, In FCCV, Lecture Notes in Computer Science. Springer, 2022.
[12] Kuan-Chih Huang. Tsung-Han Wu, Hung-Ting Su. and Winston H. Hsu. Monodtr: Monocular 3d object detection with depth-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4012-4024 June 2022.
[13] Jinrang Jia, Yifeng Shi, Yuli Qu, Rui Wang. Xing Xu, and Hai Zhang. Competition for roadside camera monocular 3D object detection. National Science Review, 05 2023. nwad121.
[14] Bo Ju. Wei Yang, Jinrang Jia, Xiaoqing Ye. Qu Chen, Xiao Tan, Hao Sun, Yifeng Shi, and Errui Ding. Danet: Dimension apart network for radar object detection. In Proceedings of the 2021 International Conference on Multimedia Retrieval, ICMR '21, page 533-539, New York, NY, USA, 2021. Association for Computing Machinery.
[15] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference
on Computer Vision and Pattern Recognition, pages 1725-1732 1732, 2014.
[16] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision?
Advances in neural information processing systems, 30, 2017.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412. 6980, 2014.
[18] Xianghao Kong. Wentao Jiang, Jinrang Jia, Yifeng Shi, Runsheng Xu, and Si Liu. Dusa: Decoupled unsupervised sim2real adaptation for vehicle-to-everythingg collaborative perception. In Proceedings of the 31st ACM International Conference on Multimedia, October 2023,
[19] Jason Ku, Alex D. Pon, and Steven L. Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). June 2019.
[20] Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth cquivariant network for monocular 3d object detection. In European Conference on Computer Vision,
pages 664 044-683.555 Springer. 2022.
[21] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijhom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June 2019.
[22] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zem- ing Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206. 10092, 2022.
[23] Zhuoling Li, Zhan Qu, Yang Zhou, Jianzhuang Liu, Haoqian Wang, and Lihui Jiang. Diversity matters:
Fully exploiting depth clues for reliable monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2791-2800, June 2022.
11

---

[24] Zhiqi Li. Wenhai Wang. Hongyang Li, Enze Xie. Chonghao Sima. Tong Lu. Yu Qiao, and Jifeng Dai. Leamning bird' seee representation from multi-camera images via spatiotemporal trans- formers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27.
Bevformer:
2022, Proceedings, Part IX, pages 1-18, 2022.
[25] Zechen Liu, Zizhang Wu, and Roland Toth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops, June 2020.
[26] Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang. Autoshape: Real-time shape-aware monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer
Vision (ICCV), pages 15641-15650, October 2021.
[27] Yan Lu, Xinzhu Ma, Lei Yang. Tianzhu Zhang. Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3111-3121, October 2021.
[28] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking pseudo-lidar representation. In Proceedings of the European Conference on Computer Vision (ECCV),
2020.
[29] Xinzhu Ma, Zhihui Wang. Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular
3d object detection via color-embedded 3d reconstruction for autonomous driving. In Proceedings of the IEEE/CVF International Conference
Computer Vision (ICCV). October 2019.
[30] Xinzhu Ma. Yinmin Zhang. Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4721 4721-4730, June 2021.
[31] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorth,, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[32] Zhixiang Min, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Enrique Dunn, and Manmohan Chandraker. Neurocs: Neural nocs supervision for monocular 3d object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21404-21414, June 2023.
[33] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Puttern
Recognition (CVPR), July 2017.
[34] Krishna Murthy, G.V V. Sai Krishna, Falak Chhaya, and K. Madhava Krishna. Reconstructing vehicles from a single image: Shape priors for road scene understanding. In 2017 IEEE International Conference
Robotics and Automation (ICRA), pages 724-731, 2017.
[35] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEER/CVF International Conference Computer
Vision (ICCV), pages 3142-3152, Octoher 2021.
[36] Liang Peng. Fei Liu, Zhengxu Yu, Senbo Yan, Dan Deng, Zheng Yang, Haifeng Liu, and Deng Cai. Lidar point cloud guided monocular 3d object detection. In European Conference on Computer Vision, 2022. [37] Liang Peng. Xiaopei Wu, Zheng Yang, Haifeng Liu, and Deng Cai. Did-m3d: Decoupling instance depth
[38] Zengyi Qin. Jinglu Wang, and Yan Lu. Monogrnet: geometric reasoning network for monocular 3d Earoeeaan
for monocular 3d object detection. In European Conference on Computer Vision, 2022.
object localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8851-8858, 2019.
[39] Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 8555-8564, June 2021.
[40] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view ceneral-purposs 3d object detection. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2397-2406, January 2022.
[41] Xuepeng Shi, Qi Ye, Xiaozhi Chen, Chuangrong Chen, Zhixiang Chen, and Tae- Kim. Geometry- based distance decomposition for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15181,, 2021.
[42] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
pages
[43] Li Wang, Li Zhang, Yi Zhu, Zhi Zhang, Tong He, Mu Li, and Xiangyang Xue. Progressive coordinate transforms for monocular 3d object detection. Advances in Neural Information Processing Systems, 34: 13334--133377 2021.
[44] Tai Wang, Xinge Zhu, Jiangmiao Pang. and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Workshops, pages 913 3-922, October 2021.
[45] Tai Wang, Xinge ZHU, Jiangmiao Pang, and Dahua Lin. Prohabilistic and geometric depth: Detecting objects in perspective. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1475-1485. PMLR, 08-11 11 Nov 2022.
12

---

[46] Yan Wang, Wei-Lun Chao, Divyansh Garg. Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June
2019.
[47] Xiaopei Wu, Liang Peng. Honghui Yang. Liang Xie, Chenxi Huang, Chengqi Deng, Haifeng Liu, and Deng Cai. Sparse fuse dense: Towards high quality 3d detection with depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5418-5427, June 2022.
[48] Tianfu Wu Xianpeng Liu, Nan Xue. Learning auxiliary monocular contexts helps monocular 3d object detection. 36th AAAI Conference on Artifical Intelligence (AAAI), Feburary 2022.
[49]
Lei Yang. Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang. Xinyu Zhang, and Peng Chen. Bevheight: A robust framework for bssoon-baed roadside 3d object detection. In JEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), Mar. 2023.
[50] Lei Yang. Xinyu Zhang, Li Wang, Minghan Zhu, Chuan- Fang Zhang, and Jun Li. Mix-teaching: simple, unified and effective semi-supervised learning framework for monocular 3d object detection, ArXiv, 0as/2207.044453 2022.
[51] Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
October 2019.
[52] Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng. Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In European Conference on Computer
Vision, pages 17-34. Springer. 2020.
[53] Xiaoqing Ye, Mao Shu, Hanyu Li, Yifeng Shi, Yingying Li, Guangjic Wang, Xiao Tan, and Errui Ding. Rope3d: The roadside perception dataset for autonomous driving and monocular 3d object detection task. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11341-21350, June 2022.
[54] Fisher Yu, Dequan Wang. Evan Shelhamer, and Trevor Darrell. Deep layer aggregation, In Proceedings of Computer Vision and Pattern Recognition (CVPR), June 2018
the IEEE Conference
[55] Haibao Yu. Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang. Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. D2i: large-scale dataset for vehicle-infraste infrassttiuree cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21361-21370 June 2022.
[56] Yunpeng Zhang. Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3289 3298, June 2021.
[57] Xingyi 7hou, Dequan Wang, and Philipp Krähenbüüh. 1eXiv:/94.007000
Objects as points. In arXiv preprint
2019.
[58] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang, Monocular 3d object detection: An extrinsic parameter free approach. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7556-7566 2021,
[59] Yunsong Zhou, Quan Liu, Hongzi Zhu, Yunzhe Li, Shan Chang, and Minyi Guo. Mogde: Boosting mobile monocular 3d object detection with ground depth estimation, 2023.
[60] Yunsong Zhou, Hongzi Zhu Quan Liu, Shan Chang. and Minyi Guo. Monoatt: Online monocular 3d object
detection with adaptive token transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Partern Recognition (CVPR), pages 17493-17503, June 2023.
[61] Zhikang Zou, Xiaoqing Ye, Liang Du, Xianhui Cheng, Xiao Tan, Li Zhang, Jianfeng Feng, Xiangyang Xue, and Errui Ding. The devil is in the task: Exploiting reciprocal appearance-localizzen features for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2713 2722, October 2021.
13


Reviews:
[{"review_id": "6AreMRIjKq", "reviewer": "Reviewer_rxxf", "summary": "This paper proposes a unified architecture for vehicle and infrastructure-based monocular 3D object detection network. At its core, the paper puts forth the concept of normalized depth that is independent of camera intrinsic focal length and extrinsic pitch angle w.r.t the ground plane. As such, the network is applicable to cameras with varying focal length and mounting angle, while not affected by the ambiguity. Following DID-M3D, the framework decomposes the center depth into cube depth and the so-called bias depth. The experiments demonstrate significantly better performance in Rope3D and DAIR datasets.", "questions": "The motivation and the implication of approximating v_p with v_c are the main question I have. I hope the authors could address this in the rebuttal. ", "limitations": "The paper has discussed the paper properly, i.e. the method currently requires separate training on vehicle and infrastructure dataset.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents interesting new insights towards the problem of monocular 3D object detection under varying focal length and pitch mounting angle. The varying focal length problem has been tackled with normalized depth by existing works in the context vehicle-based 3D object detection, but handling pitch angle is as yet under-explore. The paper for the first time derived the normalized depth of object center that is independent of the pitch angle. \n2. The method is simple yet effective. The experiments demonstrate that the proposed method yields significantly superior performance over the state-of-the-art, as shown in Table. 2.\n3. The proposed framework is a unified framework for both vehicle-based and infrastructure-based 3D detection. This open doors to the new possibility in combining the research and data across these two domains. While the method currently requires separate training on each, it holds the potential for joint training that may improve both. \n", "weaknesses": "1. In Line 158, the paper makes approximation of the angle \\delta by replacing v_p with v_c, but the paper does not discuss the implication of this approximation in practice. It seems the approximation error could possibly be large especially for nearby objects and with smaller pitch angle in camera mounting. How would this impact the performance? In addition, why not let the network to predict v_p as well? If the network is able to prediction of position of object center, i.e. v_c, what prevents it from predicting the position of bottom center v_p?\n2. While the performance gain on infrastructure cameras are significant, it is small in vehicle cameras, i.e. KITTI. In particular, the paper only compares with [1,24,44,29] while omitting other stronger existing methods, such as CMKD ECCV 2022 and LPCG ECCV 2022. It is fine that the method does not outperform these methods given the unique advantage of being a unified framework, but the paper should acknowledge this for readers’ better understanding. There are also concurrent works such as NeurOCS and Mix-Teaching with better accuracy, which are not necessary to compare against but would be good to discuss in related works for completeness. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the motivation, significance, and potential of our method. We are pleased that you support our insight on the problem of monocular 3D object detection under varying focal length and pitch mounting angle. As you have stated, our network firstly simultaneously avoids the ambiguity problem introduced by diverse focal lengths and pitch angles. We are also glad that you agree with the simplicity and effectiveness. Finally, we sincerely thank you for providing insightful suggestions. We will try to explain your concerns in the following part of this comment and are willing to discuss them further.\n\n**The motivation and implication of approximating v_p with v_c:** Firstly, we extend our gratitude for your thorough review and insightful suggestions. As you recommended, utilizing the model-predicted v_p is a method with reduced errors. In our initial experimental procedure, for the sake of simplification, we directly replaced v_p with v_c. To our pleasant surprise, this replacement resulted in a notable enhancement of 2.18% (Table 5 (a) and (c) in the original paper). Consequently, we inadvertently disregarded the influence of its inherent approximation error and the potential for additional improvements.\n\nThese days, we conducted three experiments to comprehensively analyze this problem. Exp1 proved that the original scheme would introduce an average relative error of 2.9% and a maximum relative error of 9.1%. Exp 2 proved that predicting v_p can indeed further improve the performance. Exp3 shows that the original scheme is indeed relatively weak in near distance, but using predicting v_p can alleviate the problem.\n\nThe details are as follows:\n\n* **Exp1 (Statistical Error Analysis) :** On Rope3D dataset, we measured the v pixel distance between the center and bottom center of all obstacles. The maximum distance is 98.32 pixels (for the nearest vehicle), with an average of 31.7 pixels. With the minimum focal length at about 2100 pixels, the highest error in calculating $\\tan(\\delta)$ using v_c instead of v_p is 98.32/2100=0.0468, and the average error is 31.7/2100=0.015. The $\\tan(\\delta)$ range is [-540/2100, 540/2100]= [-0.257, 0.257], and the pitch angle range is roughly [10, 15]. This affects the cosine (0.9848 to 0.9659) and sine (0.1736 to 0.2588) of the pitch angle, resulting in a normalized pitch range of [0.8994, 1.0324]. The maximum absolute error from $\\delta$ is 0.0468 * 0.2588=0.0121, averaging at 0.015 * 0.2588 =0.0039. The highest relative error is 0.0121/(1.0324 - 0.8994)=0.091, and the average relative error is 0.0039/(1.0324 - 0.8994)=0.029.\nAlthough under the existence of this error, the normalization of the pitch angle has brought about a performance improvement of 2.18%, but there is still room for improvement.\n\n* **Exp2 (Predicting v_p):** We added a head to predict the vertical coordinate of the bottom center point P, and used the predicted v_p to calculate the normalized depth. On Rope3D, the AP_3D increased from 81.55 to 82.63. Combined with cube depth, we added a head to predict the v coordinates on the second stage. As shown in Table 5 in global rebuttal, the AP_3D was increased from 92.45 to 92.61.\n\n* **Exp3 (Rope3D Evaluation by Distance):** As shown in Table 5 in global rebuttal, we split the evaluation into different dimensions according to the distance between obstacle and camera, in order to explore the detection performance at different distances. The original scheme (v_c instead of v_p) has a small improvement (92.69->92.84) due to the introduction of errors for nearby obstacles, while using the model to directly predict v_p has a more obvious improvement (92.69->93.17). We believe that the slight drop in the 30-60m (93.10->93.05) is caused by fluctuation between two independent trainings.\n\nIn summary, there are other ways to compute v_p, like having the model predict obstacle corner points and deducing v_p geometrically. However, due to time limitations, we haven't pursued this approach. In the revised paper, we'll detail various v_p solution methods (v_c substitution, model-predicted v_p, and geometric solutions) along with respective experimental results, enhancing reader understanding. \n\n**More methods on vehicle side:** Sorry for the lack of methods. As shown in the Table 3 in global rebuttal, we have added the latest vehicle side methods (such as CMKD, LPCG and MonoATT) for comparison. As you mentioned, our method as a unified framework is valuable, although not be able to surpass all vehicle-side methods. In fact, in terms of the vehicle-side method, our method has a large increase compared with our baseline method GUPNet (15.02->16.73). While compared with the DID-M3D of a similar framework, MonoUNI has an increase (16.29->16.73) and has the advantage of not requiring additional data. We will also add more concurrent work such as NeurOCS and Mix-Teaching to related work.\n\n**Limitations on separate training:** In the Limitation of our paper, we objectively acknowledge that our method doesn't include specific adjustments for the mixed training between vehicle and infrastructure sides. We believe that addressing mixed training requires domain adaptation for the distinct appearance features between vehicle and infrastructure sides, deviating from our paper's emphasis on unified optimization target and training pipline. We appreciate your agreement with this perspective. While our method doesn't address mixed training directly, based on comments from reviewer Lnrw, we've conducted mixed training experiments to comprehensively evaluate our approach. As seen in Table 4 of the global rebuttal, our MonoUNI mixed training has fewer dropped points than GUPNet and SMOKE, indicating that our method can better alleviate the additional complexity and visual ambiguity introduced by mixed training.\n"}, {"review_id": "FORJLeAqad", "reviewer": "Reviewer_YJqg", "summary": "This paper proposes an optimization target which unifies 3D detection problems for vehicle and infrastructure sides, by taking into account the diversity of camera pitch angles and focal lengths. Furthermore, the paper develops 3D normalized cube depth of obstacle to promote the learning of depth information.\nThe authors provide extensive experimental results on several monocular 3D detection benchmarks to prove the effectiveness of the proposed approach on both vehicle and infrastructure scenarios.", "questions": "1.It is mentioned in sec 3 L#157 that the pitch angle theta can be calculated from the ground equation. I suppose the accuracy of pitch angle is severely affected by the accuracy of ground plane estimation. So I wonder how the parameters of the ground plane equation are estimated.\n2.The derivation of the normalized depth are done in a degenerated 2d view instead of 3d. It would be better if the authors can provide simple explanation or prove that neglecting the effect of camera yaw angle is reasonable.", "limitations": "This paper mentioned that one model file to support 3D detection for both sides rather than separate training for the two sides. This is a work worth investing in, and it will solve the problem of vehicle-road collaborative perception very well. I look forward to the subsequent output of the authors.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors take notice of that depth can be ambiguous under the influence of focal length given similar visual features and conduct experiments to verify their claim. \n2. They propose to decouple depth from focal length and the camera optical axis orientation, specifically, to learn a normalized depth which is unaffected by focal length and axis orientation, from which the real depth can be recovered. I think the proposed idea is novel, reasonable and also proved to be effective, which eases the model from predicting ill-defined depth from visual feature.", "weaknesses": "The idea of normalized depth is good, but the article seems like semi-finished and needs to be completed carefully.\n- In sec 3, the definition of H' is missing, though it can be speculated from figure 3.\n- L#159 principle->principal.\n- Many key components of the proposed method are only described by text or even only be found in the figure.\nFor instance in sec 3.3, L#193, the authors say \"Compared with only supervising the depth of the center point and corner points, the 3D cube depth is a sufficient way to utilize the depth information\". So I guess you supervise the depth of all the points on the visible surface of the obstacle? How is it implemented? Mathematically, what are the output of the model and how are they supervised? I fail to find any formular definition of your training loss and the \"depth uncertainty\" in fig 2. As a reader, I can roughly guess the meaning but I would appreciate it if the authors could complete all the missing definitions. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the motivation, novelty, rationality and effectiveness of our method. We especially thank you for supporting our insight on learning a normalized depth that decouples depth from the focal length and the camera’s optical axis orientation. As you have stated, our normalized depth can ease the model from predicting ill-defined depth from visual features. We also thank you for providing insightful suggestions. We will try to explain your concerns in the following part of this comment and are willing to discuss them further.\n\n**About the article seeming like semi-finished:** First and foremost, we extend our gratitude for your meticulous review, and we sincerely apologize for any hurried writing in the initial submission. We have diligently revisited our paper multiple times, ensuring that the revised version is comprehensive and reader-friendly. Each new module or concept is thoroughly elucidated to prevent any reader confusion. Below are our responses to your comments.\n\n* Added the explanation of H’ in section3: Extend a line from the obstacle's center point C along the camera's imaging plane direction, intersecting line OP at point P'. The distance CP' is denoted as H.\n* Principle point was corrected to principal point.\n* We will introduce a new subsection, 3.4, to provide a comprehensive and detailed account of our loss design. Your understanding of the 3D cube depth supervision approach is accurate: indeed, we supervise the depth of all points on the visible surface of the obstacle. During training, this is achieved by supervising points with 3D cube depth ground truth on the obstacle-level feature (7x7 size) after ROI-align while the remaining points without 3D cube depth ground truth are unsupervised. The methodology for supervising Bias Depth mirrors that of Cube Depth. Concurrently, we will include explanations and relevant references concerning Depth Uncertainty, initially introduced in [1]. This concept adds an extra layer of uncertainty to each depth prediction of the model, capturing observation noise from input data. As emphasized in [1], this approach enhances the loss's robustness against noisy inputs in regression tasks. During inference, each obstacle will produce a 7x7 cube depth, a 7x7 cube depth uncertainty, a 7x7 bias depth, and a 7x7 bias depth uncertainty. Cube depths and bias depths will be weighted based on their respective uncertainties to yield the unique cube depth and bias depth, which will be added to obtain the final actual depth.\n\n**About the ground plane equation:**\n* For the vehicle-side, we default the camera optical axis to be parallel to the ground (pitch angle=0), so no additional ground plane equation data is required.\n* For the infrastructure-side, the ground plane equation is provided by the Rope3D and Dair-V2X datasets. Their methods for estimating the surface equation are the same, that is, least squares plane fitting. During making datasets, they used autonomous vehicles with LIDAR to scan the collected scenes with point clouds and used the obtained dense point clouds for ground extraction to obtain dense ground point cloud data. These ground point cloud data are included in the datasets. Additionally, the datasets supply the ground plane equation (defined by coefficients a, b, c, and d) obtained through least squares plane fitting on this data. The Rope3D and Dair-V2X datasets themselves are actively willing users to use ground equation data and original ground point clouds, and recommend users to explore different usage methods, because these data also be used in real industrial applications. The baseline methods proposed in both Rope3D and Dair-V2X papers also use these data.\n\n**The effect of camera yaw angle for normalized depth:** The paper's schematic diagram (Figure 3) represents a simplified scenario where the vehicle orientation and camera optical axis are coplanar (parallel). Even when an angle exists between the camera's optical axis and the vehicle's orientation (i.e., yaw != 0), the derivation remains applicable. This is due to the fact that the normalized depth solely depends on two points: the depth calculation point (center point C of the obstacle) and its corresponding vertical projection point on the ground (bottom center point P of the obstacle). Irrespective of the camera's yaw angle, the scenario can be visualized as the vehicle rotating around PC while the camera is stationary. In this process, PC is not changed, thus the geometric modeling process remains unaltered and the normalized depth remains constant. We will incorporate this explanation into subsection 3.3.\n\n**Limitations on separate training on vehicle and infrastructure dataset:** In the Limitation of our paper, we objectively acknowledge that our method doesn't include additional design for the mixed training between vehicle and infrastructure sides. Because we believe that mixed training should be solved from adaptation for different appearance features between vehicles and infrastructures. This is somewhat different from the unified optimization targets and training pipline emphasized in our paper, and requires independent additional method to solve. We appreciate your agreement with this opinion. Although no method is designed to solve mixed training directly, based on comments from reviewer Lnrw, we've conducted mixed training experiments to comprehensively evaluate our approach. As seen in Table 4 of the global rebuttal, our MonoUNI mixed training has fewer dropped points than GUPNet and SMOKE, indicating that our method can better alleviate the additional complexity and visual ambiguity introduced by mixed training.\n\n[1] What uncertainties do we need in bayesian deep learning for computer vision."}, {"review_id": "IdhMVO93Rw", "reviewer": "Reviewer_LnrW", "summary": "The paper proposes a new approach called MonoUNI for monocular 3D object detection of both vehicle and infrastructure sides in autonomous driving. The approach addresses the challenge of constructing algorithms for the two sides based on different prior knowledge, by taking into account the diversity of pitch angles and focal lengths. The proposed method uses a unified optimization target called normalized depth to unify the 3D detection problems for the two sides. Additionally, a 3D normalized cube depth of obstacle is developed to promote the learning of depth information and enhance the accuracy of monocular 3D detection. The paper presents extensive experiments on three widely used monocular 3D detection benchmarks, and the proposed method achieves state-of-the-art performance on all three benchmarks without introducing any extra information.", "questions": "My main concern is that the authors lack the necessary motivation and explanation for their design of the unified depth and the cube depth. The authors need to provide a detailed explanation of the rationale and purpose of the proposed designs. Currently, the proposed design approach does not appear to have significant differences from previous works.", "limitations": "The main limitation of this paper is that it does not implement training a single model to handle different settings across multiple datasets. Currently, the proposed approach cannot be considered as a solution to the problem of different settings between the vehicle and infrastructure sides.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a novel approach that unifies monocular 3D object detection for both vehicle and infrastructure sides, which is an important problem in autonomous driving. \n2. The proposed approach addresses the challenge of constructing algorithms for the two sides based on different prior knowledge, by taking into account the diversity of pitch angles and focal lengths. The use of a new optimization target called normalized depth and the 3D normalized cube depth of obstacle as an additional supervision clue helps to improve the accuracy of monocular 3D detection. \n3. The extensive experiments on three widely used benchmarks show that the proposed method achieves state-of-the-art performance on all three benchmarks without introducing any extra information.", "weaknesses": "1. Lack of key details. In formula (4), the authors mentioned the use of the definition of normalized depth to simplify the depth detection task. It is unclear whether MonoUNI needs to be built on the assumption of known pitch angle and focal length during model inference. If it does, then the work does not truly address the problem of different conditions for vehicle and infrastructure sides. If it does not, the authors need to explain why adopting a unified form would reduce the difficulty of learning, and the inference in line 161 is still unconvincing.\n2.Insufficient contribution. The proposed 3D normalized cube depth is inspired by AutoShape and DID-M3D, and extends from regressing corner coordinates to regressing the depth within the foreground 3D box. The proposed method is too straightforward and lacks the necessary motivation to explain why the depth on the surface is important, which needs to be supported by experiments. Introducing surface depth may not make a significant difference compared to regressing corner depth, and may introduce additional errors due to the irregularity of foreground instances.\n3. Mismatch between the motivation and method of the paper. The motivation of the paper is to design a new depth annotation, i.e., normalized depth, to unify monocular 3D object detection for both vehicle and infrastructure sides. However, according to line 268, if the two scenarios need to be trained separately, the proposed method's significance is limited, and it seems unable to achieve the desired Mono3D scheme that unifies vehicle and infrastructure sides.\n4. Cross-dataset experiments. The paper aims to establish a unified depth detection scheme for both vehicle and infrastructure sides. It is desirable for the authors to demonstrate the model's cross-dataset capability and the effect of multi-dataset mixed training. Additionally, using extrinsic perturbation experiments on a single dataset seems to be a necessary verification method.\n5. Minor errors. Each element in the formula needs to have a corresponding explanation, such as (5).\n6. Missing relevant papers.\n[1] MonoEF: Extrinsic parameter free monocular 3D object detection.\n[2] Mogde: Boosting mobile monocular 3D object detection with ground depth estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the significance of the unify problem in autonomous driving.  As you have stated, our MonoUNI achieves state-of-the-art performance on all three benchmarks without introducing any extra information. We also thank you for providing insightful suggestions. We will try to explain your concerns in the following part of this comment and are willing to discuss them further.\n\n**Lack of key details:** We apologize for our inappropriate representation. MonoUNI requires known pitch angle and focal length. Whether it is the vehicle-side real dataset (KITTI, Waymo or nuScenes) or the infrastructure-side dataset (Rope3D and DAIR-V2X), the pitch angle and focal length are known and willing to be used (vehicle-side pitch angle default 0). Even in industrial applications, they are actively used as prior knowledge. Using focal length and pitch angle is a way to address the problem of different conditions for vehicle and infrastructure sides, and they are usually not counted as additional data (many 3D detection methods use internal parameters for 2d-3d conversion). Both reviewers rxxf and YJqg acknowledged the value and significance of our solution.\n\n**Method is “straightforward” and insufficient contribution:** Regarding contributions, we believe that simple and effective methods are worth advocating. Despite looking \"straightforward\", to the best of our knowledge, we are the first to directly introduce 3D bbox depth supervision in 3D detection without using any additional data. Both using GUPNet as the baseline, our results are superior to DID-M3D using additional data (16.29->16.73), and only need a simple change to improve 1.71% compared to GUPNet (15.02->16.73). Reviewer rxxf also thinks that our simple but effective is a strength.\n\n**Motivation:** Compared with regressing corner depth, regressing cube depth is denser supervision. Dense depth supervision is beneficial to 3D detection. Reviewer yykW also acknowledged this and many existing papers have also proved it, including DID-M3D using obstacle surface depth for supervision, AutoShape using dense CAD information, and MonoDDE using 20 depths. That's where our motivation comes from. Essentially, our paper is analyzing what information the model in 3D detection learns, which we consider to be geometric information. The normalized depth is actually disambiguating the geometric information, while the cube depth is increasing the geometric information for supervision. MonoUNI uses more geometric information to supervise in unambiguous scenes, which has a positive impact on the both vehicle and infrastructure sides.\n\n**Experimental proof:** The obvious positive results of Table 5 (a)-(e) and (d)-(f) in our original paper prove that dense deep supervision cube depth is very important. And we also added the experimental results of Waymo and nuScenes in the Global rebuttal. Cumulatively so far, we have proven our method on three vehicle-side and two roadside real datasets.\n\nRegarding the irregularity of foreground instances, we have not observed such phenomenon in our experiments.  We also believe that if it exists for the regression cube depth, then the regression corner depth still has this problem, because the corner depth belongs to the corner point of the 3D bounding box, and it also does not really exist in the physical world. In comparison, regressing cube depth is still superior to corner depth.\n\n**Mismatch between the motivation and method:** We believe there is no mismatch problem. The purpose of our work is to unify the 3D detection tasks of the vehicle and infrastructure from the perspective of regression target and training pipeline, and we have achieved this goal. Both reviewers rxxf and YJqg acknowledged this.\n\nIn Limitation part, we objectively acknowledge that our method doesn't include additional design for the mixed training between vehicle and infrastructure sides. Because we believe that mixed training should be solved from adaptation for different appearance features between vehicles and infrastructures. This is somewhat different from the unified optimization targets and training pipline emphasized in our paper, and requires independent additional method to solve. Both reviewers rxxf and YJqg acknowledged this. Although no method is designed directly, we've conducted mixed training experiments to comprehensively evaluate our approach. As seen in Table 4 of the global rebuttal, our MonoUNI mixed training has fewer dropped points than GUPNet and SMOKE, indicating that our method can better alleviate the additional complexity and visual ambiguity introduced by mixed training.\n\n**Cross-dataset experiments:** We validate our method on Waymo val and perform cross-validation evaluation on nuScenes, as done in DEVIANT. On Waymo, as shown in Table 1 in global rebuttal, our approach achieves overall superiority over GUPNet and DEVIANT. For example, our method has a 0.51% improvement on Level1 under IOU=0.7 compared to the DEVIANT. On nuScenes, we use the KITTI val model for inference. The results are shown in Table 2 in global rebuttal. Although it does not exceed DEVIANT, it still has competitive capabilities. The results of mixed training are also explained in the answer above. Regarding the external disturbance, since the data enhancement of the pitch angle is irreversible (it is a 3d->2d process, the appearance generated by 2d->3d does not conform to the real situation), so we did not add this experiment. Any further guidance on this experiment would be greatly appreciated.\n\n**Minor errors:** We have diligently revisited our paper multiple times, ensuring that the revised version is comprehensive and reader-friendly. Each new module or concept is thoroughly elucidated to prevent any reader confusion. \n\n**Missing relevant papers:** Sorry for the lack of methods. We will add more excellent related work, such as MonoEF, MoGDE and MonoATT."}, {"review_id": "o9IXfwPfIy", "reviewer": "Reviewer_yykW", "summary": "This work focuses on the monocular 3D object detection task and proposes a unified 3D detection framework for both vehicle and infrastructure sides. In particular, to unify the diversity of pitch angles and focal lengths of multiple cameras, the authors propose a unified optimization target named normalized depth. Besides, they also design the 3D normalized cube depth of obstacle to promote the learning of depth information. They conduct extensive experiments on three datasets, including Rope3D, DAIR-V2X-I, and KITTI, and get promising results on them, especially on Rope3D and DAIR-V2X-I.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main idea is easy-to-follow and the paper is well-written.\n3. The proposed model gets SOTA performance on the infrastructure side, ie. Rope3D and DAIR-V2X-I datasets, and competitive results on the vehicle side, ie. the KITTI dataset.\n4. Code will be open-sourced.", "weaknesses": "The authors claim they proposed a unified 3D detection pipeline for both infrastructure and vehicle sides. Although the proposed method shows promising peroformance on Rope3D and DAIR-V2X-I datasets, it cannot well genelized to the vehicle side. \n\n - Their proposed method can about 16 ap (moderate setting) on KITTI testing, while the existing models can get 17+, such as [1].\n - The authors only evaluate their model on KITTI, which only contains 7K images, and evaluating it on the largescale dataset such as nuScenes and Waymo is required to show the effectiveness in vehicle side.\n - In table 5, the experiments (a -> c) and (b->d) show the pitch-based depth normailization is harmful to the vehicle-centric detection.\n - Also, table 5 shows the effectiveness of the 'cube' design to the vehicle-centric detection. Note KITTI is a small dataset with sparse depth supervision and the 'cube' design densify the depth supervision in fact, which maybe the underlying reason why it works .  However, whether this design works or not when large-scale training data avaiable is still unclear. So I recommand the authors conduct more experiemnt on larger dataset again.\n\nBased on the above points, I am not sure designing a unified pipeline for both infrastructure and vehicle sides is meaningful or not, and more experiemnts are required to support the claim.\n\n[1] Online Monocular 3D Object Detection with Adaptive Token Transformer, CVPR'23\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the quality, clarity, and effectiveness on infrastructure side of our paper.  As you have stated, our method is easy-to-follow, and achieves SOTA performance on the infrastructure side and competitive results on the vehicle side. We also thank you for providing insightful suggestions. We will try to explain your concerns in the following part of this comment and are willing to discuss them further.\n\n**Performance on vehicle side:** Sorry for the lack of methods. As shown in the Table 3 in global rebuttal, we have added the latest vehicle side methods (including CMKD, LPCG, MonoATT and DEVIANT) for comparison. Our method as a unified framework is valuable, although not be able to surpass all vehicle-side methods. Reviewer rxxf have also acknowledged this point. In fact, in terms of the vehicle-side method, our method has a large increase compared with our baseline method, such as GUPNet (15.02->16.73). While compared with the DID-M3D of a similar framework, MonoUNI has an increase (16.29->16.73) and has the advantage of not requiring additional data.\n\n**More results on Waymo and nuScenes:** Thank you for your sincere advice. We validate our method on Waymo val and perform cross-validation evaluation on KITTI-nuScenes, as done in MonoRCNN (ICCV2021) [1] and DEVIANT (ECCV 2022 oral) [2]. On Waymo, as shown in Table 1 in global rebuttal, our approach achieves overall superiority over GUPNet and DEVIANT. For example, our method has a 0.51% improvement on Level1 under IOU=0.7 compared to the DEVIANT. On nuScenes, we use the KITTI val model for inference. The results are shown in Table 2 in global rebuttal. Although it does not exceed DEVIANT, it still has competitive capabilities.\n\n**Pitch-based depth normalization on vehicle-centric detection:** Pitch-based depth normalization is not harmful on vehicle-centric detection. In fact, since the optical axis of the camera on the vehicle side is parallel to the ground and the pitch angle is equal to 0, the pitch-based depth normalization does not take effect on the vehicle side, which is equivalent to not having this module. In fact, the difference between (a->c) and (b->d) in the KITTI data results of Table 5 in our original paper is the fluctuation caused by two identical experiments due to different random seeds.\n\n**”Cube” design densify the depth supervision:** Your understanding is correct, which is also the motivation for our design of cube depth. We believe that dense depth supervision will deepen the model's understanding of geometric information. We have added Waymo validation and KITTI-nuScenes cross-dataset validation following your suggestion.\n\n**Significance of unified vehicle and infrastructure pipeline:** We believe a unified framework is meaningful. Both reviewers rxxf and Yjqg acknowledged this.\nThe main reasons are as follows:\n* This unified framework eliminates the influence of pitch angles and focal length, so that subsequent 3D detection, whether it is on the infrastructure side or on the vehicle side, can use the same regression target for supervision;\n* Essentially, our paper is analyzing what information the model in 3D detection learns, which we consider to be geometric information. The normalized depth is actually disambiguating the geometric information, while the cube depth is increasing the geometric information for supervision. MonoUNI uses more geometric information to supervise in unambiguous scenes, which has a positive impact on the both vehicle and infrastructure sides.\n* Following your comments, we have added more experiments to prove the effectiveness of our ideas and methods, including the results of Table 1 on Waymo, the results of Table 2 on nuScenes and the results of Table 4 for vehicle-infrastructure joint training. Cumulatively so far, we have proven our method on three vehicle-side and two infrastructure-side real datasets. \n* In Limitation part, we objectively acknowledge that our method doesn't include additional design for the mixed training between vehicle and infrastructure sides. Because we believe that mixed training should be solved from adaptation for different appearance features between vehicles and infrastructures. This is somewhat different from the unified optimization targets and training pipline emphasized in our paper, and requires independent additional method to solve. Both reviewers rxxf and YJqg acknowledged this. Although no method is designed directly, we've conducted mixed training experiments to comprehensively evaluate our approach. As seen in Table 4 of the global rebuttal, our MonoUNI mixed training has fewer dropped points than GUPNet and SMOKE, indicating that our method can better alleviate the additional complexity and visual ambiguity introduced by mixed training.\n\n**Missing relevant papers:** Sorry for the lack of methods. We will add more excellent related work, such as MonoEF, MoGDE and MonoATT.\n\n[1] MonoRCNN-Geometry-Based_Distance_Decomposition_for_Monocular_3D_Object_Detection.\n\n[2] DEVIANT-Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection."}]

Author rebuttal:
**Global Rebuttal**
  
  
**Table 1: Monocular 3D detection performance of Vehicle category on Waymo val set**
|$\mathbf{IOU_{3D}}$ |Difficulty|Method|Reference|Extra|$\mathbf{AP_{3D}}$(all) | $\mathbf{AP_{3D}}$(0-30m) | $\mathbf{AP_{3D}}$(30-50m) | $\mathbf{AP_{3D}}$(50m+) | $\mathbf{APH_{3D}}$(all) | $\mathbf{AP_{3D}}$(0-30m) | $\mathbf{AP_{3D}}$(30-50m) | $\mathbf{AP_{3D}}$(50m+) |
|:---:|:---:|---|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|0.7|Level_1|CaDDN|CVPR2021|LIDAR|5.03|15.54|1.47|0.10|4.99|14.43|1.45|0.10|
|0.7|Level_1|GUPNet|ICCV2021|None|2.28|6.15|0.81|0.03|2.27|6.11|0.80|0.03|
|0.7|Level_1|DEVIANT|ECCV2022|None|2.69|6.95|0.99|0.02|2.67|6.90|0.98|0.02|
|0.7|Level_1|**MonoUNI**|None|None|3.20|8.61|0.87|0.13|3.16|8.50|0.86|0.12|
|0.7|Level_2|CaDDN|CVPR2021|LIDAR|4.49|14.50|1.42|0.09|4.45|14.38|1.41|0.09|
|0.7|Level_2|GUPNet|ICCV2021|None|2.14|6.13|0.78|0.02|2.12|6.08|0.77|0.02|
|0.7|Level_2|DEVIANT|ECCV2022|None|2.52|6.93|0.95|0.02|2.50|6.87|0.94|0.02|
|0.7|Level_2|**MonoUNI**|None|None|3.04|8.59|0.85|0.12|3.00|8.48|0.84|0.12|
|0.5|Level_1|CaDDN|CVPR2021|LIDAR|17.54|45.00|9.24|0.64|17.31|44.46|9.11|0.62|
|0.5|Level_1|GUPNet|ICCV2021|None|10.02|24.78|4.84|0.22|9.94|24.59|4.78|0.22|
|0.5|Level_1|DEVIANT|ECCV2022|None|10.98|26.85|5.13|0.18|10.89|26.64|5.08|0.18|
|0.5|Level_1|**MonoUNI**|None|None|10.98|26.63|4.04|0.57|10.73|26.30|3.98|0.55|
|0.5|Level_2|CaDDN|CVPR2021|LIDAR|16.51|44.87|8.99|0.58|16.28|44.33|8.86|0.55|
|0.5|Level_2|GUPNet|ICCV2021|None|9.39|24.69|4.67|0.19|9.31|24.50|4.62|0.19|
|0.5|Level_2|DEVIANT|ECCV2022|None|10.29|26.75|4.95|0.16|10.20|26.54|4.90|0.16|
|0.5|Level_2|**MonoUNI**|None|None|10.38|26.57|3.95|0.53|10.24|26.24|3.89|0.51|
  
  
**Table 2: Cross-dataset evaluation of the KITTI val model on KITTI val and nuScenes frontal val cars with depth MAE.**
|Method|KITTI VAL(0-20m)|KITTI VAL(20-40m)|KITTI VAL(40m+)|KITTI VAL(all)|nuScenes VAL(0-20m)|nuScenes VAL(20-40m)|nuScenes VAL(40m+)|nuScenes VAL(all)|
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|M3D-RPN|0.56|1.33|2.73|1.26|0.94|3.06|10.36|2.67|
|MonoRCNN |0.46|1.27|2.59|1.14|0.94|2.84|8.65|2.39|
|GUPNet|0.45|1.10|1.85|0.89|0.82|1.70|6.20|1.45|
|DEVIANT|0.40|1.09|1.80|0.87|0.76|1.60|4.50|1.26|
|MonoUNI|0.38|0.92|1.79|0.865|0.72|1.79|4.98|1.43|
  
  
**Table 3: Monocular 3D detection performance of Car category on Rope3D val, DAIR-V2X-I val
and KITTI test sets.**
|Method|Reference|Extra Data|$\mathbf{AP_{3D}}$(Rope3D)|$\mathbf{R_{score}}$(Rope3D)|Easy(DAIR)|Mod(DAIR)|Hard(DAIR)|Easy(KITTI)|Mod(KITTI)|Hard(KITTI)|
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|M3D-RPN|ICCV2019|Depth/None|67.17|73.14|-|-|-|14.76|9.71|7.42|
|MonoDLE|CVPR2021|Depth/None|77.50|80.84|-|-|-|7.23|12.26|10.29|
|MonoFlex|CVPR2021|Depth/None|59.78|66.66|-|-|-|19.94|13.89|12.07|
|DID-M3D|ECCV2022|None/Depth|-|-|-|-|-|24.40|16.29|13.75|
|**CMKD**|ECCV2022|None/Depth|-|-|-|-|-|25.09|16.99|15.30|
|**LPCG+MonoFlex**|ECCV2022|None/Depth|-|-|-|-|-|25.56|17.80|15.38|
|**MonoEF**|CVPR2021|None/None|-|-|-|-|-|21.29|13.87|11.71|
|**DEVIANT**|ECCV2022|None/None|-|-|-|-|-|21.88|14.46|11.89|
|MonoCon|AAAI2022|None/None|-|-|-|-|-|22.50|16.46|13.95|
|**MonoATT**|CVPR2023|None/None|-|-|-|-|-|24.72|17.37|15.00|
|**MoGDE**|NeurIPS2022|None/None|-|-|-|-|-|27.07|17.88|15.66|
|Kinematic3D|ECCV2020|None/None|50.57|58.86|-|-|-|19.07|12.72|9.17|
|SMOKE|CVPR2020|None/None|72.13|76.26|66.03|62.24|60.71|14.03|9.76|7.84|
|GUPNet|CVPR2021|None/None|66.52|70.14|62.22|55.94|55.90|22.26|15.02|13.12|
|Imvoxelnet|CVPR2022|None/None|-|-|44.78|37.58|37.55|17.15|10.97|9.15|
|BEVFormer|ECCV2022|None/None|50.62|58.78|61.37|50.73|50.73|-|-|-|
|BEVDepth|AAAI2023|None/None|69.63|74.70|75.50|63.58|63.67|-|-|-|
|BEVHeight|CVPR2023|None/None|74.60|78.72|77.78|65.77|65.85|-|-|-|
|MonoUNI|None|None/None|92.45|92.63|90.92|87.24|87.20|24.75|16.73|13.49|
  
  
**Table 4: Multi-dataset mixed training under KITTI and Rope3D datasets.** "**mixed training**" means using KITTI + Rope3D training sets together for mixed training. The evaluation is performed on separate Rope3D and KITTI val sets.
|Method|Rope3D(mixed training)|KITTI(mixed training)|Rope3D(only training under Rope3D)|KITTI(only training under KITTI)|
|---|:---:|:---:|:---:|:---:|
|SMOKE|63.24|6.65|72.13|12.85|
|GUPNet|43.82|4.89|66.52|16.46|
|MonoUNI|87.89|13.62|92.45|17.18|
  
  
**Table 5: Rope3D Evaluation by different distance.**
|Method|$\mathbf{AP_{3D}}$(0-30m)|$\mathbf{AP_{3D}}$(30-60m)|$\mathbf{AP_{3D}}$(60-90m)|$\mathbf{AP_{3D}}$(90m+)|$\mathbf{AP_{3D}}$(all)|
|---|:---:|:---:|:---:|:---:|:---:|
|MonoUNI(without pitch normailization)|92.69|92.41|89.66|84.52|90.97|
|MonoUNI(v_p with v_c)|92.84|93.10|91.49|88.70|92.45|
|MonoUNI(model-predicted v_p)|93.17|93.05|91.52|88.62|92.61|


Meta-review: The paper proposes a unified pipeline for both vehicle side and infra side to detect objects in the driving scenario. They proposed a normalized depth module that is independent of camera parameters. The experiments show a great improvement in performance compared to previous methods. 

All reviewers reach concensus that the paper should be accepted, despite a borderline reject score (short review, no feedback after author rebuttal). AC agrees with reviewers that the paper should be accepted. Please revise the paper accordingly based on comments and incorporate new experiments provided in the rebuttal.
Decision: Accept (poster)