Title: MonoUNI: A Unified Vehicle and Infrastructure-side Monocular 3D Object Detection Network with Sufficient Depth Clues
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D detection

deep learning

autonomous driving
Abstract: Monocular 3D detection of vehicle and infrastructure sides are two important topics in autonomous driving. Due to diverse sensor installations and focal lengths, researchers are faced with the challenge of constructing algorithms for the two topics based on different prior knowledge. In this paper, by taking into account the diversity of pitch angles and focal lengths, we propose a unified optimization target named normalized depth, which realizes the unification of 3D detection problems for the two sides. Furthermore, to enhance the accuracy of monocular 3D detection, 3D normalized cube depth of obstacle is developed to promote the learning of depth information. We posit that the richness of depth clues is a pivotal factor impacting the detection performance on both the vehicle and infrastructure sides. A richer set of depth clues facilitates the model to learn better spatial knowledge, and the 3D normalized cube depth offers sufficient depth clues. Extensive experiments demonstrate the effectiveness of our approach. Without introducing any extra information, our method, named MonoUNI, achieves state-of-the-art performance on five widely used monocular 3D detection benchmarks, including Rope3D and DAIR-V2X-I for the infrastructure side, KITTI and Waymo for the vehicle side, and nuScenes for the cross-dataset evaluation.

MonoUNI: A Unified Vehicle and Infrastructure-side Monocular 3D Object Detection Network with Sufficient Depth Clues

Abstract

Monocular 3D detection of vehicle and infrastructure sides are two important topics in autonomous driving. Due to diverse sensor installations and focal lengths, researchers are faced with the challenge of constructing algorithms for the two topics based on different prior knowledge. In this paper, by taking into account the diversity of pitch angles and focal lengths, we propose a unified optimization target named normalized depth, which realizes the unification of 3D detection problems for the two sides. Furthermore, to enhance the accuracy of monocular 3D detection, 3D normalized cube depth of obstacle is developed to promote the learning of depth information. We posit that the richness of depth clues is a pivotal factor impacting the detection performance on both the vehicle and infrastructure sides. A richer set of depth clues facilitates the model to learn better spatial knowledge, and the 3D normalized cube depth offers sufficient depth clues. Extensive experiments demonstrate the effectiveness of our approach. Without introducing any extra information, our method, named MonoUNI, achieves state-of-the-art performance on five widely used monocular 3D detection benchmarks, including Rope3D and DAIR-V2X-I for the infrastructure side, KITTI and Waymo for the vehicle side, and nuScenes for the cross-dataset evaluation.

1 Introduction

Accurate 3D detection [14, 21, 25] is crucial for autonomous driving. Although LIDAR sensors [8, 18, 47, 51] provide high precision, camera sensors [7, 39, 52] are cost-effective and have a wider range of perception. Typically, autonomous driving systems employ frontal-view cameras mounted on the vehicle for 3D detection. As intelligent transportation continues to advance, there is increasing interest in using infrastructure-side cameras for 3D detection [13, 49, 53].

Due to the different installations and focal lengths between the vehicle and infrastructure-side cameras, researchers usually design algorithms to solve these two problems separately, which adds an additional application limitation. Fig. 1 illustrates the imaging process and the corresponding visual feature under different installations and focal lengths. With respect to the different installations, most vehicle-side cameras are installed on the top of the vehicle with a near-zero pitch angle, leading to a prior assumption that the optical axis is parallel to the ground. Most vehicle-side methods [4, 19, 27] are designed based on this assumption. In contrast, the infrastructure-side cameras, which are mounted on poles, typically have large pitch angles, rendering most of the existing vehicle-side

Camera  1

Camera  2

Camera  3

𝑃𝑙𝑎𝑛𝑒

!

Image captured by camera 1

Image captured by camera 2

Image captured by camera 3

𝜃

!

𝜃

"

𝜃

#

𝑓

!

𝑓

"

𝑓

#

𝑃𝑙𝑎𝑛𝑒

#

𝑃𝑙𝑎𝑛𝑒

"

Image captured by camera 4

𝑓

$

𝑃𝑙𝑎𝑛𝑒

$

Camera  4

𝜃

$

≈ 0

𝑧

!

𝑧

"

𝑧

#

𝑧

$

𝜃

"

> 𝜃

!

≈ 𝜃

#

> 𝜃

$

≈ 0

𝑓

!

> 𝑓

#

≈ 𝑓

"

𝑓

$

≈ 𝑓

%&'

%&'()*+

𝑧

!

> 𝑧

#

≈ 𝑧

"

methods unsuitable for direct application, and the diversity of pitch angles also further increases the difficulty of object detection. According to the various focal lengths, the type of vehicle-side cameras are relatively uniform, and the focal lengths are thus similar, while the infrastructure-side cameras have a wide range of focal lengths. This results in a new challenge of 1-to-N ambiguity for depth estimation in monocular 3D detection, which has already been an ill-posed problem.

In this work, to address the above problems, we developed the obstacle projection models of the vehicle and infrastructure sides, and found that the former can actually be regarded as a special case of the latter, where the pitch angle is approximately 0 and the focal length is fixed. Consequently, we propose a unified optimization target: normalized depth, which is independent of focal length and pitch angle, so that the prediction of the obstacle depth on the two sides is no longer disturbed by the diversity of focal length and pitch angle. Furthermore, to enhance the performance of 3D detection, we draw inspiration from methods like AutoShape [26] and DID-M3D [37], which introduce new depth clues. For instance, AutoShape uses dense key points on the vehicle surface to establish geometric constraints, while DID-M3D employs depth maps to enable the network to predict object surface depth. They both add spatially correlated depth clues to the model in different ways with auxiliary data. We suggest that incorporating rich depth clues has a greater impact on depth estimation accuracy. Thus, we use the geometric relationship to create a 3D normalized cube depth for the obstacle, allowing the model to predict the normalized depth of the obstacle's 3D bounding box without introducing any extra data. By predicting the bias depth from each point on the 3D bounding box to the obstacle center, we can obtain the final depth conveniently. The approach of predicting obstacle 3D normalized cube depth maximizes depth clues and improves detection performance without additional data.

In summary, incorporating the above techniques, our method MonoUNI first achieves state-of-the-art (SOTA) performance on both vehicle and infrastructure-side monocular 3D detection. The main contributions of this work are as follows: 1) We proposed a unified optimization target, namely normalized depth, to address the differences between vehicle and infrastructure-side 3D detection caused by the diversity of pitch angle and focal length. This optimization target can be considered as

the standard solution for future monocular 3D detection of both sides. 2) We posit that rich depth clues are crucial in 3D detection, and thus propose the use of a 3D normalized cube depth to facilitate the learning of depth information. 3) Without using any additional information, MonoUNI ranks 1st in both the Rope3D [53] and DAIR-V2X-I [55] infrastructure-side benchmarks. Moreover, we apply our method to the vehicle-side benchmarks, such as KITTI [10] and Waymo [42], and also achieve competitive results. Cross-dataset evaluation of the KITTI val model on the nuScenes [3] val set demonstrates the generalizability of our method. The code is available at https://github.com/TrafficX/MonoUNI.

2 Related Work

Vehicle-side Monocular 3D Object Detection. Vehicle-side monocular 3D detection refers to the process of analyzing a single image captured by a camera mounted on a vehicle to predict the 3D locations, dimensions, and orientations of the interest obstacles. These methods can be mainly categorized into two groups based on whether additional data is utilized. The first type of method uses additional data, such as depth maps [9, 28, 38], CAD models [5, 26, 35, 34] or LIDAR [6, 39] to enhance the detection accuracy. Pseudo-LIDAR methods [29, 46] utilize the depth map predicted by the additional network to assist the monocular image to generate a pseudo point cloud and then adopt existing LIDAR-based 3D object detection pipeline. DID-M3D [37] uses a dense depth map to generate visual depth for powerful data augmentation. CaDDN [39] uses LIDAR points to supervise additional monocular network estimates dense depth map and converts the feature to BEV perspective for prediction. AutoShape [26] utilizes CAD models to generate dense key points to alleviate the sparse constraints. NeurOCS [32] attempts to introduce the Neural Radiance Field (NeRF) [31] to solve the problem of lack of supervision information. Mix-Teaching [50] utilizes unlabeled data to achieve semi-supervised learning. Although utilizing additional information allows these methods to achieve improved performance, it inevitably leads to increased labeling and computational costs. Moreover, obtaining such data is challenging in various scenarios, especially when it comes to the infrastructure side.

The second kind of method only uses a single image as input without any extra information. Some methods [23, 27, 45, 56] use geometric projection assumptions to improve the accuracy of 3D detection. GUPNet [27] uses the 2D and 3D heights of the object to construct similar triangles to assist in regression depth. MonoFlex [56] and MonoDDE [23] further extend this similar triangle relationship using the position of key corner points to assist regression depth. PGD [45] constructs geometric relation graphs across predicted objects and uses the graph to facilitate depth estimation. Due to the installed pitch angle of infrastructure-side cameras, these geometric projections cannot be applied to infrastructure-side monocular 3D detection. Other methods [25, 30, 44] directly predict the dimensions, orientations, and locations of obstacles without the aid of geometric projections. However, these methods are designed for the vehicle side, ignoring the pitch angle diversity and the ambiguity introduced by the focal length gap between different cameras on the infrastructure side.

Infrastructure-side Monocular 3D Object Detection. Compared to vehicle-side monocular 3D object detection, which is typically limited to short-range perception, infrastructure-side 3D detection can overcome the problem of frequent occlusion in vehicle-based detection by increasing the sensor installation height, thereby providing long-range perception capabilities. Recently, DAIR-V2X-I [55] and Rope3D [53] are proposed to promote the development of 3D perception in infrastructure-side scenarios. Nonetheless, the challenges mentioned above, including the focal length difference and the variable pitch angle of the cameras, make it arduous to transfer the vehicle-side 3D detection method to the infrastructure-side setting, which results in relatively sluggish progress in infrastructure-side monocular 3D detection. BEVHeight [49] mitigates the issues arising from variations in camera pose parameters by directly predicting object height instead of object depth and achieves competitive performance in Rope3D and DAIR-V2X-I.

3 Method

3.1 Overview

Monocular 3D object detection extracts features from a single RGB image, predicting the category and 3D bounding box for each object in the image. The 3D bounding box can be further divided

Cube  Depth

uncertainty

Cube  Depth

2D Detector

𝐿

!"#$

uncertainty

𝐿

#%&'

MonoUNI

or

Infrastructure-side

Vehicle-side

Location

Dimension

Orientation

2D box

Norm cube

depth

Cube depth

Normalize

Normalized Cube Depth Generation

Other

properties

Object label

Pitch angle

Focal length

Camera

parameters

Calculate the bias

Object feature

into 3D center location ( x , y , z ), dimension ( h , w , l ) and orientation (yaw angle) θ . Among these properties, dimension, and orientation are easily learned by the network due to their strong correlation with visual features [15], but the 3D location is challenging because depth estimation is ill-posed.

The main idea of MonoUNI is to unify 3D detection targets for both vehicle and infrastructure sides, and further achieve accurate 3D location. The overall framework is depicted in Fig. 2. We use CenterNet [57] as the base model to generate discriminative representation, with DLA34 [54] serving as the backbone for feature extraction from images. Several network heads are established in MonoUNI to predict object properties, such as categorical heatmap, 2D bounding box, 3D offset, dimension, orientation, 3D normalized cube depth, bias depth, and depth uncertainty items [16, 27]. Depth uncertainty is widely used in 3D detection, which can enhance the loss's robustness against noisy inputs. The loss function is roughly similar to DID-M3D [37]. In the inference process, by leveraging the predicted 3D normalized cube depth and bias depth, and utilizing the known pitch angle and focal length information, our method can obtain the final obstacle depth.

3.2 Unified Optimization Target

Problem Analysis. In order to construct a unified optimization target for the vehicle and infrastructure sides, we analyze the significant difference in monocular 3D detection between them. First, the camera on the infrastructure side is typically installed at an elevated position with a specific pitch angle (the angle between the optical axis of the camera and the ground) to capture a wider view and detect more potential obstacles, which invalidates the commonly used assumption on the vehicle side that the optical axis of the camera is parallel to the ground. Not only that, due to different scenarios and installation methods, the pitch angle of the installed camera is varied. For instance, in the Rope3D dataset, the pitch angle of the camera ranges from 5 to 20 degrees. Second, to cope with different situations, the internal parameters of different infrastructure-side cameras usually have a relatively large difference. For example, the camera focal length of Rope3D ranges from 2100 to 2800 pixels, while the camera focal lengths of the KITTI dataset are all between 715 pixels and 721 pixels. The huge gap in focal length can lead to an ambiguity that obstacles of similar visual features in two images captured by cameras with different focal lengths can have different depths. To explore the influence of focal length diversity, we conducted experiments on Rope3D using both the popular single-stage and two-stage monocular object detection methods. Since the focal length of most images is centered around 2100 and 2700 pixels, we further split the dataset into train_2100, train_2700, val_2100, and val_2700. The experimental results are shown in Table 1.

As a consequence of the aforementioned ambiguity, utilizing two train sets for mixed training will decrease the accuracy of both validation sets when compared to training the network solely with images in a single focal length range. In particular, the average precision (AP) of GUPNet experiences a significant reduction of 70% on the val_2700. This decrease in accuracy can be attributed to the dominance of images with a focal length of around 2100 pixels during the training process.

Normalized Depth. In this subsection, we first build the simple projection models of the vehicle (Fig. 3 (a)) and infrastructure sides (Fig. 3 (b)). Taking the infrastructure-side projection model as an example, O in the figure is the optical center of the camera, the ray OZ denotes the optical axis (z-axis), point C represents the center point of the obstacle, P represents the intersection point

𝛼𝑥 + 𝛽𝑦 + 𝛾𝑧 + 𝑑 = 0

Z

O

z

C

P

h

𝜃

𝐻

𝐻

!

Y

𝛿

f

𝑦 + 𝑑 = 0

Z

O

z

C

P

h

𝜃 ≈ 0

𝐻 = 𝐻′

Y

𝛿

f

(𝑏)

Infrastructure-side

projection model

(𝑎)

Vehicle-side projection model

Imaging Plane

Imaging Plane

𝑃

!

between the vertical line from the center point C to the ground and the ground plane, H denotes the 3D distance of PC , and h is the pixel distance from the center point C to the ground in the imaging plane, θ is the pitch angle of the camera, z denotes the depth of center point C , f represents the focal length and δ is the included angle between the line connecting the point P to the optical center O and OZ . Extend a line from the obstacle's center point C along the camera's imaging plane direction, intersecting line OP at point P ′ . The distance CP ′ is denoted as H ′ . According to the parallel relationship, the following can be easily deduced:

After a simple geometric calculation, the following equation can be derived 3 :

Substitute it into equation (1) and we get:

Among them, H and h are easy to learn based on visual features, while δ , θ , and f are challenging to directly learn from the image. Therefore, learning z directly requires the model to have a strong ability to infer focal length and angle information. However, from the above analysis, it is difficult or even ambiguous. Typically, the focal length f can be obtained as prior knowledge, the pitch angle θ can be calculated as θ = arctan( γ/β ) from the ground equation αx + βy + γz + d = 0 , and δ can be calculated through simple calculation δ = arctan(( v p -c y ) /f ) , where v p is the y-axis pixel coordinates of Point P and c y is the principal point in y-axis. In particular, v p can be simply and roughly replaced by v c which may introduce an average relative error of 2 . 9% (statistics based on the

5

6

7

8

Dimension

Orientation

Location

2D Box

𝑍

!

𝑋

!

𝑌

!

𝑂

!

1

𝑋

"#$

𝑍

"#$

𝑌

"#$

2

3

4

7

𝑍

!

𝑋

!

𝑌

!

𝑂

!

1

2

3

4

6

5

8

𝑍

!

𝑋

!

𝑌

!

𝑂

!

7

1

2

3

4

6

5

8

Corner Points Generate

Planes Fitting

𝑃𝑙𝑎𝑛𝑒

!

𝛼

!

𝑥 + 𝛽

!

𝑦 + 𝛾

!

𝑧 + 𝑑

!

= 0

𝑃

𝑃

5

𝑃

$

𝑃𝑙𝑎𝑛𝑒

"

3D Cube Depth

Original Label

𝑃𝑙𝑎𝑛𝑒

!

𝑃𝑙𝑎𝑛𝑒

"

Rope3D dataset), or be predicted by the model. Therefore, a simple transformation of the formula (3) yields:

Formula (4) represents the new unified optimization target: normalized depth, which makes depth prediction independent of pitch angle and focal length, simplifying the task difficulty. When θ is equal to 0, the normalized depth degenerates into a situation where only the focal length is used for normalization ( z f ), that is, the vehicle-side projection model. In fact, simplifying the normalized depth with either the focal length ( z f ) or pitch angle ( z cos θ -sin θ ∗ tan δ ) can also partially reduces the learning difficulty and improve detection accuracy, which we will analyze in detail in the ablation experiments in section 4.4. During inference, the model outputs normalized depth, and we multiply it by the denominator in the formula (4) to get the final instance depth.

3.3 3D Normalized Cube Depth

To enrich depth clues and further improve detection performance, we utilize original labels to generate 3D cube depth for each obstacle. Based on the analysis in section 3.2, our 3D cube depth need to be normalized and become 3D normalized cube depth finally. The generation process of the 3D cube depth is shown in Fig. 4. Firstly, we use the location, orientation, and dimension information from the label to obtain the positions of the 8 corner points in the 3D bounding box in the camera coordinate system. These points are then marked in a specific sequence (1-8). Secondly, we calculate the plane equation α i x + β i y + γ i z + d i = 0 for each surface of the obstacle. To determine the equation, we select any 3 of the 4 vertices of the surface since a plane can be uniquely defined by 3 non-collinear points. Thirdly, we generate the 3D cube depth for the obstacle and project it onto the imaging plane. Specifically, each pixel ( u, v ) in the 2D bounding box is traversed to determine whether it belongs to any surface in the 2D image. Then, utilizing the internal parameters of the camera, the intersection point ( x, y, z ) of the pixel's ray with the corresponding plane can be computed and the depth z is thus achieved. The mathematical expression for this computation is as follows:

where f x and f y are the focal length of the camera, and c x and c y are the principal points. i denotes the index of the surface on the 3D bounding box. The depth z can be calculated as:

In cases where a pixel belongs to multiple faces in the 2D image, we select the smallest depth value as the final depth. This is because, during the camera imaging process, points with larger depth values are obstructed by ones closer to the camera. The same strategy is also applied when a pixel simultaneously belongs to multiple obstacles. Once the 3D cube depth is obtained, we apply Equation 3 to each depth value in the cube, resulting in a 3D normalized cube depth that can be used for

supervision. In order to get the center point depth, we also supervise the bias depth from each point on the cube depth to the center point depth. The utilization of cube depth can maximize the enrichment of depth clues based on existing labels without additional information such as depth maps, CAD, or LIDAR points. Compared with only supervising the depth of the center point and corner points, the 3D cube depth is a sufficient way to utilize the depth information.

4 Experiments

4.1 Dataset and Evaluation Metrics

Rope3D. Rope3D [53] is a comprehensive real-world benchmark for infrastructure-side 3D detection, featuring over 50,000 images and more than 1.5 million 3D objects in diverse scenes. The benchmark comprises various settings, such as different cameras with ambiguous mounting positions, diverse camera specifications, as well as different environmental conditions. For the evaluation metrics, we follow the official settings using AP 3 D | R 40 and Rope score [53], which is a combined metric of the 3D AP and other similarities, such as Average Ground Center Similarity (ACS). We follow the proposed homologous setting to utilize 70% of the images as training, and the remaining as validating. All images are randomly sampled.

DAIR-V2X-I. DAIR-V2X-I [55] is a subset of DAIR-V2X, which is a large-scale multimodal vehicle-infrastructure collaborative perception dataset. It contains 10,000 infrastructure-side images and 493,000 3D bounding boxes with 10 classes. We follow the official train/val division for evaluation, and use AP 3 D | R 40 as the main metric, with an IOU threshold of 0.5.

KITTI. KITTI [10] is a widely used benchmark consisting of 7481 training images and 7518 testing images for vehicle-side monocular 3D detection. The dataset contains three classes: Car, Pedestrian, and Cyclist, each with three difficulty levels: Easy, Moderate, and Hard. Moderate difficulty is considered the official rank level of the KITTI leaderboard. To ensure a fair comparison with previous methods, results are submitted to an official server for evaluation on the test set. AP 3 D | R 40 is used as the main metric with IOU thresholds of 0.7 for cars, and 0.5 for Pedestrians and Cyclists respectively.

Waymo. Waymo [42] assesses objects using a dual-tiered categorization: Level 1 and Level 2. The assessment is performed across three distance intervals: [0, 30), [30, 50), and [50, ∞ ) meters. Waymo employs the APH 3 D percentage metric, which integrates heading data into the AP 3 D , as a reference benchmark for evaluation.

nuScenes. nuScenes [3] comprises 28,130 training and 6,019 validation images captured from the front camera. We use validation split for cross-dataset evaluation.

4.2 Implementation Details

Our proposed MonoUNI is trained on 4 Tesla V100 GPUs with a batch size of 16 for 150 epochs. We use Adam [17] as our optimizer with an initial learning rate 1 . 25 × e -3 . Images are all resized to the same size of 960 × 512 for infrastructure side and 1280 × 384 for vehicle side. Following [37], the ROI-Align size d × d is set to 7 × 7 . Inspired by [33], we adopt the multi-bin strategy for heading angle and depth prediction in our baseline. Random crop and expand along principal points are used to achieve more physical data augmentation.

4.3 Main Results

Results of Car Category. Since there was no previous method designed for supporting both the vehicle and infrastructure sides at the same time, the results of some methods were reproduced by us or other published papers (BEVHeight [49], Rope3D [53] and DAIR-V2X [55]). As shown in Table 2, our proposed MonoUNI achieves superior performance than previous methods on infrastructure-side benchmarks, even those with extra data. Specifically, compared with BEVHeight which is the recent top1-ranked image-only method for infrastructure side, MonoUNI gains significant improvement of 17.85%/13.91% in AP 3 D and R score on Rope3D benchmark. According to DAIR-V2X, our method achieves over 21% improvement compared with BEVHeight on Moderate difficulty. For vehicle-side benchmark, although our method did not rank first, it reached a competitive result on Easy difficulty. On the Moderate and Hard difficulties, MonoUNI is slightly inferior to methods such as MonoATT [60] and MoGDE [59], probably because the gain of the 3D cube depth is weakened in the case of far-distance or severe truncation.

Results of Other Categories. In Table 3, we present the results of pedestrians and cyclists on the test set of KITTI. MonoUNI outperforms all methods by a large margin. This may be because the

introduction of depth clues promotes pedestrians and cyclists better learn spatial features. Table 4 shows the results of Big Vehicle on the Rope3D benchmark.

4.4 Ablation Study

Effectiveness of Different Components in MonoUNI on Rope3D and KITTI val set for Car category. We evaluate the effectiveness of the normalized depth and the 3D cube depth through ablations on two datasets. As shown in Table 5, the normalized depth, particularly the focal length, has a significant improvement on the infrastructure side, but only a minor effect on the vehicle side. This is due to the fact that the focal length of the camera is similar across the KITTI data, and the pitch angle is close to 0. On the other hand, the cube depth improves results for both the vehicle and infrastructure sides, providing evidence of its efficacy.

Analysis for different focal lengths. Table 6 presents the performance of MonoUNI at different focal lengths. Compared with the results in Table 1, our method has obvious benefits in solving the problem of focal length diversity. In the case of using all training data, our method can improve 19 . 53% and 20 . 43% AP respectively compared with GUPNet and SMOKE.

Rope3D

KITTI

771

Cross-dataset evaluation. Tabel 7 shows the result of our KITTI val model on the KITTI val and nuScenes [3] frontal val images, using mean absolute error (MAE) of the depth [41]. MonoUNI is better than GUPNet [27] and achieves similar competitive performance to DEVIANT [20]. This is because DEVIANT is equivariant to the depth translations and is more robust to data distribution changes.

Performance of cube depth on large dataset Waymo. In order to fully verify the effectiveness of cube depth, we verified MonoUNI on Waymo, which is a large-scale vehicle-side 3D detection dataset, as shown in Table 8.

4.5 Qualitative Results

We visualize the detection results of the MonoUNI on the both vehicle and infrastructure sides in Fig. 5. As can be seen, the MonoUNI can accurately estimate the 3D position of objects, even those not labeled by the annotators due to severe occlusion in the first row of KITTI. However, for far-distance and severely truncated obstacles, our model suffers from missed detections.

5 Conclusion

In this paper, we propose a new optimization target named normalized depth to unify monocular 3D detection for both vehicle and infrastructure sides, addressing the problem of focal length and pitch angle diversity. Furthermore, we introduce 3D Cube Depth as an additional supervision clue to improve the 3D detection performance. Experiments on five datasets (Rope3D, DAIR-V2X-I, KITTI, Waymo and nuScenes) fully demonstrate the effectiveness of our method.

6 Limitations and Future Work

Although the normalized depth unifies the optimization target of the vehicle and infrastructure sides, separate training for the two sides is still necessary instead of direct hybrid training. In the future, we aim to enable one model file to support 3D detection for both sides which would hold significant value for industrial applications, such as the domain gap problem in the vehicle-infrastructure collaborative perception and fusion. In addition, global depth clues such as depth similarity (i.e., contrastive learning) or the global topological relationship can be further developed.

References

Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection in monocular video. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16 , pages 135-152, 2020.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

Yingjie Cai, Buyu Li, Zeyu Jiao, Hongsheng Li, Xingyu Zeng, and Xiaogang Wang. Monocular 3d object detection with decoupled structured polygon estimation and height-guided depth estimation, 2021.

Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Celine Teuliere, and Thierry Chateau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10379-10388, June 2021.

Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Ziming Chen, Yifeng Shi, and Jinrang Jia. Transiff: An instance-level feature fusion framework for vehicleinfrastructure cooperative 3d detection with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 18205-18214, October 2023.

Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2020.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354-3361. IEEE, 2012.

Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In ECCV , Lecture Notes in Computer Science. Springer, 2022.

Kuan-Chih Huang, Tsung-Han Wu, Hung-Ting Su, and Winston H. Hsu. Monodtr: Monocular 3d object detection with depth-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4012-4021, June 2022.

Jinrang Jia, Yifeng Shi, Yuli Qu, Rui Wang, Xing Xu, and Hai Zhang. Competition for roadside camera monocular 3D object detection. National Science Review , 05 2023. nwad121.

Bo Ju, Wei Yang, Jinrang Jia, Xiaoqing Ye, Qu Chen, Xiao Tan, Hao Sun, Yifeng Shi, and Errui Ding. Danet: Dimension apart network for radar object detection. In Proceedings of the 2021 International Conference on Multimedia Retrieval , ICMR '21, page 533-539, New York, NY, USA, 2021. Association for Computing Machinery.

Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 1725-1732, 2014.

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Xianghao Kong, Wentao Jiang, Jinrang Jia, Yifeng Shi, Runsheng Xu, and Si Liu. Dusa: Decoupled unsupervised sim2real adaptation for vehicle-to-everything collaborative perception. In Proceedings of the 31st ACM International Conference on Multimedia , October 2023.

Jason Ku, Alex D. Pon, and Steven L. Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Abhinav Kumar, Garrick Brazil, Enrique Corona, Armin Parchami, and Xiaoming Liu. Deviant: Depth equivariant network for monocular 3d object detection. In European Conference on Computer Vision , pages 664-683. Springer, 2022.

Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206.10092 , 2022.

Zhuoling Li, Zhan Qu, Yang Zhou, Jianzhuang Liu, Haoqian Wang, and Lihui Jiang. Diversity matters: Fully exploiting depth clues for reliable monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2791-2800, June 2022.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 1-18, 2022.

Zechen Liu, Zizhang Wu, and Roland Toth. Smoke: Single-stage monocular 3d object detection via keypoint estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2020.

Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang. Autoshape: Real-time shape-aware monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15641-15650, October 2021.

Yan Lu, Xinzhu Ma, Lei Yang, Tianzhu Zhang, Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry uncertainty projection network for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3111-3121, October 2021.

Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking pseudo-lidar representation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Xinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4721-4730, June 2021.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Zhixiang Min, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Enrique Dunn, and Manmohan Chandraker. Neurocs: Neural nocs supervision for monocular 3d object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21404-21414, June 2023.

Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

J. Krishna Murthy, G.V. Sai Krishna, Falak Chhaya, and K. Madhava Krishna. Reconstructing vehicles from a single image: Shape priors for road scene understanding. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 724-731, 2017.

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3142-3152, October 2021.

Liang Peng, Fei Liu, Zhengxu Yu, Senbo Yan, Dan Deng, Zheng Yang, Haifeng Liu, and Deng Cai. Lidar point cloud guided monocular 3d object detection. In European Conference on Computer Vision , 2022.

Liang Peng, Xiaopei Wu, Zheng Yang, Haifeng Liu, and Deng Cai. Did-m3d: Decoupling instance depth for monocular 3d object detection. In European Conference on Computer Vision , 2022.

Zengyi Qin, Jinglu Wang, and Yan Lu. Monogrnet: A geometric reasoning network for monocular 3d object localization. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 8851-8858, 2019.

Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8555-8564, June 2021.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2397-2406, January 2022.

Xuepeng Shi, Qi Ye, Xiaozhi Chen, Chuangrong Chen, Zhixiang Chen, and Tae-Kyun Kim. Geometrybased distance decomposition for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15172-15181, 2021.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2446-2454, 2020.

Li Wang, Li Zhang, Yi Zhu, Zhi Zhang, Tong He, Mu Li, and Xiangyang Xue. Progressive coordinate transforms for monocular 3d object detection. Advances in Neural Information Processing Systems , 34:13364-13377, 2021.

Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages 913-922, October 2021.

Tai Wang, Xinge ZHU, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 1475-1485. PMLR, 08-11 Nov 2022.

Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Xiaopei Wu, Liang Peng, Honghui Yang, Liang Xie, Chenxi Huang, Chengqi Deng, Haifeng Liu, and Deng Cai. Sparse fuse dense: Towards high quality 3d detection with depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5418-5427, June 2022.

Tianfu Wu Xianpeng Liu, Nan Xue. Learning auxiliary monocular contexts helps monocular 3d object detection. In 36th AAAI Conference on Artifical Intelligence (AAAI) , Feburary 2022.

Lei Yang, Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang, Xinyu Zhang, and Peng Chen. Bevheight: A robust framework for vision-based roadside 3d object detection. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , Mar. 2023.

Lei Yang, Xinyu Zhang, Li Wang, Minghan Zhu, Chuan-Fang Zhang, and Jun Li. Mix-teaching: A simple, unified and effective semi-supervised learning framework for monocular 3d object detection. ArXiv , abs/2207.04448, 2022.

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng, Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In European Conference on Computer Vision , pages 17-34. Springer, 2020.

Xiaoqing Ye, Mao Shu, Hanyu Li, Yifeng Shi, Yingying Li, Guangjie Wang, Xiao Tan, and Errui Ding. Rope3d: The roadside perception dataset for autonomous driving and monocular 3d object detection task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21341-21350, June 2022.

Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21361-21370, June 2022.

Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3289-3298, June 2021.

Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. In arXiv preprint arXiv:1904.07850 , 2019.

Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang. Monocular 3d object detection: An extrinsic parameter free approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7556-7566, 2021.

Yunsong Zhou, Quan Liu, Hongzi Zhu, Yunzhe Li, Shan Chang, and Minyi Guo. Mogde: Boosting mobile monocular 3d object detection with ground depth estimation, 2023.

Yunsong Zhou, Hongzi Zhu, Quan Liu, Shan Chang, and Minyi Guo. Monoatt: Online monocular 3d object detection with adaptive token transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17493-17503, June 2023.

Zhikang Zou, Xiaoqing Ye, Liang Du, Xianhui Cheng, Xiao Tan, Li Zhang, Jianfeng Feng, Xiangyang Xue, and Errui Ding. The devil is in the task: Exploiting reciprocal appearance-localization features for monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2713-2722, October 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the monocular 3D object detection task and proposes a unified 3D detection framework for both vehicle and infrastructure sides. In particular, to unify the diversity of pitch angles and focal lengths of multiple cameras, the authors propose a unified optimization target named normalized depth. Besides, they also design the 3D normalized cube depth of obstacle to promote the learning of depth information. They conduct extensive experiments on three datasets, including Rope3D, DAIR-V2X-I, and KITTI, and get promising results on them, especially on Rope3D and DAIR-V2X-I.\n\nSTRENGTHS:\n1. The main idea is easy-to-follow and the paper is well-written.\n3. The proposed model gets SOTA performance on the infrastructure side, ie. Rope3D and DAIR-V2X-I datasets, and competitive results on the vehicle side, ie. the KITTI dataset.\n4. Code will be open-sourced.\n\nWEAKNESSES:\nThe authors claim they proposed a unified 3D detection pipeline for both infrastructure and vehicle sides. Although the proposed method shows promising peroformance on Rope3D and DAIR-V2X-I datasets, it cannot well genelized to the vehicle side. \n\n - Their proposed method can about 16 ap (moderate setting) on KITTI testing, while the existing models can get 17+, such as [1].\n - The authors only evaluate their model on KITTI, which only contains 7K images, and evaluating it on the largescale dataset such as nuScenes and Waymo is required to show the effectiveness in vehicle side.\n - In table 5, the experiments (a -> c) and (b->d) show the pitch-based depth normailization is harmful to the vehicle-centric detection.\n - Also, table 5 shows the effectiveness of the 'cube' design to the vehicle-centric detection. Note KITTI is a small dataset with sparse depth supervision and the 'cube' design densify the depth supervision in fact, which maybe the underlying reason why it works .  However, whether this design works or not when large-scale training data avaiable is still unclear. So I recommand the authors conduct more experiemnt on larger dataset again.\n\nBased on the above points, I am not sure designing a unified pipeline for both infrastructure and vehicle sides is meaningful or not, and more experiemnts are required to support the claim.\n\n[1] Online Monocular 3D Object Detection with Adaptive Token Transformer, CVPR'23\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach called MonoUNI for monocular 3D object detection of both vehicle and infrastructure sides in autonomous driving. The approach addresses the challenge of constructing algorithms for the two sides based on different prior knowledge, by taking into account the diversity of pitch angles and focal lengths. The proposed method uses a unified optimization target called normalized depth to unify the 3D detection problems for the two sides. Additionally, a 3D normalized cube depth of obstacle is developed to promote the learning of depth information and enhance the accuracy of monocular 3D detection. The paper presents extensive experiments on three widely used monocular 3D detection benchmarks, and the proposed method achieves state-of-the-art performance on all three benchmarks without introducing any extra information.\n\nSTRENGTHS:\n1. The paper proposes a novel approach that unifies monocular 3D object detection for both vehicle and infrastructure sides, which is an important problem in autonomous driving. \n2. The proposed approach addresses the challenge of constructing algorithms for the two sides based on different prior knowledge, by taking into account the diversity of pitch angles and focal lengths. The use of a new optimization target called normalized depth and the 3D normalized cube depth of obstacle as an additional supervision clue helps to improve the accuracy of monocular 3D detection. \n3. The extensive experiments on three widely used benchmarks show that the proposed method achieves state-of-the-art performance on all three benchmarks without introducing any extra information.\n\nWEAKNESSES:\n1. Lack of key details. In formula (4), the authors mentioned the use of the definition of normalized depth to simplify the depth detection task. It is unclear whether MonoUNI needs to be built on the assumption of known pitch angle and focal length during model inference. If it does, then the work does not truly address the problem of different conditions for vehicle and infrastructure sides. If it does not, the authors need to explain why adopting a unified form would reduce the difficulty of learning, and the inference in line 161 is still unconvincing.\n2.Insufficient contribution. The proposed 3D normalized cube depth is inspired by AutoShape and DID-M3D, and extends from regressing corner coordinates to regressing the depth within the foreground 3D box. The proposed method is too straightforward and lacks the necessary motivation to explain why the depth on the surface is important, which needs to be supported by experiments. Introducing surface depth may not make a significant difference compared to regressing corner depth, and may introduce additional errors due to the irregularity of foreground instances.\n3. Mismatch between the motivation and method of the paper. The motivation of the paper is to design a new depth annotation, i.e., normalized depth, to unify monocular 3D object detection for both vehicle and infrastructure sides. However, according to line 268, if the two scenarios need to be trained separately, the proposed method's significance is limited, and it seems unable to achieve the desired Mono3D scheme that unifies vehicle and infrastructure sides.\n4. Cross-dataset experiments. The paper aims to establish a unified depth detection scheme for both vehicle and infrastructure sides. It is desirable for the authors to demonstrate the model's cross-dataset capability and the effect of multi-dataset mixed training. Additionally, using extrinsic perturbation experiments on a single dataset seems to be a necessary verification method.\n5. Minor errors. Each element in the formula needs to have a corresponding explanation, such as (5).\n6. Missing relevant papers.\n[1] MonoEF: Extrinsic parameter free monocular 3D object detection.\n[2] Mogde: Boosting mobile monocular 3D object detection with ground depth estimation.\n\nQUESTIONS:\nMy main concern is that the authors lack the necessary motivation and explanation for their design of the unified depth and the cube depth. The authors need to provide a detailed explanation of the rationale and purpose of the proposed designs. Currently, the proposed design approach does not appear to have significant differences from previous works.\n\nLIMITATIONS:\nThe main limitation of this paper is that it does not implement training a single model to handle different settings across multiple datasets. Currently, the proposed approach cannot be considered as a solution to the problem of different settings between the vehicle and infrastructure sides.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an optimization target which unifies 3D detection problems for vehicle and infrastructure sides, by taking into account the diversity of camera pitch angles and focal lengths. Furthermore, the paper develops 3D normalized cube depth of obstacle to promote the learning of depth information.\nThe authors provide extensive experimental results on several monocular 3D detection benchmarks to prove the effectiveness of the proposed approach on both vehicle and infrastructure scenarios.\n\nSTRENGTHS:\n1. The authors take notice of that depth can be ambiguous under the influence of focal length given similar visual features and conduct experiments to verify their claim. \n2. They propose to decouple depth from focal length and the camera optical axis orientation, specifically, to learn a normalized depth which is unaffected by focal length and axis orientation, from which the real depth can be recovered. I think the proposed idea is novel, reasonable and also proved to be effective, which eases the model from predicting ill-defined depth from visual feature.\n\nWEAKNESSES:\nThe idea of normalized depth is good, but the article seems like semi-finished and needs to be completed carefully.\n- In sec 3, the definition of H' is missing, though it can be speculated from figure 3.\n- L#159 principle->principal.\n- Many key components of the proposed method are only described by text or even only be found in the figure.\nFor instance in sec 3.3, L#193, the authors say \"Compared with only supervising the depth of the center point and corner points, the 3D cube depth is a sufficient way to utilize the depth information\". So I guess you supervise the depth of all the points on the visible surface of the obstacle? How is it implemented? Mathematically, what are the output of the model and how are they supervised? I fail to find any formular definition of your training loss and the \"depth uncertainty\" in fig 2. As a reader, I can roughly guess the meaning but I would appreciate it if the authors could complete all the missing definitions.\n\nQUESTIONS:\n1.It is mentioned in sec 3 L#157 that the pitch angle theta can be calculated from the ground equation. I suppose the accuracy of pitch angle is severely affected by the accuracy of ground plane estimation. So I wonder how the parameters of the ground plane equation are estimated.\n2.The derivation of the normalized depth are done in a degenerated 2d view instead of 3d. It would be better if the authors can provide simple explanation or prove that neglecting the effect of camera yaw angle is reasonable.\n\nLIMITATIONS:\nThis paper mentioned that one model file to support 3D detection for both sides rather than separate training for the two sides. This is a work worth investing in, and it will solve the problem of vehicle-road collaborative perception very well. I look forward to the subsequent output of the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified architecture for vehicle and infrastructure-based monocular 3D object detection network. At its core, the paper puts forth the concept of normalized depth that is independent of camera intrinsic focal length and extrinsic pitch angle w.r.t the ground plane. As such, the network is applicable to cameras with varying focal length and mounting angle, while not affected by the ambiguity. Following DID-M3D, the framework decomposes the center depth into cube depth and the so-called bias depth. The experiments demonstrate significantly better performance in Rope3D and DAIR datasets.\n\nSTRENGTHS:\n1. The paper presents interesting new insights towards the problem of monocular 3D object detection under varying focal length and pitch mounting angle. The varying focal length problem has been tackled with normalized depth by existing works in the context vehicle-based 3D object detection, but handling pitch angle is as yet under-explore. The paper for the first time derived the normalized depth of object center that is independent of the pitch angle. \n2. The method is simple yet effective. The experiments demonstrate that the proposed method yields significantly superior performance over the state-of-the-art, as shown in Table. 2.\n3. The proposed framework is a unified framework for both vehicle-based and infrastructure-based 3D detection. This open doors to the new possibility in combining the research and data across these two domains. While the method currently requires separate training on each, it holds the potential for joint training that may improve both.\n\nWEAKNESSES:\n1. In Line 158, the paper makes approximation of the angle \\delta by replacing v_p with v_c, but the paper does not discuss the implication of this approximation in practice. It seems the approximation error could possibly be large especially for nearby objects and with smaller pitch angle in camera mounting. How would this impact the performance? In addition, why not let the network to predict v_p as well? If the network is able to prediction of position of object center, i.e. v_c, what prevents it from predicting the position of bottom center v_p?\n2. While the performance gain on infrastructure cameras are significant, it is small in vehicle cameras, i.e. KITTI. In particular, the paper only compares with [1,24,44,29] while omitting other stronger existing methods, such as CMKD ECCV 2022 and LPCG ECCV 2022. It is fine that the method does not outperform these methods given the unique advantage of being a unified framework, but the paper should acknowledge this for readers’ better understanding. There are also concurrent works such as NeurOCS and Mix-Teaching with better accuracy, which are not necessary to compare against but would be good to discuss in related works for completeness.\n\nQUESTIONS:\nThe motivation and the implication of approximating v_p with v_c are the main question I have. I hope the authors could address this in the rebuttal.\n\nLIMITATIONS:\nThe paper has discussed the paper properly, i.e. the method currently requires separate training on vehicle and infrastructure dataset.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}