MoGDE: Boosting Mobile Monocular 3D Object Detection with Ground Depth Estimation
Authors: Yunsong Zhou
Quan Liu
Hongzi Zhu
Yunzhe Li
Shan Chang
Minyi Guo
Venue: NeurIPS 2022 Accept (2022)
Keywords: Monocular 3D Object Detection
Transformer
Ground Depth Estimation.

Abstract: Monocular 3D object detection (Mono3D) in mobile settings (e.g., on a vehicle, a drone, or a robot) is an important yet challenging task. Due to the near-far disparity phenomenon of monocular vision and the ever-changing camera pose, it is hard to acquire high detection accuracy, especially for far objects. Inspired by the insight that the depth of an object can be well determined according to the depth of the ground where it stands, in this paper, we propose a novel Mono3D framework, called MoGDE, which constantly estimates the corresponding ground depth of an image and then utilizes the estimated ground depth information to guide Mono3D. To this end, we utilize a pose detection network to estimate the pose of the camera and then construct a feature map portraying pixel-level ground depth according to the 3D-to-2D perspective geometry. Moreover, to improve Mono3D with the estimated ground depth, we design an RGB-D feature fusion network based on the transformer structure, where the long-range self-attention mechanism is utilized to effectively identify ground-contacting points and pin the corresponding ground depth to the image feature map. We conduct extensive experiments on the real-world KITTI dataset. The results demonstrate that MoGDE can effectively improve the Mono3D accuracy and robustness for both near and far objects. MoGDE yields the best performance compared with the state-of-the-art methods by a large margin and is ranked number one on the KITTI 3D benchmark.

MoGDE: Boosting Mobile Monocular 3D Object Detection with Ground Depth Estimation
Yunsong Zhou
Quan Liu'
Hongzi Zhu
Yunzhe Li
Shan Chang 2 Minyi Guo
Shaanhhii Jiao Tong University DDonghua University
htw.ai/u.....ogg.i.uneiieiiiiii l changshan @ dhu.edu.cn
Abstract
Monocular 3D object detection (Mono3D) in mobile settings (e.g., on a vehicle, a drone, or a robot) is an important yet challenging task. Due to the near-far disparity phenomenon of monocular vision and the ever-changing camera pose, it is hard acquire high detection accuracy, especially for far objects. Inspired by the insight that the depth of an object can be well determined according to the depth of the ground where it stands, in this paper, we propose novel Mono3D framework, called MoGDE, which constantly estimates the corresponding ground depth of an image and then utilizes the estimated ground depth information to guide Mono3D. To this end, we utilize pose detection network estimate the pose of the camera and then construct feature map portraying pixel-level ground depth according to the 3D-to-2D perspective geometry. Moreover, to improve Mono3D with the estimated ground depth, we design RGB-D feature fusion network based the transformer structure, where the long-range self-attention mechanism is utilized effectively identify mround-contactingg points and pin the corresponding ground depth to the image feature map. We conduct extensive experiments on the real- world KITTI dataset. The results demonstrate that MoGDE the Mono3D accuracy and robustness for both near and far objects. MoGDE yields the best performance compared with the state-of-the-ar methods by large margin and ranked number one on the KITTI 3D benchmark.
effectively improve
1 Introduction
Building on the promising progress achieved 2D object detection in recent years [41, 40], 3D object detection, particularly academia as an important component many applications, ranging from autonomous vehicles [17] and drones, to robotic manipulation and augmented reality applications. Compared to LiDAR-based [12, 36, 42, 43, 23, 63] and stcreo-based [10, 11, 21, 35, 38, 54] methods, a much cheaper, more energy-efficient, and easier-to-deploy alternative, i.e., monocular 3D object detection (Mono3D), remains an open and challenging research field. A practical Mono3D detector for moving agents should meet the following two requirements: 1) the 3D bounding box produced by the Mono3D detector should be accurate enough, not only for near objects but also for very distant objects, to secure, for instance, high-prioric driving safety applications; 2) the Mono3D detector should remain robust in mobile scenarios, where the camera pose inevitably changes along with the movement of a mobile agent.
moving agents, has received increasing attention from both industry and
In the literature, resent Mono3D methods with complex network structures [62, 33, 52, 37]48] have achieved considerably high accuracy for near objects, but the predicted 3D bounding boxes for far objects are often ill- ill-posed due to the lack of depth cues. This huge disparity between near and far objects lies in the nature of monocular vision. Specifically, depicted in Figure I (a), equal
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pase
BP
J
(a) Near-far disparity phenomenon
Camera pose variation
(c) Self-attention
Figure I: (a) Equal distances of different depths from the camera (e.g. Asi = A22) have a distinct number of pixels in the image (e.g., Auj > Aug and Av1 > A02), which is referred to as the near-far disparity phenomenon of monocular vision, making the detection of far object susceptible to pixel rounding errors. (b) The camera pose variance, caused by the movement of mobile agent, can eventually result in a large offset both in form of 3D boxes and in the bird's 's eye view. (c) Each color block represents its attention value with the centroid of the vehicle. The attention mechanism of the transformer network can be well leveraged for this long-range relationship modeling.
distances of different depth from the camera (e.g., AZi = Az2) have a distinct number of pixels in the image (e.g., Auj Auz and Avi > Au2), which makes the pixel rounding errors have a non-negligible impact for detecting far objects. Furthermore, as illustrated in Figure 1(b), the camera pose variance can eventually result in large offset both in form of 3D boxes and in the bird's eye view [15]. To the best of our knowledge, existing Mono3D methods such as geometric constraint based [9, 34, 6, 533, LiDAR based [29, 30, 50, 54, 37, 16, 28]. and pure image based [41, 45, 61, 3, 25, 49, 13, 22, 4, 62, 60, 27, 59], have not taken into account the issue of inevitable camera pose changes in mobile scenarios.
In this paper, we propose a novel Mono3D method, called MoGDE, which fixates on improving detection accuracy and robustness in mobile settings. We have one key insight that the depth of an object in 3D space can be well determined uccording to the depth of the ground where it stands. Given the pinhole model and the pose of camera, the depth of each pixel corresponding to the ground can be accurately derived. Based on this insight. the core idea of MoGDE is first to constantly estimate the ground depth while moving and then to utilize the estimated ground depth information to guide a Mono3D detector.
There are two main challenges in designing MoGDE. First, how to detect varying camera pose (e.g., the pitch and roll angles) from an image, which is dynamically changing in mobile scenes, and how to ultimately obtain accurate ground depth information are non-trivial. It is clear that different camera poses correspond to distinct ground depth estimates. To tackle this challenge, we introduce a pose detection network to extract the vanishing point and horizon information in an image to estimate the instant camera pose corresponding to this image. After the view direction of the camera is decided, we then construct a feature map portraying pixel-level depth clues. Specifically, we envision virtual 3D scene containing only the sky and the ground and project this virtual scene to an image where each pixel is associated with depth uniquely derived according to the 3D-to-2D 2D perspective geometry. Therefore, MoGDE can obtain the dynamic ground depth information as prior knowledge for guiding Mono3D.
Second, how to incorporate the estimated ground depth into the image features to enhance the detection accuracy is challenging. Based on our aforementioned insight, it is essential for the Mono3D detector to identify those grownd-contacting points of an object on the image. For example in Figure 1 (a), these blue dots denote the counddcontacting points of a vehicle. To this end, we design an RGB-D feature fusion network based on the transformer structure to tie the ground depth feature to the image feature. Specifically, captures the feature of pixels close to the centroid of an object and identifies those ground-contacting points using the attention mechanism. It then attaches depth values with weights to compute new feature map containing object location information. As a result, accurate 3D detection results can be obtained via conventional Mono3D detector using the fused feature map.
illustrated in Figure I(c), the feature fusion network
Experiments on KITTI dataset [17] demonstrate that our method outperforms the SOTA methods by a large margin. Such a framework can be applied to existing detectors and is practical for industrial applications. The proposed MoGDE is ranked number one on the KITTI 3D benchmark by submission. The whole suite of the code base will be released and the experimental results will

---

be posted to the public leaderboard. We highlight the main contributions made in this paper as follows: 1) A novel Mono3D detector in mobile setting is introduced, leveraging the dynamically estimated ground depth as prior knowledge to improve the detection accuracy and robustness for both near and far objects. 2) transformer-based feature fusion network designed, which utilizes the long-range attention mechanism to effectively identify ground-contacting points and pin the corresponding ground depth to the image feature map. 3) Extensive experiments on the real-world KITTI dataset are conducted and the results demonstrate the efficacy of MoGDE.
2
Related Work
Monocular 3D Object Detection. The monocular 3D object detection aims to predict 3D bounding boxes from a single given image. Existing Mono3D methods can be roughly divided into the following
three categories. 1) Geometric constraint based methods: Extra information of prior 3D vehicle shapes is widely used, such as vehicle computer aided design (CAD) models [9, 34, 6, 53, 26] or key points [1]. By this means, extra labeling cost is inevitably required. 2) Depth assist methods: A stand- alone depth map of the monocular image is predicted at the first stage. Such prior knowledge can be derived in various ways, such as a depth map generated by LiDAR point cloud (or Pseudo-LiDAR) [50, 29, 8, 391, monocular depth predictors [37, 28, 16]. or disparity map generated by stereo cameras [54]. However, such external data is not easily available in all scenarios. In addition, the inference time increases significantly due to the prediction of these dense heatmaps. 3) Pure image-based methods: Without requiring extra side-channel information, such methods [20, 18, 24, 44] take only a single image as input and adopt center-based pipelines following conventional 2D detectors [41, 61, 45]. M3D-RPN [3] reformulates the monocular 3D detection problem as a standalone 3D region proposal network. With very few handcrafted modules. SMOKE [25] and FCOS3D [49] predict 3D bounding box by combining& a concise one-stage keypoint estimation with regressed 3D variables based on CenterNet [61] and FCOS [45], respectively. To further strengthen monocular detectors, current SOTA methods have introduced more effective but complicated geometric priors. MonoPair [13] improves the modeling of occluded objects by considering the relationship of paired samples and parses their spatial relations with uncertainty. Kinematic3D [4] proposes a novel method for monocular video- based 3D object detection, which uses kinematic motion to improve the accuracy of 3D localization. MonoEF [62] first proposes a novel method to capture the camera pose in order to formulate detectors that are not subject camera extrinsic perturbations. MonoFlex [60] conducts an uncertainty secettaint--uuddedd depth ensemble and categorizes different objects for distinctive processing. GUPNet [27] solves the error amplification problem by geometry-guided depth uncertainty and collocates hierarchical learning strategy to reduce the training instability. MonoDETR [59] introduces a simple monocular object detection framework that makes the vanilla transformer to be depth-aware and enforces the whole detection process guided by depth. The above geometrically dependent designs largely promote the overall performance of center-based methods, but the underlying problem still exists, namely, the detection accuracy for distant objects is still not satisfactory.
Object Detection with Transformer. 2D object detectors [61, 45] have achieved excellent per- formance in recent years but are equipped with cumbersome post- post-processing, e.g. non-maximum suppression (NMS) [41]. To circumvent it, the pioneering work DETR [5] constructs a novel and simple framework by adapting the powerful transformer [47] to the field of vision detection. DETR detects objects on images by encoding-decoding paradigm, which improves the detection performance by using the long-range attention mechanism. DETR is further enhanced by designing deformable attention [64]. placing anchors [51], setting conditional attention [31], embedding dense prior [57], and so on [32]. Some recent works have tried to apply transformer to some other tasks related to monocular scene reconstruction, depth prediction, etc. Transformerfusion [2] leverages the trans- former architecture so that the network learns to focus on the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. MT-SfMLLarner [46] first demonstrates how to adapt vision transformers for self-supervised monocular depth estimation focusing on improving the robustness of natural corruptions. While these methods have made a demonstration of how to apply a transformer to a monocular camera model [14, 55, 56). they all rely on other branches (either the environment reconstruction or the depth map), which will not be available in typical Mono3D task based on RGB images.
3

---

Ground Depth Esi.
Seare construction
Pase Network
Depth gradient
Ground Depth
Vaniahing
Ground Depth Fusion
Monocular Det.
Input image
Feature map
3D resulls
hoatmon
2D detaction
dim.
offset
rntation
depth
Ti Transformer
Network
3D deleciion
Figure 2: MoGDE consists three main components, i.e., ground depth estimation (GDE), ground depth fusion (GDF) and monocular 3D detection (M3D). In GDE, the pose network predicts the ground plane as well as the vanishing point. The derived pose information then used to construct a virtual scene and obtain pose-specific ground depth feature map. In GDF, transformer network is leveraged to fuse the image features with the ground depth feature map, resulting ground-aware fused feature map. M3D employs standard Mono3D detector the underlying detection core.
Our MoGDE inherits DETR's s superiority for non-local encoding and long-range attention. Specif- ically, we endow the transformer to be ground-aware by pinning ground depth to image features leveraging the encoder-decoder architecture to improve the detection accuracy for far objects.
3 Design of MoGDE
3.1
Overview
The core idea of MoGDE to utilize dynamically estimated ground depth information to improve Mono3D so that goals can be achieved: 1) superior ground-aware image features obtained to increase Mono3D accuracy for both near and far objects; 2) the impact of camera pose variation is diminished to enhance Mono3I) robustness in mobile settings. Figure 2 depicts the architecture of our framework. Specifically, MoGDE first adopts the DLA-34 [58] as its backbone, which takes monocular image size (W x Hl x 3) input and outputs feature map of size (W, x 11, x C) after down-sampling with an s-factor. Then, the feature map is fed into three components as follows:
Ground Depth Estimation (GDE). GDF mainly integrates two functions, i.e., camera pose detection (CPD), and virtual scene construction (VSC). Specifically, CPD estimates the camera pose (i.e., the pitch and roll angles) based on the predicted vanishing point and ground plane extracted by pose detection network. VSC establishes a 2D) ground depth feature map based on a pose-specific virtual 3ID scene containing only the sky and the ground.
Ground Depth Fusion (GDF), GDF leverages the attention mechanism of a transformer network to fuse the image features with the ground depth feature map, resulting superior ground-aware fused feature map.
Monocular 3D Detection (M3D). MoGDE employs GUPNet [27]. a SOTA CenterNet [61] based SOTA monocular 3D object detector as its underlying detection core.
3.2
Ground Depth Estimation
3.2.1
Camera Pose Detection
In order to generate ground depth estimate, key to detect the camera pose given an image feature map. We have the following proposition:
Proposition Given a benchmurk camera coordinate system p which is aligned with the ground plane coordinate systems, and the current camera coordinate system P which is not aligned with po due to camera movement, there exists a transformation matrix A between P and po that can be uniquely determined by pitch 6p and roll 6p angle changes of the camera.)

---

Therefore, we introduce the subsequent neural network to learn the pitch 6g and roll Op angle changes of the camera when the camera coordinate system changes from p to P Specifically, in addition to the regular regression tasks in CenterNet [61] based network, we introduce regression branch for pose detection following MonoEF [62]. Since the camera pose is a feature that is implicit for images, we chose two physical quantities with clear meaning for detection: the ground plane (associated with roll angle) and the vanishing point (associated with pitch angle). Following the state-of-the- odometer framework in DeepVP [7]. we represent regression task with L1 loss as:
RD y vp] p (H):
(1)
pose A g ($ KD vvp)ll,
where H is the input image feature; frose is the CNN architecture used for horizon and vanishing point detection in the work [19]: y RD and Vvp are the predicted ground plane and vanishing point; g mapping function g (R2 TR.2) + A3x3 which turns pitch and roll angles into matrix A. The regression network is supervised by pose and can be trained jointly with other Mono3D branches.
3.2.2 Virtual Scene Construction
We envision such a virtual scene, where there is vast and infinite horizontal plane in the camera coordinate system Pa and have the following proposition:
Proposistion 2: Given the camera coordinate system Pt, the virtual horizontal plane can be projected on the image plane of the camera according to the ideal pinhole camera model and the depth corresponding each pixel the image is determined by the camera intrinsic parameter K. and pose matrix A from p0 to P
We first construct the ground depth feature map in the camera coordinate system P Specifically, as illustrated in Figure for each pixel on the depth image locating at (u e.)) with an estimated depth g0, it can be projeproee to a point (zldd Dld 29) in the 3D scene:
20
Cy 20
3d
V3d
fy
where fa and fy are the focal lengths represented in the units of pixels along the x- and y-axis of the image plane and C₂ and Cy are the possible displacement between the image center and the foot point. These are referred to as the camera intrinsic parameters K. We omit the camera extrinsic T for the sake of simplicity, and the depth corresponding to each pixel on the image is solely determined by the camera intrinsic parameter K under p
Assume that the elevation of the camera from the ground, denoted as EL, is known (for instance, the mean height of all vehicles in the KITTI dataset, including ego vehieles, 1.65m [17]), the depth of a point on the depth feature map (u", o²) can be calculated as:
fy EL
0
(3)
6 Cy
Note that (3) is not continuous when the point is near the vanishing point, i.e., 60 = Cy. and does not physically hold when 6d < Cg. To address this problem, similar to the KITTI stereo setup, we encode the depth gradient value as an associated feature map using a virtual stereo setup with baseline B = 0.54m. We represent the ground depth d in the following form:
J - Cyy
d = ReLU(ff B fy -,,-,L+ -EL+
(4)
where b is constant to prevent the value of d from being too large. The ReLU activation is applied to suppress ground depth values smaller than zero, which not physically feasible for monocular cameras. As result, the ground depth feature map becomes spatially continuous and consistent.
Finally, to obtain the ground depth feature map in P the model needs to convert the 3D coordinate system first, and then just apply (4). We omit the formula derivation due to page limitation.
5

---

Optical axis
Virtual image plane
Encoder
Decoder
Output
Add Norm
Object
Ty
(ox
Add & Norm
Core-Based FFN
20 Reshape
po
Reshape
Add & Narm
Camera
EL
20
2p
Add Norm
Muli-teed SeffAAtention
Graund plane
Add & Nerr
Figure 3: Perspective geometry for ground depth estimation. In the camera coordinate system po, given the camera intrinsic param- eters K and the elevation of the camera from the ground EL, the depth of point on the ground depth feature map can be calculated. Moreover, by estimating the transformation matrix A between oo and an arbitrary P ground depth feature map in P can obtained. To utilize the ground depth feature, is key to locate
XN
Muti Head Seff Amention
Image features Positional encoding Location queries
The architecture of ground-aware trans- former. The encoder uses self-attention to encode the non-local mutual correlation of image pixels (i.e., object centers and ground points). The ground depth estimate used to generate location queries which are thereby fed to the decoder along with the location encoding. The cross-attention in the decoder prompts each query to consider the image and depth features
Figure 4:
points of an object (e.g.. the dark point) to get accu- rate depth (e.g., ). On the contrary, misuse of the depth in the ground depth feature cor- responding to other points on the object (e.g., the bright point) leads obvious depth esti- mation error (e.g., 2²).
of its associated points.
3.3 Ground Depth Fusion
In real-world scenarios, as depicted in Figure 3, objects have height. To fuse the image feature and the ground depth feature, it is key to locate ground-contacting points of object (e.g., the dark point) to get an accurate depth (e.g., =²)). On the contrary, misuse of the depth in the ground depth feature corresponding to other points the object (e.g., the bright point) leads to obvious depth estimation error (e.g., ).). Specifically, the relation between the estimated depth of an object g and the pixel displacement locating ground-contacting points, denoted as Ty can be calculated as,
EL fy 24 EL fy
(5)
20.1 T
It can be seen from (5) that Tr, can cause inaccurate 2" However, how to acquire Ty is non-trivial,
Inspired by the great success of transformer 64, 51, 31, 57, 32] in adaptive long-range relational modeling, we propose ground-awar feature fusion method based on transformer structure as depicted Figure 4, leveraging its attention mechanism automatically locate goound-contcciigg points of an object and fuse the corresponding depth feature with the image feature of that object.
Encoder. Our transformer encoder aims encode the correlation between image features using a self-attention mechanism. The input of the transformer encoder is the flattened image features Hing NxC with position encoding and the output is the embedding vectors H. R to be sent to the decoder. Following the self-attention pipeline, given the input matrix calculated from the image features: query E R key K E NxC and value V E N with sequence length N W x H, the output of + 1-th layer of self-attention can be briefly formulated as:
Q', K. V Embedding(HH, W 0r W W!), imgg
Attention Q', K. v = softmax (Q'K' T /VC) M'V!
(6)
img
Here, M is the mask used to constrain the visible range of attention. The introduction M' is to take advantage of the ground-aware property (i.e., the depth of each object should be related to the

---

depth of the object's location) so that each pixel will only consider information within window around that location. The encoded feature obtained through then re-transformed into a 2D feature map format and fed into convolution- feed-forward network (FFN). The 2D reshape as well as convolution-based FFN are necessary because image data is two-dimensional, unlike one-dimensional serialized data.
sellf-attention operation is
Decoder. The proposed transformer decoder aims to determine for each location its depth information, using the cross-correlation between the ground depth and the image features. We propose utilizing the ground depth as the location query the decoder instead of learnable embedding (object query), which is different from the common usage in previous encoder-decode vision transformer works 646. The main reason that the simple learnable embedding is hard to fully represent the object's property and handle complex depth variant situations in the Mono3D task. In contrast, plentiful dwste cues are hidden the ground depth features, which will give the transformer a baseline estimate of the expected depth at each location. To this end, the decoder can leverage the power of cross-attention in the transformer to efficiently model the correlation between the target pixel point and the point of interest (i.e., the grounded point), thus achieving the ground-awareness for higher performance.
Specifically, the input of the decoder is the flattened ground depth Haep E RN-x1 with position encoding and embedding vectors Hg E NC obtained from the encoder. In addition, the output the aggregated feature map H E IR Nx The ground depth is first embedded upon the standard self-attention architecture following (6). For the cross-attention module, its input Q is derived from the self-attention part upstream in the decoder, and its K is derived from the encoder. The input V is a concatenation of two sources from both the encoder and decoder. The purpose of this concatenation to make the decoder take into account both the information from the image and the depth during decoding.
4 Performance Evaluation
We conduct experiments on the widely-adopted KITTI3D dataset and KITTI Odometry dataset [17]. We report the detection results with three-level difficulties, iee. easy, moderate, and hard, in which the moderate scores are normally for ranking and the hard category is generally distant objects that is difficult to distinguish.
4.1 Quantitative and Qualitative Results
We first show the performance of our proposed MoGDE on KITTI 3D object detection benchmark for car. Comparison results with other state-of-the-ane (SOTA) monocular 3D detectors are shown in Table For the official rest set, we achieve the highest score for all kinds of samples and are ranked
Test, AP3o Mod.
Test, APsEV Mod.
Method
Extra data
Fasy
Hard
Fasy
Hard
PatchNet [28]
15.68
11.12
10.17
22.97
16.86
14.97
Depth
D4LCN [16]
16.65
11.72
9.51
22.51
16.02
12.55
Kinematic3D [4]
falti-frames
19.07
12.72
9.17
26.69
17.52
13.10
MnnoRUn [8]
19.65
12.30
10.58
27.94
17.34
15.24
Lidar
CaDDN [39]
19.17
13.41
11.46
27.94
18.91
17.19
AutoShape [26]
CAD
22.47
14.17
11.36
30.66
20.08
15.59
SMOKE [25]
14.03
9.76
7.84
20.83
14.49
12.75
MonoFlex [60]
19.94
13.89
12.07
28.23
19.75
16.89
None
GUPNet [27]
20.11
14.20
11.77
MonoDETR [59]
23.65
15.92
12.99
32.08
21.44
17.85
MoGDE (Ours)
27.07
17.88
15.66
38.38
25.60
22.91
Improvement
v.s. second-best
43.42
+1.96
+2.67
+6.30
+4.16
+5.06
Table APAo scores(%) of the car category on KITTI test set 0.7 IoU threshold referred from the KITTI benchmark website. We utilize bold to highlight the best results, and color the second-best ones and our performance gain over them in blue. Our model is ranked NO. on the benchmark.
http::/wwwww..i.ts.....dddasssss object benchmark=3d
7

---

Figure 5: Qualitative results KITTI Odometry dataset. The predicted 3D bounding boxes of our proposed MoGDE are shown in the first row. The second row shows the detection results the bird's eye view (z-direction from right to left). The green dashed boxes and red solid boxes are the prediction results of our MoGDE and the comparison baseline (GUPNet visualizes the results of the attention map in the transformer's
the ground truth, and the blue
[27]), respectively. The third encoder, where the purple point is the location of the query point; the yellow dashed box is the range of the encoder's mask; the brightness of the image represents the attention value between the query point and that pixel.
No. I among all existing methods with different additional data inputs on metrics. Compared to the second-best models, MoGDE surpasses them under easy, moderate, and hard levels respectively by +3.42, +1 19,9, and +2.67 in AP3D. especially achieving a significant increase (17%) in the hard level. The comparison fully proves the effectiveness of the proposed oracle fusion for images with prior depth knowledge.
Figure shows the qualitative results on the KITTI Odometry dataset. Compared with the baseline model without the aid of ground depth, the predictions from MoGDE are much closer the ground truth, especially for distinct objects. It shows that the consideration of sight-based supporting depth clues can help locate the object precisely.
4.2 Ablation Study
Pose -guided
Conv. Fusion
Tran. Fusion
Effectiveness of each proposed component. In Table conduct ablation study analyze the effectiveness of the proposed components: (a) Baseline: only using image features for 3D object
Fasy
Mod.
Hard
22.76
16.46
13.72
22.78
14.04
22.82
17.22
14.51
detection, i.e., without concerning posed variance and proposed ground-aware modules. (b) Consid- ering the camera pose variations implied in the images, we use the method described in [62] to apply a "projection transform" to the input image to remove the perturbations.
(d)
22.93
18.42
15.46
23.07
15.73
(1)
23.35
20.35
17.71
Table 2: Effectiveness different components
of approach the KITTI val set for car category. The first column is whether the model takes into account the pose variance. The second and third columns show which way the model chooses fuse the ground depth information.
(c) Considering the use of ground plane clues, generate a depth oracle about the scene (assuming constant pose) using a convolutional neural network. (d) With the proposed ground-aware transformer, this model has the ability to model mangrange relationships of pixels. (e) Full model except that we use a convolutional neural network for oracle fusion. (f) Full model (MoGDE).
First, we can observe from (a + b, c - e, and + f) that there is an implicit uncalibrated pose variation in the KITTI dataset, and considering it is necessary to improve the detection accuracy. Besides, by observing (b - e), we illustrate that leveraging ground depth brings an improvement in accuracy in hard level, but the improvement is limited because fusion by convolution is clumsy.

---

Poss Var.
Mathod
Easy
Val. AD
Mod.
Hard
Easy
Val. Aloa
Mod.
Hand
Methol
Easy
Val, AP30
Easy
Val, APssy
w/o
20.64
14.87
12.47
27.97
20.78
17.29
MED RPN [3]
14:53
11:07
3.65
20.85
15.62
11.88
Tiny
w/
29.86
MSD-RPP-
19,8
16,84
17.39
Iep.
+1.66
44.55
+3.50
Inp.
+5.32
45.77
+5.97
44.31
45.03
45.51
w/b
17.76
12.98
10.78
24.43
18.05
15.46
MonaPair (13)
16.28
12:30
10.42
24.12
18:17
15.76
Medium
w/
21,86
19.08
16.61
29.23
23.70
22.84
MonaPair Ours
19.20
15.42
13.16
27.33
21.71
18.68
Imp.
+4. 10
+6.10
+5.83
+4.81
+5.64
+7.38
Imp.
+2.92
+3.12
+2:74
43.21
43.54
42.92
w/o
13.29
3.05
18.05
15:34
11.57
Kinematic [4]
15.10
Targe
28.33
22.98
22.06
Kinemattic 3D Owrs
29.80
22,80
17.96
Ingp
+7.81
48.75
+8.05
+10.22
+9.64
+10.48
Inp.
+1.83
+2.44
+2.30
+1.97
+3.08
+2.86
Table 3: Robustness test of our approach on the KITTI val set for car category. Tiny, medium, and large correspond to three different degrees of pos- ture change, i.e., the camera pitch and roll angles vary with Gaussian distribution with mean 0 and
Table 4: Extension of MoGDE to existing image- only monocular 3D object detectors. We show the AP4o scores(%) evaluated on KITTI3D val set. +Ours indicates that we apply the GDE and GDF modules the original methods. All models benefit from the MoGDE design.
standard deviation 1, 2, and 3, respectively.
In contrast, (c - f) indicates the effectiveness of the transformer, which helps the model to understand the long-range attention relationship between pixel points and the ground plane.
Visualization of attention. To facilitate the understanding of our ground-aware transformer, we visualize the depth self-attention map in the encoder and paint the query points red and mask region yellow in the third row Figure 5, As shown the figure, it can be seen that, within the relevant region of each query, areas that are interfacing the object and the ground have the highest attention scores, In contrast, for non-ground pixels of the object, the lower attention values indicate that the query is not relevant to them, even if they have similar image features and geographically adjacent. This implics that under the transformer's s attention mechanism guidance, the query is able to borrow depth information from regions of interest (i.e., the ground planc), which helps the fused feature map produce more accurate prediction results.
Simulation experiments on robustness. In order to verify the robustness of our proposed MoGDE against camera pose variance, we set three cases of variances (tiny, medium, and large) to compare the accuracy degradation of MoGDE with that of the baseline. In Table 3, it can be noticed that the baseline is quite sensitive to pose variance, with very severe performance degradation, while our model only has a slight performance drop. Moreover, our model performs more robustly especially in the hard case, gaining higher performance improvement. This demonstrates the effectiveness of our proposed pose- specific ground depth in handling camera pose variance for mobile scenes.
Plugging into existing methods. Our proposed approach is flexible to extend to existing image-only Mono3D detectors. We respectively plug the Ground Depth Estimation and the Ground Depth Fusion components to three popular Mono3D detectors, which is shown in Table 4. can be seen that, with the aid of our proposed ground depth fusion, these detectors can achieve further improvements on KITTI3D vul set, demonstrating the effectiveness and flexibility of our approach. Particularly, MoGDE-enabled models tend to achieve more performance gains on the hard category. For example, for Kinematic3D, the AP AAp/VP gain is +1. +1.83/+1. .97 on the easy category and +2.30/+2.8 86 on the hard category.
Conclusion
In this paper, we have proposed a Mono3D framework, called MoGDE, which can effectively utilize the estimated ground depth as prior knowledge to improve Mono3D in mobile settings. The advantages of MoGDE are two-fold: 1) it can significantly improve the Mono3D accuracy, especially for far objects, which is an open issue for Mono3D; 2) it can improve the robustness of Mono3D detectors when applied in more appealing mobile applications. Nevertheless, MoGDE still has two main limitations as follows: 1) it heavily relies on pose detection, which directly affects the accuracy of the ground depth estimation; 2) also counts on the detection of ground-contacting points. In cases when such points are uncertain or ambiguous due to occlusion and truncation, it is hard for the proposed ground-aware feature fusion method to obtain accurate results. These limitations also direct our future work. We have implemented Mono3D and conducted extensive experiments on the real-world KITTI dataset. MoGDE yields the best performance compared with the sfate-of-the-an methods by large margin and is ranked number one on the KITTI 3D benchmark.

---

Acknowledgements
This research was supported in part by National Key RD Program of China (Grants No. 2018YFC1900700, National Natural Science Foundation of China (Grants No. 61872240 and 61972081), and the Natural Science Foundation of Shanghai (Grant No. 22ZRI400200).
References
[1] Ivan Barabanau, Alexey Artemov, Evgeny Burnaev, and Vyacheslav Murashkin. Monocular 3D Object Detection via Geometric Reasoning on Keypoints. 090iv [cs], May 2019, arXiv: 1905.05618.
[2] Aljaz Bozic, Pablo Palafox. Justus Thies, Angela Dai, and Matthias Nießner. FransformerFusion:: Monocu- lar RGB Scene Reconstruction using Transformers. Advances in Neural Information Processing Systems,
34, 2021.
[3] Garrick Brazil and Xiaoming Liu. M3D-RPN: Monocular 3D Region Proposal Network for Object Detection. In The IEEE International Conference Computer Vision (ICCV). October 2019, 2, 3.
[4] Garrick Brazil, Gerard Pons- Moll. Xiaoming Liu, and Bert Schiele. Kinematic 3D Object Detection in
Monocular Video. arXiv preprint (0Xiv:0077099540 2020. 2, 7,9
[5] Nicolas Carion. Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. In European conference computer vision,
pages 213-222 213.229. Springer, 2020. 3, 6,6,
[6] Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Céline Teulière, and Thierry Chateau. Deep MANTA: A Coarse-to-Fine Many- Task Network for Joint 2D and 3D Vehicle Analysis from Monocular Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2040-2049, 2017. 2,
[7] Cain- Chang, Jiaping Zhao, and Laurent Itti. DccpVP: Deep Learning for Vanishing Point Detection Million Street View Images. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8. IEEE, 2018.
[8] Hansheng Chen, Yuyao Huang. Wei Tian, Zhong Gao, and Lu Xiong. MonoRUn:
Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation. Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages (U379-10388, 2021. 3.
[9] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3D Object Detection for Autonomous Driving. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2147-2156, Las Vegas, NV. USA. June 2016. IEEE, 2,
[10] Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3D Object Proposals for Accurate Object Class Detection. In Advances in Neural Information
Processing Systems, pages 424-432, 2015. 1
[11] Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Huimin Ma. Sanja Fidler, and Raquel Urtasun. 3D Object Proposals Using Stereo Imagery for Accurate Object Class Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): (((5))1259-1277)) 1272, May 2018. 1
[12] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-View 3D Object Detection Network for Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
pages 1907-1915, 2017. 1
[13] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 12093-12102, 2020. 2, 3,3.9
[14] Zeyu Cheng. Yi Zhang, and Chengkai Tang. Swin-Depth: Using Transformers and Multi-Scale Fusion for Monocular- Based Depth Estimation. IEEE Sensors Journal, (1(23):26912-2699 2021,
[15] Tom van Dijk and Guido de Croon. How Neural Networks See Depth in Single Images? Proceedings
of the IEEE/CVF International Conference on Computer Vision, pages 2183-2191, 2019. 2
[16] Mingyu Ding. Yuqi Huo, Hongwei Yi, Zhe Wang. Jianping Shi, Zhiwu Lu, and Ping Luo. Learning Depth- Guided Convolutions for Monocular 3D Object Detection. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition Workshops, pages 1000-1001, 2020. 2, 7
[17] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Puttern Recognition (CVPR), 2012. 1, 2, 5.7
[18] Tong He and Stefano Soatto. Mono3D++: Monocular 3D Vehicle Detection with Two-Scale 3D Hypotheses and Task Priors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8409-8416, 2019.
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. Communications of the ACM, 60(6)84-90, 2017. 5
[20] Abhijit Kundu, Yin Li, and James Rehg. 3D-RCNN: Instance-Level 3D Object Reconstruction via
Gender-and-Comparre In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3559-3568, 2018.
[21] Peiliang Li, Xiaozhi Chen, and Shaojie Shen. Stereo R- CNN Based 3D Object Detection for Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
10

---

7644-7652. 2019. 1
[22] Peixuan Li, Huaici Zhao, Pengfei Liu, and Feidao Cao. RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving. In European Conference on Computer Vision, pages 644-660. Springer, 2020.
[23] Ming Liang, Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep Continuous Fusion for Multi-sensor
3D Object Detection. In Vittorio Ferrari, Martial Hebert. Cristian Sminchisescu, and Yair Weiss, editors.
Computer Vision - EECCV Lecture Notes in Computer Science, pages 663-678. Springer International
Publishing, 2018. 1
[24] Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou. Deep Fitting Degree Scoring Network for Monocular 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1057-1066, 2019.
[25] Zechen Liu, Zizhang Wu, and Roland Tóth. SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation. In Proceedings of the IERF/CVF Conference on Computer Vision and Pattern
Recognition Workshops, pages 996-997, 2020. 2, 3,7
[26] Zongdai Liu. Dingfu Zhou, Feixiang Lu, Jin Fang. and Liangjun Zhang. AutoShape: Real-Time Shape- Aware Monocular 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 15641-15650, 2021. 3,
[27] Yan Lu, Xinzhu Ma, Lei Yang. Tianzhu Zhang. Yating Liu, Qi Chu, Junjie Yan, and Wanli Ouyang. Geometry Uncertainty Projection Network for Monocular 3D Object Detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3111-3121, 2021. 2.
[28] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Rethinking
Pscudo-Lidar Representation. arXiv preprint arXiv:2008.04: 04582, 2020. 2, 7,7
[29]
Xinzhu Ma, Zhihui Wang. Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate Monocular
3D Object Detection via Color-Embers Embedded 3D Reconstruction for Autonomous Driving. In The IEEE
International Conference on Computer Vision (ICCV), October 2019. 2,3
[30] Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape. In The IEFF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2019. 2
[31] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional DETR for Fast Training Convergenec. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660, 2021. 3,
[32] Ishan Misra, Rohit Girdhar, and Armand Joulin. An End-to-End Transformer Model for 3D Object Detection, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906-
2917, 2021. 3.6
[33] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3D Bounding Box Estimation Using Deep Learning and Geometry. In Proceedings of the IEEF Conference on Computer Vision and Puttern Recognition. pages 7074-7082, 2017.
[34] Krishna Murthy, GV Sai Krishna, Falak Chhaya, and K Madhava Krishna. Reconstructing Vehicles from Single Image: Shape Priors for Road Scene Understanding. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pages 724-731. IEEE, 2017. 2,3
[35] Cuong Cao Pham and Jae Wook Jeon. Robust Object Proposals Re-ranking for Object Detection in Autonomous Driving using Convolutional Neural Networks. Signal Processing: Image Communication.
53:110-122, Apr. 2017. 1
[36] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum Pointnets for 3D Object Detection from RGB-D Data. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 918-927, 2018. 1
[37] Zengyi Qin, Jinglu Wang, and Yan Lu. MonoGRNet: A Geometric Reasoning Network for Monocular 3D Object Localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8851-8858, 2019. 1, 3,
[38] Zengyi Qin. Jinglu Wang. and Yan Lu. Triangulation Learning Network: From Monocular to Stereo 3D Object Detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2019. 1
[39] Cody Reading, Ali Harakeh, Julia Chae, and Steven Waslander. Categorical Depth Distribution Network for Monocular 3D Object Detection, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8555-8564, 2021. 3,
[40] Joseph Redmon and Ali Farhadi. Yolov3: An Incremental Improvement. arXiv preprint 104.:7180
2018.1
[41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks. Advances in neural information processing systems, 28, 2015.
2. 3
[42] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-779, 2019.
[43] Kiwoo Shin, Youngwook Paul Kwon, and Masayoshi Tomizuka. RoarNet: A Robust 3D Object Detection Based on Region Approximation Refinement. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 2510-2515. IEEE, 2019.
11

---

[44] Andrea Simonelli. Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Aniequera, and Peter Kontschieder. Disentangling Monocular 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1991-1999, 2019.
[45] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully Convolutional One-Stage Object Detection. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9627-9636, 2019. 2,
[46]
Arnay Varma, Hemang Chawla, Bahram Zonooz, and Elahe Arani. Transformers in Sele-Super Monocular Depth Estimation with Unknown Camera Intrinsics. arXiv preprint arXXv:2202.033131 2022. 3 [47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in neural information processing systems, 30, 2017.
[48] Jean Marie Uwabeza Vianney. Shubhra Aich, and Bingbing Liu. RefinedMPL: Refined Monocular PseudoL.iDAK for 3D Object Detection in Autonomous Driving. arXiv preprint arXiv: 1,9911,, 2019.
[49] Tai Wang. Xinge Zhu. Jiangmiao Pang, and Dahua Lin. FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 913-922, 2021. 2,
[50] Yan Wang. Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Camphell, and Kilian Q Weinberger. Pseudo-LiDAR From Visual Depth Estimation:: Bridging the Gap in 3D Object Detection for Autonomous Driving. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
1445-8453, 2.3
[51] Yingming Wang. Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor DETR: Query Design for Transformer-
Based Object Detection. arXiv preprint arXiv:2 109.07 2021. 3,
[52] Xinshuo Weng and Kris Kitani. Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[53] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Subcategory-Awart Convolutional Neural Networks for Object Proposals and Detection. (rXiv:160400469 [cs]. Mar. 2017. arXiv:
2. [54] Bin Xu and Zhenzhong Chen. Multi-level Fusion Based 3D Object Detection from Monocular Images. In Computer Vision and Pattern Recognition, pages 2345-2355. Salt Lake
2018 IEEE/CVF Conference
City, UT, USA, June 2018 IEEE. 1, 2,
[55] Guanglei Yang. Hao Tang. Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-Based Attention Networks Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16269-16279, 2021. 3
for Continuous Pixel-Wi Wise Prediction.
[56] Guanglei Yang, Hao Tang, Mingli Ding. Nicu Sebe, and Elisa Ricci. Transformers Solve Limited Receptive Field for Monocular Depth Prediction. arXiv e-prints, pages arXiv-2103, 2021. 3
[57] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient DETR: Improving End-to-End Object Detector with Dense Prior. arXiv preprint arXiv:
wwXiv:2104.00 224,0022 118801 2021. 3, 6
[58] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep Layer Aggregation. In Proceedings
Computer Vision and Puttern Recognition, pages 2403-2412, 2018. 4 [59] Renrui Zhang, Han Qiu, Tai Wang. Xuanzhuo Xu, Ziyu Guo. Yu Qiao, Peng Gao. and Hongsheng Li. Mon- oDETR: Depth-Awar Transformer for Monocular 3D Object Detection. arXiv preprint ar Xrx:::2203. 13310,
of the IEEE Conference
2022. 3,7
[60] Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects Are Different: Flexible Monocular 3D Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3289-
3298. 2021. 2, 3.
[61] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects As Points. arXiv: 000999944407770 [cs], Apr.
2019. arXiv: 1904.07850. 2, 3, 4, 5
[62] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang. Monocular 3D Object Detection: An Extrinsic Parameter Free Approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7556- 7566,7566,665 2021. 1, 2, 5,8.
[63] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490-4499,
2018.
[64] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for End-to-End Object Detection. arXiv preprint 20X10:2010.041599 2020. 3, 6,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'te 'r using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "OeMLEuYsnXG", "reviewer": "Reviewer_wuFE", "summary": "This paper utilizes ground depth as geometry cue to improve monocular 3D detection. The ground depth is estimated by camera pose and vanishing point. And the ground depth information is further integrated into the network by the transformer. The experiments demonstrate the effectiveness of the proposed components. And it achieves state-of-art results on the KITTI benchmark. ", "questions": "1. The paper mentions the Mono3D performance of far objects many times. How about the accuracy of objects at long distances (>50m)?\n\n2. The experiments are only performed on the KITTI dataset. Is this method evaluated on other public datasets (e.g. Waymo, Nuscenes)?", "limitations": "- As mentioned in the conclusion, the ground depth heavily relies on pose estimation. And it is difficult to ensure the accuracy of pose estimation.\n\n- The method assumes that the ground is flat. However, the ground at the long distance and at the side is inconsistent with the nearby place. Therefore the virtual scene constructed from the infinite plane is not convinced. \n\n- The formulas adopted in the equations are not unified. For example, the EL is suggested to be replaced by a single letter.\n", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The motivation is good. The ground depth is more reliable in the Mono3D task than other geometry clues. \n\n+ The fusion of ground depth is well-structured, in which the ground-aware transformer locates the ground-contacting points and integrates their features. \n\n+ The method is easy to follow. It is flexible to extend to other Mono3D models.\n\n+ Extensive experiments show the effectiveness of the proposed components. And the method achieves SOTA results on the KITTI leaderboard. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "KRnZuSOAZBv", "reviewer": "Reviewer_EteJ", "summary": "This paper proposes a framework, namely MoGDE, to consider ground depth cues for more accurate and robust monocular 3D object detection. Specifically, it estimates the camera pose at first and constructs ground depth map to guide the depth estimation of objects. The fusion of ground depth and 2D features is achieved with a depth-aware DETR formulation. Finally, the overall framework achieves state-of-the-art results on the KITTI benchmark, and the qualitative results support the claim that the depth information of the ground assists the localization of 3D objects.", "questions": "See the weaknesses.", "limitations": "The author adequately analyzes the limitation of the proposed methods in the Conclusion Section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper has a clear motivation, i.e., to use ground depth cues to assist the depth estimation of 3D objects.\n- The basic idea is easy to follow, indicating that the presentation of this paper is overall smooth.\n- The methodology part is overall clear and rigorous, although there are some points that require clarification.\n- The performance on the KITTI test set is impressive, and the qualitative results well support the core idea of methodology.\n\nWeaknesses:\n- There are some confusing points in the methodology part:\n\n    a. line 143: The camera pose typically consists of 6 DoF? including 3-DoF translation and 3-DoF rotation. Nevertheless, the author directly takes it as pitch and roll angles, which can be inaccurate and confusing especially when using this concept for the first time.\n\n    b. line 170: How do the \"predicted ground plane and vanishing point\" formulated? It seems that there is no explanation for their specific definition.\n\n    c. line 171: The matrix A not need any extra annotations? May need to clarify how the matrix is involved in the training procedure, at least more clearly in the main paper.\n\n    d. line 182: The projection matrix in KITTI involves several offset factors, such as bx, by, bz to describe the baseline offset between camera 2 and the referenced camera. How do you deal with such details when deriving the projection equations?\n\n    e. line 193: Why can the virtual stereo setup address the problem that the point near the vanishing point is not continuous? Here needs many words to clarify.\n\n- Experiments:\n\n    a. There seems no ablation or detailed results to reveal that the proposed method has boosted the detection performance for far-away objects, but the author takes this point as an important contribution in the presentation.\n\n    b. Why need experiments on the Odometry Benchmark and why only show the qualitative results on that? Is there any difference between 3D detection Benchmark and Odometry Benchmark? How about the quantitative results on the Odometry Benchmark?\n\n    c. line 267: lack of the explanation for the experiment group (d) (seems there is a typo)\n\n    d. Why does the proposed method perform a little worse on the Easy split of the validation set?\n\n    e. Would be better and more convincing to have experiments on a second dataset, such as nuScenes or Waymo.\n\n- Minor comments:\n\n    a. Can further polish the paper and fix some minor typos, such as need -> needs in line 198.\n\n    b. Related works can be included:\n        Ground-aware Monocular 3D Object Detection for Autonomous Driving, RA-L 2021.\n        Probabilistic and Geometric Depth: Detecting Objects in Perspective, CoRL 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DtJNhp4WjdQ", "reviewer": "Reviewer_nXFG", "summary": "Targeting monocular 3d object detection, this paper exploits depth information by estimating ground plane and fuses that information with image features via attention layers. Experimental results on KITTI are very strong, outperforming MonoDETR by 2-4points.", "questions": "The paper is well written. For failure cases when ground-contacting points are hard to detect, does the method break fully or can it still output reasonable detections (i.e. it can learn to ignore those features)? including visualization of some failure cases would help better understand the effectiveness.", "limitations": " As the authors mentioned, the proposed method is limited by ground-contacting points detection and could fail in cases where those points are occluded or truncated.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The proposed method is intuitive and clearly presented (i.e. it exploits depth information using ground plane). The paper provides a clear explanation and derivation of how the ground depth is estimated and used in the model. \n+ Experiments are comprehensive and the results are strong. It outperforms previous sota by large margin (Tab.1), provides detailed ablation of different components (Tab.2), and gives good visualization of the output with attention maps.\n+ Code is provided with high quality.\n\nWeaknesses: the proposed method could be upbounded by ground-contacting point detection and assumes the ground plane is flat, while it works well on KITTI mono 3d detection, it could be less effective in general.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8yS_347rvin", "reviewer": "Reviewer_Nv1S", "summary": "The main idea in the paper is to assume a flat ground plane and leverage known camera height above it etc to get depths for the ground points. The method refines the camera pose relative to this plane via horizon/vanishing point estimation in the images by following the DeepVP method[7] thus deriving depths for ground pixels.  For 3D detection, the idea is to fuse these depths with the image features using transformer layers, which pick suitable ground points for the object to derive its depth. For the final detector, the GUPNet head is used. \n\nThe results are claimed to be top on monocular 3D image detection on KITTI, which would be an impressive achievement. Suitable ablations for the method components are done. Additional results, showing the applicability of the introduced pose estimation components to detectors other than GUPNet are also presented. ", "questions": "How exactly did you know where the vanishing point is on KITTI? I don't fully understand how you get the GT. \n\nPlease comment on flat plane / curvy road limitations of the method, if any. \n\nAny way one can see which papers are monocular detection, on the KITTI leaderboard? \n\n", "limitations": "No specific limitations apart from what I mention in Questions above. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- Insightful use of transformers to fuse estimated depths for object detection. \n- Intuitive overall idea to leverage the flat ground plane assumption and estimate the camera pose relative to it -- it seems to make a difference in KITTI. \n- Strong results on KITTI and meaningful ablations. \n\nWeaknesses:\n- The need for pose estimation on KITTI, what quantities were given and which were not, was not particularly clear. As a result, Sec 3.2 was a bit difficult to read. Consider clarifying this better in a subsequent version. \n- The limitations of the method are not well explained. It assumes a flat ground plane which may not suit hilly scenes. Without reading the papers, it's not particularly clear to me how vanishing point estimation works when the road makes a turn. No experimental results showing model failures in such cases (or any cases really) are presented, in order to better understand the limitations / problems with the model. \n- It is difficult to track based on the links presented, whether this method is #1 on KITTI for monocular depth or not. I am willing to believe it and I see MoGDE represented on the leaderboard here: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. It clearly outperforms GUPNet and MonoDETR, so perhaps the claim is correct. \n- There are other CVPR 2022 methods on the leaderboard stronger than that are not necessarily cited (e.g. Diversity Matters: Fully Exploiting Depth Clues for Reliable Monocular 3D Object Detection, which has better numbers than MonoDETR). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept