Progressive Coordinate Transforms for Monocular 3D Object Detection
Authors: Li Wang
Li Zhang
Yi Zhu
Zhi Zhang
Tong He
Mu Li
Xiangyang Xue
Venue: NeurIPS 2021 Poster (2021)
Keywords: monocular 3d detection
progressive coordinate transform
feature fusion

Abstract: Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment. While significant progress has been achieved with expensive LiDAR point clouds, it poses a great challenge for 3D object detection given only a monocular image. While there exist different alternatives for tackling this problem, it is found that they are either equipped with heavy networks to fuse RGB and depth information or empirically ineffective to process millions of pseudo-LiDAR points. With in-depth examination, we realize that these limitations are rooted in inaccurate object localization. In this paper, we propose a novel and lightweight approach, dubbed {\em Progressive Coordinate Transforms} (PCT) to facilitate learning coordinate representations. Specifically, a localization boosting mechanism with confidence-aware loss is introduced to progressively refine the localization prediction. In addition, semantic image representation is also exploited to compensate for the usage of patch proposals. Despite being lightweight and simple, our strategy allows us to establish a new state-of-the-art among the monocular 3D detectors on the competitive KITTI benchmark. At the same time, our proposed PCT shows great generalization to most coordinate-based 3D detection frameworks.

Progressive Coordinate Transforms for Monocular 3D Object Detection
Li Wang!* Li Zhang
Yi Zhu?
Zhi Zhang
Tong He²
Mu Li Xiangyang Xue
Fudan University
Amazon Inc.
Abstract
Recognizing and localizing objects in the 3D space is a crucial ability for an Al agent perceive its surrounding environment. While significant progress has been achieved with expensive LiDAR point clouds, it poses a great challenge for 3D object detection given only a monocular image. While there exist different alternatives for tackling this problem, it is found that they are either equipped with heavy networks fuse RGB and depth information or empirically ineffective process millions of pseudo-LiDAR points. With in-depth examination, we realize that these limitations are rooted inaccurate object localization. In this paper, we propose a novel and lightweight approach, dubbed Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. Specifically, localization boosting mechanism with confidence-aware loss is introduced progressively refine the localization prediction. addition, semantic image repre- sentation is also exploited to compensate for the usage of patch proposals. Despite being lightweight and simple, our strategy leads to superior improvements on the KITTI and Waymo Open Dataset monocular 3D detection benchmarks. At the same time, our proposed PCT shows great generalization to most coordinate- based 3D detection frameworks. The code is available transforms
ttttta://gtthee com/
amazon- cectoon-eeeearee rww.ereeeeeeeeeeeee wwtppporrrsseeeoooooooooooo oe
Introduction
Object detection is fundamental and challenging task scene understanding applications. Recently, 3D object detection has received increasing attention and found applications in a wide range of scenarios such as autonomous driving, roboties, visual navigation and mixed reality. Despite the great progress from the area of 2D object detection [34 49 40 18 4 3D object detection remains largely unsolved problem aims to predict the object location in the 3D space alongside 3D object dimension and orientation.
for 3D object detection largely rely on LiDAR sensors, which provide accurate 3D point clouds of the scene. Although these approaches achieve superior performance, the dependence on expensive equipment severely limits their applicability to generic 3D perception. There also exists a cheaper alternative that takes a single-view RGB image as input, termed as monocular 3D object detection. However, its performance is far from satisfactory as itself is an ill-posed problem due to the loss of depth information in 2D image planes. Hence, several recent attempts introduce depth information to help monocular 3D detection. Such attempts can be roughly categorized into two directions, pixel-based and coordinate- based. Pixel- based
Existing prevalent approaches
50
35
Work done during internship at Amazon.
Li Zhang (lizhangfd @ anddan.ddu.n) and Xiangyang Xue (xyxue @ fudan. edu.cn) are the corresponding authors. Li Zhang is with School of Data Science, Fudan University. Li Wang and Xiangyang Xue are with School of Computer Science, Fudan University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table Probing investigation on coordinate- methods, PatchNet 2 and Pseudo-LiDAR 42] We examine the potential improvement by replacing the predicted factor with the corresponding ground truth. * indicates our reproduced performance. We can see that coordinate- bordeinate-baeed methods mostly suffer from inaccurate localization.
P
[AP AP BEV
Pseudo-LiDARR an/ AP REV]
Factor
Mod.
Easy
Hard
Mod.
Easy
Hard
Baseline
26.31/34.14
36.40/46.80
21.07/28.04
23.04/31.06
32.27/42.45
19.67/25.61
dimension
27.26/34.62
40.32/47.24
24.29/28.38
25.88/31.97
36.09/44.35
20.88/26,66
rotation
26.25/34.04
36.09/46.25
23.49/27.99
23.88/31.31
32.42/42.74
19.85/26.07
X
32.80/41.43
45.60/56.22
27.38/34.63
28.36/36.77
39.69/50.78
25. 25.08/29.92
y
30.16/34.14
40.94/46.80 80
24.58/28.04
25.53/31.06
35.19/42.45
20.69/25.61 .67
2
42.42/53.48
55,42/68.25 29
35,54/45.60
38.37/50.81
50.04/63.96
32.24/43.33 .2
location(xyz)
72.58/75227
81441/85.14
57.69/66.10
64.36/73.37
79.13/83.77
55.64/58.27
approaches 99 36 311319 41] turn use estimated depth map additional input for improved detection performance. But at the same time, this leads to heavy computational burden and large memory footprint since they often operate the entire image. Coordinated-bast approaches 42 29 46 27
pursue the coordinate representations as in LiDAR- to convert the monocular image pixels to 3D coordinate representations, then apply 3D detector on the converted coordinates. In particular, they are often lightweight since their network inputs are object proposals generated by 2D detectors 29 27.. However, the performance of coordinated- methods lags far behind LiDAR- methods. So we ask, can we identify the bottleneck that holds back the 3D detection accuracy of noordinate-based methods and how
methods. They use the predicted depth map
improve them?
In order to determine the bottleneck, we conduct an investigation on two widely adopted coordinate- based methods, PatchNet [27 and Pseudo-LiDAR (42. Specifically, for each prediction target, we examine the potential improvement by replacing its value with the corresponding ground truth, and then re-compute the 3D detection accuracy. As shown in Table [ using ground truth dimension and rotation do not bring significant improvements over the baseline. But using ground truth location (i.e., x/y/z values of the objects) almost triples detection accuracy. This indicates that coordinate- baadeatte-baeed methods mostly suffer from inaccurate localization even with the assistance of estimated depth maps.
Based on this observation, we focus on improving the accuracy of 3D center localization. In this work, we propose lightweight and generalized approach, called Progressive Coordinate Transforms (PCT), enhance the localization capability for coordinate-baase methods. First of all, since the localization regression network most coordinate-basse methods is less accurate but lightweight, we propose to progressively refine its prediction similar gradient boosting 12 137 To be specific, a localization regression network can be seen as a weak learner, and we progressively train multiple consecutive networks such that each network fits the regression residual from the previous networks. These networks share the same lightweight structure so that the computation overhead is negligible. We also predict confidence score for each network help stabilize the end-to-end training. We term this progressive refining strategy as confidence-awan localization boosting (CLB). Compared to image-only or pixel-based methods, coordinated- based methods suffer from the problem of missing global context information due to the use of patched input. In order to further improve the localization accuracy, we exploit semantic image representations from 2D detector. We term this module as global context encoding (GCE). We find that GCE can not only improve center localization accuracy, but also contribute the final 3D box estimation.
Through extensive experiments, our progressive coordinate transforms, consisting of CLB and GCE, is shown to improve popular coordinate- based models 42 27 by generating
accurate localiza- tion. Without bells and whistles, we achieve state-of-the-aa monocular 3D detection performance on KITTI [16] 17 15 with strong base method (27) Additionally, this also leads superior improvements on Waymo Open Dataset (38 compared with the base method PatchNet.
Related work
2.1 Monocular 3D object detection
Existing paradigms for monocular 3D object detection can be categorized into two types: image-only methods and depth-assisted methods.

---

For image-onil methods, they often adapt architectures and good practices from popular 2D detec- However, locating objects in 3D space is much more challenging without depth information. Hence, several works 30
tors 34 49 40
49 integrate geometry consistency into the training strategy to constrain the localization prediction. Deep3DBox [3]] divides orientation into multi-bins to stably regress them, and combines the 2D-3D box constraint to recover accurate 3D object pose. M3D-RPN [2] utilizes the geometric relationship between 2D and 3D perspectives by sharing the prior anchors and classification targets. MonoPair [6] leverages the spatial relationships between paired objects to improve accuracy on occluded objects. To further improve the performance of truncated objects, MonoFlex [48] decouples the features learning and prediction of truncated objects, and formulates an depth estimation to adaptively combine independent estimator based on uncertainty, 33 designs CaDDN fully differentiable end-to-end approach for joint depth estimation and object detection.
2
25
6
Depth-assisted methods often estimate a depth map given input image, and use it in different ways. Some pixel-basse brse-based approaches p 26] directly feed images and estimated depth maps into networks to generate depth-aware features and enhance the 3D detection performance. Some other coordinate- based approaches first transform the pixels of input images to 3D coordinates by leveraging the depth and camera information, then feed the coordinate proposals to 3D detector. Pioneering work Lieud 42 imitates the process of LiDAR-based approaches, which uses LiDAR-basee 3D input fusion embed
detector upon coordinates proposals. AM3D [29] explores the the complementary RGB cue into the network. Recently, PatchNet [27] points out that the efficacy of Dscudo-LiDAR representation comes from the coordinate transform, instead of sophisticated LiDAR-based networks. Hence, they design a simple 2D CNN to perform 3D detection. In this work, follow the research of coordinate-basee directly with a single lightweight network, we propose to progressively refine the prediction inspired by gradient boosting. We also incorporate RGB image information to complement patch proposals and enhance global context modeling. Different from AM3D [29], we utilize the RGB features from the 2D detector directly which can share the same context, and we do not need to train another RGB network from scratch.
methods 42 27
Instead of regressing 3D localization
2.2 Gradient boosting
Gradient boosting is a well-known greedy algorithm proposed in 7. which trains a sequence of learners and progressively improves the prediction results. It is a general learning framework, and has been verified to be formidable force when applied with lightweight learners Meanwhile, when each learner in the sequence is heavy, the computation cost becomes high and the performance is not
beneficial [24]] Early works in 2D detection area [99 21 22 also adopt the boosting mechanism following standard cascade paradigm, and achieve improved performance. Li et al. [2] treat face detection as an image retrieval task and improve it with boosted exemplar- based face detector. Karianakis et al. [9] and Li et al. (22) feed convolutional features of proposals instead of hand-craftie features boosted classifiers and distinguish objects from backgrounds. We can also find the usage of gradient boosting in other computer vision tasks (1]
To our best knowledge, we are the first explore the boosting mechanism in coordinate- based methods for 3D object detection. We perform this mechanism in two folds. First, instead of the entire 3D detection pipeline, we only progressively boost the localization regression network as its computational cost is insignificant comparing to the entire pipeline. Second, we refine the localization with additional confidence score in the boosting procedure, such that the loss is balanced. These choices greatly improves the performance with small extra parameters.
3 Background
Before diving into the details, we first revisit recent coordinate-bate based monocular 3D detection methods and present visual depiction of its common pipeline in Figure [] The framework usually consists of four main components: 2D bounding box generation, depth map estimation, data transformation and 3D box estimation. Specifically, given an image 7, the process can be described
2D hounding box generation. An off-the-shelf 2D object detector Fea such as Faster R-CNN employed image to generate region of interests (Rols), R F2a(1).
3

---

20 bbox
2D Detector
generation
Data
Coordinate proposals
3D Detector
30 bbox
Depth
Depth map
estimator
Figure I: A common pipeline of coordinate- based monocular 3D detectors. It consists of four steps to predict the final 3D boxes: 2D bounding box generation, depth map estimation, data transformation and 3D box estimation. In this work, focus improving the last step: 3D box estimation.
Depth map estimation. An off-the-shell depth estimator F& such as DORN [4] is applied on image to predict its depth map, Z (2(1).
Data transformation. To convert pixel (u, v) within a Rol to 3D space, the associated depth ((u, v) is used to transform into its 3D coordinates (Oz: Cy. Cz) by
(u u')
0')
Cz
C2 =
fu
fe
Here, (Czy Cyr Cz) pixel in the generated 3D coordinate patch c. (u', v') the camera principal point. fu and fo are the focal length along horizontal and vertical axis, respectively. u', v' fus fo are usually provided by the datasets.
3D box estimation. Once the 3D coordinates for each Rol are available, the final step is to predict 3D boxes with their center location, rotation and dimension. Different networks Faa such as Frustum PointNet [32 can be employed to conduct 3D box prediction, = Faa(c). Here, includes the center location (x, y, 2), rotation (0), and dimensions (wr h, of the 3D box.
Since the first two steps use off-the-shelf models and the third step can be computed analytically, in this paper, we focus on improving the last step of coordinated-bass methods. In particular, our goal to improve the accuracy localization prediction motivated by the observation Table
Method
In this section, we present our progressive coordinate transforms (PCT) for improved 3D detection. In order to obtain more accurate localization predictions, we introduce confidence-aware localization boosting mechanism (CLB) in Sec. t.T progressively refine the prediction. Then Sec. 4.2] we incorporate RGB image information by global context encoding (GCE) strategy to compensate for the drawbacks using patch proposals. In the end, we illustrate the overall framework of PCT in Figure 2 (a).
4.1
Confidence-aware localization boosting
divide the last step 3D box estimation into major components. The first component is lightweight 3D localization regressor F, whose input is 3D coordinate proposals generated from data transformation. The second component is relatively heavy network G used regress the final 3D box B. Recalling the results in Table [ 3D localization performance the weakest point a coordinate-base for up 50 AP loss when all other modules keep intact. Therefore, can find an efficient way to improve the accuracy of localization prediction and also generalizes to other coordinated-ban methods?
Following Frustum PointNet 32
most coordinate-base methods 42 45 27
43
model, accounting
Gradient boosting 12 13 is general learning framework that combines multiple weak learners into single strong in iterative fashion. Let C(x) be the risk of ensemble models, the algorithm devotes to seek an approximation h(x) to minimize L(y", x) = ( h(x)), where y a target value, (-) is the loss function. h((x:) is a linear combination of set of weak (base) learners fi f(()) from some class J, ie.., h(xx) E (x) const. Here, the total training iterations and 7 is the corresponding weight for each weak learner. To minimize the empirical risk, the algorithm starts with model Ag (x), and then incrementally expands in a greedy manner. This process manages to fit new weak learner to the residual errors made by the previous set of learners. Mathematically, the

---

8 (a (2.
2D Detector
Rol
features
G(G)
Global Context Encoder
T iter
coordinates
Pr(Cz-1)
Coordinate
Localization regressor
Box estimator
proposals
a (o
Estimated
(t-1(Ge-2)
depth
Mo.y,.)
8p-1
(a) Schematic Illustration of proposed PCT (green)
Detail flow CLB.
Figure 2: (a) Schematic illustration of proposed Coordinate Transforms (PCT). We treat F weaker learner, and perform coordinate transform 7 steps via confidence-awe aware localization boosting (CLB module) to obtain a better localization. Then the refined coordinate proposals combined with corresponding encoded RGB features (GCE module) are fed into network G to generate final 3D bounding boxes. (b) The data flow of CLB mechanism in detail. For step I, it takes refined coordinate patches ct-1 as input, which is transformed based on predicted (2 M/-1, 21-1). Then network Fi generate the residual localization for the next step, and confidence 8 is also generated during the training process.
optimization can be formulated as
ho(x) arg min Lo(y", :2);
20
(x))) = hx-1 + arg min Ly (x-1(z) f((x)).
fee
Inspired by gradient boosting, we imitate its optimization procedure to progressively adapt localization prediction by multiple localization regressors instead of single used in previous works 42] 29
46,27] 27). To be specific, we treat localization network as a weak learner and stack multiple of them shown in Figure (b). Given F is lightweight network, the extra computational cost brought by gradient boosting is insignificant. After the data transformation step, each 2D bounding box obtains its corresponding coordinate patch We take the coordinate patch the input of weak learner F to regress the center localization residual A ( y, =) based on the prediction from previous stage,
A (xt, yt, 21) Fr(c-11;; where C-1 = co To (zo you 20)
Σ Aa Dis
We denote cp and (æo, No. =u) to be the initial coordinate input patch and object location, respectively. Coordinate input patch ct-1 is then transformed according to the localization residual prediction, and fed into the next weak learner.
Thus the risk at stage can be written as,
CF. ((0" v0,,vvv22") (=))) v((x", ,f vo No, 20)
Σ
7 A 2))
(4)
where (x*, ,y",²") 2²) indicates the ground truth location. At this point, we can easily see that Eq 4 is natural derivation from Eq 2 After T iterations, the final adjusted prediction ex is fed into the network G' to estimate the 3D box B, i.e.B = G(er).
Confidence-aware network lass In the case that the target of weak learner f is differentiable. gradient boosting solves the optimization problem in forward greedy manner as shown Bq For each iteration, it first fits the weak learner to the residual error, and then the optimal value of the coefficient weight 7 determined for this weak learner. The optimization procedures train iteratively for I' iterations.
5

---

However, for our center regression task, we would like to train T weak localization networks F,(c--1,, E I.... I and 3D box prediction network G in an end-to-end manner instead of bootstrapping. This is challenging in terms of both computational cost and optimization stability, given the simultaneous training first simplify the problem by treating all "/ the same and only focus on optimizing the localization networks. However, the contribution from each weak learner I' may not be the same during end-to- end training, which leads to unstable optimization. Hence, we tailor confidence-aware boosting loss to facilitate network training, by learning confidence score 8 for each localization loss function CF. The confidence score & is learned from small decoder and followed self-balancing formulation closely coupled to the network loss. The overall loss function is defined as
set of weak learners and their coefficient weights. Therefore, we
C(B", co) = Σ CF. ((x" 20,2)), A II st) + LG(B" eT),
(5)
1=1
where B" is the 3D box ground truth, prediction after sigmoid, which represents the confidence of the localization regression loss tth stage, and II (1 sr) is the penalty on the network uncertainty. In other words, if st is approaching which means the network is confident about localization refinement 4dh stage, then no penalty will be applied. Otherwise, the uncertainty of regression loss is high, thus triggers higher penalty.
B and X is the balance weight. 8, is the
4.2 Global context encoding
Typical coordinate- methods 42
45 27 43
perform 3D detection based on 2D Rols. which is
similar to two-stage 2D detection frameworks, such as Faster R-CNN 34) In a two-stage 2D object detection framework, the second stage reuses the features from the first stage via RolPooling [34 RolAlign operators (18) guided by ROI proposals, and then a small decoder is used for localization refinement. However, in 3D detection, only cropped patches with coordinates information are fed to the network for 3D box regression. Neither RGB information nor global context is included.
Considering that the RGB information is vital visual clue, explore its aggregation in the last 3D box estimation step. Similar to two-stage 2D detectors, we obtain the RGB information by directly cropping the corresponding features from a 2D detector Fza- Then the input 3D box estimator G can be formulated as e ([D(u, u), A(u, w)], V(u, u) E R]. Here, D(-) represents the data transformation function and A(-) represents the RolAlign operation. Both operations performed upon the generated regions of interest from R.
After RolAlign operation, features are of size C K K, where C' is the number of channels and K K is the corresponding width and height, respectively. small feature encoder is then used to encode cropped features into vectors with the dimension of C. A feature fusion is followed to integrate coordinate representations with the obtained image representations. large receptive fields of image feature representations, 3D box estimator Benefiting from the have access to global context. Besides, directly cropping on RGB features also avoids learning image representations of Rols from scratch and reduces the overall network parameters.
Experiments
Two monocular 3D detection benchmarks are introduced in Sec. 5.1 and Sec. 5.2 while experimental implementation details are described in Sec. 5.3 In Sec. 5.. we conduct main analysis on KITTI dataset [16] 17 [5] with base method PatchNet [27] given its current best performance. More
experiments on Waymo Open Dataset (38] also demonstrated to further verify the generality of
our proposed PCT in Sec. 5.5
5.1 KITTI setup
We first evaluate our method on the KITTI benchmark 16 17 15 which contains 7,481 and 7,518 images for training and testing respectively. We follow 5 to split the 7,481 training images into 3712 12 for training and 3,769 for validation.

---

Table 2: Ablative analysis KITTI validation set for 3D and AP at IoU 0.7. Experiment
Group (I) is the baseline method. Different experiment settings are explored: (II) applying localization boosting without confidence constraint, (IIII performing confidence aware localization boosting algorithm, (IV) adding global context encoding on Group (II), (V) our full approach.
APan
APBRV Easy
Group
Localization Boosting
Uncertainty
GCB
Mod.
Easy
Hard
Mod.
Hard
25.88
36.07
20.99
33.34
46.39
27.54 37.54
II
26.78
37.29
24.11
34.39
47.08
28.28
III IV
27.24
38.32
24.39
33.92
46.77
27.98
27.12
37.38
24.11
34.46
46.70
28.32
V
V
27.53
38.39
24.44
34.65
47.16
28.47
Precision-recal curves are adopted for evaluation, and we report the average precision (AP) results of 3D and Bird's eye view (BEV) object detection KITTI validation and test set. For fair comparison to previous literature, the 40 recall positions-bar based metric AP R R1o is reported on test set while AP|RI R11 reported validation set. Three levels of difficulty are defined in the benchmark according the 2D bounding box height, occlusion, and truncation degree, namely, "Fasy", "Mod.", and "Hard". The KITTI benchmark ranks all methods based on the APaD of "Mod.". In particular, focus the "Car" category as in 42 277, and we adopt IoU = 0.7 as threshold for evaluation.
5.2 Waymo setup
We also carry out experiments large-scale, high quality and diverse dataset, Waymo open dataset [8 It provides pre-defined 798 training sequences and 202 validation sequences from different scenes, and another 150 test sequences without labels. The dataset contains camera images from five high-resolution pinhole cameras, and we only consider images with their 3D labels from front camera for monocular 3D detection task. We sample every third frame from the training sequences (total 52,386 images) as CaDDN [33 to form the training set due to its large scale. And validation set contains all the 39,848 images from 202 different scenes.
For evaluation, we adopt the officially released evaluation [39] to calculate the mean average precision (mAP) and the mean average precision weighted by heading (mAPH). Two levels are included according to difficulty rating, which are defined by LiDAR points. 3D labels without any points are ignored, LEVEL is assigned to examples when it contains equal or lesser than 5 points, while the rest of the examples are assigned to LEVEL_ 1. Additionally, three distances (0 30m, 30 500m, 50m (e) sensor are considered during evaluation.
5.3
Implementation details
Our overall framework of PCT can be visualized in Figure 2 In terms of implementation details, we instantiate our algorithm on two widely adopted coordinate-baut based methods with public released code (28) PatchNet and Pseudo LiDAR. Bearing efficiency in mind, we use real-time 2D detector RTM3D 23 with DLA-34 [47 as backbone. For the sake of fair comparison, literature. Since there no published depth results on Waymo open dataset, we adopt most recent monocular depth estimator AdaBins [I] trained on Waymo training set. For the CLB mechanism, we inherit the original localization regression framework in each method. Each F, shares the same structure. The corresponding confidence is generated following the last convolutional layers Fe with three linear layers and a Sigmoid function. set for the following experiments except for the ablation study on boosting
adopt depth
predictor DORN 14 on KITTI dataset as in most depth-
T' = 3 and A iterations. For GCE module, we get the corresponding input image features by performing RolAlign on the features from last convolutional layer of 2D detector. We set the output of RolAlign as 16 x 16. As 2D detector use DLA-34 as backbone, the obtained image feature representations have the size of 64 16 x 16 and entitle arbitrary sized receptive field theoretically due the embedded deformable convolution (8]. The structure of feature encoder in global context module is two common 64 x 3 x 3 convolutional layers (stride=4) and a 64 x convolutional layer (stride=l). Hence, image feature representations are encoded to a vector with 64-dim. The obtained features are then concatenated with the coordinate feature vectors from the final global pooling of box prediction network G. With the lightweight structure, we are able to optimize the network end-to-end on single Nvidia V100
7

---

23
6.
-0.05 8.00
-35
Figure 3: The statistic analysis and comparison different Localization boosting stage when 7 = 3. The vertical axis of the chart represents the number of samples after normalization. "loc. 1/2/3" denotes the 1/2/3th step of localization errors F and "loc.4" the final localization errors G. Note that when the curve is more thin, tall, and closer to zeros, the localization is more accurate.
GPU with 16G memory. The training criterion for network and G and other training settings follow the corresponding base methods for fair comparison.
5.4 Method analysis on KITTI dataset
Main ablative analysis In Table [ we conduct ablation studies analyze the effectiveness of our contributions: 1) Without any localization regression network F, network G generates 3D box prediction directly. II This configuration only contains localization boosting part without confidence constraint. III) The entire CLB mechanism is included to progressively regress center localization. IV) GCE module is added to the network based on the localization boosting block without confidence constraint since the feature fusion can be performed on either the localization regression networks F or 3D box prediction network G. V) Our full method with all the components.
As depicted in T we can observe that the performance continues to grow with the addition of every component. From group II, localization boosting brings a noticeable improvement on all settings especially "Hard" set, which confirms its effectiveness in increasing localization accuracy. Group III shows that balancing training loss by adding confidence leads to better and more stable optimization. Group IV reveals that the proposed GCE module can effectively equip RGB information and global context with 3D coordinate representations. the proposed CLB mechanism and GCE module, leading to an improvement from 25.88/36.07/20.99 to 27. 37.53388.39/24444 compared with Group I.
the end, Group V demonstrates the complementarity of
Table 3: Comparison of different boosting iteration tings on KITTI validation split set. Localization
Table 4: Evaluation of different ordinate feature fusion with GCE on KITTI validation set. Baseline is the Group (II) in Table
APaD/ /AP BEV
Boost (T)
Mod.
Easy
Hard
25.88/33.34
36.07/46.39
20.99/27.54
Method
APan an Easy
26.31/34.14
36.40/46.80
21.07/28.04 04
Mod.
Hard
2
26.69/34.06
37.17/46.42
24.04/28.03
Baseline
26.78
37.29
24.11
3
26.78/34.39
37.29/47.08
24.11/28.28
F+ GCE
27.08
37.33
24.07
4
26.77/34.21
37.12/47.00
23.48/28.22 223
G GCE
27.12
37.38
24.11
5
26.64/34.43
37.24/47.04
23.89/28.27
All GCE
27.07
37.43
24.18
Localization boosting iteration settings We explore the effect of different localization boosting iteration settings this part. For a fair comparison, we do not perform the confidence constraint on regression loss. As illustrated in Table 3 when boosting iteration T = 3, achieve the best 3D detection performance. More iterations of boosting do not bring improvements, which might be caused by overfitting with the increasing of network parameters.
To verify the improvement each step in boosting procedure, we conduct the comparison of localization errors at iteration T on the specific metries (location "xyz") of the ground truth. In particular, three stacked localization networks generate intermediate localization "loc. 1/2/3" and G output the final localization "!oc.4", As shown in Figure 3
that the distributions of "x".

---

Qualitative comparison of ground truth (green), base method PatchNet (blue), and our method (red) on KITTI val set. The first and second rows show RGB and BEV images respectively.
Figure 4:
Table 5: Comparison of generalization on KITTI validation set.
denotes that the method is
reproduced by ourselves.
APaD Easy
APBEV Easy
Method
Mod.
Hard
Mod.
Hard
Pseudo-LiDAR 42
23.04
32.27
19.67
31.06
42.45
25.67
DsDAR + CLB
24.14
34,46
20.16
32.41
44.98
26.82
Pseudo-LiDAR CLB GCE
24.43
34.34
20.18
32.50
45.35
26.91
"y" and "z" errors tend to distributed zero with localization boosting iterating. For instance, the red line in left chart is narrow and tall near zero along horizontal axis compared with other lines, which means that the corresponding coordinate is more accurate than others. This further suggests that localization boosting is useful for object localization. The corresponding BEV visualization will be shown in Supplementary Material.
Impact of global context encoding We also explore where the global context representation fusion operates. We take Group II as the baseline, and perform feature fusion localization regression network F (row one), 3D estimation network Gi (row two) or on both (final row). Rol features are encoded into vector with GCE and then concatenate with the feature vectors from network (F or G) global pooling. As shown in Table 4 the operation on G outperforms it on F, which indicates that image representation is more suitable for the overall box prediction rather than only localization as it contains the additional semantic appearance information. Although operation on all networks achieves lightly higher than it on G on the "Easy" and "Hard" set, introducing parameters is much larger due to operation on stacked localization networks. Hence, we only apply GCE on network G in our approach for lightweight network and avoid overfitting during training.
Generally applicable other coordinate-basd based algorithm In this section, we demonstrate the generalization capability of our algorithm to classic coordinate-based methods Pseudo-LiDAR [2]. As shown in Table 5 each component of our algorithm improves the original methods a lot. Specially, our approach improves Pseudo-LiDAR by 1.43/2.06/0.51 while 1.22/1 091///9933366 gains on PatchNet.
Comparison with state-of-the-arts We build our test detector on the current state-of-the-an coordinate-basee based method PatchNet, and results are shown in Table Quantitatively, our method achieves the highest performance on "Mod." set with 22 FPS on NvidiaTesla v100 including 2D detector inference time, which is the main setting for ranking on the benchmark. Specially, large 3D detection and 2. 1177766600099
BEV, are observed over the base method PatchNet with only additional 3.41M parameters. Besides, our methods also outperforms the pixel-based ofte-of-theeane methods Liu et al. 26 especially "Hard" set.
margins, 2.25/5.32/1.14
Qualitative comparisons are shown Figure 4 The ground truth, base method (PatchNet), and our method are colored in green, blue, and red, respectively. For better visualization, the first and second rows show RGB images and BEV images, respectively. Compared with the base method, our algorithm can produce higher-quality 3D bounding boxes in different kinds of scenes.

---

Table 6: Comparison with SoTA methods the KITTI test set IoU = 0.7. Our algorithm achieves
new SoTA performance. "Depth" means the method belongs to depth-assisted methods or not. "Type" indicates the method input pattern, "Pixel" denotes the methods with image as inputs directly while "Coordinate" means the coordinate-based methods with 3D coordinates as inputs.
AP Easy
APBEV
Method
Depth
Type
Mod.
Hard
Mod.
Easy
Hard
AM3D [29]
yes yes
Coordinate
10.74
16.50
9.52
17.32
25.30
14.91
PatchNet (27
Coordinate
11.12
15.68
10.17
16.86
22.97
14.97
GrooMeD-NMS 20
no
Pixel
12.32
18.10
9.65
18.27
26.19
14.05
Kinematic3D 3
yes
Pixel
12.72
19.07
9.17
17.52
26.69
13.10
DDMP-3D [1
yes
Pixel
12.78
19.71
9.80
17.89
28.08
13.44
Liu et al. [26] PCT
yes
Pixel
13.25
21.65
9.91
17.98
29.81
13.08
yes
Coordinate
13.37
21.00
11.31
19.03
29.65
15.92
5.5 Results on Waymo Open Dataset
Table 7 shows the results of base method PatchNet 22 and our proposed PCT. It can be observed that our method consistently outperforms the base method mAP/mAPH of 0.50%/0.5 % and 0.28% on the LEVEL_I and LEVEL_2 difficulties respectively under loU = 0.7. Again, our method is efficient, e.g. takes days complete training on large scale Waymo dataset with 8-GPU node. More qualitative results can be seen at Supplementary Material.
Table 3D performance on Waymo validation set. We demonstrate results of base method Patch- Net [27] and corresponding PCT at IoU = 0.7 and IOU 0.5, Our proposed PCT achieves consistent improvements on all settings.
3D mAP/ 3D MAPH
Difficulty
Threshold
Method
Overall
0 30m
30-50m
50
PatchNet
0.39 0.37
1.67 / 1.63
0.12
0.03 / 0.03
loU=0.7
PCT
0.89/ 0.88
3.18/ 3.15
0.27 0.227
0.07 0.07
LEVEL_1
PatchNet
2.927 2.74
10.03 / 9.75
1.0970 7.996
0.2370.18
IoU=0.5
PCT
4.20/4.15
14.70 / 14.54
1.78 1.75
0.39 0.33
PatchNet
0.38/
1.67 1.63
0.13
0.03 0.03
loU=0.7
PCT
0.66/ 0.66
3.18/ 3.15
0.27 /.226
0.07 /0.07
LEVEL_2
PatchNet
2,4272.28
(0.0179.73
1.07 70.94
0,2270.16
IoU=0.5
PCT
4.93/3.99
14.67 / 14.51
1.71
0.36 /.33
6 Conclusions
In this paper, have introduced novel approach PCT to address the inaccurate localization problem for monocular 3D object detection. This is achieved by iteratively transforming the coordinate representation with confidence-aware booting mechanism. Meanwhile, global context is introduced to compensate for the missing of semantic image representation in coordinated-basee Through extensive experiments, have shown that our proposed PCT substantially improve the model by a large margin, and achieve state-of-the-art monocular
methods.
performance of the coordinate-ban 3D detection performance on KITTI test set. Moreover, we also show consistent improvements compared to the strong bascline on the large-scale Waymo Open dataset.
There are several limitations that could indicate the possible directions for future work. First, the performance of off- the-shelf 2D detector directly influences the accuracy of coordinate- based methods, hence how effectively design the 3D box estimation algorithm to fit with existing 2D detectors is important. Second, we only concentrate on the lightweight coordinate- based methods. It requires further exploration extend our approach to pixel-based methods. Finally, our proposed global context encoding is a simple module. Despite working well, a more tailored feature fusion strategy between coordinate representation and RGB image representation is worth exploring.
Potential impacts. Our method focuses on the monocular detection which can be applied in autonomous driving field. One potential social problem of our work is that it may aggravate the employment crisis of human servants and drivers, which replaces human with autonomous robots and intelligent systems.
10

---

Acknowledgments
This work was supported by Shanghai Municipal Science and Technology Major Projects (No.2021SH2D22X11033 and No.20188HZDD2001,
References
[1] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. CVPR, 2021.
[2] Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV, 2019.
[3] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection in monocular video. arXiv preprint, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals for accurate object class detection. In NeurIPS, 2015.
[6] Yongjian Chen, Lei Tai, Kai Sun, and Mingyang Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. In CVPR, 2020.
[7] Corinna Cortes, Mehryar Mohri, and Umar Syed. Deep boosting. In ICML, 2014.
[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
[9] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3d object detection. In CVPR, 2019.
[10] Liang Du, Xiaoqing Ye, Xiao Tan, Jianfeng Feng, Zhenbo Xu, Errui Ding, and Shilei Wen. Associate-3ddett. Perceptual-to-coneeptue association for 3d point cloud object detection. In CVPR, 2020.
[11] Liang Du, Xiaoqing Ye, Xiao Tan, Edward Johns, Bo Chen, Errui Ding, Xiangyang Xue, and Jianfeng Feng. Ago-net: Association-guiaa 3d point cloud object detection network. TPAMI, 2021.
[12] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, 2001.
[13] Jerome H Friedman. Stochastic gradient boosting. Computational statistics data analysis, 2002.
[14] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. CVPR, 2018.
[15] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. httt://www.cvlitt pec//attastt//tttt//rraaooooiie wwtp:pppppopppoobaaaaaeae 3d 2013.
[16] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.
[17] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[19] Nikolaos Karianakis, Thomas Fuchs, and Stefano Soatto. Boosting convolutional features for robust object proposals. arXiv preprint, 2015.

---

[20] Abhinav Kumar, Garrick Brazil, and Xiaoming Liu. Groomed- Grouped mathematically differentiable nms for monocular 3d object detection. CVPR, pages 5973-8983. 2021.
[21] Haoxiang Li, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Gang Hua. Efficient boosted
exemplar-based face detection. In CVPR, 2014.
[22] Haoxiang Li. Zhe Lin, Xiaohui Shen, Jonathan Brandt, and Gang Hua. A convolutional neural network cascade for face detection. In CVPR, 2015.
[23] Peixuan Li, Huaici Zhao, Pengfei Liu, and Feidao Cao. Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving. In ECCV, 2020.
[24] Xuchun Li, Lei Wang, and Eric Sung. A study of adaboost with svm based weak learners, In IJCNN, 2005.
[25] Lijie Liu, Jiwen Lu, Chunjing Xu, Qi Tian, and Jie Zhou. Deep fitting degree scoring network for monocular 3d object detection. In CVPR, 2019.
[26] Y. Liu, Y. Yuan, and M. Liu, Ground-aware monocular 3d object detection for autonomous driving. IEEE Robotics and Automation Letters, 2021.
[27] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. Re- thinking pseudo-lidar representation. In ECCV, 2020.
[28] Xinzhu Ma, Shinan Liu, Zhiyi Xia, Hongwen Zhang, Xingyu Zeng, and Wanli Ouyang. https
//github. Tigithubccom/ze wwwpt/uuuaaaaachhhaecee 2020.
[29] Xinzhu Ma. Zhihui Wang, Haojie Li, Pengbo Zhang, Wanli Ouyang, and Xin Fan. Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving. In ICCV, 2019.
[30] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In CVPR, 2017.
[31] Erli Ouyang. Li Zhang, Mohan Chen, Anurag Arnab, and Yanwei Fu. Dynamic depth fusion and transformation for monocular 3d object detection. In ACCV, 2020.
[32] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018
[33] Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribu- tionnetwork for monocular 3d object detection. In CVPR, 2021.
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[35] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[36] Xuepeng Shi, Zhixiang Chen, and Tae- Kyun Kim. Distance-normalized unified representation for monocular 3d object detection. In ECCV, 2020.
[37] Jeany Son, Ilchae Jung, Kayoung Park, and Bohyung Han. Tracking-by besgmmenttiii with online gradient boosting decision tree. In ICCV, 2015.
[38] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[39] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. htttps://githut com/ waymo-reseen wwwearch.wwymooooee..... open -atasstt 2020.
[40] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In 2019.
12

---

[41] Li Wang, Liang Du, Xiaoqing Ye. Yanwei Fu, Guodong Guo, Xiangyang Xue, Jianfeng Feng. and Li Zhang. Depth-condition dynamic message propagation for monocular 3d object detection. In CVPR, 2021.
[42] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.
[43] Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pseudo-lide point cloud. ICCV workshops, 2019.
[44] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[45] Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng, Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In ECCV, 2020.
[46] Yurong You, Yan Wang. Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo- Accurate depth for 3d object detection in autonomous driving. arXiv preprint, 2019.
[47] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018.
[48] Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. CVPR, 2021.
[49] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint, 2019.
[50] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
[51] Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R. Manmatha, Mu Li, and Alexander Smola. Improving semantic segmentation via self-training. arXiv preprint arXiv:2004.149600 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We have described exactly in abstract section and
Sec.
(b) Did you describe the limitations of your work? [Yes] See Sec. 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? (Ys]] We demonstrate
the codes in Github website. The dataset is public and URL [5] 39 is also attached in
Sec. 5
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. [..3
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [No] We report experiments results at the fixed seed.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The type of resources can be seen at Sec. 5.3 And each GPU can run two experiments, thus, our included experiments (total number of 15) paper require about 8 Nvidia Tesla v100 GPUs (16G).
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec.
(b) Did you mention the license of the assets? [Yes] We conduct experiments on KITTI
(license: CC BY-NC-SA 3.0) and Waymo (license: Custom (non-commercial),
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you' using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We conduct experiments only on public datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rq2z4TM8Qd", "reviewer": "Reviewer_rVMM", "summary": "This paper proposes progressive coordinate transforms to solve the problems in monolar 3D object detection. But there are some problems with workload and experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The main contribution of this paper focuses on the application of gradient enhancement to localization prediction. Global context encoding seems just a small improvement to former algorithms. Therefore, the overall contribution is not particularly large.\n\n2. The experiment in this paper is carried out on the KITTI data set. The results on the test set in Table 6 should be obtained from the official website submitted to the official website, but the corresponding data (anonymous) is not seen on the KITTI website. So where did the experimental results in Table 6 come from? If it was obtained by KITTI, please submit it anonymously; if not, then there is a problem with the authenticity of the data. In addition, KITTI includes three types of detection object. Why does the experimental result only include the detection result of the car? How about the other two types of test results?\n\n3. The experiment is only carried out on the KITTI dataset, but more experiments on data sets such as waymo and nuscences should be carried out. In addition, the method in this paper is actually a 3D object detection algorithm based on monocular images, so it should be compared with other algorithms of the same type (3D object detection based on monocular image), such as GrooMeD-NMS[1], DDMP-3D[2], CaDDN[3], etc, but this paper does not compare them.\n\n[1] A. Kumar, G. Brazil and X. Liu: GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection. CVPR 2021.\n[2] L. Wang, L. Du, X. Ye, Y. Fu, G. Guo, X. Xue, J. Feng and L. Zhang: Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection. CVPR 2020.\n[3] C. Reading, A. Harakeh, J. Chae and S. Waslander: Categorical Depth Distribution Network for Monocular 3D Object Detection. CVPR 2021.\n\n4. This paper claims to design a lightweight coordinate-based network, so what is the amount of calculation and complexity of the algorithm in this paper? How does it compare to other coordinate-based networks? The comparison between the results of this part and other methods is also very important, but it does not appear.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "OZdNJeAnc_", "reviewer": "Reviewer_gTZB", "summary": "The paper first analyzes where the main performance loss is in the\ninference of 3D bounding boxes from 2D images and concludes it is\nin the 3D box center estimation phase. The introduced method\nis an add-on iterative refinement which can boost performance of\nseveral algorithms, and improves state-of-art when boosting\nPatchNet, as applied to the KITTI dataset.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces an iterative box center coordinate refinement stage\ninto the 3D BBox estimation (based on 2D box detections and monocular\ndepth estimates).\nConfidence weights are estimated for each progressive refinement.\nSemantic information is also added, which also boosts box location\nestimation. An ablation study shows that each of the 3 added components\nboosts performance a little. The proposed addition is also added to\nthe Pseudo-LiDAR algorithm, also improving performance.\nComparison of the number of iterations of the localization boost was\ndone to select the best number of iterations (3).\n\nThe paper is good: some nice analysis and ideas, good experiments, clearly written, but is also incremental; hence the proposed score of 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I saw no obvious weaknesses. I was curious about the magnitude of the\nweights learned.\nThere was a slight confusion between T=3 and +2 in table 3. Presumably\nthey mean the same thing, so it might be better to use the T values\nrather than the increments.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are general ethical issues about the impact of autonomous vehicles.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1rp_TMkCIJt", "reviewer": "Reviewer_vnDo", "summary": "This paper presents a \"progressive coordinate transform\" strategy for improving monocular 3D object detection. Specifically, the authors identify that the main bottleneck in performance for monocular 3D object detection is the localization performance and tailor a solution towards progressively refining the coordinate proposals with multiple stacked modules before making the final 3D bbox prediction. The authors also propose a \"Global context encoding\" which processes and concatenates the visual features from the object proposal before the final 3D bbox prediction. This empirically results in an improvement in overall 3D bbox detection performance - achieving state of the art of the KITTI monocular 3D object detection benchmark. The primary contribution of this paper is identifying localization performance as a key bottleneck for 3D monocular object detection and tailoring a simple solution to improve performance over prior work.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n-  Table 1 in the paper is most insightful in my opinion and identifies localization as a bottleneck (vs size or rotation prediction). This is also somewhat expected because the variation in dimensions for cars is not expected to be high and orientation is also expected to be in a constrained range in driving scenarios. However quantifying this still has value. The authors build from this simple intuition and propose an iterative refinement method after the coordinate proposal module in standard 3D monocular object detection methods. The simplicity is key.\n\n*Weaknesses*\n- The two major concerns I have with the paper are technical novelty and significance of improvement over prior work. The major contribution in this work is really the \"CLB\" module which is primarily an iterative refinement module which predicts residuals between the ground truth and the previous prediction at each stage. The GCE module which appends a visual feature vector computed from the detector backbone isnt necessarily novel and has relatively low effect on performance (<0.4AP in table 4).\n\n- The final improvement on the validation set seems significant for the \"Hard\" subset (~3.5AP in AP3D) but might not be very significant in my opinion on the other subsets and metrics. This is true when comparing against non coordinate-based methods on the KITTI test set in Table 6. \n\n- The figures in the paper often do not add much to the discussion. Fig 1 lacks more visuals and primarily a set of blocks. The caption for Fig 2(a) does not do a good job of explaining it and one has to follow the text very closely to understand the schematic here. The top row if FIg 3(a) is also hard to follow. The authors mention \"peakier\" red distribution curves indicating decreasing localization error over iterations but this is not readily visible to me. The BEV visualization and image bbox visualization in Fig 4 are also very small and the boxes themselves are hard to see. Overall, the figures in the manuscript need quite some improvement.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors are encouraged to comment on the technical novelty and significance of improvements over prior work on the KITTI dataset.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: At the time of rebuttal, the reviewers had widely varying opinions on the paper. Major concerns included general clarity as well as a set of fairly concrete experimental issues. The authors responded to the reviews with detailed new experiments to address the reviewer concerns. Reviewer vnDo was persuaded by these experiments and raised their score to an acceptance. Reviewer rVMM did not participate in post-rebuttal discussion. The AC examined rVMM's requests and the authors' responses. The AC cannot speak on behalf of rVMM; however, in the AC's view, much of the reviewers' concerns are ostensibly addressed directly with reasonable experiments. On balance, given that the consensus of the active reviewers is acceptance, and the rebuttal seems likely to address the remaining reviewer's concerns, the AC is strongly inclined to recommend acceptance.

The AC would suggest that the authors:
- Incorporate, to the best of their ability, the results presented in the rebuttal. If they do not fit, the authors should report them in the supplement.
- Take Reviewer vnDO's comments on figures and clarity seriously, and update them and their captions. 
- As noted by the reviewers, the authors should extend their discussion of social impacts.

Both promises/new results were instrumental in the paper's acceptance, and the authors will maximize the impact of their paper if they make these changes.
Decision: Accept (Poster)