Progressive Coordinate Transforms for Monocular 3D Object Detection

Abstract
Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment. While significant progress has been achieved with expensive LiDAR point clouds, it poses a great challenge for 3D object detection given only a monocular image. While there exist different alternatives for tackling this problem, it is found that they are either equipped with heavy networks to fuse RGB and depth information or empirically ineffective to process millions of pseudo-LiDAR points. With in-depth examination, we realize that these limitations are rooted in inaccurate object localization. In this paper, we propose a novel and lightweight approach, dubbed Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. Specifically, a localization boosting mechanism with confidence-aware loss is introduced to progressively refine the localization prediction. In addition, semantic image representation is also exploited to compensate for the usage of patch proposals. Despite being lightweight and simple, our strategy leads to superior improvements on the KITTI and Waymo Open Dataset monocular 3D detection benchmarks. At the same time, our proposed PCT shows great generalization to most coordinatebased 3D detection frameworks. The code is available at: https://github.com/ amazon-research/progressive-coordinate-transforms.

1 Introduction
Object detection is a fundamental and challenging task in scene understanding applications. Recently, 3D object detection has received increasing attention and found applications in a wide range of scenarios such as autonomous driving, robotics, visual navigation and mixed reality. Despite the great progress from the area of 2D object detection [34, 49, 40, 18, 4], 3D object detection remains a largely unsolved problem as it aims to predict the object location in the 3D space alongside 3D object dimension and orientation.
Existing prevalent approaches [50, 44, 35, 10, 11] for 3D object detection largely rely on LiDAR sensors, which provide accurate 3D point clouds of the scene. Although these approaches achieve superior performance, the dependence on expensive equipment severely limits their applicability to generic 3D perception. There also exists a cheaper alternative that takes a single-view RGB image as input, termed as monocular 3D object detection. However, its performance is far from satisfactory as itself is an ill-posed problem due to the loss of depth information in 2D image planes. Hence, several recent attempts introduce depth information to help monocular 3D detection. Such attempts can be roughly categorized into two directions, pixel-based and coordinate-based. Pixel-based ∗Work done during an internship at Amazon. †Li Zhang (lizhangfd@fudan.edu.cn) and Xiangyang Xue (xyxue@fudan.edu.cn) are the corresponding authors. Li Zhang is with School of Data Science, Fudan University. Li Wang and Xiangyang Xue are with School of Computer Science, Fudan University.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
approaches [9, 36, 31, 41] turn to use estimated depth map as additional input for improved detection performance. But at the same time, this leads to heavy computational burden and large memory footprint since they often operate on the entire image. Coordinated-based approaches [42, 29, 46, 27] pursue the coordinate representations as in LiDAR-based methods. They use the predicted depth map to convert the monocular image pixels to 3D coordinate representations, then apply a 3D detector on the converted coordinates. In particular, they are often lightweight since their network inputs are object proposals generated by 2D detectors [29, 27]. However, the performance of coordinated-based methods lags far behind LiDAR-based methods. So we ask, can we identify the bottleneck that holds back the 3D detection accuracy of coordinate-based methods and how can we improve them?
In order to determine the bottleneck, we conduct an investigation on two widely adopted coordinatebased methods, PatchNet [27] and Pseudo-LiDAR [42]. Specifically, for each prediction target, we examine the potential improvement by replacing its value with the corresponding ground truth, and then re-compute the 3D detection accuracy. As shown in Table 1, using ground truth dimension and rotation do not bring significant improvements over the baseline. But using ground truth location (i.e., x/y/z values of the objects) almost triples detection accuracy. This indicates that coordinate-based methods mostly suffer from inaccurate localization even with the assistance of estimated depth maps.
Based on this observation, we focus on improving the accuracy of 3D center localization. In this work, we propose a lightweight and generalized approach, called Progressive Coordinate Transforms (PCT), to enhance the localization capability for coordinate-based methods. First of all, since the localization regression network in most coordinate-based methods is less accurate but lightweight, we propose to progressively refine its prediction similar to gradient boosting [12, 13]. To be specific, a localization regression network can be seen as a weak learner, and we progressively train multiple consecutive networks such that each network fits the regression residual from the previous networks. These networks share the same lightweight structure so that the computation overhead is negligible. We also predict a confidence score for each network to help stabilize the end-to-end training. We term this progressive refining strategy as confidence-aware localization boosting (CLB). Compared to image-only or pixel-based methods, coordinated-based methods suffer from the problem of missing global context information due to the use of patched input. In order to further improve the localization accuracy, we exploit semantic image representations from 2D detector. We term this module as global context encoding (GCE). We find that GCE can not only improve center localization accuracy, but also contribute to the final 3D box estimation.
Through extensive experiments, our progressive coordinate transforms, consisting of CLB and GCE, is shown to improve popular coordinate-based models [42, 27] by generating more accurate localization. Without bells and whistles, we achieve state-of-the-art monocular 3D detection performance on KITTI [16, 17, 15] with a strong base method [27]. Additionally, this also leads to superior improvements on Waymo Open Dataset [38] compared with the base method PatchNet.

2.1 Monocular 3D object detection
Existing paradigms for monocular 3D object detection can be categorized into two types: image-only methods and depth-assisted methods.
For image-only methods, they often adapt architectures and good practices from popular 2D detectors [34, 49, 40]. However, locating objects in 3D space is much more challenging without depth information. Hence, several works [30, 2, 25, 6, 49] integrate geometry consistency into the training strategy to constrain the localization prediction. Deep3DBox [30] divides orientation into multi-bins to stably regress them, and combines the 2D-3D box constraint to recover accurate 3D object pose. M3D-RPN [2] utilizes the geometric relationship between 2D and 3D perspectives by sharing the prior anchors and classification targets. MonoPair [6] leverages the spatial relationships between paired objects to improve accuracy on occluded objects. To further improve the performance of truncated objects, MonoFlex [48] decouples the features learning and prediction of truncated objects, and formulates an depth estimation to adaptively combine independent estimator based on uncertainty. [33] designs CaDDN as a fully differentiable end-to-end approach for joint depth estimation and object detection.
Depth-assisted methods often estimate a depth map given a input image, and use it in different ways. Some pixel-based approaches [9, 26] directly feed images and estimated depth maps into networks to generate depth-aware features and enhance the 3D detection performance. Some other coordinatebased approaches first transform the pixels of input images to 3D coordinates by leveraging the depth and camera information, then feed the coordinate proposals to a 3D detector. Pioneering work Pseudo-LiDAR [42] imitates the process of LiDAR-based approaches, which uses LiDAR-based 3D detector upon coordinates proposals. AM3D [29] explores the multi-modal input fusion to embed the complementary RGB cue into the network. Recently, PatchNet [27] points out that the efficacy of pseudo-LiDAR representation comes from the coordinate transform, instead of sophisticated LiDAR-based networks. Hence, they design a simple 2D CNN to perform 3D detection. In this work, we follow the research of coordinate-based methods [42, 27]. Instead of regressing 3D localization directly with a single lightweight network, we propose to progressively refine the prediction inspired by gradient boosting. We also incorporate RGB image information to complement patch proposals and enhance global context modeling. Different from AM3D [29], we utilize the RGB features from the 2D detector directly which can share the same context, and we do not need to train another RGB network from scratch.

2.2 Gradient boosting
Gradient boosting is a well-known greedy algorithm proposed in [7], which trains a sequence of learners and progressively improves the prediction results. It is a general learning framework, and has been verified to be a formidable force when applied with lightweight learners. Meanwhile, when each learner in the sequence is heavy, the computation cost becomes high and the performance is not beneficial [24]. Early works in 2D detection area [19, 21, 22] also adopt the boosting mechanism following a standard cascade paradigm, and achieve improved performance. Li et al. [21] treat face detection as an image retrieval task and improve it with a boosted exemplar-based face detector. Karianakis et al. [19] and Li et al. [22] feed convolutional features of proposals instead of hand-crafted features to boosted classifiers and distinguish objects from backgrounds. We can also find the usage of gradient boosting in other computer vision tasks [37, 51].
To our best knowledge, we are the first to explore the boosting mechanism in coordinate-based methods for 3D object detection. We perform this mechanism in two folds. First, instead of the entire 3D detection pipeline, we only progressively boost the localization regression network as its computational cost is insignificant comparing to the entire pipeline. Second, we refine the localization with an additional confidence score in the boosting procedure, such that the loss is balanced. These choices greatly improves the performance with small extra parameters.

3 Background
Before diving into the details, we first revisit recent coordinate-based monocular 3D detection methods and present a visual depiction of its common pipeline in Figure 1. The framework usually consists of four main components: 2D bounding box generation, depth map estimation, data transformation and 3D box estimation. Specifically, given an image I , the process can be described as:
2D bounding box generation. An off-the-shelf 2D object detector F2d such as Faster R-CNN [34] is employed on image I to generate region of interests (RoIs),R = F2d(I).
Depth map estimation. An off-the-shelf depth estimator Fz such as DORN [14] is applied on image I to predict its depth map, Z = Fz(I). Data transformation. To convert a pixel (u, v) within a RoI to 3D space, the associated depth z = Z(u, v) is used to transform it into its 3D coordinates (cx, cy, cz) by
cx = (u− u′)× z
fu ; cy = (v − v′)× z fv ; cz = z (1)
Here, (cx, cy, cz) is a pixel in the generated 3D coordinate patch c. (u′, v′) is the camera principal point. fu and fv are the focal length along horizontal and vertical axis, respectively. u′, v′, fu, fv are usually provided by the datasets.
3D box estimation. Once the 3D coordinates for each RoI are available, the final step is to predict 3D boxes with their center location, rotation and dimension. Different networks F3d such as Frustum PointNet [32] can be employed to conduct 3D box prediction, B = F3d(c). Here, B includes the center location (x, y, z), rotation (θ), and dimensions (w, h, l) of the 3D box.
Since the first two steps use off-the-shelf models and the third step can be computed analytically, in this paper, we focus on improving the last step of coordinated-based methods. In particular, our goal is to improve the accuracy of localization prediction motivated by the observation in Table 1.

4 Method
In this section, we present our progressive coordinate transforms (PCT) for improved 3D detection. In order to obtain more accurate localization predictions, we introduce a confidence-aware localization boosting mechanism (CLB) in Sec. 4.1 to progressively refine the prediction. Then in Sec. 4.2, we incorporate RGB image information by a global context encoding (GCE) strategy to compensate for the drawbacks of using patch proposals. In the end, we illustrate the overall framework of PCT in Figure 2 (a).

4.1 Confidence-aware localization boosting
Following Frustum PointNet [32], most coordinate-based methods [42, 45, 27, 43] divide the last step of 3D box estimation into two major components. The first component is a lightweight 3D localization regressor F , whose input is 3D coordinate proposals generated from data transformation. The second component is a relatively heavy network G used to regress the final 3D box B. Recalling the results in Table 1, 3D localization performance is the weakest point of a coordinate-based model, accounting for up to 50 AP loss when all other modules keep intact. Therefore, can we find an efficient way to improve the accuracy of localization prediction and also generalizes to other coordinated-based methods?
Gradient boosting [12, 13] is a general learning framework that combines multiple weak learners into a single strong one in an iterative fashion. Let L(x) be the risk of ensemble models, the algorithm devotes to seek an approximation h(x) to minimize L(y∗, x) = Ψ(y∗, h(x)), where y∗ is a target value, Ψ(·) is the loss function. h(x) is a linear combination of a set of weak (base) learners ft(x) from some class F , i.e., h(x) = ∑t=T t=1 γtft(x)+const. Here, T is the total training iterations and γt is the corresponding weight for each weak learner. To minimize the empirical risk, the algorithm starts with a model h0(x), and then incrementally expands it in a greedy manner. This process manages to fit a new weak learner to the residual errors made by the previous set of learners. Mathematically, the
optimization can be formulated as
h0(x) = arg min γ0
L0(y∗, x);
...
ht(x) = ht−1(x) + arg min ft∈F
Lt(y∗, ht−1(x) + ft(x)). (2)
Inspired by gradient boosting, we imitate its optimization procedure to progressively adapt localization prediction by multiple localization regressors instead of a single one used in previous works [42, 29, 46, 27]. To be specific, we treat localization network F as a weak learner and stack multiple of them as shown in Figure 2 (b). Given F is a lightweight network, the extra computational cost brought by gradient boosting is insignificant. After the data transformation step, each 2D bounding box obtains its corresponding coordinate patch c. We take the coordinate patch as the input of weak learner F to regress the center localization residual ∆(x, y, z) based on the prediction from previous stage,
∆(xt, yt, zt) = Ft(ct−1); where ct−1 = c0 − γ0(x0, y0, z0)− t−1∑ i=1 γi∆(xi, yi, zi). (3)
We denote c0 and (x0, y0, z0) to be the initial coordinate input patch and object location, respectively. Coordinate input patch ct−1 is then transformed according to the localization residual prediction, and fed into the next weak learner.
Thus the risk at stage t can be written as,
LFt((x∗, y∗, z∗), ct−1) = Ψ((x∗, y∗, z∗), γ0(x0, y0, z0) + t−1∑ i=1 γi∆(xi, yi, zi)) (4)
where (x∗, y∗, z∗) indicates the ground truth location. At this point, we can easily see that Eq 4 is a natural derivation from Eq 2. After T iterations, the final adjusted prediction cT is fed into the network G to estimate the 3D box B, i.e.B = G(cT ). Confidence-aware network loss In the case that the target of weak learner f is differentiable, gradient boosting solves the optimization problem in a forward greedy manner as shown in Eq 2. For each iteration, it first fits the weak learner f to the residual error, and then the optimal value of the coefficient weight γ is determined for this weak learner. The optimization procedures train iteratively for T iterations.
However, for our center regression task, we would like to train T weak localization networks Ft(ct−1), t ∈ 1, ..., T and a 3D box prediction network G in an end-to-end manner instead of bootstrapping. This is challenging in terms of both computational cost and optimization stability, given the simultaneous training of a set of weak learners and their coefficient weights. Therefore, we first simplify the problem by treating all γt the same and only focus on optimizing the localization networks. However, the contribution from each weak learner F may not be the same during end-toend training, which leads to unstable optimization. Hence, we tailor a confidence-aware boosting loss to facilitate network training, by learning confidence score st for each localization loss function LFt . The confidence score st is learned from a small decoder and followed a self-balancing formulation closely coupled to the network loss. The overall loss function is defined as
L(B∗, c0) = T∑ t=1 st ∗ LFt((x∗, y∗, z∗), ct−1) + λs T∏ t=1 (1− st) + LG(B∗, cT ), (5)
where B∗ is the 3D box ground truth, (x∗, y∗, z∗) ∈ B∗ and λs is the balance weight. st is the prediction after sigmoid, which represents the confidence of the localization regression loss at tth
stage, and ∏T t=1(1−st) is the penalty on the network uncertainty. In other words, if st is approaching 1, which means the network is confident about localization refinement at tth stage, then no penalty will be applied. Otherwise, the uncertainty of regression loss is high, thus triggers a higher penalty.

4.2 Global context encoding
Typical coordinate-based methods [42, 45, 27, 43] perform 3D detection based on 2D RoIs, which is similar to two-stage 2D detection frameworks, such as Faster R-CNN [34]. In a two-stage 2D object detection framework, the second stage reuses the features from the first stage via RoIPooling [34] or RoIAlign operators [18] guided by ROI proposals, and then a small decoder is used for localization refinement. However, in 3D detection, only cropped patches with coordinates information are fed to the network for 3D box regression. Neither RGB information nor global context is included.
Considering that the RGB information is a vital visual clue, we explore its aggregation in the last 3D box estimation step. Similar to two-stage 2D detectors, we obtain the RGB information by directly cropping the corresponding features from a 2D detector F2d. Then the input to 3D box estimator G can be formulated as c = {[D(u, v),A(u, v)],∀(u, v) ∈ R}. Here, D(·) represents the data transformation function and A(·) represents the RoIAlign operation. Both operations are performed upon the generated regions of interest fromR. After RoIAlign operation, features are of size C ×K ×K, where C is the number of channels and K ×K is the corresponding width and height, respectively. A small feature encoder is then used to encode cropped features into vectors with the dimension of C. A feature fusion is followed to integrate coordinate representations with the obtained image representations. Benefiting from the large receptive fields of image feature representations, 3D box estimator can now have access to global context. Besides, directly cropping on RGB features also avoids learning image representations of RoIs from scratch and reduces the overall network parameters.

5 Experiments
Two monocular 3D detection benchmarks are introduced in Sec. 5.1 and Sec. 5.2, while experimental implementation details are described in Sec. 5.3. In Sec. 5.4, we conduct main analysis on KITTI dataset [16, 17, 15] with base method PatchNet [27] given its current best performance. More experiments on Waymo Open Dataset [38] are also demonstrated to further verify the generality of our proposed PCT in Sec. 5.5.

5.1 KITTI setup
We first evaluate our method on the KITTI benchmark [16, 17, 15], which contains 7,481 and 7,518 images for training and testing respectively. We follow [5] to split the 7,481 training images into 3712 for training and 3,769 for validation.
Precision-recall curves are adopted for evaluation, and we report the average precision (AP) results of 3D and Bird’s eye view (BEV) object detection on KITTI validation and test set. For fair comparison to previous literature, the 40 recall positions-based metricAP |R40 is reported on test set whileAP |R11 is reported on validation set. Three levels of difficulty are defined in the benchmark according to the 2D bounding box height, occlusion, and truncation degree, namely, “Easy”, “Mod.”, and “Hard”. The KITTI benchmark ranks all methods based on the AP3D of “Mod.”. In particular, we focus on the “Car” category as in [42, 27], and we adopt IoU = 0.7 as threshold for evaluation.

5.2 Waymo setup
We also carry out experiments on large-scale, high quality and diverse dataset, Waymo open dataset [38]. It provides pre-defined 798 training sequences and 202 validation sequences from different scenes, and another 150 test sequences without labels. The dataset contains camera images from five high-resolution pinhole cameras, and we only consider images with their 3D labels from front camera for monocular 3D detection task. We sample every third frame from the training sequences (total 52,386 images) as in CaDDN [33] to form the training set due to its large scale. And validation set contains all the 39,848 images from 202 different scenes.
For evaluation, we adopt the officially released evaluation [39] to calculate the mean average precision (mAP) and the mean average precision weighted by heading (mAPH). Two levels are included according to difficulty rating, which are defined by LiDAR points. 3D labels without any points are ignored, LEVEL_2 is assigned to examples when it contains equal or lesser than 5 points, while the rest of the examples are assigned to LEVEL_1. Additionally, three distances (0 - 30m, 30 - 50m, 50m -∞) to sensor are considered during evaluation.

5.3 Implementation details
Our overall framework of PCT can be visualized in Figure 2. In terms of implementation details, we instantiate our algorithm on two widely adopted coordinate-based methods with public released code [28], PatchNet and Pseudo LiDAR. Bearing efficiency in mind, we use a real-time 2D detector RTM3D [23] with DLA-34 [47] as backbone. For the sake of fair comparison, we adopt depth predictor DORN [14] on KITTI dataset as in most depth-assisted literature. Since there is no published depth results on Waymo open dataset, we adopt a most recent monocular depth estimator AdaBins [1] trained on Waymo training set. For the CLB mechanism, we inherit the original localization regression framework in each method. Each Ft shares the same structure. The corresponding confidence is generated following the last convolutional layers of Ft with three linear layers and a Sigmoid function. T = 3 and λs = 1 are set for the following experiments except for the ablation study on boosting iterations. For GCE module, we get the corresponding input image features by performing RoIAlign on the features from last convolutional layer of 2D detector. We set the output of RoIAlign as 16× 16. As 2D detector use DLA-34 as backbone, the obtained image feature representations have the size of 64× 16× 16 and entitle arbitrary sized receptive field theoretically due to the embedded deformable convolution [8]. The structure of feature encoder in global context module is two common 64× 3× 3 convolutional layers (stride=4) and a 64× 1× 1 convolutional layer (stride=1). Hence, image feature representations are encoded to a vector with 64-dim. The obtained features are then concatenated with the coordinate feature vectors from the final global pooling of box prediction network G. With the lightweight structure, we are able to optimize the network end-to-end on a single Nvidia V100
GPU with 16G memory. The training criterion for network F and G and other training settings follow the corresponding base methods for fair comparison.

5.4 Method analysis on KITTI dataset
Main ablative analysis In Table 2, we conduct ablation studies to analyze the effectiveness of our contributions: I) Without any localization regression network F , network G generates 3D box prediction directly. II) This configuration only contains localization boosting part without confidence constraint. III) The entire CLB mechanism is included to progressively regress center localization. IV) GCE module is added to the network based on the localization boosting block without confidence constraint since the feature fusion can be performed on either the localization regression networks F or 3D box prediction network G. V) Our full method with all the components.
As depicted in Table 2, we can observe that the performance continues to grow with the addition of every component. From group II, localization boosting brings a noticeable improvement on all settings especially “Hard” set, which confirms its effectiveness in increasing localization accuracy. Group III shows that balancing training loss by adding confidence leads to better and more stable optimization. Group IV reveals that the proposed GCE module can effectively equip RGB information and global context with 3D coordinate representations. In the end, Group V demonstrates the complementarity of the proposed CLB mechanism and GCE module, leading to an improvement from 25.88/36.07/20.99 to 27.53/38.39/24.44 compared with Group I.
Localization boosting iteration settings We explore the effect of different localization boosting iteration settings in this part. For a fair comparison, we do not perform the confidence constraint on regression loss. As illustrated in Table 3, when boosting iteration T = 3, we achieve the best 3D detection performance. More iterations of boosting do not bring improvements, which might be caused by overfitting with the increasing of network parameters.
To verify the improvement of each step in boosting procedure, we conduct the comparison of localization errors at iteration T = 3 on the specific metrics (location “xyz”) of the ground truth. In particular, three stacked localization networks F generate intermediate localization “loc.1/2/3” and G output the final localization “loc.4”. As shown in Figure 3, we can see that the distributions of “x”,
“y” and “z” errors tend to distributed to zero with localization boosting iterating. For instance, the red line in left chart is narrow and tall near zero along horizontal axis compared with other lines, which means that the corresponding x coordinate is more accurate than others. This further suggests that localization boosting is useful for object localization. The corresponding BEV visualization will be shown in Supplementary Material.
Impact of global context encoding We also explore where the global context representation fusion operates. We take Group II as the baseline, and perform feature fusion on localization regression network F (row one), 3D estimation network G (row two) or on both (final row). RoI features are encoded into a vector with GCE and then concatenate with the feature vectors from network (F or G) global pooling. As shown in Table 4, the operation on G outperforms it on F , which indicates that image representation is more suitable for the overall box prediction rather than only localization as it contains the additional semantic appearance information. Although operation on all networks achieves a lightly higher than it on G on the “Easy” and “Hard” set, introducing parameters is much larger due to operation on stacked localization networks. Hence, we only apply GCE on network G in our approach for a lightweight network and avoid overfitting during training.
Generally applicable to other coordinate-based algorithm In this section, we demonstrate the generalization capability of our algorithm to classic coordinate-based methods Pseudo-LiDAR [42]. As shown in Table 5, each component of our algorithm improves the original methods a lot. Specially, our approach improves Pseudo-LiDAR by 1.43/2.06/0.51 while 1.22/1.99/3.36 gains on PatchNet.
Comparison with state-of-the-arts We build our test detector on the current state-of-the-art coordinate-based method PatchNet, and results are shown in Table 6. Quantitatively, our method achieves the highest performance on “Mod.” set with 22 FPS on NvidiaTesla v100 including 2D detector inference time, which is the main setting for ranking on the benchmark. Specially, large margins, 2.25/5.32/1.14 on 3D detection and 2.17/6.68/0.95 on BEV, are observed over the base method PatchNet with only additional 3.41M parameters. Besides, our methods also outperforms the pixel-based state-of-the-arts methods Liu et al. [26] especially on “Hard” set.
Qualitative comparisons are shown in Figure 4. The ground truth, base method (PatchNet), and our method are colored in green, blue, and red, respectively. For better visualization, the first and second rows show RGB images and BEV images, respectively. Compared with the base method, our algorithm can produce higher-quality 3D bounding boxes in different kinds of scenes.

5.5 Results on Waymo Open Dataset
Table 7 shows the results of base method PatchNet [27] and our proposed PCT. It can be observed that our method consistently outperforms the base method on mAP/mAPH of 0.50%/0.51% and 0.28%/0.30% on the LEVEL_1 and LEVEL_2 difficulties respectively under IoU = 0.7. Again, our method is efficient, e.g, it takes 5 days to complete training on large scale Waymo dataset with a 8-GPU node. More qualitative results can be seen at Supplementary Material.

6 Conclusions
In this paper, we have introduced a novel approach PCT to address the inaccurate localization problem for monocular 3D object detection. This is achieved by iteratively transforming the coordinate representation with a confidence-aware booting mechanism. Meanwhile, global context is introduced to compensate for the missing of semantic image representation in coordinated-based methods. Through extensive experiments, we have shown that our proposed PCT substantially improve the performance of the coordinate-based model by a large margin, and achieve state-of-the-art monocular 3D detection performance on KITTI test set. Moreover, we also show consistent improvements compared to the strong baseline on the large-scale Waymo Open dataset.
There are several limitations that could indicate the possible directions for future work. First, the performance of off-the-shelf 2D detector directly influences the accuracy of coordinate-based methods, hence how to effectively design the 3D box estimation algorithm to fit with existing 2D detectors is important. Second, we only concentrate on the lightweight coordinate-based methods. It requires further exploration to extend our approach to pixel-based methods. Finally, our proposed global context encoding is a simple module. Despite working well, a more tailored feature fusion strategy between coordinate representation and RGB image representation is worth exploring.
Potential impacts. Our method focuses on the monocular 3D detection which can be applied in autonomous driving field. One potential social problem of our work is that it may aggravate the employment crisis of human servants and drivers, which replaces human with autonomous robots and intelligent systems.

Acknowledgments
This work was supported by Shanghai Municipal Science and Technology Major Projects (No.2021SHZDZX0103 and No.2018SHZDZX01).

Limitations
There are several limitations that could indicate the possible directions for future work. First, the performance of off-the-shelf 2D detector directly influences the accuracy of coordinate-based methods, hence how to effectively design the 3D box estimation algorithm to fit with existing 2D detectors is important. Second, we only concentrate on the lightweight coordinate-based methods. It requires further exploration to extend our approach to pixel-based methods. Finally, our proposed global context encoding is a simple module. Despite working well, a more tailored feature fusion strategy between coordinate representation and RGB image representation is worth exploring. Potential impacts. Our method focuses on the monocular 3D detection which can be applied in autonomous driving field. One potential social problem of our work is that it may aggravate the employment crisis of human servants and drivers, which replaces human with autonomous robots and intelligent systems.

Reviewer Summary
Reviewer_1: This paper presents a "progressive coordinate transform" strategy for improving monocular 3D object detection. Specifically, the authors identify that the main bottleneck in performance for monocular 3D object detection is the localization performance and tailor a solution towards progressively refining the coordinate proposals with multiple stacked modules before making the final 3D bbox prediction. The authors also propose a "Global context encoding" which processes and concatenates the visual features from the object proposal before the final 3D bbox prediction. This empirically results in an improvement in overall 3D bbox detection performance - achieving state of the art of the KITTI monocular 3D object detection benchmark. The primary contribution of this paper is identifying localization performance as a key bottleneck for 3D monocular object detection and tailoring a simple solution to improve performance over prior work.

Reviewer_2: The paper first analyzes where the main performance loss is in the inference of 3D bounding boxes from 2D images and concludes it is in the 3D box center estimation phase. The introduced method is an add-on iterative refinement which can boost performance of several algorithms, and improves state-of-art when boosting PatchNet, as applied to the KITTI dataset.

Reviewer_3: This paper proposes progressive coordinate transforms to solve the problems in monolar 3D object detection. But there are some problems with workload and experiments.

Reviewer Comments
Reviewer_1: Strengths
Table 1 in the paper is most insightful in my opinion and identifies localization as a bottleneck (vs size or rotation prediction). This is also somewhat expected because the variation in dimensions for cars is not expected to be high and orientation is also expected to be in a constrained range in driving scenarios. However quantifying this still has value. The authors build from this simple intuition and propose an iterative refinement method after the coordinate proposal module in standard 3D monocular object detection methods. The simplicity is key.
Weaknesses
The two major concerns I have with the paper are technical novelty and significance of improvement over prior work. The major contribution in this work is really the "CLB" module which is primarily an iterative refinement module which predicts residuals between the ground truth and the previous prediction at each stage. The GCE module which appends a visual feature vector computed from the detector backbone isnt necessarily novel and has relatively low effect on performance (<0.4AP in table 4).
The final improvement on the validation set seems significant for the "Hard" subset (~3.5AP in AP3D) but might not be very significant in my opinion on the other subsets and metrics. This is true when comparing against non coordinate-based methods on the KITTI test set in Table 6.
The figures in the paper often do not add much to the discussion. Fig 1 lacks more visuals and primarily a set of blocks. The caption for Fig 2(a) does not do a good job of explaining it and one has to follow the text very closely to understand the schematic here. The top row if FIg 3(a) is also hard to follow. The authors mention "peakier" red distribution curves indicating decreasing localization error over iterations but this is not readily visible to me. The BEV visualization and image bbox visualization in Fig 4 are also very small and the boxes themselves are hard to see. Overall, the figures in the manuscript need quite some improvement.
Limitations And Societal Impact:
The authors are encouraged to comment on the technical novelty and significance of improvements over prior work on the KITTI dataset.
Needs Ethics Review: No
Time Spent Reviewing: 2

Reviewer_2: The paper introduces an iterative box center coordinate refinement stage into the 3D BBox estimation (based on 2D box detections and monocular depth estimates). Confidence weights are estimated for each progressive refinement. Semantic information is also added, which also boosts box location estimation. An ablation study shows that each of the 3 added components boosts performance a little. The proposed addition is also added to the Pseudo-LiDAR algorithm, also improving performance. Comparison of the number of iterations of the localization boost was done to select the best number of iterations (3).
The paper is good: some nice analysis and ideas, good experiments, clearly written, but is also incremental; hence the proposed score of 8.
Limitations And Societal Impact:
I saw no obvious weaknesses. I was curious about the magnitude of the weights learned. There was a slight confusion between T=3 and +2 in table 3. Presumably they mean the same thing, so it might be better to use the T values rather than the increments.
Ethical Concerns:
There are general ethical issues about the impact of autonomous vehicles.
Needs Ethics Review: No
Time Spent Reviewing: 2

Reviewer_3: The main contribution of this paper focuses on the application of gradient enhancement to localization prediction. Global context encoding seems just a small improvement to former algorithms. Therefore, the overall contribution is not particularly large.
The experiment in this paper is carried out on the KITTI data set. The results on the test set in Table 6 should be obtained from the official website submitted to the official website, but the corresponding data (anonymous) is not seen on the KITTI website. So where did the experimental results in Table 6 come from? If it was obtained by KITTI, please submit it anonymously; if not, then there is a problem with the authenticity of the data. In addition, KITTI includes three types of detection object. Why does the experimental result only include the detection result of the car? How about the other two types of test results?
The experiment is only carried out on the KITTI dataset, but more experiments on data sets such as waymo and nuscences should be carried out. In addition, the method in this paper is actually a 3D object detection algorithm based on monocular images, so it should be compared with other algorithms of the same type (3D object detection based on monocular image), such as GrooMeD-NMS[1], DDMP-3D[2], CaDDN[3], etc, but this paper does not compare them.
[1] A. Kumar, G. Brazil and X. Liu: GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection. CVPR 2021. [2] L. Wang, L. Du, X. Ye, Y. Fu, G. Guo, X. Xue, J. Feng and L. Zhang: Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection. CVPR 2020. [3] C. Reading, A. Harakeh, J. Chae and S. Waslander: Categorical Depth Distribution Network for Monocular 3D Object Detection. CVPR 2021.
This paper claims to design a lightweight coordinate-based network, so what is the amount of calculation and complexity of the algorithm in this paper? How does it compare to other coordinate-based networks? The comparison between the results of this part and other methods is also very important, but it does not appear.
Limitations And Societal Impact:
The authors have not addressed the limitations and potential negative societal impact of their work.
Needs Ethics Review: No
Time Spent Reviewing: 8 hours