Progressive Coordinate Transforms for Monocular 3D Object Detection

1 Introduction . Object detection is a fundamental and challenging task in scene understanding applications . Recently , 3D object detection has received increasing attention and found applications in a wide range of scenarios such as autonomous driving , robotics , visual navigation and mixed reality . Despite the great progress from the area of 2D object detection [ 34 , 49 , 40 , 18 , 4 ] , 3D object detection remains a largely unsolved problem as it aims to predict the object location in the 3D space alongside 3D object dimension and orientation . Existing prevalent approaches [ 50 , 44 , 35 , 10 , 11 ] for 3D object detection largely rely on LiDAR sensors , which provide accurate 3D point clouds of the scene . Although these approaches achieve superior performance , the dependence on expensive equipment severely limits their applicability to generic 3D perception . There also exists a cheaper alternative that takes a single-view RGB image as input , termed as monocular 3D object detection . However , its performance is far from satisfactory as itself is an ill-posed problem due to the loss of depth information in 2D image planes . Hence , several recent attempts introduce depth information to help monocular 3D detection . Such attempts can be roughly categorized into two directions , pixel-based and coordinate-based . Pixel-based ∗Work done during an internship at Amazon . †Li Zhang ( lizhangfd @ fudan.edu.cn ) and Xiangyang Xue ( xyxue @ fudan.edu.cn ) are the corresponding authors . Li Zhang is with School of Data Science , Fudan University . Li Wang and Xiangyang Xue are with School of Computer Science , Fudan University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approaches [ 9 , 36 , 31 , 41 ] turn to use estimated depth map as additional input for improved detection performance . But at the same time , this leads to heavy computational burden and large memory footprint since they often operate on the entire image . Coordinated-based approaches [ 42 , 29 , 46 , 27 ] pursue the coordinate representations as in LiDAR-based methods . They use the predicted depth map to convert the monocular image pixels to 3D coordinate representations , then apply a 3D detector on the converted coordinates . In particular , they are often lightweight since their network inputs are object proposals generated by 2D detectors [ 29 , 27 ] . However , the performance of coordinated-based methods lags far behind LiDAR-based methods . So we ask , can we identify the bottleneck that holds back the 3D detection accuracy of coordinate-based methods and how can we improve them ? In order to determine the bottleneck , we conduct an investigation on two widely adopted coordinatebased methods , PatchNet [ 27 ] and Pseudo-LiDAR [ 42 ] . Specifically , for each prediction target , we examine the potential improvement by replacing its value with the corresponding ground truth , and then re-compute the 3D detection accuracy . As shown in Table 1 , using ground truth dimension and rotation do not bring significant improvements over the baseline . But using ground truth location ( i.e. , x/y/z values of the objects ) almost triples detection accuracy . This indicates that coordinate-based methods mostly suffer from inaccurate localization even with the assistance of estimated depth maps . Based on this observation , we focus on improving the accuracy of 3D center localization . In this work , we propose a lightweight and generalized approach , called Progressive Coordinate Transforms ( PCT ) , to enhance the localization capability for coordinate-based methods . First of all , since the localization regression network in most coordinate-based methods is less accurate but lightweight , we propose to progressively refine its prediction similar to gradient boosting [ 12 , 13 ] . To be specific , a localization regression network can be seen as a weak learner , and we progressively train multiple consecutive networks such that each network fits the regression residual from the previous networks . These networks share the same lightweight structure so that the computation overhead is negligible . We also predict a confidence score for each network to help stabilize the end-to-end training . We term this progressive refining strategy as confidence-aware localization boosting ( CLB ) . Compared to image-only or pixel-based methods , coordinated-based methods suffer from the problem of missing global context information due to the use of patched input . In order to further improve the localization accuracy , we exploit semantic image representations from 2D detector . We term this module as global context encoding ( GCE ) . We find that GCE can not only improve center localization accuracy , but also contribute to the final 3D box estimation . Through extensive experiments , our progressive coordinate transforms , consisting of CLB and GCE , is shown to improve popular coordinate-based models [ 42 , 27 ] by generating more accurate localization . Without bells and whistles , we achieve state-of-the-art monocular 3D detection performance on KITTI [ 16 , 17 , 15 ] with a strong base method [ 27 ] . Additionally , this also leads to superior improvements on Waymo Open Dataset [ 38 ] compared with the base method PatchNet . 2 Related work . 2.1 Monocular 3D object detection . Existing paradigms for monocular 3D object detection can be categorized into two types : image-only methods and depth-assisted methods . For image-only methods , they often adapt architectures and good practices from popular 2D detectors [ 34 , 49 , 40 ] . However , locating objects in 3D space is much more challenging without depth information . Hence , several works [ 30 , 2 , 25 , 6 , 49 ] integrate geometry consistency into the training strategy to constrain the localization prediction . Deep3DBox [ 30 ] divides orientation into multi-bins to stably regress them , and combines the 2D-3D box constraint to recover accurate 3D object pose . M3D-RPN [ 2 ] utilizes the geometric relationship between 2D and 3D perspectives by sharing the prior anchors and classification targets . MonoPair [ 6 ] leverages the spatial relationships between paired objects to improve accuracy on occluded objects . To further improve the performance of truncated objects , MonoFlex [ 48 ] decouples the features learning and prediction of truncated objects , and formulates an depth estimation to adaptively combine independent estimator based on uncertainty . [ 33 ] designs CaDDN as a fully differentiable end-to-end approach for joint depth estimation and object detection . Depth-assisted methods often estimate a depth map given a input image , and use it in different ways . Some pixel-based approaches [ 9 , 26 ] directly feed images and estimated depth maps into networks to generate depth-aware features and enhance the 3D detection performance . Some other coordinatebased approaches first transform the pixels of input images to 3D coordinates by leveraging the depth and camera information , then feed the coordinate proposals to a 3D detector . Pioneering work Pseudo-LiDAR [ 42 ] imitates the process of LiDAR-based approaches , which uses LiDAR-based 3D detector upon coordinates proposals . AM3D [ 29 ] explores the multi-modal input fusion to embed the complementary RGB cue into the network . Recently , PatchNet [ 27 ] points out that the efficacy of pseudo-LiDAR representation comes from the coordinate transform , instead of sophisticated LiDAR-based networks . Hence , they design a simple 2D CNN to perform 3D detection . In this work , we follow the research of coordinate-based methods [ 42 , 27 ] . Instead of regressing 3D localization directly with a single lightweight network , we propose to progressively refine the prediction inspired by gradient boosting . We also incorporate RGB image information to complement patch proposals and enhance global context modeling . Different from AM3D [ 29 ] , we utilize the RGB features from the 2D detector directly which can share the same context , and we do not need to train another RGB network from scratch . 2.2 Gradient boosting . Gradient boosting is a well-known greedy algorithm proposed in [ 7 ] , which trains a sequence of learners and progressively improves the prediction results . It is a general learning framework , and has been verified to be a formidable force when applied with lightweight learners . Meanwhile , when each learner in the sequence is heavy , the computation cost becomes high and the performance is not beneficial [ 24 ] . Early works in 2D detection area [ 19 , 21 , 22 ] also adopt the boosting mechanism following a standard cascade paradigm , and achieve improved performance . Li et al . [ 21 ] treat face detection as an image retrieval task and improve it with a boosted exemplar-based face detector . Karianakis et al . [ 19 ] and Li et al . [ 22 ] feed convolutional features of proposals instead of hand-crafted features to boosted classifiers and distinguish objects from backgrounds . We can also find the usage of gradient boosting in other computer vision tasks [ 37 , 51 ] . To our best knowledge , we are the first to explore the boosting mechanism in coordinate-based methods for 3D object detection . We perform this mechanism in two folds . First , instead of the entire 3D detection pipeline , we only progressively boost the localization regression network as its computational cost is insignificant comparing to the entire pipeline . Second , we refine the localization with an additional confidence score in the boosting procedure , such that the loss is balanced . These choices greatly improves the performance with small extra parameters . 3 Background . Before diving into the details , we first revisit recent coordinate-based monocular 3D detection methods and present a visual depiction of its common pipeline in Figure 1 . The framework usually consists of four main components : 2D bounding box generation , depth map estimation , data transformation and 3D box estimation . Specifically , given an image I , the process can be described as : 2D bounding box generation . An off-the-shelf 2D object detector F2d such as Faster R-CNN [ 34 ] is employed on image I to generate region of interests ( RoIs ) , R = F2d ( I ) . Depth map estimation . An off-the-shelf depth estimator Fz such as DORN [ 14 ] is applied on image I to predict its depth map , Z = Fz ( I ) . Data transformation . To convert a pixel ( u , v ) within a RoI to 3D space , the associated depth z = Z ( u , v ) is used to transform it into its 3D coordinates ( cx , cy , cz ) by cx = ( u− u′ ) × z fu ; cy = ( v − v′ ) × z fv ; cz = z ( 1 ) Here , ( cx , cy , cz ) is a pixel in the generated 3D coordinate patch c. ( u′ , v′ ) is the camera principal point . fu and fv are the focal length along horizontal and vertical axis , respectively . u′ , v′ , fu , fv are usually provided by the datasets . 3D box estimation . Once the 3D coordinates for each RoI are available , the final step is to predict 3D boxes with their center location , rotation and dimension . Different networks F3d such as Frustum PointNet [ 32 ] can be employed to conduct 3D box prediction , B = F3d ( c ) . Here , B includes the center location ( x , y , z ) , rotation ( θ ) , and dimensions ( w , h , l ) of the 3D box . Since the first two steps use off-the-shelf models and the third step can be computed analytically , in this paper , we focus on improving the last step of coordinated-based methods . In particular , our goal is to improve the accuracy of localization prediction motivated by the observation in Table 1 .

Summary: This paper presents a "progressive coordinate transform" strategy for improving monocular 3D object detection. Specifically, the authors identify that the main bottleneck in performance for monocular 3D object detection is the localization performance and tailor a solution towards progressively refining the coordinate proposals with multiple stacked modules before making the final 3D bbox prediction. The authors also propose a "Global context encoding" which processes and concatenates the visual features from the object proposal before the final 3D bbox prediction. This empirically results in an improvement in overall 3D bbox detection performance - achieving state of the art of the KITTI monocular 3D object detection benchmark. The primary contribution of this paper is identifying localization performance as a key bottleneck for 3D monocular object detection and tailoring a simple solution to improve performance over prior work.