MonoDistill: Learning Spatial Features for Monocular 3D Object Detection

1 INTRODUCTION . 3D object detection is an indispensable component for 3D scene perception , which has wide applications in the real world , such as autonomous driving and robotic navigation . Although the algorithms with stereo ( Li et al. , 2019b ; Wang et al. , 2019 ; Chen et al. , 2020a ) or LiDAR sensors ( Qi et al. , 2018 ; Shi et al. , 2019 ; 2020 ) show promising performances , the heavy dependence on the expensive equipment restricts the application of these algorithms . Accordingly , the methods based on the cheaper and more easy-to-deploy monocular cameras ( Xu & Chen , 2018 ; Ma et al. , 2019 ; 2021 ; Brazil & Liu , 2019 ; Ding et al. , 2020 ) show great potentials and have attracted lots of attention . As shown in Figure 1 ( a ) , some prior works ( Brazil & Liu , 2019 ; Simonelli et al. , 2019 ; Chen et al. , 2020b ) directly estimate the 3D bounding boxes from monocular images . However , because of the lack of depth cues , it is extremely hard to accurately detect the objects in the 3D space , and the localization error is the major issue of these methods ( Ma et al. , 2021 ) . To mitigate this problem , an intuitive idea is to estimate the depth maps from RGB images , and then use them to augment the input data ( Figure 1 ( b ) ) ( Xu & Chen , 2018 ; Ding et al. , 2020 ) or directly use them as the input data ( Figure 1 ( c ) ) ( Wang et al. , 2019 ; Ma et al. , 2019 ) . Although these two strategies have made significant improvement in performance , the drawbacks of them can not be ignored : ( 1 ) These methods generally use an off-the-shelf depth estimator to generate the depth maps , which introduce lots of computational cost ( e.g . the most commonly used depth estimator ( Fu et al. , 2018 ) need about 400ms to process a standard KITTI image ) . ( 2 ) The depth estimator and detector are trained separately , which may lead to a sub-optimal optimization . Recently , Reading et al . ( 2021 ) propose an end-to-end framework ( Figure 1 ( d ) ) for monocular 3D detection , which can also leverage depth estimator to provide depth cues . Specifically , they introduce a sub-network to estimate the depth distribution and use it to enrich the RGB features . Although this model can be trained end-to-end and achieves better performance , it still suffer from the low inference speed ( 630ms per image ) , mainly caused by the depth estimator and the complicated network architecture . Note that the welldesigned monocular detectors , like Ma et al . ( 2021 ) ; Zhang et al . ( 2021b ) ; Lu et al . ( 2021 ) , only take about 40ms per image . In this paper , we aim to introduce the depth cues to the monocular 3D detectors without introducing any extra cost in the inference phase . Inspired by the knowledge distillation ( Hinton et al . ) , which can transfer the learned knowledge from a well-trained CNN to another one without any changes in the model design , we propose that the spatial cues may also be transferred by this way from the LiDAR-based models . However , the main problem for this proposal is the difference of the feature representations used in these two kinds of methods ( 2D images features vs. 3D voxel features ) . To bridge this gap , we propose to project the LiDAR signals into the image plane and use the 2D CNN , instead of the commonly used 3D CNN or point-wise CNN , to train a ‘ image-version ’ LiDAR-based model . After this alignment , the knowledge distillation can be friendly applied to enrich the features of our monocular detector . Based on the above-mentioned motivation and strategy , we propose the distillation based monocular 3D detector ( MonoDistill ) : We first train a teacher net using the projected LiDAR maps ( used as the ground truths of the depth estimator in previous works ) , and then train our monocular 3D detector under the guidance of the teacher net . We argue that , compared with previous works , the proposed method has the following two advantages : First , our method directly learn the spatial cues from the teacher net , instead of the estimated depth maps . This design performs better by avoiding the information loss in the proxy task . Second , our method does not change the network architecture of the baseline model , and thus no extra computational cost is introduced . Experimental results on the most commonly used KITTI benchmark , where we rank the 1st place among all monocular based models by applying the proposed method on a simple baseline , demonstrate the effectiveness of our approach . Besides , we also conduct extensive ablation studies to present each design of our method in detail . More importantly , these experiments clearly illustrate the improvements are achieved by the introduction of spatial cues , instead of other unaccountable factors in CNN . 2 RELATED WORKS . 3D Detection from only monocular images . 3D detection from only monocular image data is challenging due to the lack of reliable depth information . To alleviate this problem , lots of scholars propose their solutions in different ways , including but not limited to network design ( Roddick et al. , 2019 ; Brazil & Liu , 2019 ; Zhou et al . ; Liu et al. , 2020 ; Luo et al. , 2021 ) , loss formulation ( Simonelli et al. , 2019 ; Ma et al. , 2021 ) , 3D prior ( Brazil & Liu , 2019 ) , geometric constraint ( Mousavian et al. , 2017 ; Qin et al. , 2019 ; Li et al. , 2019a ; Chen et al. , 2020b ) , or perspective modeling ( Zhang et al. , 2021a ; Lu et al. , 2021 ; Shi et al. , 2021 ) . Depth augmented monocular 3D detection . To provide the depth information to the 3D detectors , several works choose to estimate the depth maps from RGB images . According to the usage of the estimated depth maps , these methods can be briefly divided into three categories . The first class of these methods use the estimated depth maps to augment the RGB images ( Figure 1 ( b ) ) . In particular , Xu & Chen ( 2018 ) propose three fusion strategies of the RGB images and depth maps , while Ding et al . ( 2020 ) and Wang et al . ( 2021a ) focus on how to enrich the RGB features with depth maps in the latent feature space . Besides , Wang et al . ( 2019 ) ; Ma et al . ( 2019 ) propose another pipeline ( Figure 1 ( c ) ) : They back-project the depth maps into the 3D space , and then train a LiDAR-based model and use the resulting data ( pseudo-LiDAR signal ) to predict the 3D boxes . This framework shows promising performance , and lots of works ( Weng & Kitani , 2019 ; Cai et al. , 2020 ; Wang et al. , 2020a ; Chu et al. , 2021 ) are built on this solid foundation . Recently , Reading et al . ( 2021 ) propose another way to leverage the depth cues for monocular 3D detection ( Figure 1 ( d ) ) . Particularly , they first estimate the depth distribution using a sub-network , and then use it to lift the 2D features into 3D features , which is used to generate the final results . Compared with the previous two families , this model can be trained in the end-to-end manner , avoiding the sub-optimal optimization . However , a common disadvantage of these methods is that they inevitably increase the computational cost while introducing depth information . Unlike these methods , our model chooses to learn the feature representation under the guidance of depth maps , instead of integrating them . Accordingly , the proposed model not only introduces rich depth cues but also maintains high efficiency . Knowledge distillation . Knowledge distillation ( KD ) is initially proposed by Hinton et al . for model compression , and the main idea of this mechanism is transferring the learned knowledge from large CNN models to the small one . This strategy has been proved in many computer vision tasks , such as 2D object detection ( Dai et al. , 2021 ; Chen et al. , 2017 ; Gupta et al. , 2016 ) , semantic semantic segmentation ( Hou et al. , 2020b ; Liu et al. , 2019 ) . However , few work explore it in monocular 3D detection . In this work , we design a KD-based paradigm to efficiently introduce depth cues for monocular 3D detectors . LIGA Stereo . We found a recent work LIGA Stereo ( Guo et al. , 2021 ) ( submitted to arXiv on 18 Aug. 2021 ) discusses the application of KD for stereo 3D detection under the guidance of LiDAR signals . Here we discuss the main differences of LIGA Stereo and our work . First , the tasks and underlying data are different ( monocular vs. stereo ) , which leads to different conclusions . For example , Guo et al . ( 2021 ) concludes that using the predictions of teacher net as ‘ soft label ’ can not bring benefits . However , our experimental results show the effectiveness of this design . Even more , in our task , supervising the student net in the result space is more effective than feature space . Second , they use an off-the-shelf LiDAR-based model to provide guidance to their model . However , we project the LiDAR signals into image plane and use the resulting data to train the teacher net . Except for the input data , the teacher net and student net are completely aligned , including network architecture , hyper-parameters , and training schedule . Third , to ensure the consistent shape of features , LIGA Stereo need to generate the cost volume from stereo images , which is time-consuming ( it need about 350ms to estimate 3D boxes from a KITTI image ) and hard to achieve for monocular images . In contrast , our method align the feature representations by adjusting the LiDAR-based model , instead of the target model . This design makes our method more efficient ( about 35ms per image ) and can generalize to all kinds of image-based models in theory . 3 METHOD . 3.1 OVERVIEW . Figure 3 presents the framework of the proposed MonoDistill , which mainly has three components : a monocular 3D detector , an aligned LiDAR-based detector , and several side branches which build the bridge to provide guidance from the LiDAR-based detector to our monocular 3D detector . We will introduce the how to build these parts one by one in the rest of this section . 3.2 BASELINE MODEL . Student model . We use the one-stage monocular 3D detector MonoDLE ( Ma et al. , 2021 ) as our baseline model . Particularly , this baseline model adopts DLA-34 ( Yu et al. , 2017 ) as the backbone and uses several parallel heads to predict the required items for 3D object detection . Due to this clean and compact design , this model achieves good performance with high efficiency . Besides , we further normalize the confidence of each predicted object using the estimated depth uncertainty ( see Appendix A.1 for more details ) , which brings about 1 AP improvement 1 . Teacher model . Existing LiDAR-based models are mainly based on the 3D CNN or point-wise CNN . To align the gap between the feature representations of the monocular detector and the LiDAR-based detector , we project the LiDAR points into the image plane to generate the sparse depth map . Further , we also use the interpolation algorithm ( Ku et al. , 2018 ) to generate the dense depth , and see Figure 2 for the visualization of generated data . Then , we use these ‘ image-like LiDAR maps ’ to train a LiDAR-based detector using the identical network with our student model .

Summary: This paper proposes to distill the features from a LiDAR teacher model to a monocular-based student model. To align the feature maps between the teacher and student model, the teacher model uses the same networks as the student and the only difference is that the teacher model takes the sparse/dense depth map as input. Several techniques include scene-level distillation, object-level distillation in feature/outputs, feature fusion. In the experimental part, this paper extensively ablates several key analyses including cross-modal evaluation between baseline model and full model, depth estimation.