MonoDistill: Learning Spatial Features for Monocular 3D Object Detection

1 INTRODUCTION . 3D object detection is an indispensable component for 3D scene perception , which has wide applications in the real world , such as autonomous driving and robotic navigation . Although the algorithms with stereo ( Li et al. , 2019b ; Wang et al. , 2019 ; Chen et al. , 2020a ) or LiDAR sensors ( Qi et al. , 2018 ; Shi et al. , 2019 ; 2020 ) show promising performances , the heavy dependence on the expensive equipment restricts the application of these algorithms . Accordingly , the methods based on the cheaper and more easy-to-deploy monocular cameras ( Xu & Chen , 2018 ; Ma et al. , 2019 ; 2021 ; Brazil & Liu , 2019 ; Ding et al. , 2020 ) show great potentials and have attracted lots of attention . As shown in Figure 1 ( a ) , some prior works ( Brazil & Liu , 2019 ; Simonelli et al. , 2019 ; Chen et al. , 2020b ) directly estimate the 3D bounding boxes from monocular images . However , because of the lack of depth cues , it is extremely hard to accurately detect the objects in the 3D space , and the localization error is the major issue of these methods ( Ma et al. , 2021 ) . To mitigate this problem , an intuitive idea is to estimate the depth maps from RGB images , and then use them to augment the input data ( Figure 1 ( b ) ) ( Xu & Chen , 2018 ; Ding et al. , 2020 ) or directly use them as the input data ( Figure 1 ( c ) ) ( Wang et al. , 2019 ; Ma et al. , 2019 ) . Although these two strategies have made significant improvement in performance , the drawbacks of them can not be ignored : ( 1 ) These methods generally use an off-the-shelf depth estimator to generate the depth maps , which introduce lots of computational cost ( e.g . the most commonly used depth estimator ( Fu et al. , 2018 ) need about 400ms to process a standard KITTI image ) . ( 2 ) The depth estimator and detector are trained separately , which may lead to a sub-optimal optimization . Recently , Reading et al . ( 2021 ) propose an end-to-end framework ( Figure 1 ( d ) ) for monocular 3D detection , which can also leverage depth estimator to provide depth cues . Specifically , they introduce a sub-network to estimate the depth distribution and use it to enrich the RGB features . Although this model can be trained end-to-end and achieves better performance , it still suffer from the low inference speed ( 630ms per image ) , mainly caused by the depth estimator and the complicated network architecture . Note that the welldesigned monocular detectors , like Ma et al . ( 2021 ) ; Zhang et al . ( 2021b ) ; Lu et al . ( 2021 ) , only take about 40ms per image . In this paper , we aim to introduce the depth cues to the monocular 3D detectors without introducing any extra cost in the inference phase . Inspired by the knowledge distillation ( Hinton et al . ) , which can transfer the learned knowledge from a well-trained CNN to another one without any changes in the model design , we propose that the spatial cues may also be transferred by this way from the LiDAR-based models . However , the main problem for this proposal is the difference of the feature representations used in these two kinds of methods ( 2D images features vs. 3D voxel features ) . To bridge this gap , we propose to project the LiDAR signals into the image plane and use the 2D CNN , instead of the commonly used 3D CNN or point-wise CNN , to train a ‘ image-version ’ LiDAR-based model . After this alignment , the knowledge distillation can be friendly applied to enrich the features of our monocular detector . Based on the above-mentioned motivation and strategy , we propose the distillation based monocular 3D detector ( MonoDistill ) : We first train a teacher net using the projected LiDAR maps ( used as the ground truths of the depth estimator in previous works ) , and then train our monocular 3D detector under the guidance of the teacher net . We argue that , compared with previous works , the proposed method has the following two advantages : First , our method directly learn the spatial cues from the teacher net , instead of the estimated depth maps . This design performs better by avoiding the information loss in the proxy task . Second , our method does not change the network architecture of the baseline model , and thus no extra computational cost is introduced . Experimental results on the most commonly used KITTI benchmark , where we rank the 1st place among all monocular based models by applying the proposed method on a simple baseline , demonstrate the effectiveness of our approach . Besides , we also conduct extensive ablation studies to present each design of our method in detail . More importantly , these experiments clearly illustrate the improvements are achieved by the introduction of spatial cues , instead of other unaccountable factors in CNN . 2 RELATED WORKS . 3D Detection from only monocular images . 3D detection from only monocular image data is challenging due to the lack of reliable depth information . To alleviate this problem , lots of scholars propose their solutions in different ways , including but not limited to network design ( Roddick et al. , 2019 ; Brazil & Liu , 2019 ; Zhou et al . ; Liu et al. , 2020 ; Luo et al. , 2021 ) , loss formulation ( Simonelli et al. , 2019 ; Ma et al. , 2021 ) , 3D prior ( Brazil & Liu , 2019 ) , geometric constraint ( Mousavian et al. , 2017 ; Qin et al. , 2019 ; Li et al. , 2019a ; Chen et al. , 2020b ) , or perspective modeling ( Zhang et al. , 2021a ; Lu et al. , 2021 ; Shi et al. , 2021 ) . Depth augmented monocular 3D detection . To provide the depth information to the 3D detectors , several works choose to estimate the depth maps from RGB images . According to the usage of the estimated depth maps , these methods can be briefly divided into three categories . The first class of these methods use the estimated depth maps to augment the RGB images ( Figure 1 ( b ) ) . In particular , Xu & Chen ( 2018 ) propose three fusion strategies of the RGB images and depth maps , while Ding et al . ( 2020 ) and Wang et al . ( 2021a ) focus on how to enrich the RGB features with depth maps in the latent feature space . Besides , Wang et al . ( 2019 ) ; Ma et al . ( 2019 ) propose another pipeline ( Figure 1 ( c ) ) : They back-project the depth maps into the 3D space , and then train a LiDAR-based model and use the resulting data ( pseudo-LiDAR signal ) to predict the 3D boxes . This framework shows promising performance , and lots of works ( Weng & Kitani , 2019 ; Cai et al. , 2020 ; Wang et al. , 2020a ; Chu et al. , 2021 ) are built on this solid foundation . Recently , Reading et al . ( 2021 ) propose another way to leverage the depth cues for monocular 3D detection ( Figure 1 ( d ) ) . Particularly , they first estimate the depth distribution using a sub-network , and then use it to lift the 2D features into 3D features , which is used to generate the final results . Compared with the previous two families , this model can be trained in the end-to-end manner , avoiding the sub-optimal optimization . However , a common disadvantage of these methods is that they inevitably increase the computational cost while introducing depth information . Unlike these methods , our model chooses to learn the feature representation under the guidance of depth maps , instead of integrating them . Accordingly , the proposed model not only introduces rich depth cues but also maintains high efficiency . Knowledge distillation . Knowledge distillation ( KD ) is initially proposed by Hinton et al . for model compression , and the main idea of this mechanism is transferring the learned knowledge from large CNN models to the small one . This strategy has been proved in many computer vision tasks , such as 2D object detection ( Dai et al. , 2021 ; Chen et al. , 2017 ; Gupta et al. , 2016 ) , semantic semantic segmentation ( Hou et al. , 2020b ; Liu et al. , 2019 ) . However , few work explore it in monocular 3D detection . In this work , we design a KD-based paradigm to efficiently introduce depth cues for monocular 3D detectors . LIGA Stereo . We found a recent work LIGA Stereo ( Guo et al. , 2021 ) ( submitted to arXiv on 18 Aug. 2021 ) discusses the application of KD for stereo 3D detection under the guidance of LiDAR signals . Here we discuss the main differences of LIGA Stereo and our work . First , the tasks and underlying data are different ( monocular vs. stereo ) , which leads to different conclusions . For example , Guo et al . ( 2021 ) concludes that using the predictions of teacher net as ‘ soft label ’ can not bring benefits . However , our experimental results show the effectiveness of this design . Even more , in our task , supervising the student net in the result space is more effective than feature space . Second , they use an off-the-shelf LiDAR-based model to provide guidance to their model . However , we project the LiDAR signals into image plane and use the resulting data to train the teacher net . Except for the input data , the teacher net and student net are completely aligned , including network architecture , hyper-parameters , and training schedule . Third , to ensure the consistent shape of features , LIGA Stereo need to generate the cost volume from stereo images , which is time-consuming ( it need about 350ms to estimate 3D boxes from a KITTI image ) and hard to achieve for monocular images . In contrast , our method align the feature representations by adjusting the LiDAR-based model , instead of the target model . This design makes our method more efficient ( about 35ms per image ) and can generalize to all kinds of image-based models in theory . 3 METHOD . 3.1 OVERVIEW . Figure 3 presents the framework of the proposed MonoDistill , which mainly has three components : a monocular 3D detector , an aligned LiDAR-based detector , and several side branches which build the bridge to provide guidance from the LiDAR-based detector to our monocular 3D detector . We will introduce the how to build these parts one by one in the rest of this section . 3.2 BASELINE MODEL . Student model . We use the one-stage monocular 3D detector MonoDLE ( Ma et al. , 2021 ) as our baseline model . Particularly , this baseline model adopts DLA-34 ( Yu et al. , 2017 ) as the backbone and uses several parallel heads to predict the required items for 3D object detection . Due to this clean and compact design , this model achieves good performance with high efficiency . Besides , we further normalize the confidence of each predicted object using the estimated depth uncertainty ( see Appendix A.1 for more details ) , which brings about 1 AP improvement 1 . Teacher model . Existing LiDAR-based models are mainly based on the 3D CNN or point-wise CNN . To align the gap between the feature representations of the monocular detector and the LiDAR-based detector , we project the LiDAR points into the image plane to generate the sparse depth map . Further , we also use the interpolation algorithm ( Ku et al. , 2018 ) to generate the dense depth , and see Figure 2 for the visualization of generated data . Then , we use these ‘ image-like LiDAR maps ’ to train a LiDAR-based detector using the identical network with our student model .

Summary: The paper presents MonoDistill, a way to enhance RGB-based 3D object detection through knowledge distillation from a LiDAR-based teacher. A 3D detection pipeline (built around MonoDLE) is trained both on RGB images and densified LiDAR input. Three main mechanism enable teacher-student transfer: On feature level, scene-level distillation aligns affinity maps between teacher and student while masked features are trained on object-level, additional masked pseudo labels are leveraged around the teacher centre predictions. Each component is carefully ablated and experiments are conducted on the KITTI benchmark.