DAN‐Conv: Depth aware non‐local convolution for LiDAR depth completion

Electronics LettersVolume 57, Issue 20 p. 754-757 Image and vision processing and display technologyOpen Access DAN-Conv: Depth aware non-local convolution for LiDAR depth completion Lin Yan, Lin Yan orcid.org/0000-0001-5507-8989 School of Computer Science and Technology, Xidian University, Xi'an, ChinaSearch for more papers by this authorKai Liu, Corresponding Author Kai Liu kailiu@mail.xidian.edu.cn School of Computer Science and Technology, Xidian University, Xi'an, China Email: kailiu@mail.xidian.edu.cnSearch for more papers by this authorLong Gao, Long Gao School of Telecommunication Engineering, Xidian University, Xi'an, ChinaSearch for more papers by this author Lin Yan, Lin Yan orcid.org/0000-0001-5507-8989 School of Computer Science and Technology, Xidian University, Xi'an, ChinaSearch for more papers by this authorKai Liu, Corresponding Author Kai Liu kailiu@mail.xidian.edu.cn School of Computer Science and Technology, Xidian University, Xi'an, China Email: kailiu@mail.xidian.edu.cnSearch for more papers by this authorLong Gao, Long Gao School of Telecommunication Engineering, Xidian University, Xi'an, ChinaSearch for more papers by this author First published: 22 June 2021 https://doi.org/10.1049/ell2.12254AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Sparse LiDAR depth completion is a beneficial task for many robotic applications. It commonly generates a dense depth prediction from a sparse depth map and its corresponding aligned RGB image. This image-guided depth completion task mainly has two challenges: sparse data processing and multi-modality data fusion. In this letter, they are dealt with by two novel solutions: (1) To efficiently process sparse depth input, a Depth Aware Non-local Convolution (DAN-Conv) is proposed. It augments the spatial sampling locations of a convolution operation. Specifically, DAN-Conv constructs a non-local neighbourhood relationship by exploiting the intrinsic correlation among observable depth pixels. In particular, it can readily replace standard convolution without introducing additional network parameters. (2) A Symmetric Co-attention Module (SCM) is proposed to fuse and enhance features from depth and image domain. It estimates the importance of complementary features by the co-attention mechanism. Finally, a neural network built on DAN-Conv and SCM is proposed. It achieves competitive performance on the challenging KITTI depth completion benchmark. Comparing to approaches with approximate accuracy, this lightweight network requires significantly fewer learnable parameters. Introduction Owning to the limitation of LiDAR (Light Detection and Ranging) sensor, it produces an accurate but sparse 3D point cloud description of the surroundings. After projecting the point cloud onto the image plane, the observable pixels on the depth map are fairly sparse (e.g. less than 6% with depth information from KITTI [1]). The goal of completion is to predict the depth value for the non-observable pixels. Recent studies have shown that Convolution Neural Network (CNN) can achieve impressive performance on this task. These approaches can be categorized into two types: (1) Depth-only approaches [1, 2, 17] utilize the continuity of depth values to interpolate missing depth with sparse input only. These works demonstrate the strong correlation among adjacent depth points. (2) Image-guided approaches [3, 4, 15] exploit depth cues from RGB image as auxiliary guidance. The rich semantic cues and texture information from the image domain help these methods achieve relatively higher completion accuracy. In the CNN-based completion methods, the fixed receptive field of a standard 2D convolution inevitably includes irrelevant and even blank neighbours from the sparse depth map. To overcome the shortcoming of this sampling strategy, SI-Conv [1] tries to deal with it by normalizing convolution output with a binary observation mask. However, the number of valid feature points contributing to each output pixel will be non-uniform. Xu et al. [6] utilize Deformable ConvNets [5] to process depth map. It utilizes an auxiliary network to predict non-local neighbours. By visualizing the position of neighbours predicted by DeformConv, they observed that the adjacent points with approximate depth values are highly correlated. In the meanwhile, the drastic changes of depth value always appear at the boundary of an object. Based on the above facts, we speculate that the difference of depth value is sufficient to support the selection of non-local neighbours and propose DAN-Conv. In specific, DAN-Conv generates the sampling points from a fixed number of adjacent observable points with an approximate depth value. Unlike DeformConv which predicts offset of neighbours by several neural network layers, DAN-Conv does not require any extra learning configuration. It can be integrated into the existing network seamlessly. As shown in Figure 1, we select several depth points and illustrate their neighbours generated by DAN-Conv. Obviously, DAN-Conv can aggregate more homogeneous feature points than a standard convolution. Fig. 1Open in figure viewerPowerPoint Example of neighbours used by DAN-Conv. (a) RGB image. (b) Selected depth points. (c) Selected points (green +) and their neighbours (blue +) are linked by yellow lines According to Zhong et al. [7], there is an inherent relationship between the depth map and RGB image: although the completion only happens on the non-observable pixels, it is still necessary to exploit the complementary between them to enhance each other. Nevertheless, the widely used channel-wise concatenation cannot effectively exploit this prior. Zhong et al. [7] also proposed that some statistical techniques, for example, correlation analysis, can be used to learn the shared subspace between two modalities. Based on these arguments, we propose to use the co-attention mechanism [9] to jointly reason about the complementary between features from depth and image domain. In specific, we propose a Symmetric Co-attention Module (SCM) to fuse and then enhance point-wise features by adaptively estimating the importance of the feature from another modality. Built on DAN-Conv and SCM, a novel encoder-decoder structured CNN is proposed for this completion task. In the encoder part, DAN-Conv is used to process point-wise data gathered from the depth map and RGB image. The non-local neighbourhood relationship from the sparse depth map is replicated to the image domain. In the decoder part, we fuse and upscale feature maps from two modalities in a cascade manner. Thanks to the effectiveness and simplicity of this design, our model achieves state-of-the-art performance while maintaining a lightweight network structure. In summary, the major contributions of this letter can be summarized as: 1 We propose a non-local convolution operation to improve the efficiency of sparse depth map processing. 2 We propose a multi-modality fusion module that exploits the complementary between depth and image data by the co-attention mechanism. 3 The experimental results demonstrate that the competitive performance is achieved by our model with significantly fewer network parameters. Related work Depth-only Methods merely use sparse depth measurements as input to recovery the full resolution depth map. Despite the limitation of observable depth points, these approaches demonstrate that the non-observable depth areas can be interpolated by the sparse input only. Ku et al. [2] apply several image processing techniques on the inverted depth map. These operators include hole filling and blurring to round off edges and remove outlier pixels. The main disadvantage of [2] is its insufficient adaptability to various scenes under different sparsity patterns. Recently, more deep learning based methods are proposed to solve this problem. Jonas et al. [1] propose a sparse convolution based deep neural network. Their sparsity invariant convolution explicitly considers data sparsity by utilizing a binary observation mask. Eldesokey et al. [22] propagate confidences together with sparse input through a novel normalized convolution. The learning target of their network is to minimize the data error and maximize the output confidence simultaneously. It is a non-trivial task for the depth-only methods to accurately infer the missing depth values when the span between the observable depth point is too wide. Thus, the lately proposed methods use an auxiliary RGB image to guide the completion procedure. These image-guided methods require the extra RGB image to be pixel aligned with the sparse depth map. Ma et al. [15] propose a self-supervised framework to train an encoder-decoder network. Without using semi-dense annotation, they exploit the photo consistency from nearby RGB frames as the supervision signal. DeepLiDAR [4] and Xu et al. [3] further boost the completion accuracy on LiDAR depth data. They use surface normal to restrain depth generation. A synthetic dataset is used to train their network with the pre-generated surface normal. Wouter et al. [10] predict dense depth from two pathways: a global pathway uses concatenated RGB image and sparse depth map to downscale feature map and extract guidance information, while a local pathway performs completion with the help of guidance from the global pathway. Jaritz et al. [17] propose a modified NASNet [20] for depth completion and semantic segmentation tasks simultaneously. Their sparse training strategy makes their model handle various sparse density patterns. These works demonstrate that accurate depth completion requires auxiliary information such as depth cues from the image domain. Nevertheless, a small displacement between RGB image and depth map is inevitable. In the meanwhile, it is still an open problem about how to effectively fuse data from multiple modalities. Approach Depth Aware Non-local Convolution (DAN-Conv) generates non-local neighbours for each observable point and then aggregates the collected feature as a standard convolution does. Instead of applying filters with a regular structure, for example, a 3 ×3 grid, DAN-Conv avoids the aggregation of non-observable and irrelevant local neighbours to a certain extend. Figure 2 illustrates the process of DAN-Conv. The channel width of the input feature map is C i n . We first gather N point-wise features from input map where the depth measurements are available. Then, the receptive field of convolution is constructed by the indices of K neighbours generated from Algorithm 1. After that, the organized neighbourhood features are concatenated with gathered features before the multiplication of weights. The convolution weight has a shape of [ ( K + 1 ) ∗ C i n , C o u t ] . Finally, the C o u t width point-wise feature is scattered back to a blank feature map with the help of a binary observation mask M. Fig. 2Open in figure viewerPowerPoint The process of DAN-Conv The core of DAN-Conv is the generation of K neighbours for the observable pixels. As described in Algorithm 1, for each observable point, we first retrieve the indices of 2 K nearest neighbours according to their spatial locations. Then, the K output neighbours, that is, the set Ω i , are selected from them with the most approximate depth value. Thus, the output relationship Ω is represented as a form of feature indices. ALGORITHM 1. Calculation of neighbours indices We propose a Symmetric Co-attention Module (SCM) to fuse and enhance features from two modalities by learning the complementary between them. As shown in Figure 3, SCM takes point-wise feature from image and depth domain as input. It concatenates them and feeds them into a Fully Connected (FC) layer to form a fused feature representation. Inspired by the co-attention mechanism [9], this representation can be used to generate two weighting maps for attentional fusion. In specific, we pass it into two FC+Sigmoid layer combinations separately. Finally, the feature from one modality, for example, F I , is enhanced by its complementary, that is, F d , with a weight map w d in a summation manner: F I ′ = F I + w d ∗ F d . Fig. 3Open in figure viewerPowerPoint The process of symmetric co-attention module DAN-MOD module plays the role of feature fusion and feature map downscaling. It consists of DAN-Conv, SCM, and standard 2D convolution layers. The structure of DAN-MOD is shown in Figure 4. Besides feature map F i n and its corresponding observation mask M, DAN-MOD takes point-wise feature F i n ′ from another modality as input. It first uses the standard convolution to process the feature map and generates F 2 d . The stride parameter S decides whether to perform downscaling or not. Then, the DAN-Conv is employed to process point-wise features for each observable point. The neighbourhood relationship needed by DAN-Conv is computed in advance from a depth map at the specific scale. The point-wise feature F s output by SCM just covers the observable pixels. The non-observable positions are filled by the scheme proposed from [8]. Thus, the intermediate dense feature map is generated as: F = F s + ( 1 − M ) ⊙ F 2 d . Finally, a skip connection [19] from input to output is applied to avoid gradient vanishing with residual learning. Fig. 4Open in figure viewerPowerPoint The process of DAN-MOD module. symbols ⊝ , ⊕ , and ⊛ represent element-wise minus, summation, and multiplication, respectively The architecture of this proposed encoder-decoder network is depicted in Figure 5. Fig. 5Open in figure viewerPowerPoint The architecture of proposed network The encoder part composes of a depth branch and an image branch. It is built upon multiple DAN-MOD modules. For feature maps at a specific scale, a pair of DAN-MODs are employed firstly. The second pair is used to perform both feature fusion and feature map downscaling. We use the sampling method proposed by Li et al. [11] to generate downscaled depth maps. This strategy minimizes the breaking of intrinsic structure caused by max pooling. The extracted neighbourhood relationship is shared by two branches. The decoder is built by several Symmetric Gated Fusion Modules (SGFM) [18]. It has a symmetric structure to fuse multi-modal contextual representations. Finally, three dense depth maps, that is D ̂ , D ̂ d , and D ̂ I , are predicted for fusion, depth, and image branch, respectively. D ̂ is the final output. Similar to [18], the training process is mainly supervised by the MSE reconstruction loss between three predictions and the ground truth. In addition, we use a smoothness loss to encourage the local smoothing of the dense prediction. This setting can resolve the griding effects caused by transpose convolution [21]. Experiments We compare our model against the state-of-the-art methods in the KITTI depth completion benchmark [1]. KITTI provides pairs of depth maps generated by 64 line Velodyne LiDAR and aligned RGB images. It has 86,898 pairs for training, 1,000 pairs for validation, and another 1000 pairs for testing without ground truth. The raw image has a size of 1242 ×375. Since the top part of the image does not contain any depth points provided by LiDAR, we follow [10] to crop the inputs to a 1216 ×256 aspect ratio. The following data augmentation techniques are applied to the training split on the fly: 1 Flipping. Horizontally flip RGB image and depth map. 2 Depth Jittering. Apply white noise to input depth points. 3 Color Jittering. Shift brightness, contrast, or saturation of RGB image. 4 Rotation. Rotate RGB image and depth map by a degree in [ − 1.5 , 1.5 ] . The quantitative results are evaluated by four metrics: root mean squared error (RMSE), mean absolute error (MAE), root mean squared error of the inverse depth (iRMSE), and mean absolute error of the inverse depth (iMAE): RMSE ( m m ): 1 | V | ∑ u , v ∈ V | T ( u , v ) − G ( u , v ) | 2 , MAE ( m m ): 1 | V | ∑ u , v ∈ V | T ( u , v ) − G ( u , v ) | , iRMSE( 1 / k m ): 1 | V | ∑ u , v ∈ V | 1 T ( u , v ) − 1 G ( u , v ) | 2 , iMAE( 1 / k m ): 1 | V | ∑ u , v ∈ V | 1 T ( u , v ) − 1 G ( u , v ) | . The model used in this part is trained 50 epochs. The performance of our proposed network is compared with other state-of-the-art works in Table 1. It achieves competitive performance and even outperforms some approaches using additional data or labels. TABLE 1. Quantitative comparison on the TEST set of KITTI benchmark Methods RMSE ( m m ) MAE ( m m ) iRMSE ( 1 / k m ) iMAE ( 1 / k m ) #Para. Trained in 2D space with KITTI data only NConv-L2 [16] 829.98 233.26 2.60 1.03 - Sp2Dense [15] 814.73 249.95 2.80 1.21 26M Guidance [14] 807.42 253.98 2.73 1.33 - Revisit [8] 792.80 225.81 2.42 0.99 - DSPN [6] 766.74 220.36 2.47 1.03 - M-CHN [11] 762.19 220.41 2.30 0.98 - CSPN++ [12] 743.69 209.28 2.07 0.90 26M NLSPN [13] 741.68 199.59 1.99 0.84 25.84M ours 759.65 213.68 2.17 0.92 1.46M Training with additional data or labels PwP [3] 777.05 235.17 2.42 1.13 28.99M Uncert. [10] 772.87 215.02 2.19 0.93 2.55M DeepL. [4] 758.38 226.50 2.56 1.15 53.44M Training with both 2D and 3D data FuseNet [23] 752.88 221.19 2.34 1.14 - The ablation model is trained 30 epochs for evaluating the effectiveness of the SCM and the amount of neighbours K in DAN-Conv. The results in Table 2 demonstrate the improvements brought by SCM. Although the impact on the scale and complexity of our network is trivial, nevertheless, it plays an important role in determining the accuracy of its prediction. TABLE 2. Ablation studies by varying SCM and K on VAL set of KITTI SCM? K MAE ( m m ) RMSE ( m m ) FLOPs #Params × 6 214 742 39.72G 1.40M × 9 208 718 40.24G 1.44M × 12 203 711 40.75G 1.47M ✓ 6 210 701 40.07G 1.43M √ 9 199 684 40.58G 1.46M √ 12 205 689 41.09G 1.50M Conclusion In this letter, we focus on solving two challenges in image-guided sparse depth completion task: how to process sparse depth data and how to fuse features from different modalities. For the first one, we propose a novel depth aware non-local convolution to replace standard convolution. It can aggregate non-local contextual information adaptively by selecting the adjacent observable points with an approximate depth value. For the second one, a co-attention based symmetric module is proposed to exploit the complementary between features from depth and image domain. Finally, a network is built upon the above two solutions. It achieves state-of-the-art performance on challenging KITTI benchmark with 1.46M parameters only. The source code is available at https://github.com/godspeed1989/DANConv. References 1Uhrig, J., et al.: Sparsity invariant CNNs. 2017 International Conference on 3D Vision (3DV), 2017, pp. 11– 20Google Scholar 2Ku, J., Harakeh, A., Waslander, S.L.: In defense of classical image processing: Fast depth completion on the cpu. In: 15th Conference on Computer and Robot Vision, 2018, pp. 16– 22Google Scholar 3Xu, Y., et al.: Depth completion from sparse LiDAR data with depth-normal constraints. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2811– 2820. IEEE, Piscataway, NJ (2019) Google Scholar 4Qiu, J., et al.: Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse LiDAR data and single color image. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3313– 3322. IEEE, Piscataway, NJ (2019) Google Scholar 5Dai, J., et al.: Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 764– 773. IEEE, Piscataway, NJ (2017) Google Scholar 6Xu, Z., Yin, H., Yao, J.: Deformable spatial propagation networks for depth completion. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 913– 917. IEEE, Piscataway, NJ (2020) Google Scholar 7Zhong, Y., Wu, C.Y., You, S.: Deep RGB-D canonical correlation analysis for sparse depth completion. Adv. in Neu. Info. Proc. Sys. 32, 5331– 5341 (2019) Google Scholar 8Yan, L., et al.: Revisiting sparsity invariant convolution: A network for image guided depth completion. IEEE Access 8, 126323– 126332 (2020) CrossrefWeb of Science®Google Scholar 9Lu, J., et al.: Hierarchical question-image co-attention for visual question answering. In: 30th International Conference on Neural Information Processing Systems, 2016, pp. 289– 297Google Scholar 10Gansbeke, W.V., et al.: Sparse and noisy LiDAR completion with RGB guidance and uncertainty. In: 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1– 6. IEEE, Piscataway, NJ (2019) Google Scholar 11Li, A., et al.: A multi-scale guided cascade hourglass network for depth completion. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 32– 40. IEEE, Piscataway, NJ (2020) Google Scholar 12Cheng, X., Wang, P., Guan, C.: Cspn : Learning context and resource aware convolutional spatial propagation networks for depth completion. AAAI Conf. on Art. Intell. 34, 10615– 10622 (2020) Google Scholar 13Cheng, X., Wang, P., Guan, C.: Cspn : Learning context and resource aware convolutional spatial propagation networks for depth completion. AAAI Conf. on Art. Intell. 34, 10615– 10622 (2020) Google Scholar 14Lee, S., et al.: Deep architecture with cross guidance between single image and sparse LiDAR data for depth completion. IEEE Access 8, 79801– 79810 (2020) CrossrefWeb of Science®Google Scholar 15Ma, F., et al.: Self-supervised sparse-to-dense: Self-supervised depth completion from LiDAR and monocular camera. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 3288– 3295. IEEE, Piscataway, NJ (2019) Google Scholar 16Eldesokey, A., et al.: Confidence propagation through CNNs for guided sparse depth regression. IEEE Trans. on Pat. Ana. and Mach. Intell. 42(10), 2423– 2436 (2019) CrossrefPubMedWeb of Science®Google Scholar 17Jaritz, M., et al.: Sparse and dense data with CNNs: Depth completion and semantic segmentation. In: 2018 International Conference on 3D Vision (3DV), pp. 52– 60. IEEE, Piscataway, NJ (2018) Google Scholar 18Zhao, S., et al.: Adaptive context-aware multi-modal network for depth completion, pp. 5264– 5276. arXiv preprint, (2021) Google Scholar 19He, K., et al.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770– 778. IEEE, Piscataway, NJ (2016) Google Scholar 20Zoph, B., et al.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8697– 8710. IEEE, Piscataway, NJ (2018) Google Scholar 21Zhou, T., et al.: Unsupervised learning of depth and ego-motion from video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1851– 1858. IEEE, Piscataway, NJ (2017) Google Scholar 22Eldesokey, A., Felsberg, M., Khan, F.S.: Confidence propagation through cnns for guided sparse depth regression. IEEE Trans. Pattern Anal. Mach. Intell. 42(10), 2423– 2436 (2020) CrossrefPubMedWeb of Science®Google Scholar 23Chen, Y., et al.: Learning joint 2D-3D representations for depth completion. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10023– 10032. IEEE, Piscataway, NJ (2019) Google Scholar Volume57, Issue20September 2021Pages 754-757 FiguresReferencesRelatedInformation