Semantic-Guided Representation Enhancement for Self-supervised Monocular Trained Depth Estimation

1 INTRODUCTION . Depth estimation is a long standing problem in computer vision community , which offers useful information to a wide range of tasks including robotic perception , augmented reality and autonomous driving , etc . Compared with depth estimation methods which rely on active vision or multi-view paradigms ( Schonberger & Frahm , 2016 ; Li et al. , 2019 ) , estimating depth from only a single image is highly ill-posed , thus brings greater challenges for higher quality results . In recent years , monocular depth estimation has witnessed new renaissances with the advent of deep learning ( He et al. , 2016 ; Jaderberg et al. , 2015 ; Simonyan & Zisserman , 2014 ) . By learning deep representations in a supervised manner , various of networks ( Eigen & Fergus , 2015 ; Eigen et al. , 2014 ; Laina et al. , 2016 ) are capable of producing high quality depth maps thanks to the large corpus of training data . At the mean time , consider the lack of labeled training data for network training , recent advances ( Zhou et al. , 2017 ; Godard et al. , 2019 ; 2017 ) show that monocular depth estimation can be accomplished in a self-supervised way . The network can be trained by unlabeled image sequences using two-view geometric constraints , while achieving comparable results with the supervised paradigm . The learning-based methods managed to handle the highly ill-posed monocular depth estimation problem by implicitly learning the mapping between visual appearance and its corresponding depth value . However , despite the great effectiveness for learning-based depth estimation , these methods still struggle to conduct precise depth estimation on challenging image regions such as semantic category borders or thin object areas . For example , the estimated object depth usually fails to align with the real object borders , and the depth of foreground objects which have thin structures tends to be submerged in the background . We attribute these phenomena to the limited depth representation ability that ( 1 ) the pixel-wise local depth information can not be well represented by current depth network , especially on highly ambiguous , semantic border areas , ( 2 ) current depth representations are not capable of well describing depth foreground/background relationships globally . These issues lead to the wrong depth estimation from the true scene structures , which hinders the further applications in the real-world tasks . In this paper , we address this problem via enhancing both local and global depth feature representations for self-supervised monocular depth estimation via semantic guidance . As semantic segmentation conducts explicit category-level scene understandings and produces well-aligned object boundary detection , we propose an extra semantic estimation branch inside the self-supervised paradigm . The semantic branch offers rich contextual features which is fused with depth features during multiscale learning . Under this framework , we propose to enhance the depth feature using semantic guidance in a local-to-global way . To improve the local depth representations on semantic category borders , inspired by the sampling and enhancing strategies used in semantic segmentation ( Kirillov et al. , 2020 ) , our first contribution is to propose a Semantic-guided Edge Enhancement Module ( SEEM ) which is specially designed to enhance the local point-based depth representations that locate on the semantic edges . Different from the method in ( Kirillov et al. , 2020 ) , we enhance the point features using multi-level of representations from different domains under a self-supervised framework . To be specific , we sample a set of point positions lying on the binary semantic category borders , and extract the point-wise features of the corresponding edge positions from the encoded feature , depth decoding feature as well as the semantic decoding feature . We then merge and enhance the point-wise features and feed them to the final depth decoding features to promote the edge-aware depth inference for self-supervision . For global depth representation enhancement , our second contribution is to propose a semantic-guided multi-level attention module to improve the global depth representation . Different from the conventional self-attentions ( Fu et al. , 2019 ; Vaswani et al. , 2017 ) which are implemented as single modules on the bottleneck feature block , we propose to explore the self-attentions on different level of decoding layers . In this way , both semantic and depth representations can be further promoted by exploring and leveraging the pixel-wise correlations inside of their fused features . We validate our method mainly on KITTI 2015 ( Geiger et al. , 2012 ) , and the Cityscapes ( Cordts et al. , 2016 ) benchmark is also used for evaluating the generalization ability . Experiments show that the proposed method significantly improves the depth estimation on category edges and thin scene structures . Extensive quantitative and qualitative results validate the superiority of our method that it outperforms the state-of-the-art methods for self-supervised monocular depth estimation . 2 RELATED WORK . There exist extensive researches on monocular depth estimation , including geometry-based methods ( Schonberger & Frahm , 2016 ; Enqvist et al. , 2011 ) and learning-based methods ( Eigen et al. , 2014 ; Laina et al. , 2016 ) . In this paper , however , we concentrate only on the self-supervised depth training and semantic-guided depth estimation , which is highly related to the research focus of this paper . Self-supervised depth estimation . Self-supervised methods enable the networks to learn depth representation from merely unlabeled image sequences that they reformulate the depth supervision loss into the image reconstruction loss . Godard et al . ( 2017 ) and Garg et al . ( 2016 ) first propose the self-supervised method on stereo images , then Zhou et al . ( 2017 ) propose a monocular trained approach using a separate motion estimation network . Based on these frameworks , a large corpus of works seek to promote self-supervised learning from different aspects . For more robust selfsupervision signals , Mahjourian et al . ( 2018 ) propose to use 3D information as extra supervisory signal , another kind of methods leverage additional information such as the optical flow ( Ranjan et al. , 2019 ; Wang et al. , 2019b ) to strengthen depth supervision via consistency constraints . In order to solve the loss deviation problems on non-rigid motion areas , selective masks are used to filter out the these areas when computing losses . Prior works generate the mask by the network itself ( Zhou et al. , 2017 ; Yang et al. , 2017 ; Vijayanarasimhan et al. , 2017 ) , while the succeeding methods produce the mask by leveraging geometric clues ( Bian et al. , 2019 ; Wang et al. , 2019a ; Godard et al. , 2019 ; Klingner et al. , 2020 ) , which is proved to be more effective . There also exist other methods trying to enhance the network performance with traditional SfM ( Schonberger & Frahm , 2016 ) , which offer pseudo labels for depth estimation . Guizilini et al . ( 2020a ) propose a novel network architecture to improve depth estimation . In this paper , we do not consider the performance improvements contributed by this novel architecture in experimental comparisons , and compare different methods with conventional backbones . Depth estimation using semantics . Semantic information are shown to provide positive effects toward depth estimation framework in the previous works . The methods can be categorized into two groups by their way to use the semantic information . One group of the methods leverage the semantic labels directly to guide the depth learning . The works of Ramirez et al . ( 2018 ) and Chen et al . ( 2019a ) propose depth constraints by leveraging the semantic labels of the scene . Klingner et al . ( 2020 ) and Casser et al . ( 2019 ) address the non-rigid motion issues by handling the moving object areas highlighted by the semantic map . Wang et al . ( 2020 ) use a divide-and-conquer strategy to conduct depth estimation with different semantic categories , and Zhu et al . ( 2020 ) offer a depth morphing paradigm with the help of semantic foreground edges . The other group of methods enhance the depth representation by feature manipulation . Chen et al . ( 2019a ) generated depth and semantic maps by a single latent representation . Ochs et al . ( 2019 ) proposed a segmentation-like loss item for depth estimation . Guizilini et al . ( 2020b ) leveraged a pre-trained semantic segmentation network and further conduct feature merging by PAC-based ( Su et al. , 2019 ) module . In this paper , we propose an individual semantic network for depth estimation . But different with Guizilini et al . ( 2020b ) , our semantic branch shares the same encoder and is trained together with the depth network for better scene representation . We leverage the semantic information in both ways . The semantic edges are used to guide the edge-based point feature sampling for local depth depth representation enhancement . At the mean time , we utilize the semantic feature blocks to conduct multi-scale feature fusion for global depth representation enhancement . The proposed method constructs a holistic way for feature representation enhancing in self-supervised depth estimation . 3 SELF-SUPERVISED DEPTH ESTIMATION FRAMEWORK . The proposed method builds upon the self-supervised depth estimation framework . An image triplet ( It−1 , It , It+1 ) is used as the input , where It acts as the target image and It′ ∈ { It−1 , It+1 } are the source images . During training , the target image It is fed into the depth network fD to get the estimated depth Dt = fD ( It ) , while the adjacent image pairs ( It , It′ ) are put into the ego-motion network for the 6-DoF ego-motion estimations Tt′→t . Then , the synthesized target images It′→t can be estimated using the source images It′ , depth Dt and the ego-motions Tt′→t , following the formulation from Zhou et al . ( 2017 ) . The self-supervised loss can be calculated by L ( It , It′→t ) = Lp ( It , It′→t ) + λLs ( It , It′→t ) , ( 1 ) where λ is the weighting factor between the photometric loss Lp and smoothness loss Ls . Following Godard et al . ( 2019 ) , we implement the minimum photometric loss Lp ( It , It′→t ) = min t′ ( α 2 ( 1− SSIM ( It , It′→t ) ) + ( 1− α ) ‖It − It′→t‖1 ) , ( 2 ) where SSIM is the Structural Similarity item ( Wang et al. , 2004 ) , α is the weighting factor which is set to 0.85 . The smoothness loss item ( Godard et al. , 2017 ) is implemented to smooth the depth map by the consistency between the image and depth gradient Ls ( It ) = |∂xDt| e−|∂xIt| + |∂yDt| e−|∂yIt| , ( 3 ) where ∂x , ∂x denote gradient operation on x , y-axis . In our implementation , we also conduct automasking and multi-scale depth upsampling strategy as proposed in Godard et al . ( 2019 ) to further improve depth estimation . 4 THE PROPOSED METHOD . In this paper , we promote the self-supervised depth estimation paradigm via proposing a semanticguided representation enhancing depth network . The overview of the method is shown in Figure 1 . We propose a multi-task framework which consists of the depth and semantic decoding branch with shared feature encoder . During the training stage , the semantic branch feeds contextual features to the depth branch to enhance depth comprehension . Under this multi-task framework , we propose a Semantic-guided Edge Enhancement Module ( SEEM ) to enhance the local depth feature representation on semantic category edges . Meanwhile , we enhance the global depth feature representations by proposing the semantic-guided multi-level self-attention module , which improves both the semantic and depth feature representations by digging into the pixel-wise dependencies .

Summary: This paper proposed a novel framework to improve self-supervised monocular depth estimation leveraging semantic features at local and global level. The proposed framework includes a semantic-guided edge enhancement module to extract and enhance point-based features around semantic boundaries. The proposed framework also incorporates feature fusion through a self-attention module at different feature levels for global fusion. Evaluation on KITTI datasets compared with recent state-of-the-arts are provided. The proposed method outperformed other monocular depth estimation methods including the ones that also leverage semantic information.