Semantic-Guided Representation Enhancement for Self-supervised Monocular Trained Depth Estimation

1 INTRODUCTION . Depth estimation is a long standing problem in computer vision community , which offers useful information to a wide range of tasks including robotic perception , augmented reality and autonomous driving , etc . Compared with depth estimation methods which rely on active vision or multi-view paradigms ( Schonberger & Frahm , 2016 ; Li et al. , 2019 ) , estimating depth from only a single image is highly ill-posed , thus brings greater challenges for higher quality results . In recent years , monocular depth estimation has witnessed new renaissances with the advent of deep learning ( He et al. , 2016 ; Jaderberg et al. , 2015 ; Simonyan & Zisserman , 2014 ) . By learning deep representations in a supervised manner , various of networks ( Eigen & Fergus , 2015 ; Eigen et al. , 2014 ; Laina et al. , 2016 ) are capable of producing high quality depth maps thanks to the large corpus of training data . At the mean time , consider the lack of labeled training data for network training , recent advances ( Zhou et al. , 2017 ; Godard et al. , 2019 ; 2017 ) show that monocular depth estimation can be accomplished in a self-supervised way . The network can be trained by unlabeled image sequences using two-view geometric constraints , while achieving comparable results with the supervised paradigm . The learning-based methods managed to handle the highly ill-posed monocular depth estimation problem by implicitly learning the mapping between visual appearance and its corresponding depth value . However , despite the great effectiveness for learning-based depth estimation , these methods still struggle to conduct precise depth estimation on challenging image regions such as semantic category borders or thin object areas . For example , the estimated object depth usually fails to align with the real object borders , and the depth of foreground objects which have thin structures tends to be submerged in the background . We attribute these phenomena to the limited depth representation ability that ( 1 ) the pixel-wise local depth information can not be well represented by current depth network , especially on highly ambiguous , semantic border areas , ( 2 ) current depth representations are not capable of well describing depth foreground/background relationships globally . These issues lead to the wrong depth estimation from the true scene structures , which hinders the further applications in the real-world tasks . In this paper , we address this problem via enhancing both local and global depth feature representations for self-supervised monocular depth estimation via semantic guidance . As semantic segmentation conducts explicit category-level scene understandings and produces well-aligned object boundary detection , we propose an extra semantic estimation branch inside the self-supervised paradigm . The semantic branch offers rich contextual features which is fused with depth features during multiscale learning . Under this framework , we propose to enhance the depth feature using semantic guidance in a local-to-global way . To improve the local depth representations on semantic category borders , inspired by the sampling and enhancing strategies used in semantic segmentation ( Kirillov et al. , 2020 ) , our first contribution is to propose a Semantic-guided Edge Enhancement Module ( SEEM ) which is specially designed to enhance the local point-based depth representations that locate on the semantic edges . Different from the method in ( Kirillov et al. , 2020 ) , we enhance the point features using multi-level of representations from different domains under a self-supervised framework . To be specific , we sample a set of point positions lying on the binary semantic category borders , and extract the point-wise features of the corresponding edge positions from the encoded feature , depth decoding feature as well as the semantic decoding feature . We then merge and enhance the point-wise features and feed them to the final depth decoding features to promote the edge-aware depth inference for self-supervision . For global depth representation enhancement , our second contribution is to propose a semantic-guided multi-level attention module to improve the global depth representation . Different from the conventional self-attentions ( Fu et al. , 2019 ; Vaswani et al. , 2017 ) which are implemented as single modules on the bottleneck feature block , we propose to explore the self-attentions on different level of decoding layers . In this way , both semantic and depth representations can be further promoted by exploring and leveraging the pixel-wise correlations inside of their fused features . We validate our method mainly on KITTI 2015 ( Geiger et al. , 2012 ) , and the Cityscapes ( Cordts et al. , 2016 ) benchmark is also used for evaluating the generalization ability . Experiments show that the proposed method significantly improves the depth estimation on category edges and thin scene structures . Extensive quantitative and qualitative results validate the superiority of our method that it outperforms the state-of-the-art methods for self-supervised monocular depth estimation . 2 RELATED WORK . There exist extensive researches on monocular depth estimation , including geometry-based methods ( Schonberger & Frahm , 2016 ; Enqvist et al. , 2011 ) and learning-based methods ( Eigen et al. , 2014 ; Laina et al. , 2016 ) . In this paper , however , we concentrate only on the self-supervised depth training and semantic-guided depth estimation , which is highly related to the research focus of this paper . Self-supervised depth estimation . Self-supervised methods enable the networks to learn depth representation from merely unlabeled image sequences that they reformulate the depth supervision loss into the image reconstruction loss . Godard et al . ( 2017 ) and Garg et al . ( 2016 ) first propose the self-supervised method on stereo images , then Zhou et al . ( 2017 ) propose a monocular trained approach using a separate motion estimation network . Based on these frameworks , a large corpus of works seek to promote self-supervised learning from different aspects . For more robust selfsupervision signals , Mahjourian et al . ( 2018 ) propose to use 3D information as extra supervisory signal , another kind of methods leverage additional information such as the optical flow ( Ranjan et al. , 2019 ; Wang et al. , 2019b ) to strengthen depth supervision via consistency constraints . In order to solve the loss deviation problems on non-rigid motion areas , selective masks are used to filter out the these areas when computing losses . Prior works generate the mask by the network itself ( Zhou et al. , 2017 ; Yang et al. , 2017 ; Vijayanarasimhan et al. , 2017 ) , while the succeeding methods produce the mask by leveraging geometric clues ( Bian et al. , 2019 ; Wang et al. , 2019a ; Godard et al. , 2019 ; Klingner et al. , 2020 ) , which is proved to be more effective . There also exist other methods trying to enhance the network performance with traditional SfM ( Schonberger & Frahm , 2016 ) , which offer pseudo labels for depth estimation . Guizilini et al . ( 2020a ) propose a novel network architecture to improve depth estimation . In this paper , we do not consider the performance improvements contributed by this novel architecture in experimental comparisons , and compare different methods with conventional backbones . Depth estimation using semantics . Semantic information are shown to provide positive effects toward depth estimation framework in the previous works . The methods can be categorized into two groups by their way to use the semantic information . One group of the methods leverage the semantic labels directly to guide the depth learning . The works of Ramirez et al . ( 2018 ) and Chen et al . ( 2019a ) propose depth constraints by leveraging the semantic labels of the scene . Klingner et al . ( 2020 ) and Casser et al . ( 2019 ) address the non-rigid motion issues by handling the moving object areas highlighted by the semantic map . Wang et al . ( 2020 ) use a divide-and-conquer strategy to conduct depth estimation with different semantic categories , and Zhu et al . ( 2020 ) offer a depth morphing paradigm with the help of semantic foreground edges . The other group of methods enhance the depth representation by feature manipulation . Chen et al . ( 2019a ) generated depth and semantic maps by a single latent representation . Ochs et al . ( 2019 ) proposed a segmentation-like loss item for depth estimation . Guizilini et al . ( 2020b ) leveraged a pre-trained semantic segmentation network and further conduct feature merging by PAC-based ( Su et al. , 2019 ) module . In this paper , we propose an individual semantic network for depth estimation . But different with Guizilini et al . ( 2020b ) , our semantic branch shares the same encoder and is trained together with the depth network for better scene representation . We leverage the semantic information in both ways . The semantic edges are used to guide the edge-based point feature sampling for local depth depth representation enhancement . At the mean time , we utilize the semantic feature blocks to conduct multi-scale feature fusion for global depth representation enhancement . The proposed method constructs a holistic way for feature representation enhancing in self-supervised depth estimation . 3 SELF-SUPERVISED DEPTH ESTIMATION FRAMEWORK . The proposed method builds upon the self-supervised depth estimation framework . An image triplet ( It−1 , It , It+1 ) is used as the input , where It acts as the target image and It′ ∈ { It−1 , It+1 } are the source images . During training , the target image It is fed into the depth network fD to get the estimated depth Dt = fD ( It ) , while the adjacent image pairs ( It , It′ ) are put into the ego-motion network for the 6-DoF ego-motion estimations Tt′→t . Then , the synthesized target images It′→t can be estimated using the source images It′ , depth Dt and the ego-motions Tt′→t , following the formulation from Zhou et al . ( 2017 ) . The self-supervised loss can be calculated by L ( It , It′→t ) = Lp ( It , It′→t ) + λLs ( It , It′→t ) , ( 1 ) where λ is the weighting factor between the photometric loss Lp and smoothness loss Ls . Following Godard et al . ( 2019 ) , we implement the minimum photometric loss Lp ( It , It′→t ) = min t′ ( α 2 ( 1− SSIM ( It , It′→t ) ) + ( 1− α ) ‖It − It′→t‖1 ) , ( 2 ) where SSIM is the Structural Similarity item ( Wang et al. , 2004 ) , α is the weighting factor which is set to 0.85 . The smoothness loss item ( Godard et al. , 2017 ) is implemented to smooth the depth map by the consistency between the image and depth gradient Ls ( It ) = |∂xDt| e−|∂xIt| + |∂yDt| e−|∂yIt| , ( 3 ) where ∂x , ∂x denote gradient operation on x , y-axis . In our implementation , we also conduct automasking and multi-scale depth upsampling strategy as proposed in Godard et al . ( 2019 ) to further improve depth estimation . 4 THE PROPOSED METHOD . In this paper , we promote the self-supervised depth estimation paradigm via proposing a semanticguided representation enhancing depth network . The overview of the method is shown in Figure 1 . We propose a multi-task framework which consists of the depth and semantic decoding branch with shared feature encoder . During the training stage , the semantic branch feeds contextual features to the depth branch to enhance depth comprehension . Under this multi-task framework , we propose a Semantic-guided Edge Enhancement Module ( SEEM ) to enhance the local depth feature representation on semantic category edges . Meanwhile , we enhance the global depth feature representations by proposing the semantic-guided multi-level self-attention module , which improves both the semantic and depth feature representations by digging into the pixel-wise dependencies .

Summary: The authors tackle the problem of self-supervised depth estimation and particularly address the issue of poor depth estimation on object boundaries. The introduction motivates the problem well, and the related work covers most of the relevant papers. The authors propose two main modifications allowing them to leverage an off-the-shelf semantic segmentation network: SEEM (semantic guided edge enhancement module) and a multi-level self-attention mechanism that fuses depth and semantic features at different levels. The combination of these contributions along with an off-the-shelf semantic segmentation network achieves impressive results, especially on the  boundaries of objects.