Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

1 INTRODUCTION . Accurate depth estimation is a key problem in computer vision and robotics , as it is instrumental for perception , navigation , and planning . Although perceiving depth typically requires dedicated sensors ( e.g. , stereo rigs , LiDAR ) , learning to predict depth from monocular imagery can provide useful cues for a wide array of tasks ( Michels et al. , 2005 ; Kendall et al. , 2018 ; Manhardt et al. , 2019 ; Lee et al. , 2019 ) . Going beyond supervised learning from direct measurements ( Eigen et al. , 2014 ) , self-supervised methods exploit geometry as supervision ( Guo et al. , 2018 ; Pillai et al. , 2019 ; Zou et al. , 2018 ; Yang et al. , 2017 ) , therefore having the potential to leverage large scale datasets of raw videos to outperform supervised methods ( Guizilini et al. , 2019 ) . Although depth from a single image is an ill-posed inverse problem , monocular depth networks are able to make accurate predictions by learning representations connecting the appearance of scenes and objects with their geometry in Euclidean 3D space . Due to perspective , there is indeed an equivariance relationship between the visual appearance of an object in 2D and its depth , when conditioned on the object ’ s category . For instance , a car 25 meters away appears smaller ( on the image plane ) than a car only 5 meters away but bigger than a truck 50 meters away . Current depth estimation methods either do not leverage this structure explicitly or rely on strong semantic supervision to jointly optimize geometric consistency and a semantic proxy task in a multi-task objective ( Ochs et al. , 2019 ; Chen et al. , 2019 ) , thus departing from the self-supervised paradigm . In this paper , we explore how we can leverage semantic information to improve monocular depth prediction in a self-supervised way . Our main contribution is a novel architecture that uses a fixed pre-trained semantic segmentation network to guide geometric representation learning in a selfsupervised monocular depth network . In contrast to standard convolutional layers , our architecture uses pixel-adaptive convolutions ( Su et al. , 2019 ) to learn semantic-dependent representations that can better capture the aforementioned equivariance property . Leveraging semantics may nonetheless introduce category-specific biases . Our second contribution is a two-stage training process where we automatically detect the presence of a common bias on dynamic objects ( projections at infinity ) and resample the training set to de-bias it . Our method improves upon the state of the art in selfsupervised monocular depth estimation on the standard KITTI benchmark ( Geiger et al. , 2013 ) , both on average over pixels , over classes , and for dynamic categories in particular . †Source code and pretrained models are available on https : //github.com/TRI-ML/packnet-sfm 2 RELATED WORK . Since the seminal work of Eigen et al . ( 2014 ) , substantial progress has been done to improve the accuracy of supervised depth estimation from monocular images , including the use of Conditional Random Fields ( CRFs ) ( Li et al. , 2015 ) , joint optimization of surface normals ( Qi et al. , 2018 ) , fusion of multiple depth maps ( Lee et al. , 2018 ) , and ordinal classification ( Fu et al. , 2018 ) . Consequently , as supervised techniques for depth estimation advanced rapidly , the availability of largescale depth labels became a bottleneck , especially for outdoor applications . Garg et al . ( 2016 ) and Godard et al . ( 2017 ) provided an alternative self-supervised strategy involving stereo cameras , where Spatial Transformer Networks ( Jaderberg et al. , 2015 ) can be used to geometrically warp , in a differentiable way , the right image into a synthesized left image , using the predicted depth from the left image . The photometric consistency loss between the resulting synthesized and original left images can then be minimized in an end-to-end manner using a Structural Similarity term ( Wang et al. , 2004 ) and additional depth regularization terms . Following Godard et al . ( 2017 ) and Ummenhofer et al . ( 2017 ) , Zhou et al . ( 2017 ) generalized this to the purely monocular setting , where a depth and a pose networks are simultaneously learned from unlabeled monocular videos . Rapid progress in terms of architectures and objective functions ( Yin & Shi , 2018 ; Mahjourian et al. , 2018 ; Casser et al. , 2019 ; Zou et al. , 2018 ; Klodt & Vedaldi , 2018 ; Wang et al. , 2018 ; Yang et al. , 2018 ) have since then turned monocular depth estimation into one of the most successful applications of self-supervised learning , even outperforming supervised methods ( Guizilini et al. , 2019 ) . The introduction of semantic information to improve depth estimates has been explored in prior works , and can be broadly divided into two categories . The first one uses semantic ( or instance ) information to mask out or properly model dynamic portions of the image , which are not accounted for in the photometric loss calculation . Güney & Geiger ( 2015 ) leveraged object knowledge in a Markov Random Field ( MRF ) to resolve stereo ambiguities , while Bai et al . ( 2016 ) used a conjunction of instance-level segmentation and epipolar constraints to reduce uncertainty in optical flow estimation . Casser et al . ( 2019 ) used instance-level masks to estimate motion models for different objects in the environment , and account for their external motion in the resulting warped image . The second category attempts to learn both tasks in a single framework , and uses consistency losses to ensure that both are optimized simultaneously and regularize each other , so the information contained in one task can be transferred to improve the other . For instance , Ochs et al . ( 2019 ) estimated depth with an ordinal classification loss similar to the standard semantic classification loss , and used empirical weighting to combine them into a single loss for optimization . Similarly , Chen et al . ( 2019 ) used a unified conditional decoder that can generate either semantic or depth estimates , and both outputs are used to generate a series of losses also combined using empirical weighting to generate the final loss to be optimized . Our approach focuses instead on representation learning , exploiting semantic features into the selfsupervised depth network by using a pretrained semantic segmentation network to guide the generation of depth features . This is done using pixel-adaptive convolutions , recently proposed in Su et al . ( 2019 ) and applied to tasks such as depth upsampling using RGB images for feature guidance . We show that different depth networks can be readily modified to leverage this semantic feature guidance , ranging from widely used ResNets ( He et al. , 2016 ) to the current state-of-the-art PackNet ( Guizilini et al. , 2019 ) , with a consistent gain in performance across these architectures . 3 SELF-SUPERVISED STRUCTURE-FROM-MOTION . Our semantically-guided architecture is developed within a self-supervised monocular depth estimation setting , commonly known as structure-from-motion ( SfM ) . Learning in a self-supervised structure-from-motion setting requires two networks : a monocular depth model fD : I → D , that outputs a depth prediction D̂ = fD ( I ( p ) ) for every pixel p in the target image I ; and a monocular ego-motion estimator fx : ( It , IS ) → xt→S , that predicts the 6 DoF transformations for all s ∈ S given by xt→s = ( R t0 1 ) ∈ SE ( 3 ) between the target image It and a set of temporal context source images Is ∈ IS . In all reported experiments we use It−1 and It+1 as source images . 3.1 THE SELF-SUPERVISED OBJECTIVE LOSS . We train the depth and pose networks simultaneously , using the same protocols and losses as described in Guizilini et al . ( 2019 ) . Our self-supervised objective loss consists of an appearance matching term Lp that is imposed between the synthesized Ît and original It target images , and a depth regularization term Ls that ensures edge-aware smoothing in the depth estimates D̂t . The final objective loss is averaged per pixel , pyramid scale and image batch , and is defined as : L ( It , Ît ) = Lp ( It , Ît ) + λ1 Ls ( D̂t ) ( 1 ) where λ1 is a weighting coefficient between the photometricLp and depth smoothnessLs loss terms . Following Godard et al . ( 2017 ) and Zhou et al . ( 2017 ) , the similarity between synthesized Ît and original It target images is estimated using a Structural Similarity ( SSIM ) term ( Wang et al. , 2004 ) combined with an L1 loss term , inducing the following overall photometric loss : Lp ( It , Ît ) = α 1− SSIM ( It , Ît ) 2 + ( 1− α ) ‖It − Ît‖ ( 2 ) In order to regularize the depth in low gradient regions , we incorporate an edge-aware term similar to Godard et al . ( 2017 ) . This loss is weighted for each of the pyramid levels , decaying by a factor of 2 on each downsampling , starting with a weight of 1 for the 0th pyramid level . Ls ( D̂t ) = |δxD̂t|e−|δxIt| + |δyD̂t|e−|δyIt| ( 3 ) We also incorporate some of the insights introduced in Godard et al . ( 2018 ) , namely auto-masking , minimum reprojection error , and inverse depth map upsampling to further improve depth estimation performance in our self-supervised monocular setting . 3.2 DEPTH AND POSE NETWORKS . Our baseline depth and pose networks are based on the PackNet architecture introduced by Guizilini et al . ( 2019 ) , which proposes novel packing and unpacking blocks to respectively downsample and upsample feature maps during the encoding and decoding stages . This network was selected due to its state-of-the-art performance in the task of self-supervised monocular depth estimation , so we can analyze if our proposed architecture is capable of further improving the current state-of-the-art . However , there are no restrictions as to which models our proposed semantically-guided architecture can be applied to , and in Section 5.4 we study its application to different depth networks . 4 SEMANTICALLY-GUIDED GEOMETRIC REPRESENTATION LEARNING . In this section , we describe our method to inject semantic information into a self-supervised depth network via its augmentation with semantic-aware convolutions . Our proposed architecture is depicted in Figure 2 and is composed of two networks : a primary one , responsible for the generation of depth predictions D̂ = fD ( I ( p ) ) ; and a secondary one , capable of producing semantic predictions . Only the first network is optimized during self-supervised learning ; the semantic network is initialized from pretrained weights and is not further optimized . This is in contrast to the common practice of supervised ( ImageNet ) pretraining of depth encoders ( Godard et al. , 2018 ; Casser et al. , 2019 ; Zou et al. , 2018 ) : here instead of fine-tuning from pre-trained weights , we preserve these secondary weights to guide the feature learning process of the primary depth network . Our approach also differs from learning without forgetting ( Li & Hoiem , 2017 ) by leveraging fixed intermediate feature representations as a way to maintain consistent semantic guidance throughout training . 4.1 SEMANTICALLY-GUIDED DEPTH FEATURES . We leverage the information from the pretrained semantic network in the depth network through the use of pixel-adaptive convolutions ( Su et al. , 2019 ) . They were recently proposed to address some limitations inherent to the standard convolution operation , namely its translation invariance making it content-agnostic . While this significantly reduces the number of parameters of the resulting network , this might also lead to sub-optimal solutions under certain conditions important for geometric representation learning . For example , spatially-shared filters globally average the loss gradients over the entire image , forcing the network to learn weights that can not leverage location-specific information beyond their limited receptive fields . Content-agnostic filters are unable to distinguish between different pixels that are visually similar ( i.e . dark areas due to shadows or black objects ) or generalize to similar objects that are visually different ( i.e . cars with varying colors ) . In this work , we use pixel-adaptive convolutions to produce semantic-aware depth features , where the fixed information encoded in the semantic network is used to disambiguate geometric representations for the generation of multi-level depth features . As shown in Figure 2 , we extract multi-level feature maps from the semantic network . For each feature map , we apply a 3 × 3 and a 1 × 1 convolutional layer followed by Group Normalization ( Wu & He , 2018 ) and ELU non-linearities ( Clevert et al. , 2016 ) . These processed semantic feature maps are then used as guidance on their respective pixel-adaptive convolutional layers , following the formulation proposed in Su et al . ( 2019 ) : v′i = ∑ j∈Ω ( i ) K ( fi , fj ) W [ pi − pj ] vj + b ( 4 ) In the above equation , f ∈ RD are processed features from the semantic network that will serve to guide the pixel-adaptive convolutions from the depth network , p = ( x , y ) T are pixel coordinates , with [ pi − pj ] denoting 2D spatial offsets between pixels , Wk×k are convolutional weights with kernel size k , Ωi defines a k×k convolutional window around i , v is the input signal to be convolved , and b ∈ R1 is a bias term . K is the kernel used to calculate the correlation between guiding features , here chosen to be the standard Gaussian kernel : K ( fi , fj ) = exp ( −1 2 ( fi − fj ) TΣ−1ij ( fi − fj ) ) ( 5 ) where Σij is the covariance matrix between features fi and fj , here chosen to be a diagonal matrix σ2 ·ID , with σ as an extra learnable parameter for each convolutional filter . These kernel evaluations can be seen as a secondary set of weights applied to the standard convolutional weights , changing their impact on the resulting depth features depending on the content stored in the guiding semantic features . For example , the information contained in depth features pertaining to the sky should not be used to generate depth features describing a pedestrian , and this behavior is now captured as a larger distance between their corresponding semantic features , which in turn produces smaller weights for that particular convolutional filter . Note that the standard convolution can be considered a special case of the pixel-adaptive convolution , where ∀ ij , K ( fi , fj ) = 1 .

Summary: The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. The semantic features are predicted by a pretrained network rather than relying on a ground truth. Moreover, a two-stage training process in proposed in order to filter out images leading to erroneous SfM predictions. The method is evaluated with different networks on the KITTY dataset.