Pseudo Label-Guided Multi Task Learning for Scene Understanding

1 INTRODUCTION . Scene understanding has become increasingly popular in both academia and industry as an essential technology for realizing a variety of vision-based applications such as robotics and autonomous driving . 3D geometric and semantic information of a scene often serve as a basic building block for high-level scene understanding tasks . Numerous approaches have been proposed for inferring a depth map ( Garg et al. , 2016 ; Godard et al. , 2019 ) or grouping semantically similar parts ( Chen et al. , 2017 ; Yuan et al. , 2019 ) from a single image . In parallel with such a rapid evolution for individual tasks , several approaches ( Chen et al. , 2019 ; Zhang et al. , 2018 ; Guizilini et al. , 2020b ; Liu et al. , 2019 ) have focused on boosting the performance through joint learning of the semantic segmentation and monocular depth estimation tasks by considering that the two tasks are highly correlated . For instance , pixels with the same semantic segmentation labels within an object are likely to have similar ( or smoothly-varying ) depth values . An abrupt change of depth values often implies the boundary of two objects containing different semantic segmentation labels . These properties have been applied to deep networks to enhance the semantic segmentation and monocular depth estimation tasks in a synergetic manner . In ( Chen et al. , 2019 ) , they proposed a joint learning model that learns semantic-aware representation to advance the monocular depth estimation with the aid of semantic segmentation . A depth map is advanced by making use of loss functions designed for the purpose of bonding geometric and semantic understanding . The method in ( Guizilini et al. , 2020b ) proposed a new architecture that improves the accuracy of monocular depth estimation through the pixel-adaptive convolution ( Su et al. , 2019 ) using semantic feature maps computed from pre-trained semantic segmentation networks . Despite the improved monocular depth accuracy over a single monocular depth network , the performance improvement of the semantic segmentation task by the aid of geometrical representation has not been verified ( Chen et al. , 2019 ) , or even the semantic segmentation network was fixed with pretrained parameters ( Guizilini et al. , 2020b ) . A generic computational approach for multi-task learning ( MTL ) was proposed in ( Zamir et al. , 2018 ) , which models the structure across twenty six tasks , including 2D , 2.5D , 3D , and semantic tasks , by finding first and higher-order transfer learning dependencies across them in a latent space to seamlessly reuse supervision among related tasks and/or solve them in a single network without increasing the complexity significantly . This was further extended by imposing a cross-task consis- tency based on inference-path invariance on a graph of multiple tasks ( Zamir et al. , 2020 ) . Though these approaches provide a generic and principled way for leveraging redundancies across multiple tasks , there may be limitations to improving the performance of individual tasks in that it is difficult to consider task-specific architectures and loss functions in such unified frameworks . With the same objective yet with a different methodology , the method in ( Liu et al. , 2019 ) proposes a novel MTL architecture consisting of task-shared and task-specific networks based on task-attention modules , aiming to learn both generalizable features for multiple tasks and features tailored to each task . They validated the performance in the joint learning of monocular depth and semantic segmentation . In this paper , we propose a novel MTL architecture for monocular depth estimation and semantic segmentation tasks , called pseudo label-guided multi-task learning ( Pseudo-MTL ) . The proposed architecture leverages geometrically- and semantically-guided representations by introducing pseudo ground truth labels . When a pair of stereo images is given as inputs , our method first generates pseudo ground truth left and right depth maps by using existing pre-trained stereo matching networks ( Pang et al. , 2017 ; Chang & Chen , 2018 ) . To prevent inaccurate depth values from being used , a stereo confidence map ( Poggi & Mattoccia , 2016 ) is used together as auxiliary data that measures the reliability of the pseudo depth labels . These are leveraged for supervising the monocular depth network , obtaining substantial performance gain over recent self-supervised monocular depth estimation approaches ( Godard et al. , 2017 ; 2019 ) . More importantly , the pseudo depth labels are particularly useful when imposing a cross-view consistency across left and right images . The estimated monocular depth and segmentation maps of two views are tied from a geometric perspective by minimizing the cross-view consistency loss , alleviating the mismatch problem incurred by inconsistent prediction across two views significantly . We will verify through an intensive ablation study that the proposed cross-consistency loss leads to a substantial improvement on both tasks . Experimental results also show that our approach achieves an outstanding performance over state-of-the-arts . In short , our novel contributions can be summarized as follows . • We propose a novel MTL approach that jointly performs monocular depth estimation and semantic segmentation through pseudo depth labels . • The cross-view consistency loss based on the pseudo depth labels and associated confidence maps is proposed to enable consistent predictions across two views . • An intensive ablation study is provided to quantify the contribution of the proposed items to performance improvement . 2 RELATED WORK . Monocular Depth Estimation While early works for monocular depth estimation are based on supervised learning , self-supervised learning has attracted increasing interest in recent approaches ( Godard et al. , 2017 ; 2019 ; Watson et al. , 2019 ) to overcome the lack of ground truth depth labels . Here , we review works mostly relevant to our method . Godard et al . ( Godard et al. , 2017 ; 2019 ) proposed the deep network that infers a disparity map using the image reconstruction loss and left-right consistency loss from a pair of stereo images or monocular videos . Chen et al . ( Chen et al. , 2019 ) infers both disparity and semantic segmentation maps by enforcing the cross consistency across stereo images to address the mismatch problem of ( Godard et al. , 2017 ) . Several approaches have focused on improving the monocular depth estimation through the aid of segmentation networks , e.g. , by stitching local depth segments from instance segmentation with respect to scale and shift ( Wang et al. , 2020 ) or leveraging pretrained semantic segmentation networks to guide the monocular depth estimation ( Guizilini et al. , 2020b ) . Semantic Segmentation A deep convolutional encoder-decoder architecture for semantic segmentation proposed in ( Badrinarayanan et al. , 2017 ) has been widely used as backbone . The pyramid pooling module was proposed for leveraging global context through aggregation of different regionbased contexts ( Zhao et al. , 2017 ) . Some segmentation works attempted to combine different tasks to improve segmentation performance . Gated-SCNN ( Takikawa et al. , 2019 ) refines segmentation results by fusing semantic-region features and boundary features . FuseNet ( Hazirbas et al. , 2016 ) proposed to fuse features from color and depth images for improving the segmentation performance . Multi-task learning In ( Chen et al. , 2019 ; Takikawa et al. , 2019 ; Zhang et al. , 2018 ) , they proposed to leverage task-specific loss functions to tie up two ( or more ) tasks within the MTL architecture . For instance , Chen et al . ( Chen et al. , 2019 ) attempted to improve a monocular depth accuracy by using the loss functions that measure the consistency between geometric and semantic predictions . The generic computational approach for MTL was proposed by leveraging redundancies across multiple tasks in a latent space in ( Zamir et al. , 2018 ; 2020 ) . The task-attention modules were introduced to extract features for individual tasks in ( Misra et al. , 2016 ; Liu et al. , 2019 ; Jha et al. , 2020 ) . In this work , we focus on improving the performance of the MTL architecture for monocular depth estimation and semantic segmentation tasks by using the cross-view consistency loss based on pseudo labels . 3 PROPOSED METHOD . 3.1 OVERVIEW AND ARCHITECTURE DESIGN . Our Pseudo-MTL approach focuses on improving the performance of the monocular depth estimation and semantic segmentation tasks through task-specific losses defined based on the pseudo depth labels generated by using pre-trained stereo matching networks ( Pang et al. , 2017 ) . The stereo confidence maps are used together as auxiliary data to compensate for estimation errors in the pseudo depth labels . These are effective in mitigating undesired artifacts of errors that may exist in the pseudo depth labels . In our work , we chose the CCNN ( Poggi & Mattoccia , 2016 ) for calculating the confidence map , but more advanced confidence estimation approaches can also be used . As shown in Figure 1 , the proposed Pseudo-MTL network is based on the encoder-decoder architecture , in which a single encoder takes an image and two decoders predict the monocular depth and semantic segmentation maps . The encoder network E consists of the convolutional layers of the VGG network ( Simonyan & Zisserman , 2015 ) . Two decoders , Dd for monocular depth estimation and Ds for monocular depth estimation , are designed symmetrically with the encoder . While two tasks share the encoder , the task-specific decoder branches are used for each task . The pseudo depth labels and the segmentation label maps of stereo images are used for supervising the proposed architecture . The monocular depth and segmentation maps of left and right images are estimated by passing each image to the proposed architecture , as shown in Figure 1 . The cross-view consistency loss is then imposed on the prediction results of two views . To be specific , the estimated monocular depth maps of left and right images are warped and tested using the pseudo depth labels for ensuring inference-view invariance on the monocular depth estimation , and a similar procedure is also applied to the semantic segmentation . Using the pseudo depth labels for training the proposed model is advantageous at various aspects . The pseudo depth labels of stereo images , filtered out by its confidence map , provides a better supervision ( Choi et al. , 2020 ) than recent self-supervised monocular depth estimation approaches . More importantly , the cross-view consistency based on the pseudo depth labels mitigates the mismatch problem by inconsistent prediction results of two views , leading to a substantial performance gain . Our method aims at advancing the two tasks via task-specific losses based on pseudo ground truth labels , and existing MTL architectures , e.g . based on task-specific attention modules and adaptive balancing ( Liu et al. , 2019 ; Jha et al. , 2020 ) , can be used complementarily with our loss functions . 3.2 LOSS FUNCTIONS . Loss functions are divided into two parts , 1 ) supervised loss for depth and segmentation networks and 2 ) pseudo depth-guided reconstruction loss for cross-view consistency . Note that the supervised loss used for monocular depth estimation relies on the pseudo depth labels generated from a pair of stereo images . 3.2.1 LOSS FOR MONOCULAR DEPTH AND SEMANTIC SEGMENTATION . Depth maps di for i = { l , r } , predicted by the decoder Dd for monocular depth estimation , are used for measuring the depth regression loss Ld as follows : Ld = ∑ i= { l , r } Lreg ( ci , di , d pgt i ) , where Lreg ( ci , di , d pgt i ) = 1 Zi ∑ p∈Φ ci ( p ) · |di ( p ) − dpgti ( p ) |1 , ( 1 ) where ci and d pgt i indicate the confidence map and pseudo ground truth depth map of left ( i = l ) or right ( i = r ) images , respectively . The loss is normalized with Zi = ∑ p ci ( p ) . Φ represents a set of all pixels . The confidence map serves to exclude inaccurate depth values of dpgti when calculating the depth regression loss Ld . This can be used in various ways , including the hard thresholding ( Cho et al. , 2019 ; Tonioni et al. , 2020 ) and the soft thresholding ( Choi et al. , 2020 ) . Among them , the soft thresholded confidence map ( Choi et al. , 2020 ) is shown to be effective in the monocular depth estimation . Our work chose to threshold the confidence map through the soft-thresholding of ( Choi et al. , 2020 ) . We found that the pretrained threshold network already provides satisfactory results , and thus it was fixed during our network training . A supervised loss for semantic segmentation is defined with the standard cross-entropy H : Ls = ∑ i= { l , r } H ( si , s gt i ) , ( 2 ) si and s gt i denote the segmentation map , predicted by the decoder Ds for semantic segmentation , and ground truth segmentation map , respectively . The supervised loss for both tasks is defined as LS = αdLd + αsLs with loss weights αd and αs .

Summary: The paper presents a joint learning strategy for simultaneous semantic segmentation and monocular depth estimation. The main idea is to exploit stereo pairs in training and introduce pseudo-depth label estimated from pre-trained stereo-matching networks. Given the pseudo-depth with confidence estimation, the method proposes a cross-view consistency loss for both depth and semantic predictions, which augments the standard segmentation loss. The proposed method is evaluated on KITTI and Cityscapes datasets with comparisons to prior work and ablative study.