Generative Modeling for Multitask Visual Learning

Generative modeling has recently shown great promise in computer vision , but it has mostly focused on synthesizing visually realistic images . In this paper , motivated by multi-task learning of shareable feature representations , we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks . Correspondingly , we propose a general multi-task oriented generative modeling ( MGM ) framework , by coupling a discriminative multi-task network with a generative network . While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios , our framework enables us to use synthesized images paired with only weak annotations ( i.e. , image-level scene labels ) to facilitate multiple visual tasks . Experimental evaluation on challenging multi-task benchmarks , including NYUv2 and Taskonomy , demonstrates that our MGM framework improves the performance of all the tasks by large margins , especially in the low-data regimes , and our model consistently outperforms state-of-the-art multi-task approaches . 1 INTRODUCTION . Seeing with the mind ’ s eye — creating internal images of objects and scenes not actually present to the senses , is perhaps one of the hallmarks in human cognition ( Pelaprat & Cole , 2011 ) . For humans , this visual imagination integrates learning experience and facilitates learning by solving different problems ( Egan , 2014 ; Pearson , 2019 ; Pelaprat & Cole , 2011 ; Egan , 1989 ) . Inspired by such ability , there has been increasing interest in building generative models that can synthesize images ( Goodfellow et al. , 2014 ) . Yet , most of the effort has focused on generating visually realistic images ( Brock et al. , 2018 ; Zhang et al. , 2019 ) , which are still far from useful for machine perception tasks ( Shmelkov et al. , 2018 ; Borji , 2019 ; Wu et al. , 2016 ) . Even though recent work has started improving the “ usefulness ” of synthesized images , this line of investigation is often limited to a single specific task ( Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ; Zhu et al. , 2018 ; Souly et al. , 2017 ) . Could we guide generative models to benefit multiple visual tasks ? While similar spirits of shared feature representations have been widely studied as multi-task learning or meta-learning ( Finn et al. , 2017 ; Zamir et al. , 2018 ) , here we take a different perspective — learning a shared generative model across various tasks ( as illustrated in Figure 1 ) . Leveraging multiple tasks allows us to capture the underlying image generation mechanism for more comprehensive object and scene understanding than being done within individual tasks . Taking simultaneous semantic segmentation , depth estimation , and surface normal prediction as an example ( Figure 1 ) , successful generative modeling requires understanding not only the semantics but also the 3D geometric structure and physical property of the input image . Meanwhile , a learned generative model facilitates the flow of knowledge across tasks , so that they benefit one another . For instance , the synthesized images provide meaningful variations in existing images and could work as additional training data to build better task-specific models . These variations are especially critical when the data is limited . This paper thus explores multi-task oriented generative modeling ( MGM ) , by coupling a discriminative multi-task network with a generative network . To make them cooperate with each other , a straightforward solution would be to synthesize both RGB images and corresponding pixel-level annotations ( e.g. , pixel-wise class labels for semantic segmentation and depth map for normal estimation ) . In the single task scenario , existing work trains a separate generative model to synthesize paired pixel-level labeled data ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) and produce an augmented set . However , these models are still highly task-dependant , and extending them to multi-task scenarios becomes difficult . A natural question then is : Do we actually need to synthesize paired image and multi-annotation data to be useful for multi-task visual learning ? Our MGM addresses this question by proposing a general framework that uses synthesized images paired with only weak annotations ( i.e. , image-level scene labels ) to facilitate multiple visual tasks . Our key insight is to introduce auxiliary discriminative tasks that ( i ) only require image-level annotation or no annotation , and ( ii ) correlate with the original multiple tasks of interest . To this end , as additional components of the discriminative multi-task network , we introduce a refinement network and a self-supervision network that satisfies these properties . Through joint training , the discriminative network explicitly guides the image synthesis process . The generative network also contributes to further refining the shared feature representation . Meanwhile , the synthesized images of the generative network are used as additional training data for the discriminative network . In more detail , the refinement network performs scene classification on the basis of the multitask network predictions , which requires only scene labels . The self-supervision network can be operationalized on both real and synthesized images without reliance on annotations . With these two modules , our MGM is able to learn from both ( pixel-wise ) fully-annotated real images and synthesized ( image-level ) weakly labeled images . We instantiate MGM with the state-of-the-art encoder-decoder based multi-task network ( Zamir et al. , 2018 ) , self-attention GAN ( Zhang et al. , 2019 ) , and contrastive learning-based self-supervision network ( Chen et al. , 2020 ) . Note that our framework is agnostic to the choice of these model components . We evaluate our approach on standard multi-task benchmarks , including the NYUv2 ( Nathan Silberman & Fergus , 2012 ) and Taskonomy ( Zamir et al. , 2018 ) datasets . Consistent with the previous works ( Sun et al. , 2019 ; Standley et al. , 2020 ) , we focus on three tasks of great practical importance : semantic segmentation , depth estimation , and normal prediction . The evaluation shows that : ( 1 ) our MGM consistently outperforms state-of-the-art multi-task approaches by large margins , especially in the low-data regime . ( 2 ) With the increasing of generated samples , the performance of MGM consistently improves and it also almost reaches the performance upper-bound that trains with weakly annotated real images . ( 3 ) Our model can be extended to more visual tasks . 2 PILOT STUDY . This pilot study provides an initial experimentation , which shows the importance of our proposed problem ( multi-task oriented generative modeling ) and motivates the development of our method . Specifically , we show that directly using images synthesized by an off-the-shelf generative model that is trained with the realistic objective is not helpful for downstream pixel-level perception tasks . Experimental Design : For ease of analysis , here we focus on a single task – semantic segmentation , and use the Tiny-Taskonomy dataset ( Zamir et al. , 2018 ) . The dataset split and evaluation metric are the same as our main experiments ( See Section 4 for details ) . We train a self-attention GAN ( Zhang et al. , 2019 ) on Tiny-Taskonomy , and use it to generate the same number of synthesized images as the real images to augment the training set . How to generate pixel-level annotations ? One remaining question is how to generate pixel-level annotations for these synthesized images produced by the self-attention GAN . While prior work has explored synthesizing both images and their pixel-level annotations for some specific tasks ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) , these annotations are still not reliable . For ease of analysis , in this study , we factor out the effect of annotations and assume that we have an oracle annotator . We use the annotator from Taskonomy ( Zamir et al. , 2018 ) , which is a state-of-the-art large fully supervised semantic segmentation network . In fact , the ground-truth of semantic segmentation on Taskonomy is produced as the output of this network rather than labeled by humans . By doing so , we ensure that the annotations of the synthesized images are “ accurate ” . Comparisons : Single-Task ( ST ) model is our baseline which follows the architecture of Taskonomy single task network . ST is trained on real images only . STG is the ST model trained on the augmented set . Table 1 reports the results of these two models , and STG is worse than ST . Does synthesized images help downstream tasks ? From Table 1 , the answer is NO . Even though the images are synthesized by one of the state-of-the-art generative models and are labeled by the oracle annotator , they still can not benefit the downstream task . This is probably because these images are synthesized off the shelf without “ knowing ” the downstream task . Our key insight then is that we need to explicitly use the downstream task objective to guide the image synthesis process . Moreover , here we focused on a single task and assumed that we had the oracle annotations . However , an oracle annotator is difficult to obtain in practice , especially for multiple tasks . Also , existing work can not synthesize paired images and pixel-level annotations for multiple tasks ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) . To overcome these challenges , in what follows we demonstrate how to facilitate multiple visual tasks with synthesize images that only need image-level scene labels . 3 METHOD . We propose multi-task oriented generative modeling ( MGM ) to leverage generative networks for multi-task visual learning , as summarized in Figure 2 . In this section , we first formalize the novel problem setting of MGM . Then , we explain the general framework and an instantiation of the MGM model with state-of-the-art multi-task learning and image generation approaches . Finally , we discuss the detailed training strategy for the framework . 3.1 PROBLEM SETTING . Multi-task discriminative learning : Given a set of n visual tasks T = { T1 , T2 , · · · , Tn } , we aim to learn a discriminative multi-task model M that is able to address all of these tasks simultaneously : M ( x ) → ŷ = ( ŷ1 , ŷ2 , · · · , ŷn ) , where x is an input image and ŷi is the prediction for task Ti . Here we focus on the type of per-pixel level prediction tasks ( e.g. , semantic segmentation or depth estimation ) . We treat image classification as a special task , which provides global semantic description ( i.e. , scene labels ) of images and only requires image-level category annotation c. Therefore , the set of fully annotated real data is denoted as Sreal = { ( xj , y1j , y2j , · · · , ynj , cj ) } . Generative learning : Meanwhile , we aim to learn a generative model G that produces a set of synthesized data but with only corresponding image-level scene labels ( weak annotation ) : G ( c , z ) → x̃ , where z is a random input , and x̃ is a synthesized image . The scene label of x̃ is denoted as c̃ = c. We denote the set of synthesized images and their corresponding scene labels as S̃syn = { ( x̃k , c̃k ) } . Cooperation between discriminative and generative learning : Our objective is that the discriminative model M and the generative model G cooperate with each other to improve the performance on the multiple visual tasks T . During the whole process , the full model only gets access to the real fully-labeled data Sreal , then the generative network is trained to produce the synthesized set S̃syn . Finally , M effectively learns from both Sreal and S̃syn . Note that , unlike most of the existing work on image generation ( Brock et al. , 2018 ; Zhang et al. , 2019 ) , we do not focus on the visual realism of the synthesized images x̃ . Instead , we hope G to capture the underlying mechanism that benefits M .

Summary: This paper proposes a multi-task network for pixel-wise estimation tasks such as semantic segmentation and depth estimation. The authors additionally introduced an GAN-based image generation network, a self-supervised network and a scene category classification network (which was called a refinement network in the paper). As an image generation network, any conditional GANs including SA-GAN and DCGAN can be used. As a self-supervised network, SimCLR was used in the proposed network. The refinement network classifies a scene category from the outputs of multitask networks (i.e. estimated segmentation masks and estimated depth images).  In the experiments with two dataset, the proposed model, MGM, consistently outperformed the results of the baselines and ablated methods.  In addition, additional experiments with six tasks and high resolution images (256x256) were performed and confirmed the effectiveness of the proposed method as well.