Generative Modeling for Multitask Visual Learning

Generative modeling has recently shown great promise in computer vision , but it has mostly focused on synthesizing visually realistic images . In this paper , motivated by multi-task learning of shareable feature representations , we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks . Correspondingly , we propose a general multi-task oriented generative modeling ( MGM ) framework , by coupling a discriminative multi-task network with a generative network . While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios , our framework enables us to use synthesized images paired with only weak annotations ( i.e. , image-level scene labels ) to facilitate multiple visual tasks . Experimental evaluation on challenging multi-task benchmarks , including NYUv2 and Taskonomy , demonstrates that our MGM framework improves the performance of all the tasks by large margins , especially in the low-data regimes , and our model consistently outperforms state-of-the-art multi-task approaches . 1 INTRODUCTION . Seeing with the mind ’ s eye — creating internal images of objects and scenes not actually present to the senses , is perhaps one of the hallmarks in human cognition ( Pelaprat & Cole , 2011 ) . For humans , this visual imagination integrates learning experience and facilitates learning by solving different problems ( Egan , 2014 ; Pearson , 2019 ; Pelaprat & Cole , 2011 ; Egan , 1989 ) . Inspired by such ability , there has been increasing interest in building generative models that can synthesize images ( Goodfellow et al. , 2014 ) . Yet , most of the effort has focused on generating visually realistic images ( Brock et al. , 2018 ; Zhang et al. , 2019 ) , which are still far from useful for machine perception tasks ( Shmelkov et al. , 2018 ; Borji , 2019 ; Wu et al. , 2016 ) . Even though recent work has started improving the “ usefulness ” of synthesized images , this line of investigation is often limited to a single specific task ( Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ; Zhu et al. , 2018 ; Souly et al. , 2017 ) . Could we guide generative models to benefit multiple visual tasks ? While similar spirits of shared feature representations have been widely studied as multi-task learning or meta-learning ( Finn et al. , 2017 ; Zamir et al. , 2018 ) , here we take a different perspective — learning a shared generative model across various tasks ( as illustrated in Figure 1 ) . Leveraging multiple tasks allows us to capture the underlying image generation mechanism for more comprehensive object and scene understanding than being done within individual tasks . Taking simultaneous semantic segmentation , depth estimation , and surface normal prediction as an example ( Figure 1 ) , successful generative modeling requires understanding not only the semantics but also the 3D geometric structure and physical property of the input image . Meanwhile , a learned generative model facilitates the flow of knowledge across tasks , so that they benefit one another . For instance , the synthesized images provide meaningful variations in existing images and could work as additional training data to build better task-specific models . These variations are especially critical when the data is limited . This paper thus explores multi-task oriented generative modeling ( MGM ) , by coupling a discriminative multi-task network with a generative network . To make them cooperate with each other , a straightforward solution would be to synthesize both RGB images and corresponding pixel-level annotations ( e.g. , pixel-wise class labels for semantic segmentation and depth map for normal estimation ) . In the single task scenario , existing work trains a separate generative model to synthesize paired pixel-level labeled data ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) and produce an augmented set . However , these models are still highly task-dependant , and extending them to multi-task scenarios becomes difficult . A natural question then is : Do we actually need to synthesize paired image and multi-annotation data to be useful for multi-task visual learning ? Our MGM addresses this question by proposing a general framework that uses synthesized images paired with only weak annotations ( i.e. , image-level scene labels ) to facilitate multiple visual tasks . Our key insight is to introduce auxiliary discriminative tasks that ( i ) only require image-level annotation or no annotation , and ( ii ) correlate with the original multiple tasks of interest . To this end , as additional components of the discriminative multi-task network , we introduce a refinement network and a self-supervision network that satisfies these properties . Through joint training , the discriminative network explicitly guides the image synthesis process . The generative network also contributes to further refining the shared feature representation . Meanwhile , the synthesized images of the generative network are used as additional training data for the discriminative network . In more detail , the refinement network performs scene classification on the basis of the multitask network predictions , which requires only scene labels . The self-supervision network can be operationalized on both real and synthesized images without reliance on annotations . With these two modules , our MGM is able to learn from both ( pixel-wise ) fully-annotated real images and synthesized ( image-level ) weakly labeled images . We instantiate MGM with the state-of-the-art encoder-decoder based multi-task network ( Zamir et al. , 2018 ) , self-attention GAN ( Zhang et al. , 2019 ) , and contrastive learning-based self-supervision network ( Chen et al. , 2020 ) . Note that our framework is agnostic to the choice of these model components . We evaluate our approach on standard multi-task benchmarks , including the NYUv2 ( Nathan Silberman & Fergus , 2012 ) and Taskonomy ( Zamir et al. , 2018 ) datasets . Consistent with the previous works ( Sun et al. , 2019 ; Standley et al. , 2020 ) , we focus on three tasks of great practical importance : semantic segmentation , depth estimation , and normal prediction . The evaluation shows that : ( 1 ) our MGM consistently outperforms state-of-the-art multi-task approaches by large margins , especially in the low-data regime . ( 2 ) With the increasing of generated samples , the performance of MGM consistently improves and it also almost reaches the performance upper-bound that trains with weakly annotated real images . ( 3 ) Our model can be extended to more visual tasks . 2 PILOT STUDY . This pilot study provides an initial experimentation , which shows the importance of our proposed problem ( multi-task oriented generative modeling ) and motivates the development of our method . Specifically , we show that directly using images synthesized by an off-the-shelf generative model that is trained with the realistic objective is not helpful for downstream pixel-level perception tasks . Experimental Design : For ease of analysis , here we focus on a single task – semantic segmentation , and use the Tiny-Taskonomy dataset ( Zamir et al. , 2018 ) . The dataset split and evaluation metric are the same as our main experiments ( See Section 4 for details ) . We train a self-attention GAN ( Zhang et al. , 2019 ) on Tiny-Taskonomy , and use it to generate the same number of synthesized images as the real images to augment the training set . How to generate pixel-level annotations ? One remaining question is how to generate pixel-level annotations for these synthesized images produced by the self-attention GAN . While prior work has explored synthesizing both images and their pixel-level annotations for some specific tasks ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) , these annotations are still not reliable . For ease of analysis , in this study , we factor out the effect of annotations and assume that we have an oracle annotator . We use the annotator from Taskonomy ( Zamir et al. , 2018 ) , which is a state-of-the-art large fully supervised semantic segmentation network . In fact , the ground-truth of semantic segmentation on Taskonomy is produced as the output of this network rather than labeled by humans . By doing so , we ensure that the annotations of the synthesized images are “ accurate ” . Comparisons : Single-Task ( ST ) model is our baseline which follows the architecture of Taskonomy single task network . ST is trained on real images only . STG is the ST model trained on the augmented set . Table 1 reports the results of these two models , and STG is worse than ST . Does synthesized images help downstream tasks ? From Table 1 , the answer is NO . Even though the images are synthesized by one of the state-of-the-art generative models and are labeled by the oracle annotator , they still can not benefit the downstream task . This is probably because these images are synthesized off the shelf without “ knowing ” the downstream task . Our key insight then is that we need to explicitly use the downstream task objective to guide the image synthesis process . Moreover , here we focused on a single task and assumed that we had the oracle annotations . However , an oracle annotator is difficult to obtain in practice , especially for multiple tasks . Also , existing work can not synthesize paired images and pixel-level annotations for multiple tasks ( Sandfort et al. , 2019 ; Choi et al. , 2019 ) . To overcome these challenges , in what follows we demonstrate how to facilitate multiple visual tasks with synthesize images that only need image-level scene labels . 3 METHOD . We propose multi-task oriented generative modeling ( MGM ) to leverage generative networks for multi-task visual learning , as summarized in Figure 2 . In this section , we first formalize the novel problem setting of MGM . Then , we explain the general framework and an instantiation of the MGM model with state-of-the-art multi-task learning and image generation approaches . Finally , we discuss the detailed training strategy for the framework . 3.1 PROBLEM SETTING . Multi-task discriminative learning : Given a set of n visual tasks T = { T1 , T2 , · · · , Tn } , we aim to learn a discriminative multi-task model M that is able to address all of these tasks simultaneously : M ( x ) → ŷ = ( ŷ1 , ŷ2 , · · · , ŷn ) , where x is an input image and ŷi is the prediction for task Ti . Here we focus on the type of per-pixel level prediction tasks ( e.g. , semantic segmentation or depth estimation ) . We treat image classification as a special task , which provides global semantic description ( i.e. , scene labels ) of images and only requires image-level category annotation c. Therefore , the set of fully annotated real data is denoted as Sreal = { ( xj , y1j , y2j , · · · , ynj , cj ) } . Generative learning : Meanwhile , we aim to learn a generative model G that produces a set of synthesized data but with only corresponding image-level scene labels ( weak annotation ) : G ( c , z ) → x̃ , where z is a random input , and x̃ is a synthesized image . The scene label of x̃ is denoted as c̃ = c. We denote the set of synthesized images and their corresponding scene labels as S̃syn = { ( x̃k , c̃k ) } . Cooperation between discriminative and generative learning : Our objective is that the discriminative model M and the generative model G cooperate with each other to improve the performance on the multiple visual tasks T . During the whole process , the full model only gets access to the real fully-labeled data Sreal , then the generative network is trained to produce the synthesized set S̃syn . Finally , M effectively learns from both Sreal and S̃syn . Note that , unlike most of the existing work on image generation ( Brock et al. , 2018 ; Zhang et al. , 2019 ) , we do not focus on the visual realism of the synthesized images x̃ . Instead , we hope G to capture the underlying mechanism that benefits M .

Summary: The paper proposes a multi-task learning framework that incorporates a generative model, which is designed as an in-the-loop data augmenter that could improve the downstream tasks, especially in the low-data regime. The authors made strong emphasis on using a generative modeling as an effective form of data augmentation during training.  The major contributions, as perceived by the reviewer, are the following: 1. Showed that using an off-the-shelf image generator does not improve the down stream tasks.  2. Proposed a jointly learning framework, which includes an in-the-loop generator, a self-supervision block, a refinement block and an EM style optimization scheme that outperforms single-task and multi-task baselines. 3. Made ablation studies that shows the importance of different design choices.