Generative Modeling for Multitask Visual Learning
Authors: Zhipeng Bao
Yu-Xiong Wang
Martial Hebert
Venue: ICLR 2022 Submitted (2022)

Abstract: Generative modeling has recently shown great promise in computer vision, but it has mostly focused on synthesizing visually realistic images. In this paper, motivated by multi-task learning of shareable feature representations, we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. Correspondingly, we propose a general multi-task oriented generative modeling (MGM) framework, by coupling a discriminative multi-task network with a generative network. While it is challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios, our framework enables us to use synthesized images paired with only weak annotations (i.e., image-level scene labels) to facilitate multiple visual tasks. Experimental evaluation on challenging multi-task benchmarks, including NYUv2 and Taskonomy, demonstrates that our MGM framework improves the performance of all the tasks by large margins, especially in the low-data regimes, and our model consistently outperforms state-of-the-art multi-task approaches.

Under review as a conference paper at ICLR 2022
GENERATIVE MODELING FOR MULTI-TASK VISUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Generative modeling has recently shown great promise in computer vision, but it has mostly focused on synthesizing visually realistic images. In this paper, moti- vated by multi-task learning of shareable feature representations, we consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. Correspondingly, we propose a general multi-task oriented generative modeling (MGM) framework, by coupling a discriminative multi-task network with a generative network. While it challenging to synthesize both RGB images and pixel-level annotations in multi-task scenarios, our framework enables to use synthesized images paired with only weak annotations (iee. image-level scene labels) to facilitate multiple visual tasks. Experimental evaluation on chal- lenging multi-task benchmarks, including NYUv2 and Taskonomy, demonstrates that our MGM framework improves the performance of all the tasks by large mar- gins, especially in the low-data regimes, and our model consistently outperforms state-of-the thethe--tttttt art multi-task approaches.
INTRODUCTION
Seeing with the mind's eye creating internal images of objects and scenes not actually present to the senses, is perhaps one of the hallmarks in human cognition (Pelaprat & Cole, 2011), 1). For humans, this visual imagination integrates learning experience and facilitates learning by solving different problems (Egan, 2014; Pearson, 2019; Pelaprat & Cole, 2011; Egan, 1989). Inspired by such ability, there has been increasing interest in building generative models that can synthesize images (Goodfellow al., 2014). Yet, most of the effort has focused on generating visually realistic images (Brock et al., 2018; Zhang et al., 2019), which are still far from useful for machine perception tasks (Shmelkov et al., 2018; Borji, 2019; Wu et al., 2016). Even though recent work has started improving the "usefulness" of synthesized images, this line of investigation is often limited single specific task (Nguyen- Phuucc et al., 2018; Sitzmann et al., 2019; Zhu et al., 2018; Souly et al., 2017). Could we guide generative models to benefit multiple visual tasks?
While similar spirits of shared feature representations have been widely studied as multi-task learning or meta-learning (Finn et al., 2017; Zamir et al., 2018), here we take a different perspective- - learning shared generative model across various tasks (as illustrated in Figure 1). Leveraging multiple tasks allows us to capture the underlying image generation mechanism for more comprehensive object and scene understanding than being done within individual tasks. Taking simultaneous semantic segmentation, depth estimation, and surface normal prediction as example (Figure 1), successful generative modeling requires understanding not only the semantics but also the 3D geometric structure and physical property of the input image. Meanwhile, a learned generative model facilitates the flow of knowledge across tasks, so that they benefit one another. For instance, the synthesized images provide meaningful variations in existing images and could work as additional training data to build better task-specific models. These variations are especially critical when the data is limited.
This paper thus explores multi-task oriented generative modeling (MGM). by coupling discrim- inative multi-task network with a generative network. To make them cooperate with each other, a straightforward solution would be to synthesize both RGB images and corresponding pixel-level annotations (e.g., pixel-wise class labels for semantic segmentation and depth map for normal esti- mation). In the single task scenario, existing work trains a separate generative model to synthesize paired pixel-level labeled data (Sandfort et al., 2019; Choi et al., 2019) and produce an augmented set. However, these models are still highly daseddep,,,,,,, and extending them to multi-task scenarios becomes difficult. A natural question then is: Do we actually need to synthesize paired image and multi-annotation data to be useful for multi-task visual learning?

---

Under review as a conference paper at ICLR 2022
herral
Nerral
Naki
Nodd
(0her Voxal Tadko)
(Ml Niea tasls)
Dipch baeease
Dipck
(A) Trddioonal aask Banework
oriened modellig
Figure 1: Left:: Traditional multi-task learning framework (that learns shared feature representations) vs. Right: our proposed multi-task oriented generative modeling (that learns a shared generative model across various visual perception tasks)
Our MGM addresses this question by proposing a general framework that uses synthesized images paired with only weak annotations (i.e.. image-level scene labels) to facilitate multiple visual tasks. Our key insight is introduce auxiliary discriminative tasks that (i) only require image-level annotation or no annotation, and (ii) correlate with the original multiple tasks of interest. To this end, as additional components of the discriminative multi-task network, we introduce refinement network and self- seeeevuppeviioon network that satisfies these properties. Through joint training, the discriminative network explicitly guides the image synthesis process. The generative network also contributes to further refining the shared feature representation. Meanwhile, the synthesized images of the generative network are used as additional training data for the discriminative network.
In more detail, the refinement network performs scene classification on the basis of the multi- task network predictions, which requires only scene labels. The supervipevviiion network can be operationalized on both real and synthesized images without reliance on annotations. With these two modules, our MGM is able to learn from both (pixel-wise fully-annotated real images and synthesized (image-level) weakly labeled images. We instantiate MGM with the state-of-thh-ar art encoder-decoded based multi-task network (Zamir et al., 2018), self afffatteenton GAN (Zhang et al., 2019), and contrastive learning-baa based self-supervision network (Chen et al., 2020). Note that our framework is agnostic to the choice of these model components.
We evaluate our approach on standard multi-tasi benchmarks, including the NYUv2 (Nathan Sil- berman & Fergus, 2012) and Taskonomy (Zamir et al., 2018) datasets. Consistent with the previous works (Sun et al.. 2019; Standley et al., 2020), we focus on three tasks of great practical importance: semantic segmentation, depth estimation, and normal prediction. The evaluation shows that: (1) our MGM consistently outperforms state-of ofte--of-hhe-rt multi-task approaches by large margins, especially in the low-data regime. (2) With the increasing of generated samples, the performance of MGM consistently improves and it also almost reaches the performance upper-bound that trains with weakly annotated real images. (3) Our model can be extended more visual tasks.
2 PILOT STUDY
This pilot study provides initial experimentation, which shows the importance of proposed problem (multi-task oriented generative modeling) and motivates the development of our method. Specifically, we show that directly using images synthesized by an off-the-shee generative model that is trained with the realistic objective is not helpful for downstream pixel-level perception tasks.
Experimental Design: For ease of analysis, here we focus on a single task semantic segmentation, and use the LinyTTakonoomy dataset (Zamir et al., 2018). The dataset split and evaluation metric are the same as our main experiments (See Section for details). We train a self-attention GAN (Zhang et al., 2019) on Tiny- Tayy-TTskonmy and use it to generate the same number of synthesized images as the real images to augment the training set.
One remaining question is how generate pixel-level annotations for these synthesized images produced by the self-attention GAN. While prior work has explored synthesizing both images and their pixel-level annotations for some specific tasks (Sandfort al., 2019; Choi et al., 2019), these annotations are still not reliable. For ease of analysis, in this study, we factor out the effect of annotations and assume that we have an orucle annotator. We use the annotator from Taskonomy (Zamir al., 2018), which is state-of-the-ee large fully supervised semantic segmentation network. In fact, the ground-truth of semantic segmentation on Taskonomy is produced as the output of this network rather than labeled by humans. By doing so, we ensure that the annotations of the synthesized images are "accurate".
How to generate pixel-level annotations?
Comparisons: Single-Tan Task (ST) model is our baseline which follows the architecture of Taskonomy single task network. ST is trained on real images only. STc is the ST model trained on the augmented set. Table reports the results of these two models, and ST is worse than ST.
2

---

Under review as a conference paper at ICLR 2022
Latret Encodes
B
loage
Cookeel
Dos
leage
Self-saperrien Network
Sharl
Image
Lawa
*Dining (Scara Label)
Image Gencration Netwark:
Refincasent Network: R
Figure 2: Architecture of our proposed multi-task oriented generative modeling (MGM) framework. There are four main components in the framework: Multi-task network to address the target multiple pixel-level prediction tasks; self-supervision network to facilitate representation learning using images without any annotation; refinement network to perform scene classification using weak annotation; image generation network to synthesize useful images that benefit multiple tasks.
Model
ST STg
moss (L) 0.111
0.148
Table 1: Pilot experiment for semantic segmentation on the Tiny-Taskooomy dataset. Directly using images synthesized by off-the-shelf generative model (self-attention GAN) may hurt the performance on the downstream task. ST: single-task semantic segmentation model trained on real images only; STg: the same model trained on both real and synthesized images.
Does synthesized images help downstream tasks? From Table 1, the answer NO. Even though the images are synthesized by one the state-of-the-art generative models and are labeled by the oracle annotator, they still cannot benefit the downstream task. This is probably because these images are synthesized off the shelf without "knowing" the downstream task. Our key insight then that we need to explicitly use the downstream task objective to guide the image synthesis process. Moreover, here focused annotator difficult to obtain in practice, especially for multiple tasks. Also, existing work cannot synthesize paired images and pixel-level annotations for multiple tasks (Sandfort et al., 2019; Choi al., 2019). To overcome these challenges, what follows we demonstrate how to facilitate multiple visual tasks with synthesize images that only need image-level scene labels.
single task and assumed that had the oracle annotations. However, oracle
3 METHOD
We propose multi-task oriented generative modeling (MGM) to leverage generative networks for multi-task visual learning, as summarized in Figure 2. In this section, first formalize the novel problem setting MGM. Then, we explain the general framework and an instantiation the MGM model with state-of-the -ftth--rt multi-task learning and image generation approaches. Finally, we discuss the detailed training strategy for the framework.
3.1 PROBLEM SETTING
we aim to learn discriminative multi-tass model M that is able address all of these tasks simultaneously: M(x) + y = (Ø² 72 ... T where æ is an input image and T is prediction for task T. Here we focus on the type of per-pixel level prediction tasks (e.g.. semantic segmentation or depth special task, which provides global semantic description (i.e., scene labels) images and only requires image-level category annotation c. Therefore, the set of fully annotated real data is denoted as S = {(x,. y/. V² ... VV" c;)).
Multi-task discriminative learning: Given a set of n visual tasks T T2,,T2....
estimation). We treat image classification
Generative learning: Meanwhile, we aim to learn a generative model G that produces a set of synthesized data but with only corresponding image-level scene labels (weak annotation): G 2) - Σ, where is random input, and ž is synthesized image. The scene label of T is denoted as é c. denote the set of synthesized images and their corresponding scene labels as Sayn (( 7x)
Cooperation between discriminative and generative learning: Our objective is that the discrimi- native model M and the generative model G cooperate with each other to improve the performance

---

Under review as a conference paper at ICLR 2022
on the multiple visual tasks T. During the whole process, the full model only gets access to the real fully-labeled data Steat, then the generative network is trained to produce the synthesized set Seyn- Finally, M effectively learn from both Sreal and Sayn- Note that, unlike most of the existing work on image generation (Brock et al., 2018; Zhang et al., 2019), we do not focus on the visual realism of the synthesized images Instead, we hope G to capture the underlying mechanism that benefits M.
3.2 FRAMEWORK AND ARCHITECTURE
Figure 2 shows the architecture of our proposed MGM framework. It contains four components: the main multi-task discriminative network M, the image generation network G. the refinement network R. and the self-supervision network. By introducing the refinement network and the self-supervision network, the full model can leverage both fully-labeled real images and weakly-labeled synthesized images to facilitate the learning of latent feature representation. These two networks thus allow M and R to better cooperate with each other. Notice that our MGM ís a model-agnostee framework, and here we instantiate its components with state-of-the-ar art models. In the ablation study (Sec. 4.3) and the supplementary material, we show that our MGM works well with different choices of the model components.
Multi-task Network (M): The multi-task network aims to make predictions for multiple target tasks based on an input image. Consistent with the most recent work on multi-task learning, we instantiate an encoder-decoder based architecture (Zamir et al., 2018; Zhang et al., 2019; Sun et al., 2019). Considering the trade-off between model complexity and performance, we use a shared encoder E to extract features from input images, and individual decoders for each target task. We adopt ResNet-18 18 (He et al., 2016) for the encoder and symmetric transposed decoders following Zamir et al. (2018). For each task, we have its own loss function to update the corresponding decoder and the shared encoder.
Image Generation Network (G): The generative model G is a variant of generative adversarial networks (GANs). We include the generator in our framework, but this module also has a discriminator during its own training. G takes as input a latent vector and a category label c, and synthesizes an image belonging to category c. Considering the trade-off between performance and training cost, we instantiate G with self-attention generative adversarial network (SAGAN) (Zhang et al., 2019). We achieve conditional image generation by applying conditional batch normalization (CBN) layers (De Vries et al., 2017):
fie,wh - -E(f + Be:
CBN (fr,c,b,w Ne, Bc) %
(I)
/aa (a,r, +
where fi.c,h, is an extracted e-channel 2D feature for the i-th sample, and is small value avoid collapse. %e and 3e are two parameters to control the mean and variance of the normalization, which are learned by the model for each class. We use hinge loss for the adversarial training. Notice that the proposed framework is flexible with different generative models, and we also show the effectiveness of using DCGAN (Radford et al., 2015) in the supplementary.
Refinement Network (R): As one of our key contributions, we introduce the refinement network R further refine the shared representation using the global scene category labels. takes the predictions of the multi-task network as input and predicts the category label of the input image. Importantly, because R only requires category labels, it can be effortlessly operationalized on the "weakly-annotated synthesized images. Meanwhile, R also enforces the semantic consistency of the synthesized images with G.
We upply an algorithm inspired by Expectation Sopeetation-Maaximmn (EM) (Dempster al., 1977) to train the refinement network R. For the fully-annotated real images (a, y, c), we use the scene classification loss to update R and refine the encoder E in the multi-task network M. Then for the synthesized images (T. 2), since their multi-task predictions produced by M might not be reliable, we only refine E with R frozen using the scene classification loss. Through refining the share feature representation with the synthesized images, this process also provides implicit guidance to the image generation network.
More specifically, we model the whole multi-task network and refinement network as a joint proba- bility graph:
II P (vf 1;; o= P(c y 8'
Plz,v.c,0,00 = -P(x)
(2)
4

---

Under review as a conference paper at ICLR 2022
Guidance
JMM
Multi-ask
Embedking
Generator
Layer
Refinement
Figure 3: Joint training of the multi-task network and the image generation network. The multi-task network provides useful feature representation guide the image generation process, while the generation network refines the shared representation through back- propagation.
Algorithm 1:
The training procedure of MGM
Initialization:
maz_epoch: Maximum epoch for the training;
M: Multi-task Network, G: Image Generation Network;
E: Multi-task Encoder, R: the Refinement Network;
Exet:: Self-supervision Network Encoder;
N: minibatch size;
for epoch t- 1 max_epoch do
Split Steat into minibatches with size N: Sminli
for (x, y, c) Smin do
y = M(x)
Emalts (y. ý) - update M:
è= R(ÿ)
Lce(c, é) update R. E;
Sample 2N augmented images Taug
Nert-Xent(Tavu -> update E, Egelf;
Use LG to train G;
(7, 2) G c)
Lor(c, [()(2))
update
Sample 2N augmented images the synthesized data Zang Xxnt (Taug) > update E.
end end
where an input image, z is the vector of multi-task predictions, cs the scene label, is the vector of parameters of the multi-task network, and 0' is the vector of parameters of the refinement network. The parameters 9 and 6' are learned to maximize the joint probability. For data samples in Steal, maximize the joint probability and update 0' to train the refinement network.
g argmax P(Ck y; 0').
6'
For data samples in Sayu: we update the parameters of M (0) in an EM-like manner. During the step, we estimate the latent multi-task ground-truth by:
y argmax P (v) Tx. 0) P(ee y')
(4)
Then for the M step, we back crokpppreeaatee the error between yt and ÿ (the multi-task predictions) to the multi-task encoder.
0 argmax P (у² 0):
(5)
We use cross-entropy as the classification loss function.
Self-supervision Network: The self-supervision network facilitates the representation learning of the encoder E by performing self-supervised learning tasks on images without any annotation so that can be operationalized on both real and synthesized images. We modify SimCLR (Chen et al., 2020), one of the sfate-of-the-are approaches, our self sulf-supervision network.
This network contains an additional embedding network Egetr, working on the output of the multi-task encoder E, to obtain 1D latent feature of the input image: = Eger((((r)). Then, it performs contrastive learning with these latent vectors. Specifically, given minibatch of N images, this network first randomly samples two transformed views of each source image as augmented images (See supplementary material for the detailed transformations), resulting in 2N augmented images. For each augmented image, there is only one pair of positive augmented examples from the same source image, and other 2 1) negative pairs. Then the network jointly minimizes the distance of positive pairs and maximizes the distance of negative pairs in the latent space, through the normalized

---

Under review as a conference paper at ICLR 2022
Data Setting Models
100% Dara Seting
50%
Dura Serting
25% Dara Setting
ST
MT
MGM
ST
MT
MGM
MGM.
ST
MT
MGM
MGMM
NYU Y2
85-m100(1)
0.249
+0,00
00000
0.256
+0.005
0.264
+00000
0.230
0.237
=00006
+ =0005
0,251
0.258
+004
+0004
10004
000007
0.229
0.004
0.257
0.005
DE-MARSEC)
# 100019
0.218
0.708
0.021
100014
0.698
4 500077
0.837
= 0,018
0.819
1 0.011
0.734
+00000
0.225
+0017
0960
0.37
00015
0,844
0.011
0.827
2,009
SN-mAD()
0.06
0.008
1: 0.010
0:255
1 0.008
0.309
0.291
0.010
1: 0.009
0,270
0.006
1: 0.007
0.312
0,596
0.007
0.277
0.006
0.005
Tiny
SS-mL.oss()
+ 0.002
0.137
0.003
0.003
+ 0.003
0.138
0.002
+ 0.003
0.772
0.002
+ +0000
0.341
0.002
0.002
+ +.0002
Tack- onomy
(E-mLosss())
1.716
1.0006
1.584
0.008
1.472
0.006
1,768
0.007
1.595
0.009
1499
0.008
1.378
0,007
4 1.0.00
1,795
1892
0.008
1,585
0.009
1.380
0,008
ml
# 0.0003
0.555
= 0.003
0.153
0.145
0.002
0.157
0.002
0.156
0.002
+: -0.002
0.740
0.000
# 0.0002
0.152
0.002
0.148
0.003
0.702
0.002
Table 2: Main results (mean + std) on the NYUv2 and Tiny- Taskonomy datasets. SS: semantic
segmentation: DE: depth estimation; SN: surface normal prediction. t means higher is better; +
means lower is better. We use different metrics on the two datasets, following existing protocol. Our MGM consistently and significantly outperforms both single-task (ST) and multi-task (MT) basclines,
reaching the performance upper-bound of training with weakly annotated real images (MGM,).
Data Setting
100% Data Setting
50%
Data Settigg
25%
Data Setting
Models
MGM
MGM
MGM
MGM
MGM
MGM
MGM
MGM,j
MGM
SS-mlOU(t)
0.261
0.262
0.264
0.243
0.243
0.251
0.215
0.220
0.229
NYU
DF-mABSHC)
0.707
0.701
0.698
0.799
0.763
0.734
0.868
0.860
0.844
v2
SN-mAIXL)
0.262
0.259
0.255
0.287
0.281
0.273
0.292
0.286
0,277
Tiny
SS-mLoss(()
0,108
0.108
0.106
0.116
0.115
0.114
0.119
0.121
0.117
Task-
DE-mLoss(
1,491
1,488
1,472
1.527
1.523
1,499
1.636
1.616
1,585
anomy
SN-mLoss(()
0.151
0.151
0.145
0.153
0.152
0.147
0,154
0.152
0.148
Table 3: Comparison of our MGM model with its variants. /GM/G: without generating synthesized
images; MGM, without joint learning- Our MGM outperforms single-task and multi-task baselines even without synthesized data, showing its effectiveness as general multi-task learning framework. The model performance further improves with joint learning.
temperature--caled cross-entropy (NT-Xen()) loss (Chen et al., 2020):
exp (dis (14,11))
L. log 2N
(6)
1 (Aye exp (dis (14. /T)
-=
where ls,y is the NT-Xent loss for a positive pair of examples in the latent space (14, 1 [(ae]] E 0, an indicator function evaluating to if A i, and is a temperature parameter. dis (14, j45) is a distance function, and we use cosine distance following Chen et al. (2020). This loss is further paok-propeggtte to refine the multi-task encoder E. Notice that other types of self-supervised tasks are applicable as well. To demonstrate this, in Sec. 4.3 we also report the result with another task image reconstruction.
3.3 INTERACTION AMONG NETWORKS
Cooperation Through Joint Training: We propose a simple but effective joint training algorithm shown in Figure 3. The image generation network G takes the transferred feature representation of the multi-task encoder E, added with some Gaussian noise, as the latent input 2 to conduct conditional image generation. Hence, the generation network obtains additional, explicit guidance (i.e., extra effective features) from the multi-task network to facilitate the generation of "better images" images that may not look more realistic but are more useful for the multiple target tasks. Then, the generation error of G will be back- propagated to E to further refine the shared representation. This process can be also viewed as introducing image generation as additional task in the multi-task learning framework.
Training Procedure: We describe the procedure in Algorithm and further explain it in the supple- mentary material.
4 EXPERIMENTS
To evaluate our proposed MGM model and investigate the impact of each component, we conduct variety of experiments on two standard multi-task learning datasets. We also perform detailed analysis and ablation studies.
4.1 DATASETS AND COMPARED METHODS
Datasets: Following the work of Sun et al. (2019) and Standley et al. (2020), we mainly focus on three representative visual tasks in the main experiments: semantic segmentation (SS), surface normal prediction (SN), and depth estimation (DE). At the end of this section, we will show that our approach is scalable to an additional number of tasks. We evaluate all the models two widely-
6

---

Under review as a conference paper at ICLR 2022
Tol kango
35 and Timor
99% DE and Time
99% SN and Foree
99% SS and Finor
98% Diland Froo
59% S% and Time
ST
MT
MCM
MGM
(Das)
(Oan)
Guard
Tralh
Figure 4: Visualization and error comparison of the multi-task prediction outputs in the 50% data setting. The prediction results of MGM is quite close the ground ground-truth, significantly outperforming the tatte-of-the-art results.
benchmarked datasets:
NYUv2 (Nathan Silberman & Fergus, 2012; Eigen & Fergus, 2015) and Tiny-Taskonomy (Zamir et al., 2018). See Sec. B in the appendix for more details.
Compared Methods: We mainly focus on our comparison with two state-of-the-arr discriminative baselines: Single- Task (ST) model follows the architecture of Taskonomy single task network (Zamir et al., 2018), and address each task individually; Multi-Task (MT) model refers to the sub-ne network for the three tasks of interest in Standley et al. (2020). These two baselines can be viewed as using our multi-task network without the proposed refinement, self-suuervision, and generation networks. Note that our work is the first that introduces generative modeling for multi-task learning, and there is no existing baseline in this direction.
Our MGM is the full model trained with both fully-labeled real data and weakly-labeled synthesized data, which are produced by the generation network through joint training. In addition, to further validate the effectiveness of our MGM model, we consider its variant model MGM, that is trained with both fully and weakly labeled real data. MGM, is used to show the performance upper bound in the supervised learning scenario, where the synthesized images are replaced by the real images in the dataset. The resolution is set to 128 for all the experiments. For all the compared methods, we use ResNet-18 like architecture to build the encoder and use the standard decoder architecture of Taskonomy (Zamir et al., 2018).
Data Settings: We conduct experiments with three different data settings: (1) 100% data setting; (2) 50% data setting; and (3) 25% data setting. For each setting, we use 100%, 50%, or 25% of the entire labeled training set to train the model. For MGM, we add another 50% or 25% of weakly- labeled real data in the last two settings. For MGM, we include the same number of weakly-labeled synthesized data in all three settings.
For NYUv2, following the metrics in Eigen & Fergus (2015); Sun et al. (2019), we measure the mean Intersection- Overr- Union (mIOU) for the semantic segmentation task, the mean Absolute Error (mABSE) for the depth estimation task, and the mean Angular Distance (mAD) for the surface normal estimation task. For Tiny- Tan-Toskonomy, we follow the evaluation metrics of previous work (Zamir et al., 2018; Standley et al., 2020; Sun et al., 2019) and report the averaged loss values on the test set.
Evaluation Metrics:
Implementation Details: See appendix Sec. A for the training details and the sensitivity of the hyper-parameters.
4.2 MAIN RESULTS
Quantitative Results: We run all the models for 5 times and report the averaged results and the standard deviation on the two datasets in Table 2. From this table, we have the following key observations that support the effectiveness of our approach. (1) Existing discriminative multi-task learning approaches may not consistently benefit all the three individual tasks. However, our MGM consistently and significantly outperforms both the single-task and multi-task baselines across all the scenarios. (2) By using weakly-labeled synthesized data, the results of our model in the 50% data setting are even better than those of baselines in the 100% data setting. (3) More interesting,
the performance of our MGM is close to MGM,. which indicates that our synthesized images are comparably usefuul as real images for improving multiple visual perception tasks. (4) The performance gap between the two models especially minimal in the 25% labeled data setting, suggesting that our proposed MGM model is, in particular, helpful with limited data.
Qualitative Results: We also visualize the prediction results on the three tasks for ST, MT, and MGM in the 50% data setting in Figure 4. While obvious defects can be found for all the baselines, the results of our proposed method are quite close to the ground-truth.
How Does Generative Modeling Benefit Multi-tasks? To have a better understanding of how the generative modeling and joint learning mechanism benefit multi-task visual learning, we also
7

---

Under review as a conference paper at ICLR 2022
Model
SS-mIOU(1)
DE-mABSE (J)
SN-mAD ())
MGM/aell Jaelf
0,239
0.776
0.279
MGM /retine
0.254
0.808
0.290
MGM
0.241
0.768
0.285
MGM
0.251
0.734
0.273
Table 4: Ablation study. (I) MGM /self: /sett without self-supervision task; (2) MGM frefime: without
classification refinement network; and (3) MGMrecon with simple reconstruction task self- supervision. The two proposed components are complementary and both benefit the multiple tasks. The refinement network works better for surface normal; the self-supervision network works better for semantic segmentation. Their combination achieves the best.
consider two variants of our MGM model and evaluate their performance. MGM / is the MGM model trained with Steal only (without generative modeling), which shows the performance of our proposed multi-task learning framework in general (with the help from the auxiliary refinement and self self-supervision networks), and helps to understand the gain of leveraging generative modeling. MGM/3 /j is trained with synthesized images produced by pre-trained SAGAN withou/ the joint training mechanism. Table shows the results on the two datasets.
Combining the results of Tables and 2, we find: (1) MGM outperforms both ST and MT baseline even without generative modeling, indicating the benefit of the superviee task and the refine- ment network. (2) By introducing synthesized images that are trained separately, the multi-task performance slightly improves, which shows the effectiveness of involving generative modeling into multi-tasks, under the assistance of our refinement and self- superviiiiviii networks. (3) The joint learn- ing mechanism further improves the cooperation between generative modeling and discriminative learning, thus enabling the generative model to better facilitate multi-task visual learning.
4.3 ABLATION STUDY
For all the experiments in this section, models are trained in the 50% data setting, unless specifically mentioned.
Impact of Self-supervision Task and Refinement Network: Two important components of the proposed framework are the self-supervision task and the refinement network. We evaluate their impact individually in Table 4. MGM /self is the model trained without the self-supervision task; MGM frefine is the model without the refinement network; for MGM1econ we replace the SimCLR based self-supervision method with weaker reconstruction task, and use Mean Square Error as the loss function.
We could see that the refinement network works better for the surface normal task, and the self- supervision task works better for the semantic segmentation task; they are complementary to each other, and combining them generally achieves the best performance. In addition, the model could still gain some benefit even when we use some weak sulf-vupervi tasks like reconstruction, which indicates the generalizability and robustness of our MGM model.
Number of Synthesized Images vs. Real images: From the previous results, we have found that the synthesized images could benefit the target multi-tasks in a way similar to weakly labeled real images. To further investigate the impact of the number of synthesized images, we vary if from 25% to 125% during multi-task training on NYUv2 in the 25% real data setting, Figure 5 summarizes the result. First, we can see that the performance gap between MGM (without joint training) and MGM becomes larger for higher ratio of weakly labeled data, which indicates the importance of our joint learning mechanism. More importantly, while the real images are constrained in number due to the human collection effort, our generation network is able to synthesize unlimited amounts of images. This is demonstrated in the comparison between MGM, (with real images) and MGM: the performance of our MGM keeps improving with respect to the number of synthesized images, achieving results almost comparable to that of MGM, when MGM, uses all the available weakly labeled real images.
We also provide ablation studies on the generalizability, impact of parameters, and the resolution of images Sec. D in the appendix.
4.4 EXTENSION
Experiments with More Tasks: MGM also flexible and scalable with different tasks. In addition to the three tasks addressed in the main experiments, here we add three extra tasks: Edge Texture (ET), Reshading (Re), and Principal Curvature (PC), leading to six tasks in total. We evaluate the performance of all the compared models on Tiny-taskonomy in the 50% data setting, and report the mean test loss for all the tasks. The result is reported in Table 5. Again, our proposed method still outperforms state-of-the of-the-art baselines.
8

---

Under review as a conference paper at ICLR 2022
010
PGP
PGP
210
512
Mai
MC74
Figure 5: Performance change with different ratios of weakly labeled data. Joint learning significantly
improves the performance. The performance of MGM keeps increasing with the number of the weakly labeled synthesized images, achieving results almost comparable to that of MGM, trained
with all the available weakly labeled real images.
Model
SS (4)
DE (L)
SN (4)
ET (4)
Re (4)
PC (4)
ST 0.120
1.768
0.157
0.228
0.703
0.462
MT
0.1
1.747
0.169
0.241
0.704
0.436
MGM
0.108
1.715
0.152
0.201
0.699
0.417
Table 5: Mean Test losses for six tasks on Tiny-Taskonomy. Again, our MGM ouiperforms the baselines, indicating its flexibility, generability, and scalability.
5 RELATED WORK
Multi-task Learning and Task Relationship: Multi-task learning (MTL) aims to leverage informa-
tion coming from signals of related tasks so that each individual task can gain benefit (Doersch & Zisserman, 2017). Ruder (2017) identifies that most recent works use two clusters of strategies for MTL: hard parameter sharing techniques (Kokkinos, 2017; Doersch & Zisserman, 2017: Pentina & Lampert, 2017) and soft parameter sharing techniques (Misra et al., 2016; Sener & Koltun, 2018; Chen et al., 2018b). These strategies have achieved good performance for MTL with similar tasks. Researchers have also carefully studied the task relationships among different tasks to make the best cooperations among them. Taskonomy exploits the relationships among various visual tasks to benefit the transfer or multi-task learning (Zamir et al., 2018). Standley et al. (2020) considers task cooperation and competition, and proposes a method to assign tasks to a few neural networks to balance all of them. Some other following works also explores task relationships among different types of tasks (Sun et al., 2019; Zamir et al., 2020; Armeni et al., 2019; Pal & Balasubramanian 2019; Yeo et al., 2021; Wallace et al.. 2021). These works only consider MTL with discriminative tasks. In comparison, we first introduce generative modeling multi-task visual learning.
Generative Modeling for Visual Learning: Besides the initial goal of synthesizing realistic images, some recent work has explored the potential to leverage generative models to synthesize "usefull" images for other visual tasks (Shorten & Khoshgoftaar, 2019). The most straightforward way is to generate images and the corresponding annotations as data augmentation for the target visual task (Bao et al., 2021; Sandfort et al., 2019; Choi et al., 2019; Wang et al., 2018). Another strategy to leverage generative models through well-designed error feedback or adversarial training (Luc et al., 2016; CS Kumar et al., 2018; Mustikovela et al., 2020). There have been works that apply generative models for different visual tasks including classification (Zhan et al., 2018; Frid-Adar et al., 2018; Zhu et al., 2018), semantic segmentation (Luc al., 2016) and depth estimation (Pilzer et al., 2018; Alcotti et al., 2018). Among them, Souly et al. (2017) also propose a semi-supervise framework to leverage generative models for semantic segmentation through adversarial training. Different from these methods, MGM is applicable to various multiple visual tasks and different generative networks.
Reduced-SSuperission Methods: Recent works take advantage of weakly labeled data by assigning some self-created labels (e.g.co colorization, rotation, reconstruction) (Noroozi & Favaro, 2016; Noroozi et al., 2017: Chen et al., 2020; Dosovitskiy et al., 2014; Pathak et al., 2016). Similar self-supervised techniques have been proved useful for multi-task learning (Liu et al., 2008; Ren & Jae Lee, 2018; Doersch & Zisserman, 2017: Lee et al., 2019). Among these techniques, famous one is the SoppetationnMaxxiiieee
(EM) algorithm (Dempster et al., 1977; Papandreou et al., 2015), which leverages the information of weakly or unlabelled data by iteratively estimating and refining their labels. We adopt similar spirit and introduce the refinement network for MGM framework.
CONCLUSION
This paper proposes to introduce generative modeling for multi-task visual learning. The main chal- lenge is that current generative models almost cannot synthesize both RGB images and livel-level annotations in multi-task scenarios. We address this problem by proposing multi-task oriented gener- ative modeling (MGM) framework equipped with the self-supervision network and the refinement network, which enable us to take advantage synthesized images paired with image-level scene labels to facilitate multiple visual tasks. Experimental results indicate our MGM model consistently outperforms of-thoofthhe-rt multi-task approaches.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Filippo Aleotti, Fabio Tosi, Matteo Poggi, and Stefano Mattoccia, Generative adversarial networks for unsupervised monocular depth prediction. In ECCV Workshops, 2018. 9
Iro Armeni, Zhi- Yang He, Jun Young Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik. and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In ICCV, 2019.
Zhipeng Bao. Yu-Xiong Wang, and Martial Hebert. Bowtie networks: Generative modeling for joint
few-shot recognition and novel-view synthesis. ICLR, 2021. 9
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018. 1, 4,
Jun-Cheng Chen, Wei-An Lin, Jingxiao Zheng, and Rama Chellappa. real-time multi-task single shot face detector. In ICIP, 2018a. 18
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020. 2, 5, 6, 9,
14
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ICML, 2018b. 9
Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In CVPR Workshops, 2019. 18
Jachoon Choi, Taekyung Kim, and Changick Kim. Self-ensembling with gan- based data augmentation for domain adaptation in semantic segmentation. In ICCV, 2019. 2,
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 18
Arun CS Kumar, Suchendra Bhandarkar, and Mukta Prasad. Monocular depth prediction using generative adversarial networks. In CVPRW, 2018. 9
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NeurIPS, 2017. 4, 14
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 1977.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In ICCV, 2017. 9
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NeurIPS, 2014.
Kieran Egan. Memory, imagination, and learning: Connected by the story. Phi Delta Kappan, 1989.
Kieran Egan. Imagination in teaching and learning: The middle school years. University of Chicago Press, 2014.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. ICCV, 2015. 7, 15
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In ICML. 2017.
10

---

Under review as a conference paper at ICLR 2022
Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan-based synthetic medical image augmentation for increased enn performance in liver lesion classification. Neurocomputing, 2018. 9
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual organization and recognition of indoor scenes from rgb-d images. In CVPR, 2013. 15
Hu Han, Anil Jain, Fang Wang, Shiguang Shan, and Xilin Chen. Heterogeneous face attribute estimation: A deep multi-task learning approach. PAMI, 2017. 18
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016. 4, 14
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6985 2014. 14
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, 2017. 9
Wonhee Lee, Joonil Na, and Gunhee Kim. Multi-task self-supervised object detection via recycling of bounding box annotations. In CVPR, 2019. 9
Jianshu Li, Jian Zhao, Fang Zhao, Hao Liu, Jing Li, Shengmei Shen, Jiashi Feng, and Terence Sim. Robust face recognition with deep multi-view representation learning. International Conference on Multimedia, 2016. 18
Jianshu Li, Pan Zhou, Yunpeng Chen, Jian Zhao, Sujoy Roy, Yan Shuicheng, Jiashi Feng, and Terence Sim. Task relation networks. In WACV, 2019. 18
Hao Liu, Dong Li, JinZhang Peng, Qingjie Zhao, Lu Tian, and Yi Shan. Mina: Search multi-task networks for autonomous driving. In ACCV, 2020. 18
Qiuhua Liu, Xuejun Liao, and Lawrence Carin. Semi-supervised multitask learning. In NeurlPS, 2008. 9
Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adversarial networks. arXiv preprint arXiv:1611.08408 2016. 9
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
Siva Karthik Mustikovela, Varun Jampani, Shalini De Mello, Sifei Liu, Umar Iqbal, Carsten Rother, and Jan Kautz. Self-supervised viewpoint learning from image collections. In CVPR, 2020. 9
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 2, 7, 14, 15
Thu H Nguyen-Phuoe Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep
convolutional network for differentiable rendering from 3d shapes. In NeurIPS, 2018. 1
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016. 9
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017. 9
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer. In CVPR, 2019.
George Panagopoulos. Multi-task learning for commercial brain computer interfaces. In BIBE, 2017. 18
George Papandreou, Liang-Chieh Chen, Kevin Murphy, and Alan Yuille. Weakly-and semi- supervised learning of a deep convolutional network for semantic image segmentation. In ICCV, 2015. 9
11

---

Under review as a conference paper at ICLR 2022
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Joel Pearson. The human imagination: the cognitive neuroscience of visual mental imagery. Nature
Reviews Neuroscience, 2019. 1
Etienne Pelaprat and Michael Cole. "minding the gap": Imagination, creativity and human cognition. Integrative Psychological and Behavioral Science, 2011.
Anastasia Pentina and Christoph H Lampert. Multi-task learning with labeled and unlabeled tasks. In ICML, 2017. 9
Andrea Pilzer, Dan Xu, Mihai Puscas, Elisa Ricci, and Nieu Sebe. Unsupervised adversarial depth estimation using cycled generative networks. In 3DV, 2018. 9
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks. arXiv preprint arXiv:/571.0643/ 2015. 4, 14, 17
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using
synthetic imagery. In CVPR, 2018. 9, 15, 16
Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint 17Xiv:/706.05098, 2017. 9
Veit Sandfort, Ke Yan, Perry J Pickhardt, and Ronald M Summers. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in et segmentation tasks. Scientificc reports, 2019. 1, 2, 3,
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NeurIPS, 2018. 9
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In ECCV, 2018.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of Big Data, 2019. 9
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, 2019. 1
Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. In ICCV, 2017. 1, 19
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML,, 2020. 2, 6, 7, 9, 15, 16
Ximeng Sun, Rameswar Panda, and Rogerio Feris. Adashare: Learning what to share for efficient deep multi-task learning. arXiv preprint arXiv: 1911. 2019. 2, 4, 6, 7, 9, 14, 15, 16
Bram Wallace, Ziyang Wu. and Bharath Hariharan. Can we characterize tasks without labels or features? In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1245-1254, June 2021. 9
Dan Wang, Junjie Wen, Yuyong Wang, Xiangdong Huang, and Feng Pei. End-to-end self-driving using deep neural networks with multi-auxiliary tasks. Automotive Innovation, 2019. 18
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018. 9
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the quantitative analysis of decoder-based generative models. arXiv preprint MoMi::6611.02211 2016.
Teresa Yeo, Oguzhan Fatih Kar, and Amir Zamir. Robustness via cross-domain ensembles. arXiv preprint arXiv:2 103, 10919, 2021. 9
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 1, 1,2, 4, 7, 9, 14, 15, 16
12

---

Under review as a conference paper at ICLR 2022
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In CVPR, 2020. 9
Fangneng Zhan, Shijian Lu, and Chuhui Xue. Verisimilar image synthesis for accurate detection and recognition texts in scenes. ECCV, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019. 1, 4, 14, 15, 17, 19
Chenhong Zhou, Changxing Ding, Xinchao Wang, Zhentai Lu, and Dacheng Tao. One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. TIP, 2020. 18
Xinyue Zhu, Yifan Liu, Jiahong Li, Tao Wan, and Zengchang Qin. Emotion classification with data augmentation using generative adversarial networks. In PAKDD, 2018. 1,
13

---

Under review as a conference paper at ICLR 2022
We summarize the content of the appendix follows. Section A includes additional details of model architecture of our proposed multi-task oriented generative modeling (MGM) framework. Section B describes implementation details of MGM and also the dataset settings. Section C compares MGM with other state-of-te oft--o--thhtttt art multi-task models and further investigates in the few-shot regime. Section D provides additional ablation study on the training strategies for the refinement network, training with higher resolution images, generalizibility of the shared feature representation, and different types of image generation networks such as DCGAN. Section E shows more prediction visualizations. Finally, Section F discusses the potential of applying MGM to multi-tasi learning in other specific domains.
A ADDITIONAL DETAILS OF MODEL ARCHITECTURE
Multi-task Network: The multi-task network contains a shared encoder network and separate decoder networks for target tasks. We use a ResNer-18 (He et al., 2016) 6) as the encoder network, and its architecture follows the standard Pytorch implementation!. We only change the size of the features of each layer group from [64, 128, 256. 512] to [48, 96, 192, 360), so as to better address our GPU memory constraints. The size of the final feature representation is (360, 8, 8). For the decoder network, we use the same architecture as in Taskonomy (Zamir et al., 2018).
Sule-supe Network: We adopt SimCLR (Chen et al., 2020) our self- self-supervision network. SimCLR is one of the state-of-the-ard self-supervised learning approaches based on instance-level discrimination tasks. Following (Chen et al., 2020), we randomly apply 5 types of transformations on source image obtain an augmented image. These transformations are follows: (1) random resizing and cropping followed by resizing back to the original size: (2) random horizontal flipping with probability of 0.5; (3) random color jittering with probability of 0.5; (4) random transformation of RGB images to gray-scale images with probability of 0.2; (5) random Gaussian blur with probability of 0.5. The shape of the transformed latent feature, which is used to perform contrastive learning, is (128.).
Refinement Network: The refinement network takes as input the prediction results of the multi- task network. For each individual prediction, apply ResNet- 10 (He al., 2016) as the refinement encoder to extract the features. The feature dimension of each layer group of the refinement encoder is the same as the multi-task encoder network. Then we concatenate all the features together and apply a fully-connected layer with the hidden size of 128 to obtain the final scene class prediction.
Image Generation Network: We have instantiated the image generation network with two widely used generative networks: self-attention GAN (SAGAN)² (Zhang et al., 2019) the main paper and deep convolutional GAN (DCGAN)² (Radford et al., 2015) in Section D.5 of this document. For DCGAN, we change the original natch-norrmllization layers to conditional batch normalization layers (De Vries et al., 7) to allow conditional image generation. For the additional embedding layer used for joint learning, we use a simple global averaged pooling layer followed by dense layer. e.g.
B IMPLEMENTATION DETAILS
Data Processing: For the NYUv2 (Nathan Silberman & Fergus, 2012) dataset, following Sun et al. (2019) we resize and normalize the RGB images to (-1, 1), standardize the normal ground-truth to (0, 1), and do not normalize the depth. For the Taskonomy dataset (Zamir al., 2018), we follow the standard data normalization in Zamir et al. (2018).
Additional Implementation Details: We use Adam (Kingma & Ba, 2014) optimizer for all the models. The learning rates are set to 0.001 for the multi-task, self-supervisiin, and refinement networks, 0.0001 for the SAGAN generator, and 0.0004 for the SAGAN discriminator. The batch size is set to 32. We use cross-entropy loss for semantic segmentation and the scene classification task of the refinement network, and an 4 loss for surface normal and depth estimation.
resnet py http:///hhhhi..oeeeee//////////s
'https: thub. httpttccco//ttt/eee77/////////
14

---

Under review as a conference paper at ICLR 2022
Due to the different converge time for the different modules, a three-stage strategy to perform joint training: (1) We first train the multi-task network separately with fully labeled real data; (2) We then freeze the multi-task network, and train the image generation network and the embedding network separately; (3) We do joint training with the whole network using both fully labeled real data and weakly labeled sythesized data. During the pre-training process of SAGAN, we set the batch size to 128 to train a better model following Zhang et al. (2019). Then, for the joint training, we use batch size of 32 for all the sub-networks. Additionally, for the same minibatch of data, update the image generation network times iteratively during stage (3).
Training Procedure: We summarized the training procedure in Algorithm 1 in the main paper. Here we further explain the training procedure in more details. Given a minibatch data in Steal, we conduct the following training procedure.
1. For the input images x, we predict ý M(r), and then use the task-specific losses between y and ý to update the multi-task network M.
2. We predict the scene labels by è = Ríg). and update the refinement network and the multi-task encoding network E using the cross-entropy loss between c and é.
3. We randomly sample pairs of augmented images, process them with the self-supervision network, and then update the self-supervision network and the multi-task encoder E with the NT-Xent loss in Eqn. (6).
4. We train the image generation network G through adversarial training with (x, c). and mack-propagate the adversarial error and update E at the same time.
5. We sample another minibatch of synthesized data (3r, 2), and use these data update E by performing both the Ei-like algorithm described Section 3.2 (main paper) with R and the self- eeff-supervised learning as in step 3.
B.1 DATASET SETTING
We evaluate all the models on two widely-benchmarken datasets: NYUv2 (Nathan Silberman & Fergus, 2012; Eigen & Fergus, 2015) containing 1, 1,449 images with 40 types of objects (Gupta et al., 2013); Tin-Toasknnoyy which is the standard tiny split of the Taskonomy dataset (Zamir al., 2018).
For Tiny-Tasknommy dataset, since a certain amount of images for each category is required to train generative network, we keep the images of the top 35 scene categories on Tiny-Taskonomy, with each one consisting of more than 1,000 images. This resulting dataset contains 358,426 images in total. For NYUv2, we randomly select 1,049 images as the full training set and 200 images each as the validation/test set. For tin-TTakkooomy we randomly pick 80% of the whole as the full training set and 10% each as the validation/test set.
C ADDITIONAL COMPARISONS
C.I COMPARISON WITH OTHER MULTI-TASK MODELS
In the main paper, for fair comparison we focused on comparing our MGM model with internal models (e.g.. the multi-task model upon which MGM builds). To have a more comprehensive understanding of the performance of MGM, we also compare our method with some ofate-of-the-ar multi-task models. We focus the 25% data setting for the NYUv2 dataset, where collaboration between different tasks and the utilization of data is vitally important.
We include six models in this experiments. ST is the single-task model, where the encoder and the decoder are adopted from Zamir et al. (2018). MT is the multi-task model that uses a shared encoder as ST and separate decoders. ST and MT are the baselines compared in the main paper. Groppig uses the optimal network for the three tasks concluded from Standley et al. (2020). Another two well- performing multi-task models are Cross-stitch (Ren & Jae Lee, 2018)* and AdaShare (Sun et al., 2019). MGM is our proposed model. Table 6 summarizes the results. Notably, with a simple shared
Wee modify the network architecture following Sun al. (2019) make it work for the three tasks.
15

---

Under review as a conference paper at ICLR 2022
Method
SS-mIOU(")
DE-MABSB(U
SN-mAD())
ST (Zamir et al., 2018)
0.199
0.908
0.312
MT (7amir et al., 2018)
0.207
0.874
0.296
Groupigg (Standley al., 2020)
0.215
0.853
0.292
Cross Stitch (Ren Jae Lee, 2018)
0.205
0.917
0.296
AdaShare (Sun et al., 2019)
0.211
0.875
0.289
MGM
0.229
0.844
0.277
Table 6: Comparison with state-of-the-ee multi-task models in the 25% data setting on the NYUv2 dataset. Notably, with simple shared encoder architecture, our MGM model outperforms other state-of-the-art multi-task networks with more sophisticated architectures, which indicates the benefit of introducing generative modeling for multi-task learning. addition, our MGM is model-agnostic framework and could be incorporated with these different multi-task models for further improvement.
Method ST
SS-mIOU(1) DE-mABSE(L SN-mAD())
0.162
1.004
0.337
MT
0.185
0.930
0.311
MGM
0.197
0.911
0.291
Table Comparison the few-shot regime- in the 10% data setting on the NYUv2 dataset where around 3 images for each scene is used as the training set. Again, MGM significantly outperforms the compared models, showing the benefit of generative models the extremely low-data regime.
encoder architecture, our MGM model outperforms other state-of-the-art multi-task networks with more sophisticated architectures, which indicates the benefit of introducing generative modeling for multi-task learning. In addition, our MGM is model-agnostic framework and could be incorporated with these different multi-task models for further improvement.
C.2 EXPERIMENTS WITH FEW-SHOT SETTING
Since the learned generative model facilitates flow of knowledge across tasks and provides meaningful variations in existing images, it is especially beneficial in the low-data regime. So we designed 10% data setting for NYUv2 dataset, where around images for each scene is used as the training set. We also compare our MGM model with ST and MT. From Table 7, we can see that MGM outperforms the other compared models significantly, indicating the gain of generative models in the extremely low-data regime.
D ADDITIONAL ABLATION STUDY
D.1 IMPACT OF PARAMETERS
Introducing the refinement, self- self-supervision, and image generation networks also leads to more parameters. To validate that the performance improvements come from the novel design of our architecture rather than merely increasing the number of parameters, we provide two model variants additional baselines: ST, and MT, use ResNet-34 decoder networks. These two networks have a similar amount of parameters as MGM. The result in Table 8 show that simply increasing the number of parameters cannot significantly boost performance.
the encoder network and the corresponding
D.2 TRAINING STRATEGIES FOR REHNEMENT NETWORK
In the main paper, we proposed an Expectation- Maxiimum (EM) like algorithm coordinate the training between the refinement network and the main network. Here we compare our EM-Like Training (EML) with two alternative ways of the training procedure: Plain End-to-End Training (PEoE) backwards the refinement loss update the entire network directly: Loosely Separate Training (L.SeT) trains the refinement network and the main network separately, and only backwards the error to the encoder network when dealing with weakly labeled images. Table 9 shows the comparison results on the NYUv2 dataset the 50% data setting. From this table, we could find: (1) A naive end-to-end training strategy is not able to facilitate the cooperation between different networks and thus hurts the overall performance; (2) Loosely separate training enables the communication
16

---

Under review as a conference paper at ICLR 2022
Model
SS-mlOU (1)
DE-mABSE (J)
SN-mAD (1)
ST 0,230
0.837
0.309
MT
0.237
0.819
0.291
ST 0.232
0.841
0.304
MTi 0.236
0.804
0.288
Table 8: Impact of parameters. STi and MT;: baselines with larger number of parameters (with
deeper backbones). Simply increasing the number of parameters cannot significantly boost perfor- mance.
Method MT 0.237
SS-mIOU(1)
DE-mABSE(U) SN-mAD(L)
0.815
0.291
PEoE
0.211
0.896
0.301
LSeT 0.247
0.768
0.277
EML
0.251
0.734
0.273
Table 9; Results the NYUv2 dataset in the 50% data setting with different training strategies for the refinement network. 'MT": multi-task learning baseline; "PEoE': plain end-to-end training; 'LSeTT: loosely separate training; 'EML': EM-like training (proposed in the main paper). Our EML significantly outperforms alternative strategies to train the refinement network.
between the refinement network and the main network when needed and thus outperforms the baseline; (3) Our proposed EM-like training strategy further improves over the loosely separate training and achieves the best performance.
D.3 EXPERIMENTS HIGHER IMAGE RESOLUTION
In principle, the proposed framework is agnostic to the specific types of multi-task networks and image generation networks, thus flexible with image resolutions. The practical constraint lies in that it is still challenging and resource-consumin regh-resolution images (Zhang et al., 2019; Brock et al., 2018), although the deep multi-task models normally work better with high-resolution images. the main experiments, consistent with exiting image synthesis work (Zhang et al., 2019), we focused on the resolution of 128 x 128. Here we experiments with higher resolution, 256 x 256 NYUv2. The
for modern generative models to synthesize very
further made attempt to results of all the compared models in the 50% data setting are shown in Table 10. We could find that MGM still consistently outperforms the baselines, indicating the great robustness and flexibility of our proposed framework.
D.4 GENERALIZATION OF THE SHARED FEATURE REPRESENTATION
Intuitively, our MGM achieves state-of -ft---ofthhhteeee the-ah art performance by effectively learning a shared feature representation. We further show the generalization capability of this representation by designing the following experiment: for the multi-task model and our MGM model, first learn the shared feature space with the SS and DE tasks, and we then use that learned feature space to train new decoder for the SN task. We report the results NYUv2 in Table 11. Our MGM outperforms the multi-task model all the three data settings, which means that MGM indeed learns better and robust shared feature space.
D.5 EXPERIMENTAL RESULTS WITH DCGAN
The proposed MGM model is general framework and is flexible with different choices of model components. Here we show flexibility for the image generation network: We replace SAGAN with DCGAN (Radford et al., 2015), and Table 12 shows the result on the NYUv2 dataset in the 50% data setting. Similar to the results in the main paper, MGM equipped with DCGAN ('MGM- DCGAN') consistently improves the performance on all the tasks, which indicates the robustness of the proposed MGM framework. In addition, we find that 'MGM-SAGA outperforms 'MGM- DCGAN', suggesting that powerful image generation network leads to better performance.
17

---

Under review as a conference paper at ICLR 2022
Model
SS-mIOU (1)
DE-mABSE (L)
SN-mAD (L)
ST
0.239
0.849
0.282
MT
0.244
0.834
0.313
MGM
0.257
0.819
0.275
Table 10: Experiments with 256 image resolution on the NYUv2 dataset. Our MGM still consistently outperforms the compared baselines, showing the great robustness and flexibility of the proposed framework.
Model MT
mAD-100%(L)
mAD-50%(L)
mAD-25%(L)
0.291
0.310
0.323
MGM
0.280
0.298
0.305
Table 11: Results for the SN task with pre-trained feature representations by SS and tasks. MGM consistently outperforms multi-task (MT), indicating that MGM learns more effective and generalizable feature representation.
E MORE VISUALIZATIONS
In Figure of the main paper, we visualized the prediction results. Here we provide more visual- izations of the multi-task predictions for MGM and the compared baselines in Figure 6. Our MGM model significantly outperforms both ST and MT baselines.
F DISCUSSION AND FUTURE WORK
In the main paper, we focused on applying generative networks to general multi-task learning. On the other hand, generative models can also be exploited to address multiple tasks in specific domains, such as human face modeling (Li et al., 2019; 2016; Han al., 2017; Chen et al., 201 8a), autonomous driving (Liu et al., 2020; Wang et al., 2019; Chennupati et al., 2019), and bio-medical processing (Zhou et al., 2020; Panagopoulos, 2017). We leave such exploration of leveraging our MGM model these domains as future work.
G ADDITIONAL EXPERIMENTAL EVALUATIONS
G.1 EXPERIMENTAL EVALUATION ON CITYSCAPE SUBSET
In this section, demonstrate that MGM work with datasets when image-level labels available. In an alternative may, the proposed MGM frame model can work with object labels as well since the generative network and refinement network can naturally work with multi-hot labels the refinement network can work with multi-label classifier, and the generative network can be multi-label-conditione GAN.
We conducted semantic segmentation on subset of CityScape (Cordts et al., 2016) dataset, the Zurich street scene, We focused on the semantic segmentation task, which is representative task on CityScape. We use the 30 standard multi-hot CityScape semantic object labels for the generative model and also the refinement networks. We use 80% of the data for training and 20 % for testing and compare the performance of data with random multi-hot labels using MGM. The results are shown in Table 13. MGM still outperforms the baseline ST model, indicating the robustness and generalizability of the model with multi-hot object labels.
ST and MGM for this experiment. We generate the same amount
G.2 EXPERIMENTS WITH EXTREME Low DATA AT TASKONOMY
We noticed that for Tin--Taskonomy dataset, 25% data setting is still far from low-data regime. To further explore the effectiveness with model with low-data, we further conduct experiment with a subset of Tiny-Taskonomy dataset. We randomly select 3 nodes (allensville, benevolence, and coffeen) from Tiny-Te Tan-TTakonomy dataset to build dataset with 17,404 images-aree 5% data setting compared with the full Tiny-Taskonma dataset. We then conduct experiments with ST,
18

---

Under review as a conference paper at ICLR 2022
Method
SS-mIOU(1)
DE-MABSE())
SN-mAD())
ST
0.233
0.835
0.309
MT
0.237
0.815
0.291
MGM-SAGAN
0.251
0.734
0.273
MGM-DCGAN 0.245
0.750
0.285
Table 12: Results on the NYUv2 dataset in the 50% data setting with different image generation networks. 'MGM-SAGAN: MGM equipped with SAGAN (presented in the main paper); 'MGM- DCGAN": MGM equipped with DCGAN. Both 'MGM-SAGAN' and 'MGM-DCGAN" consistently improve the performance on all the tasks and outperform single-task (ST) and multi-task (MT) baselines. This shows the robustness and flexibility of the proposed MGM framework. In addition, "MGM-SAGAN' outperforms "MGM-DCGAN, suggesting that a more powerful image generation network leads to better performance.
Method ST
SS-mIOU(T)
0.57
MGM
0.64
Table 13: Results on the CityScape Subset. MGM still outperforms the baseline ST model, indicating the robustness and generalizability of the model with multi-hot object labels.
MT and MGM for this subset. All the other experimental settings keep the same as the main paper. Table 14 shows the comparable results. Combining the results in Table 2, we can find that MGM consistently outperforms ST and MT and is robust and especially helpful in low-data regime.
G.3 ABLATION WITH SINGLE TASKS
MGM is general framework that can be applied to both single tasks and multiple tasks. In the main submission, we mainly focused on the more challenging multi-task scenario. In this subsection, we conduct experiments with single tasks. Here we add three baselines applying MGM to the single tasks (SS, DE, SN) named MGM-SS, MGM-DE and MGM-SN in the NYUv2 50% data setting. We further provide an additional baseline by using the equivalently sampled data from the above three model as the augmented data, but not jointly train the generative network, named MGM-Combine The results of these models shown in Table 15. We have the following observations: (1) The proposed MGM framework can consistently benefit each single tasks though without leveraging shared features from multiple tasks. (2) Compared with the full MGM model, the performance drops when only using single tasks jointly train with the generative model. (3) When using the individually optimized generative models, the performance is slight better than MGM,j but still could not reach MGM, indicating the importance of our joint training mechanism.
H VISUALIZATIONS OF GENERATED IMAGES
We visualize the generated images for the Taskooomy dataset of SAGAN (Zhang al., 2019) and MGM in Figure 7. From Figure 7, we observe that (1) the conventional SAGAN, trained with the realistic objective and without the guidance of the downstream tasks, produces phato-realistic images; (2) the visual quality of the generated images by MGM becomes degraded, where SAGAN is jointly trained with the multi-task learning objective and under the guidance of the downstream tasks. Interestingly, a similar phenomenon has been observed in (Souly et al., 2017), where generative model is used to facilitate the semantic segmentation task. We hypothesize this is because those synthesized images that are useful for improving downstream tasks might not be necessarily photo- realistic. While the images synthesized by MGM are not visually realistic, they may contain some crucial discriminative information that can be leveraged for addressing the downstream tasks for example, the synthesized images may contain some unseen patterns from the real images, which increases the diversity of the training data. In addition, the difference of the synthesized images between SAGAN and MGM can also partially explain the result in the pilot study the images synthesized off the shelf are quite different from the desired images for multi -aak learning, and thus they are not effective in facilitating downstream tasks.
19

---

Under review as a conference paper at ICLR 2022
ST-SS
MT-SS
MGM-SS
GT-SS
ST-DE
MT-DE
MGM-DE
GT-DE
ST-SN
MT-SN
MGM-SN
GT-SN
Figure 6: More visualizations of the multi-task predictions for MGM and the compared baselines. SS: semantic segmentation task; DE: depth estimation task; SN: surface normal prediction task; ST: single-task model; MT: multi-task model: MGM: multi-task oriented generative modeling (our proposed model); GT: ground-trudt The prediction results of our MGM model are much closer to the ground-trun truth and significantly outperform the state-of-the-ae results.
I VISUALIZATIONS OF THE SEMANTIC ANNOTATIONS
We also visualize two samples and the corresponding semantic segmentation annotations we used for the pilot study in Figure 8. The visualization shows that the annotation is accurate and using Taskonomy is sufficient for training.
20

---

Under review as a conference paper at ICLR 2022
Model ST
SS (4)
DE
SN SN)
0.137
1.836
0.161
MT 0.156
1.807
0.162
MGM
0.125
1.670
0.153
Table 14: Comparison with extreme low data in Taskonomy. In this data setting, MGM significantly outperforms both ST and MT, indicating that MGM is robust and especially helpful low-data regime.
Model
SS-mIOU (1)
DE-MABSE (1)
SN-mAD (1)
0.230
0.837
0.309
MGM-DE 0.244
MGM-SS
0.752
MGM-SN
0.277
MGM-Combine MGM
0.249
0.747
0.277
0.251
0.734
0.273
Table 15: Ablation with MGM for single tasks and stronger baseline with the learned information from the three individual tasks but without jointly training. The experiments are conducted on NYUv2 50% data setting. MGM-SS, MGM-DE, MGM-SN: variantal MGM model for single tasks. MGM-Combine: MGM variant trained with augmented images generated by the above three models. The proposed MGM framework can consistently benefit each single tasks and MGM-combine cannot reach the performance of MGM, indicating the importance of joint training mechanism.
SAGAN
MGM
Figure 7: Generated images SAGAN and MGM for the dataset. After jointly training, the images are not visually realistic, but they are helpful to improve the downstream task performance.
RGB
Segmentation
RGB
Segmentation
Figure 8: Paired RGB and semantic segmentation labels. This paired data indicates that the oracle annotator used in the pilot study is effective.
21


Reviews:
[{"review_id": "ueU_sjuOOki", "reviewer": "Reviewer_Ytpa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed framework that combines multi-task learning and generative modeling is interesting and inspiring. I also appreciate the extensive experiments (ablation studies and extensions) conducted by the authors.\n\nHowever, I am not entirely convinced by the pilot study. The experiment is carried out on a single task, and the proposed framework is suitable for multi-task. The discovery of single-task scenarios may not be applicable to multi-task scenarios.  A more convincing comparison is to show that \"using images synthesized by an off-the-shelf generative model may hurt the performance on multi-task learning\". In addition, I think using Taskonomy's output as the ground truth to train with the synthesized image is not good enough. Although Taskonomy is the state-of-the-art method, its qualitative performance is still far from being regarded as a good ground truth. It is better to visualize the sample ground truth (Taskonomy's output) in the supplement to prove that the output of using Taskonomy is sufficient for training. In general, I think the pilot study is still very different from the proposed framework: it is conducted on a single task, there is no self-supervision/contrastive learning, it does not use weak labels, but uses the output of Taskonomy as the ground truth. I think the pilot study alone is not enough to support the claim.\n\nI also don't fully believe in the benefits of introducing generative models in joint training. The role of generative models is like providing additional data augmentations. The advantage of training the generative model with the multi-task model is that it can better learn the synthetic images guided by the multi-task outputs and the downstream tasks. I think a stronger baseline is to train a separate generative model for each task on the same dataset for each experiment, and then fix the generative model weights, train a multi-task learning network with the generated model guided by the downstream tasks. Again, since I believe that the pilot study is not enough to be such a baseline, further comparisons are needed.", "summary_of_the_paper": "This paper focuses on using generative models to improve multi-task learning. The authors propose a framework that combines discriminative multi-task model and generative model, with a refinement network that performs scene classification on top of the multi-task network predictions to guide the learning. The network uses weak labels and self-supervision for end-to-end training. Compared with single-task and multi-task learning baselines, both quantitative and qualitative results show improved performance.", "summary_of_the_review": "Although I think the whole idea is interesting and worthy of discussion in the research community, I still don’t fully believe most of the claims: e.g. the pilot study that tries to support the claim \"directly using images synthesized by an off-the-shelf generative model (self-attention GAN) may hurt the performance on the downstream task\" and the performance differences between using generative models and a more complex data augmentation technique. Therefore, at this stage, I will vote for borderline reject. I will re-evaluate the score based on the authors' response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NvUxHjxtLp", "reviewer": "Reviewer_X2gh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A pilot experiment that shows off-the-shelf image generators is not suitable as data augmentation.\n\n- The paper is abundant with experiments and most of the numerical results shows good indication that the proposed method is better than existing state-of-the-art, and ablations studies somewhat justifies the design choices of the method. \n\n\nWeakness:\n\nMajor concerns:\n\n- Does it really help when in low data regime?\n\n   I don't think the remark of being helpful in low data regime is convincing according to the experiments. On the NYU data set, which is ~1000 images for training, I think it is safe to say that even 100% of data is in this 'low data' regime. Therefore I don't really think the experiments on 25% or 50% of NYU is indicative of the model's performance if the original dataset is, for example, ~100k. \n\n  For the taskonomy results, ST/MT seems to have less degradation when the training data gets more and more limited. For example, ST on semantic segmentation drops from .111 to .119, yet the proposed method drops from .106 to .117. Does this mean ST is more robust to sparser data? Also, on normal prediction, it seems that ST/MT is better at 25% of data compared to 100%. Does it mean that they are better models in the low data regime?\n\n\n- Too many components, hurts the generative model narrative; The generative modeling helps, but is it absolutely necessary? \n   \n   The narrative of the paper focuses on the generative model that is jointly trained with the multitask model. But, it seems it only brings marginal improvements as shown in table 3 with MGM/G. The performance drop without the generative model is really just marginal. Yet, in table 4 and 9, it seems that the optimization scheme and the self-supervision block have more significant impact on the performance. This really makes one wonder why is the generative model necessary at all? \n\n- Are numbers indicative?\n  \n  I would refrain from stating \"our results are quite close to GT\". According to the qualitative results shown in the main text and in the supplementary, it seems that the prediction quality is quite low for all methods. This somewhat hurts the interpretation of the numbers, since they are more effective at telling which results is better instead of being 'less bad'. But this could also be the visualization issue, where the RGB colors are blended in as well. \n\n- inconsistent metrics\n\n  I understand the authors are following the conventions for evaluation on different dataset, but it makes is hard to read the table when the metrics are different across the dataset. It would be more indicative if those are made consistent. I'm not aware of the specifics of evaluating on taskonomy, but it seems the NYU metrics are more universal. \n\nMinor concerns:\n\n- visualization of the results.\nAll the visualization of depth/normal/semantic segmentations are overlaid with the original image, which is confusing and makes it hard to compare across methods and against GT. It makes more sense to show just the original result. Also the normal seems to have weird padding around the prediction, and sometimes there's some noise pattern around the boundary. Is that a bug? if those regions are included in metric calculation, then the numbers are meaning less as those might dominate depending on the form of the metric.\n\n\n- The paper also has some minor issues in writing, but it seems those are not limiting the quality of the paper at the current state.\n\n", "summary_of_the_paper": "The paper proposes a multi-task learning framework that incorporates a generative model, which is designed as an in-the-loop data augmenter that could improve the downstream tasks, especially in the low-data regime. The authors made strong emphasis on using a generative modeling as an effective form of data augmentation during training.\n\nThe major contributions, as perceived by the reviewer, are the following:\n1. Showed that using an off-the-shelf image generator does not improve the down stream tasks. \n2. Proposed a jointly learning framework, which includes an in-the-loop generator, a self-supervision block, a refinement block and an EM style optimization scheme that outperforms single-task and multi-task baselines.\n3. Made ablation studies that shows the importance of different design choices.\n", "summary_of_the_review": "I believe this paper is slightly below the the acceptance threshold.\n\nThe paper presents a novel joint frame work that includes a generative model for multi-task learning, and quantitative experiments show that the proposed method out-performs baseline methods. \nBut, it is hard to interpret the results since:\n\n1. The proposed methods include a generative modeling block, a self-supervision block and a refinement block. Through the ablation studies, it seems that the generative blocks, brings marginal benefits while other blocks such as self-supervision blocks, brings a larger margin for improvements. \n\n2. Authors claim that the proposed method is better at low-data regime, which is a less convincing remark to give according to the experiment results: The NYU dataset is extremely small, which makes data ablation experiments on NYU really hard to interpret. Does it really make sense to train on 25% of NYU, which is ~250 images, with a deep learning network? The performance drop of ST/MT on tiny taskonomy is less significant than the proposed method, does that mean they are better at low-data regime? \n\n3. It seems the author proposed a multitask learning framework that outperforms some baseline, but it hard to interpret why. \n\nBased on theses observation, I believe this paper is slightly below acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9-0Z-SzqPbW", "reviewer": "Reviewer_zrdP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros)\nThe idea on introducing an image generation network, G,  into multi-task learning on pixel-wise estimation tasks seems novel. To use scene-class-conditionally synthesized images without GT annotations of the target multi-tasks, the authors introduced a refinement network which classifies a scene class. The scene class was known for a synthesized image, since it was generated with a class condition, which enabled training of the multi-task network,M, with generated images.  In addition, EM-style training was proposed to train the network effectively. Its effectiveness was successfully proved by the experiments. This is the biggest contribution of this paper.\n\nAlthough adding a self-supervision network is not so novel, its effectiveness was shown by the experiments. \n\nThe paper is well written, and the experiments including ones in the supplementary material are comprehensive and detailed. \nThe pilot study is helpful to understand that naive usage of GAN for this task is not effective at all.\n\nCons)\nThe proposed method needs to scene labels for all the training images. That's why MGM cannot be applied to the CityScape dataset which consists of only street scene. This property narrows applicability of the proposed method.\n\nAdditional Comments)\nTo deal with CityScape, it will be needed to introduce a multi-label classifier into the refinement network, and use a multi-label-conditional \nGAN (probably using multiple-hot instead of one-hot vector). Does the authors think if this works ? In Semi Supervised Semantic Segmentation Using Generative Adversarial Network,ICCV2017, multi-label was used. \n\nMGM does not seem to be limited to multiple tasks. It can be applied to a single task such as only semantic segmentation. Did the authors try that ?\n\nShowing the synthesized images on the supplementary material will be helpful for the readers.\n", "summary_of_the_paper": "This paper proposes a multi-task network for pixel-wise estimation tasks such as semantic segmentation and depth estimation. The authors additionally introduced an GAN-based image generation network, a self-supervised network and a scene category classification network (which was called a refinement network in the paper). As an image generation network, any conditional GANs including SA-GAN and DCGAN can be used. As a self-supervised network, SimCLR was used in the proposed network. The refinement network classifies\na scene category from the outputs of multitask networks (i.e. estimated segmentation masks and estimated depth images).\n\nIn the experiments with two dataset, the proposed model, MGM, consistently outperformed the results of the baselines and ablated methods. \nIn addition, additional experiments with six tasks and high resolution images (256x256) were performed and confirmed the effectiveness of the proposed method as well.", "summary_of_the_review": "This paper has novelty which is introducing a generative model into multitask network, and the effectiveness was successfully supported by the comprehensive experiments.  The reviewer think this paper can be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received borderline reviews. While the reviewers acknowledged good motivation, good number of experiments and good numeral results that demonstrated the proposed method outperforms the existing state of the art, there are shared concerns: the experimental setup is not really a "low data" regime, generative models jointly trained with the multi-task model only led to marginal improvements, and the prediction quality is quite low for all methods. In addition, it's unclear why the images generated by MGM have a lot of artifacts, and how the artifacts affect the performance. Overall, the reviewers were not convinced after the rebuttal.
Decision: Reject