Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-task network for pixel-wise estimation tasks such as semantic segmentation and depth estimation. The authors additionally introduced an GAN-based image generation network, a self-supervised network and a scene category classification network (which was called a refinement network in the paper). As an image generation network, any conditional GANs including SA-GAN and DCGAN can be used. As a self-supervised network, SimCLR was used in the proposed network. The refinement network classifies
a scene category from the outputs of multitask networks (i.e. estimated segmentation masks and estimated depth images).

In the experiments with two dataset, the proposed model, MGM, consistently outperformed the results of the baselines and ablated methods. 
In addition, additional experiments with six tasks and high resolution images (256x256) were performed and confirmed the effectiveness of the proposed method as well.
main_review: Pros)
The idea on introducing an image generation network, G,  into multi-task learning on pixel-wise estimation tasks seems novel. To use scene-class-conditionally synthesized images without GT annotations of the target multi-tasks, the authors introduced a refinement network which classifies a scene class. The scene class was known for a synthesized image, since it was generated with a class condition, which enabled training of the multi-task network,M, with generated images.  In addition, EM-style training was proposed to train the network effectively. Its effectiveness was successfully proved by the experiments. This is the biggest contribution of this paper.

Although adding a self-supervision network is not so novel, its effectiveness was shown by the experiments. 

The paper is well written, and the experiments including ones in the supplementary material are comprehensive and detailed. 
The pilot study is helpful to understand that naive usage of GAN for this task is not effective at all.

Cons)
The proposed method needs to scene labels for all the training images. That's why MGM cannot be applied to the CityScape dataset which consists of only street scene. This property narrows applicability of the proposed method.

Additional Comments)
To deal with CityScape, it will be needed to introduce a multi-label classifier into the refinement network, and use a multi-label-conditional 
GAN (probably using multiple-hot instead of one-hot vector). Does the authors think if this works ? In Semi Supervised Semantic Segmentation Using Generative Adversarial Network,ICCV2017, multi-label was used. 

MGM does not seem to be limited to multiple tasks. It can be applied to a single task such as only semantic segmentation. Did the authors try that ?

Showing the synthesized images on the supplementary material will be helpful for the readers.

summary_of_the_review: This paper has novelty which is introducing a generative model into multitask network, and the effectiveness was successfully supported by the comprehensive experiments.  The reviewer think this paper can be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on using generative models to improve multi-task learning. The authors propose a framework that combines discriminative multi-task model and generative model, with a refinement network that performs scene classification on top of the multi-task network predictions to guide the learning. The network uses weak labels and self-supervision for end-to-end training. Compared with single-task and multi-task learning baselines, both quantitative and qualitative results show improved performance.
main_review: The proposed framework that combines multi-task learning and generative modeling is interesting and inspiring. I also appreciate the extensive experiments (ablation studies and extensions) conducted by the authors.

However, I am not entirely convinced by the pilot study. The experiment is carried out on a single task, and the proposed framework is suitable for multi-task. The discovery of single-task scenarios may not be applicable to multi-task scenarios.  A more convincing comparison is to show that "using images synthesized by an off-the-shelf generative model may hurt the performance on multi-task learning". In addition, I think using Taskonomy's output as the ground truth to train with the synthesized image is not good enough. Although Taskonomy is the state-of-the-art method, its qualitative performance is still far from being regarded as a good ground truth. It is better to visualize the sample ground truth (Taskonomy's output) in the supplement to prove that the output of using Taskonomy is sufficient for training. In general, I think the pilot study is still very different from the proposed framework: it is conducted on a single task, there is no self-supervision/contrastive learning, it does not use weak labels, but uses the output of Taskonomy as the ground truth. I think the pilot study alone is not enough to support the claim.

I also don't fully believe in the benefits of introducing generative models in joint training. The role of generative models is like providing additional data augmentations. The advantage of training the generative model with the multi-task model is that it can better learn the synthetic images guided by the multi-task outputs and the downstream tasks. I think a stronger baseline is to train a separate generative model for each task on the same dataset for each experiment, and then fix the generative model weights, train a multi-task learning network with the generated model guided by the downstream tasks. Again, since I believe that the pilot study is not enough to be such a baseline, further comparisons are needed.
summary_of_the_review: Although I think the whole idea is interesting and worthy of discussion in the research community, I still don’t fully believe most of the claims: e.g. the pilot study that tries to support the claim "directly using images synthesized by an off-the-shelf generative model (self-attention GAN) may hurt the performance on the downstream task" and the performance differences between using generative models and a more complex data augmentation technique. Therefore, at this stage, I will vote for borderline reject. I will re-evaluate the score based on the authors' response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-task learning framework that incorporates a generative model, which is designed as an in-the-loop data augmenter that could improve the downstream tasks, especially in the low-data regime. The authors made strong emphasis on using a generative modeling as an effective form of data augmentation during training.

The major contributions, as perceived by the reviewer, are the following:
1. Showed that using an off-the-shelf image generator does not improve the down stream tasks. 
2. Proposed a jointly learning framework, which includes an in-the-loop generator, a self-supervision block, a refinement block and an EM style optimization scheme that outperforms single-task and multi-task baselines.
3. Made ablation studies that shows the importance of different design choices.

main_review: Strengths:
- A pilot experiment that shows off-the-shelf image generators is not suitable as data augmentation.

- The paper is abundant with experiments and most of the numerical results shows good indication that the proposed method is better than existing state-of-the-art, and ablations studies somewhat justifies the design choices of the method. 


Weakness:

Major concerns:

- Does it really help when in low data regime?

   I don't think the remark of being helpful in low data regime is convincing according to the experiments. On the NYU data set, which is ~1000 images for training, I think it is safe to say that even 100% of data is in this 'low data' regime. Therefore I don't really think the experiments on 25% or 50% of NYU is indicative of the model's performance if the original dataset is, for example, ~100k. 

  For the taskonomy results, ST/MT seems to have less degradation when the training data gets more and more limited. For example, ST on semantic segmentation drops from .111 to .119, yet the proposed method drops from .106 to .117. Does this mean ST is more robust to sparser data? Also, on normal prediction, it seems that ST/MT is better at 25% of data compared to 100%. Does it mean that they are better models in the low data regime?


- Too many components, hurts the generative model narrative; The generative modeling helps, but is it absolutely necessary? 
   
   The narrative of the paper focuses on the generative model that is jointly trained with the multitask model. But, it seems it only brings marginal improvements as shown in table 3 with MGM/G. The performance drop without the generative model is really just marginal. Yet, in table 4 and 9, it seems that the optimization scheme and the self-supervision block have more significant impact on the performance. This really makes one wonder why is the generative model necessary at all? 

- Are numbers indicative?
  
  I would refrain from stating "our results are quite close to GT". According to the qualitative results shown in the main text and in the supplementary, it seems that the prediction quality is quite low for all methods. This somewhat hurts the interpretation of the numbers, since they are more effective at telling which results is better instead of being 'less bad'. But this could also be the visualization issue, where the RGB colors are blended in as well. 

- inconsistent metrics

  I understand the authors are following the conventions for evaluation on different dataset, but it makes is hard to read the table when the metrics are different across the dataset. It would be more indicative if those are made consistent. I'm not aware of the specifics of evaluating on taskonomy, but it seems the NYU metrics are more universal. 

Minor concerns:

- visualization of the results.
All the visualization of depth/normal/semantic segmentations are overlaid with the original image, which is confusing and makes it hard to compare across methods and against GT. It makes more sense to show just the original result. Also the normal seems to have weird padding around the prediction, and sometimes there's some noise pattern around the boundary. Is that a bug? if those regions are included in metric calculation, then the numbers are meaning less as those might dominate depending on the form of the metric.


- The paper also has some minor issues in writing, but it seems those are not limiting the quality of the paper at the current state.


summary_of_the_review: I believe this paper is slightly below the the acceptance threshold.

The paper presents a novel joint frame work that includes a generative model for multi-task learning, and quantitative experiments show that the proposed method out-performs baseline methods. 
But, it is hard to interpret the results since:

1. The proposed methods include a generative modeling block, a self-supervision block and a refinement block. Through the ablation studies, it seems that the generative blocks, brings marginal benefits while other blocks such as self-supervision blocks, brings a larger margin for improvements. 

2. Authors claim that the proposed method is better at low-data regime, which is a less convincing remark to give according to the experiment results: The NYU dataset is extremely small, which makes data ablation experiments on NYU really hard to interpret. Does it really make sense to train on 25% of NYU, which is ~250 images, with a deep learning network? The performance drop of ST/MT on tiny taskonomy is less significant than the proposed method, does that mean they are better at low-data regime? 

3. It seems the author proposed a multitask learning framework that outperforms some baseline, but it hard to interpret why. 

Based on theses observation, I believe this paper is slightly below acceptance threshold.
review_title: null

Meta-review: The paper received borderline reviews. While the reviewers acknowledged good motivation, good number of experiments and good numeral results that demonstrated the proposed method outperforms the existing state of the art, there are shared concerns: the experimental setup is not really a "low data" regime, generative models jointly trained with the multi-task model only led to marginal improvements, and the prediction quality is quite low for all methods. In addition, it's unclear why the images generated by MGM have a lot of artifacts, and how the artifacts affect the performance. Overall, the reviewers were not convinced after the rebuttal.
Decision: Reject