Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Post-rebuttal:

I acknowledge reading the rebuttal as well as other reviewers comments. I'm satisfied with the rebuttal, I think that the authors have addressed many of my initial comments and I'm happy to increase the score of the paper to 7. If the paper gets accepted to the conference I would encourage the authors to include and expand a discussion about method limitations in the main body of the paper.
----------------------------------------------------------------
Pre-rebuttal:

The paper deals with semantic image synthesis and proposes a model that can generate images from segmentation masks. The main methodological contribution of the paper is a segmentation-based discriminator. By using such discriminator, the model is able to learn high quality image syntheses. The proposed pipeline is evaluated on multiple datasets (ADE20K, Cityscapes and COCO-stuff). The qualitative and quantitative results highlight good image quality of the proposed pipeline.


Pros:

Solid model evaluation including large number of ablations.
Simple and intuitive pipeline design that is shown to produce high quality image syntheses.
Well written paper and well structured appendix.


Cons:

Although the synthesized image quality is improved it seems to come at the expense of image diversity.
The terms used in the papers are sometimes confusing and could be improved.
Although the paper uses standard evaluation metrics for image generation (such as FID), their use for conditional image synthesis is is not fully justified.


Detailed review and comments:

Abstract:
-> "semantic image synthesis GAN" - I'm not convinced with this wording used in the paper. It mixes task (semantic image synthesis) with a type of generative model (GAN). This introduces some confusions in the text, e.g. "semantic image synthesis GAN models still greatly suffer from poor image quality when trained with only adversarial supervision" that might be a bit unclear since GANs are always trained with only adversarial supervision. I would recommend to  simply use "semantic image synthesis".

Introduction:
-> Conditional GANs references: Goodfellow et al 2014 is a good reference for unconditional GAN, not conditional GAN setup. Currently, this reference is linked with other references to conditional reconstruction.

-> Since this is an application oriented paper I think that the introduction would benefit from motivating a bit the task that is under study in this paper. e.g. why semantic image synthesis is of interest to ICLR community?

-> "Although the perceptual loss substantially improves the accuracy of previous methods, it comes with the computational overhead introduced by utilizing an extra network for training. Moreover, it usually dominates over the adversarial loss during training, introducing a bias towards ImageNet, which can have a negative impact on the diversity and quality of generated images, as we show in our experiments." - I'm not convinced if the diversity reduction when using the perceptual loss is due to the bias towards ImageNet. The reduction of diversity is rather an effect of perceptual loss overall. Could the authors comment on this?

Prior work references:
-> This reference (https://arxiv.org/pdf/2004.03590.pdf) seems relevant. Could the authors comment on this reference in the paper? Adding comparison to this work would make the paper even stronger.

Methodology:
-> I find the wording of "3D noise sampling" a bit unclear. Is it just a resized version of 1D tensor? Is there any importance on how the 3D noise sampling is structured? Could the authors clarify the idea of 3D noise sampling in the main body of the paper?

Experiments:
-> Could the authors comment why FID is a good metric for semantic image synthesis? Justifying the choice of metrics would make the evaluation section stronger.  From the discussion of the results, (diversity quality tradeoffs) it seems that the paper would benefit from reporting precision-recall like metrics for GANs (e.g. https://arxiv.org/abs/1807.09499, https://arxiv.org/abs/1905.10887, https://arxiv.org/abs/1904.06991). Adding additional metrics would further improve the paper.

-> Table 2. It is unclear to me what is the meaning of red color numbers. Could the authors add clarifications about it to the caption?

-> Table 2. It is bit surprising to see that the model trained only with adversarial loss has relatively low diversity. I would expect that the lack of perceptual loss would lead also to increased diversity in synthesized images. Could the authors comment on what is the limiting factor to increase sample diversity? Could the diversity be affected by the proportion of mixing (real vs. fake)  in LabelMix?

-> Would it be possible to add qualitative results for SPADE+? This model seems to have high diversity scores and it would be interesting to see model samples.

-> The paper lacks discussion on the limitations of the method. Could the authors comment on the potential limitations of the suggested approach?

review_title: Well executed paper, some clarifications are missing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors approach the problem of conditional image generation via generative adversarial networks. To this end, they propose an approach that utilizes only semantic segmentation annotations and adversarial loss. No perceptual loss is required. Their discriminator leverages semantic labels to improve the image generations. They evaluate their approach on a variety of datasets including ADE20K, COCO, and CityScapes. They demonstrate substantial quantitative and qualitative performance over baselines and perform an ablation analysis.

Pros:
1. The problem of mask conditioned image generation has immediate applications in computational photography and computer graphics.

2. The paper is well written and easy to understand.

3. Performance is impressive and experimental evaluation is thorough. The authors perform an ablation analysis.

4. The proposed techniques (Segment based discriminator, 3d noise vector) are useful insights into tuning GANs to perform well.

Cons:
1. Technical novelty is limited. The contributions ( segment discriminator and 3D noise) are useful engineering increments but not particularly large insights into the workings of image generation.

2. While the need for VGG based perceptual loss may not be needed here, it seems like the perceptual loss has just been shifted to the spatial realm. There is still a dependence on strong semantic constraints. Instead of using image class labels to train features, the per pixel class labels are used. 

In summary I believe the approach shows promise and important tweaks for image generation networks. Performance is good. However, the technical novelty is limited.

review_title: Good Improvement over Baselines, but Technical Novelty is Limited

final_score: rating: 7: Good paper, accept
review_content: # Post Rebuttal:

Thank you for the detailed rebuttal. The comment made about being able to use this in a semi-supervised setting is an exciting direction and I encourage the authors to pursue it on larger less-labeled datasets mentioned in the review in a future work/final submission. I am glad that removing VGG improved the results on COCO.

Ultimately, I am keeping my score at 7, accept.

# Pre-Rebuttal: 

Summary 
=============================================================================================================

This paper provides an improvement to the SPADE+ architecture that removes the need for a perceptual loss mechanism while also quantitatively improving the outputs with respect to FID/mmIOU. The paper achieves this while reducing the number of params and removing the ImageNet bias of the original perceptual VGG architecture. The authors evaluate their architecture on 3 datasets: ADE20K, Cityscapes and COCO-stuff. They do while using only adversarial supervision. They achieve this by proposing label mix to train the network by having the discriminator output class labels including an additional "fake" label in addition to a variety of other optimizations.

Overall, I vote for accepting this paper. It significantly improves on the SPADE+ architecture by removing a pre-trained component and reducing the number of params. The technique seems to improve upon prior work qualitatively, quantitatively, and performance-wise. 

Pros
==============================================================================================================
1. The paper is very well written.
1. The paper does provide a very thorough comparison with prior works.
1. This paper improves on the results of prior methods while reducing the # parameters and the need for imagenet pretraining.

Cons: 
==============================================================================================================
1. Pixel level semantic segmentations are difficult and expensive to generate. This limits the number of datasets that can be used for these work. In this sense, the data is highly supervised. 
1. That being said, I would love to the results on a more challenging dataset like LVIS and/or Open Images. For Open Images, it would be interesting to see if the performance scales with 350+ classes. For the former, it would be interesting to see how well it does on objects that appear very rarely. (I don't expect the results on LVIS to be good, but I would expect them to be better than SPADE+).
1. Architectural ablations only cover the first two datasets, any reasons why there are none for COCO-stuff?  One of the main claims of the paper, (that this technique does better without VGG), is not evaluated on COCO-stuff.
review_title: Excellent improvement of SPADE+. Would like to see ablations on COCO and potentially results on Open Images.

Meta-review: The paper received 3 reviews with positive ratings: 7,6,7. The reviewers appreciated overall quality of the manuscript, thoroughness of the evaluation, and practical importance of this work (mentioning though that the technical novelty is still not high). They also acknowledged impressive empirical performance. The authors provided detailed responses to each of the reviews separately, which seemed to have resolved the remaining concerns.
As a result, the final recommendation is to accept this work for presentation at ICLR as a poster.
Decision: Accept (Poster)