Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new paradigm for unconditional image synthesis with semantic layouts as the bottleneck. The presented approach is straightforward: we can first sample a semantic layout from a latent variable, and then perform image synthesis from this semantic layout. The proposed method is able to synthesize images that look more realistic than unsupervised image synthesis methods such as BigGAN.

Strengths: - The idea of having some predefined intermediate representation (such as semantic layout) can help improve unconditional image synthesis. 
- The approach works well on the datasets with semantic layouts, including Cityscapes and ADE20K datasets.
- The experiments are comprehensive and convincing.
- The proposed end-to-end fine-tuning can improve the visual results.

Weaknesses: 
- Apparently, there is a limitation of the proposed approach: it can only work on datasets with semantic layouts.
- This overall idea of having semantic layouts as a bottleneck is simple. Technically, semantic bottle synthesis and semantic image synthesis are based on well-known models (ProGAN+WGAN, SPADE). The technical contributions may appear limited, but I appreciate the overall idea for unconditional image synthesis.
- Why the perceptual evaluation of BigGAN in Table 3 for Cityscapes-5K is not available?
review_title: Review for SB-GAN

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper suggests a new approach for unconditional generation of complex scenes. The approach performs generation in two steps: first a semantic map is generated from noise using a conventional generator architecture, then the semantic image is turner into an RGB image by SPADE translator.

The paper has several strengths. First, the idea is clear and may make sense (though this has not been shown convincingly). Furthermore, the paper is well written and has detailed related work review (though some important papers are missed). The results are also interesting.

Despite the strenghts, I think that the paper may not be suitable for ICLR in the current form for the following reasons:

1) Novelty. The idea of two stage generation of complex images with GANs has been proposed before in a well-known paper [Wang & Gupta, Generative Image Modeling Using Style and Structure Adversarial Networks, ECCV16] . There, the image of normals rather than semantic segmentation served as an intermediate ("bottleneck") representation, otherwise the idea is very similar. It is likely that the normal map may be a better intermediate representation since it is continuous-valued and does not need to deal with discretization issues. Overall, a comparison and a proper positioning w.r.t. [Wang&Gupta] is needed.

2) Deficient comparisons with 1-step GANs. The authors for some reason chose Progressive GANs and BigGAN as the reference 1-stage GANs. This choice is totally unclear to me. StyleGAN v1 and v2 are improved versions of ProGAN and should have been tried instead. Given the use of SPADE (i.e. style based generator) in the authors' architecture, the comparison to StyleGAN would be all the more natural. I find it very likely that the result of StyleGAN can be very similar or better than the authors after similar amount of tuning, especially on Cityscapes 25K. I am therefore not convinced that the proposed idea is actually working.

3) [Minor] There is a published CVPR workshop paper with a very similar idea https://openaccess.thecvf.com/content_CVPRW_2020/html/w23/Volokitin_Decomposing_Image_Generation_Into_Layout_Prediction_and_Conditional_Synthesis_CVPRW_2020_paper.html . The results are worse and some important differences exist. However, it does undermine the novelty. Still a CVPR-workshop paper may be missed by the community, so I weigh this issue as minor.

4) The results are interesting, but they are not terrific. I wonder if the authors can scale their method to higher resolutions (which would be very useful for complex scenes), or if it would break down in some way. 

5) [Minor; suggestion not a criticism] There is an obvious use for what the authors are doing. The data they generate can be used to train semantic segmentation networks (essentially serving as dataset augmentation). I think having the evaluation of this aspect would be useful and would make the paper stronger.

6) [Minor] The phrase "In fact, due to the missing ordering relationships, generating smooth
segmentation maps cannot be enforced by smoothness among values of neighboring pixels." should be reformulated. A large body of work exist (generally associated with MRF/CRFs in computer vision; and also in statistical physics) on enforcing smoothness (in different senses) in discrete-valued maps, e.g. via Potts prior. 

To sum up, the idea is clear and well described, but the paper does not convince me that the idea is working (improves over StyleGAN; can synthesize high-res images) and in particular that semantic bottleneck is working better than other bottlenecks (e.g. proposed by [Wang&Gupta]).
review_title: Reasonable approach but validation is unconvincing

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work proposes a novel approach for unconditioned image synthesis of complex scenes by intelligently coupling two major tasks; unconditional label generation and label-conditioned image synthesis. To overcome the limitation of failing to generate high-fidelity complex scenes using current GAN-based approaches, this method proposes to divide this into two parts: unconditional segmentation map synthesis network and conditional segmentation-to-image synthesis network. The former is based on the ProGAN with some modifications in losses to deal with discrete semantic labels, and the latter leverages the existing method (Park et al. 2019) based on SPADE residual blocks. Experimental results demonstrate superior performance on the complex scene synthesis. Additionally, the latter part for segmentation-to-image synthesis task also outperforms the existing method (Park et al. 2019) thanks to joint end-to-end training with the former using ProGAN.

* Pros
1) Decomposing the complex scene synthesis into two sub-tasks (segmentation map generation and segmentation-to-image synthesis) looks novel, also validating outstanding performance over SOTA.
2) Segmentation-to-image synthesis is also boosted, thanks to joint end-to-end training with the unconditional segmentation map synthesis network.

* Cons
1) Though this paper is well-written, some parts need more details.
- In Section 3.1, it would be nice to explain how to generate semantic segmentation maps progressively by referring to Figure 1.
- Eq (3) uses loss functions from two sub-networks (semantic bottleneck synthesis network and semantic image synthesis network). 
Why did you not use L_D_SPD and L_G_SPD in (2) for training the whole network with a pair of real RGB images and real segmentation maps. 
It seems that L_1^VGG and L_1^Feat can also be used in (3).

- Semantic bottleneck synthesis in Section 3.1 needs more explanations, e.g., how to convert real segmentation maps into probability maps, using argmax and soft argmax in forward and backward passes.

review_title: This work introduces an interesting approach that intelligently weaves two networks for unconditional label generation and label-conditioned image synthesis. Experimental results support its superior performance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. Summary. This paper considers a two-stage procedure of generation complex scenes (cityscapes or living rooms, i.e. without a central object): Firstly, a noise vector is mapped to a discrete semantic map (Gumbel softmax and straight-through estimator are applied). Secondly, the obtained segmentation map is translated to an RGB image using the SPADE architecture. Both networks are pretrained separately and then finetuned in an end-to-end manner.

1. Decision. What I really like about this work is the attempt to dissect the scene generation into several steps. This may be of great interest in the case of datasets of limited size, as the authors showcase. What is more, this work tackles the problem of the generation of discrete semantic labels and has success.  However, the chosen intermediate representation is, to my mind, the main concern of the presented approach as ground-true semantic labels are not so easy to crowdsource. This makes the considered pipeline a bit limited in practical usage. A possible solution is to employ the off-the-shelf pretrained segmentation net (as in your Cityscapes-25k experiment): I believe the paper would be stronger with an additional experiment on more common datasets for scene generation (e.g., LSUN bedrooms), labeled with such a network. One more thing, I would recommend choosing a more recent and appealing baseline like StyleGAN instead of ProGAN. Summarizing, I tend to vote for accepting, though believe that the experiments could be more solid.

1. Questions
    1. Is the sampling step with the Gumbel-softmax trick crucial for the generation of semantic maps? Why the combination of simple softmax scores and straight-through estimator does not suffice?
    1. How exactly do you downscale the ground-true segmentation maps to feed the discriminator at coarse scales of progressive learning?
review_title: Need more evidence the semantic bottleneck is not a blocker for practical usage

Meta-review: The paper receives a mixed rating, with R3 rates the paper above the bar, R1 and R2 rates marginally above the bar, and R4 recommends rejection. The cited positive points include 1) decomposing image generation into first synthesizing segmentation masks and then converting segmentation masks to images, and 2) good results comparing to Progressive GAN and BigGAN. R4 raises several concerns, including the novelty concern and unconvincing experimental validation. After analyzing the papers, the reviews, and the rebuttal, the AC finds the arguments made by R4 more convincing. Decomposing image generation to a two-step approach has been illustrated in the prior work [Wang & Gupta ECCV 2016, Hong, Yang, Choi, Lee CVPR 2018]. The proposed method does not provide additional insights. The provided experimental results are not very convincing, either. As the proposed setting assuming the availability of segmentation masks, it is not surprising that it outperforms the unconditional baselines. Overall, the AC believes the paper does not have enough novelty to justify its acceptance and would recommend rejection of the paper.
Decision: Reject