Unconditional Synthesis of Complex Scenes Using a Semantic Bottleneck
Authors: Samaneh Azadi
Michael Tschannen
Eric Tzeng
Sylvain Gelly
Trevor Darrell
Mario Lucic
Venue:  (2021)
Keywords: Unconditional Image Synthesis
Complex Scene
GAN
Semantic Bottleneck

Abstract: Coupling the high-fidelity generation capabilities of label-conditional image synthesis methods with the flexibility of unconditional generative models, we propose a semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them to learn the scene structure through an unconditional progressive segmentation generation network. During inference, our model first synthesizes a realistic segmentation layout from scratch, then synthesizes a realistic scene conditioned on that layout through a conditional segmentation-to-image synthesis network. When trained end-to-end, the resulting model outperforms state-of-the-art generative models in unsupervised image synthesis on two challenging domains in terms of the Frechet Inception Distance and perceptual evaluations. Moreover, we demonstrate that the end-to-end training significantly improves the segmentation-to-image synthesis sub-network, which results in superior performance over the state-of-the-art when conditioning on real segmentation layouts.

Under review as a conference paper at ICLR 2021
UNCONDITIONAL SYNTHESIS OF COMPLEX SCENES USING A SEMANTIC BOTTLENECK
Anonymous authors Paper under double-blind review
ABSTRACT
Coupling the high-fidelity generation capabilities of label-conditional image syn- thesis methods with the ficxibility of unconditional generative models, we propose semantic bottleneck GAN model for unconditional synthesis of complex scenes. We assume pixel-wise segmentation labels are available during training and use them learn the scene structure through an unconditional progressive segmenta- tion generation network. During inference, our model first synthesizes a realistic segmentation layout from seratch, then synthesizes realistic scene conditioned synthesis network. When trained end-to-end, the resulting model outperforms titte-of-thearr gen- erative models in unsupervised image synthesis on two challenging domains in terms of the Fréchet Inception Distance and perceptual evaluations. Moreover, we demonstrate that the end-to-end training significantly improves the segmentation- to-image synthesis sub-network, which results in superior performance over the fate-offthe--al when conditioning real segmentation layouts.
on that layout through a conditional segmentation-to-imal
INTRODUCTION
Significant strides have been made on generative models for image synthesis, with variery of meth- ods based on Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) achieving state- of-the-art performance. At lower resolutions or in specialized domains, GAN-based methods are able to synthesize samples which are near-indisee sear-iddisinguishabble from real samples (Brock et al., 2019). However, generating complex, high-resolution scenes from scratch remains a challenging problem, as shown in Figure 1-(a) and (b). As image resolution and complexity increase, the coherence of synthesized images decreases - samples lack consistent local or global structures.
Stochastic decoder- based models, such as conditional GANs, some of these issues. In particular, both Pix2PixHI (Wang et al., 2018) and SPADE (Park et al., 2019) are able synthesize high-quality scenes using strong conditioning mechanism based semantic segmentation labels during the scene generation process. Global structure encoded in the segmentation layout of the scene is what allows these models focus primarily on generating convincing local content consistent with that structure.
recently proposed to alleviate
key practical drawback of such conditional models that they require full segmentation layouts as input. Thus, unlike unconditional generative approaches which synthesize images from randomly sampled noise, these models are limited to generating images from a set of scenes that is prescribed in advance, typically either through segmentation labels from an existing dataset, or scenes that are hand-crafted by experts.
Contributions
To overcome these limitations, we propose new model, the Semantic Bottleneck GAN (SB-GAN), which couples high-fidelity generation capabilities of label-conditional models with the flexibility of unconditional image generation. This in turn enables our model to synthesize an unlimited number of novel complex scenes, while still maintaining high-fidelity output charac- teristic of image-conditional models.
Our SB-GAN first unconditionally generates pixel-wise semantic label map of scene (i.e. for each spatial location it outputs a class label), and then generates a realistic scene image by condi- tioning on that semantic map, Figure 1-(d). By factorizing the task into these steps, are able

---

Under review as a conference paper at ICLR 2021
460
Sogmensttin Layout Synthests
Nom-sompiex syrthessee by RigGAN (from imagabat)
image Synthesis
2584512
complex cemple synthesized by BigDAN (frm Cityscapes)
(6) complex sample synthestzed (from Chyscapes)
Samantic Bottlenack CAN
unconditiona complex
Figure (a) Examples non-complex images from ImageNet synthesized by the state-of-the- art BigGAN model (Brock et al., 2019). Although these samples look decent, the complex scenes synthesized by BigGAN (e.g., from the Cityscapes dataset) are blurry and defective in local structure (e.g., cars are blended together) (b). Zoom in for more detail. (c) A complex scene synthesized by our model respects both local and global structural integrity of the scene. (d) Schematic of our unconditional Semantic Bottleneck GAN. We progressively train the adversarial segmentation synthesis network to generate realistic segmentation maps from scratch, then synthesize a photo- realistic image using a conditional image synthesis network. End-to-end coupling of these two components results in state-of-the-art unconditional synthesis of complex scenes.
to separately tackle the problems of producing convincing segmentation layouts (i.e. useful global structure) and filling these layouts with convincing appearances (i.e. local structure). When trained end-to-end, the model yields samples which have a coherent global structure well fine local details, e.g., Figure I-(c). Empirical evaluation shows that our Semantic Bottleneck GAN achieves new state-of-the-en on two complex datasets with relatively small number of training images, Cityscapes and ADE-Indoor, as measured both by the Fréchet Inception Distance (FID) and by per- ceptual evaluations. Additionally, we observe that the conditional component of our SB-GAN jointly trained with segmentation layout synthesis significantly improves the state-of-the-- arrt semantic image synthesis network (Park et al., 2019), resulting in higher-quality outputs when conditioning ground truth segmentation layouts.
geamentation-to-maag synthesis
Key Challenges While both unconditional generation and image-to-ime to-image translation are well- explored learning problems, fully unconditional generation of the segmentation maps is notori- ously hard task: (i) Semantic categories do not respect any ordering relationships and the network therefore required capture the intricate relationship between segmentation classes, their shapes, and their spatial dependencies. (ii) As opposed to RGB values, semantic categories are discrete, hence con-diifeereniible which poses challenge for end-to-end training (Sec. 3.2) (iii) Naively combining state-of-the-art unconditional generation and image-to-image translation models leads to poor performance. However, by carefully designing an additional discriminator component and corresponding training protocol, we not only manage to improve the performance of the end-to-end model, but also the performance of each component separately (Sec. 3.3).
We emphasize that despite these challenges approach scales to 256 x 256 resolution and 95 semantic categories, whereas existing state-of-the-are GAN models directly generating RGB images at that resolution already suffer from considerable instability (Sec. 4).
2 RELATED WORK
Generative Adversarial Networks (GANs) GANs (Goodfellow et al., 2014) are powerful class of generative models successfully applied to various image synthesis tasks such as image style trans- fer (Isola et al., 2017: Zhu et al., 2017), unsupervised representation learning (Chen et al., 2016; Pathak et al., 2016; Radford al., 2016), image super-resolution (Ledig al., 2017: Dong et al., 2016), and text-to-image synthesis (Zhang et al., 2017: ; Xu et al., 2018: Qiao et al., 2019b). Training GANs is notoriously hard and recent efforts focused Gupta, 2016; Karras et al., 2017; Zhang et al., 2019; Chen et al., 2019a), loss functions (Arjovsky al., 2017), regularization (Gulrajani al., 2017; Miyato et al., 2018 8), large-scale training (Brock
improving neural architectures (Wang &
2

---

Under review as a conference paper at ICLR 2021
Systhesi Network
Symihesta Network
Disriiiintor Image
Rancom rolee
"Wh Iorea?
Discrimiiator "Whim image is mos concidennt 20g1 rorictiiionn
Figure 2: Schematic of Semantic Bottleneck GAN. Starting from random noise, synthesize segmentation layout and use discriminator to bias the segmentation synthesis network towards re- alistic looking segmentation layouts. The generated layout is then provided as input to conditional image symthesis network to synthesize the final image. A second discriminator is used to bias the conditional image synthesis network towards realistic images paired with real segmentation layouts. Finally, a third unconditional discriminator is used to bias the conditional image synthesis network towards generating images that match the real image distribution.
et al., 2019), self-supervision (Chen et al., 2019b), and sampling (Brock et al., 2019; Azadi et al., 2019a). Improving the performance of GANs by disentangling structure and style has been stud- ied by Wang & Gupta (2016) where structure is represented by a surface normal map and style is the texture mapped onto the structure. Another compelling approach which enables generation of high-resolution images is based on progressive training: a model is trained to first synthesize lower- resolution images (e.g. 8), then the resolution is gradually increased until the desired resolution is achieved (Karras et al., 2017). Recently, Brock et al. (2019) showed that GANs significantly ben- efit from large-scale training, both terms of model size and batch size. We note that these models able synthesize high-quality images in settings where objects are very prominent and centrally placed or follow some well- well-defined structure, as the corresponding distribution is casier to capture. In contrast, when the scenes more complex and the amount of data is limited, the task becomes extremely challenging for these state-of-the-aa models. We aim to improve the performance in the context of complex scenes and small number of training examples by disentangling the image generation problem into learning the structure represented by semantic layouts and filling in the RGB details using semantic image synthesis model. A similar idea was proposed by a concurrent work (Volokitin et al., 2020) with substantial differences in the model and results.
GANs on discrete domains GANs for discrete domains have been investigated in several works (Yu al., 2017b; Lin et al., 2017: Bojchevski et al., 2018; Lu et al., 2018). Training in this domain is even more challenging as the samples from discrete distributions are not differen- tiable with respect the network parameters. This problem be somewhat alleviated by using the Gumbel-softmax distribution, which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function (Kusner & Hernández-Lobato 2016). We will show how to apply a similar principle to learn the distribution of discrete segmentation masks.
Conditional image synthesis In conditional image synthesis one aims to generate images by con- ditioning
input which be provided in the form of image (Isola et al., 2017: Zhu al., 2017; Azadi et al., 2018: 8; 2019b; Liu et al., 2017), a text phrase (Reed et al., 2016; Zhang et al., 2017: Qiao et al., 2019a; Ashual & Wolf, 2019; Hong et al., 2018), scene graph (Johnson al., 2018; Ashual & Wolf, 2019), a class label, or a semantic layout (Odena et al., 2017; Chen & Koltun,
2017; Wang et al., 2018; Park al., 2019). These conditional GAN methods learn mapping that translates samples from the source distribution into samples from the target domain.
The text-to-image synthesis models proposed in (Hong et al., 2018: Li al., 2019) decompose the synthesis task into multiple steps. As illustrated in the Appendix, given the text description, semantic layout is constructed by generating object bounding boxes and refining each box by esti- mating object shapes. Then, an image is synthesized conditionally on the generated semantic layout from the first step. Our work shares the same high-level idea of decomposing the image generation problem into the semantic layout synthesis and the conditional semantic-lyyouttoo-mnaag synthesis. However, we note that the above approaches, as opposed to ours, are conditional and require su- pervision in the form of textual descriptions. Secondly, they sequential in nature and synthesize masks of a few different objects (e.g. person, elephant), but not a fully fine-grann mnn-grained semantic map (e.g. missing sky, grass, etc.) In stark contrast, our approach unconditionally synthesizes the fial
3

---

Under review as a conference paper at ICLR 2021
semantic layout of the entire scene from noise input in end-to-ent network design. Due the above distinctions, their segmentation synthesis models differ significantly from ours in terms of architecture and design shown in Figure 6 in the Appendix.
SEMANTIC BOTTLENECK GAN (SB-GAN)
We propose an unconditional Semantic Bottleneck GAN architecture learn the distribution of complex scenes. To tackle the problems of learning both the global layout and the local structure, divide this synthesis problem into two parts: an unconditional segmentation map synthesis net- work and a conditional segmenntaiion-to-maage synthesis model. Our first network is designed to coarsely learn the scene distribution by synthesizing semantic layouts. It generates per-pixel seman- tic categories following the progressive GAN model architecture (ProGAN) (Karras et al., 2017). This fully unconditional generation of the segmentation maps is novel, very challenging, and care- ful design is crucial, as described in Section 3.1. The second network populates the synthesized semantic layouts with texture by predicting RGB pixel values using Spatially-Adaptic Normaliza- tion (SPADE), following the architecture of the state-of-the-arr semantic synthesis network in (Park et al., 2019). We assume the ground truth segmentation masks are available for all or part of the target scene dataset. In the following sections, we will first discuss semantic bottleneck synthe- sis pipeline and summarize the SPADE network for image synthesis. We will then couple these two networks in end-to-end design which we refer as Semantic Bottleneck GAN (SB-GAN).
3.1 SEMANTIC BOTTLENECK SYNTHESIS
Our goal is to learn a (coarse) estimate of the scene distribution from samples corresponding to real segmentation maps with K semantic categories. Starting from random noise, we generate a tensor Y E [1, KJ which represents a per-pixel segmentation class, with H and W indicating the height and width, respectively, of the generated map and N the batch size. In practice, we progressively train from a low to a high resolution using the ProGAN architecture (Karras et al., 2017) coupled with the Improved WGAN loss function (Gulrajani et al., 2017) on the ground truth discrete-valued segmentation maps, illustrated in Figure 1-(d). Similar to ProGAN, to increase the spatial resolution of the generated segmentation maps during training, we incrementally add layers to the generator and the discriminator. In contrast to ProGAN, in which the generator outputs continuous RGB values, we predict per- per-pixel discrete semantic class labels. This task is extremely challenging as it requires the network to capture the intricate relationship between segmentation classes and their spatial dependencies. To this end, we apply the numbel-softmas trick (Jang et al., 2017; Maddison et al., 2016) coupled with through estimator (Jang al., 2017), described in detail below.
We synthesize segmentation layouts by first generating per-pixel probability scores of belonging to each of the K semantic classes and then sampling a semantic class per pixel. The per-pixel probability scores are computed by applying a softmax function to the last layer of the generator (i.e. logits) which results in probability maps púj E [0, 1]K, with Σ PM Σk=1
for each spatial location (i,j) E [[I, HI] x [1, W]. To sample semantic class from this multinomial distribution, would ideally apply the following well-known procedure at each spatial location: (1) sample & i.i.d. samples, Gg. from the standard Gumbel distribution, (2) add these samples each logit, and (3) take the index the maximal value. This reparametrization indeed allows for an efficient forward but is not differentiable. Nevertheless, the max be replaced with the softmax function and the quality of the approximation can be controlled by varying the temperature hyperparameter T the smaller the 7, the closer the approximation is the categorical distribution (Jang et al., 2017):
$ expi PM + Gx)/T} 1 (log P.J + G.)/T}
(1)
Similar to the real samples, the synthesized samples fed to the GAN discriminator should still con- tain discrete category labels. As result, for the forward pass, compute arg max, Sy. while for the backward pass, we use the soft predicted scores S₃ directly, a strategy known as straight- estimation (Jang al., 2017).
4

---

Under review as a conference paper at ICLR 2021
3.2 SEMANTIC IMAGE SYNTHESIS
Our second sub-network converts the synthesized semantic layouts into photo-realistic images using spatially-adaptis normalization (Park et al., 2019). The segmentation masks are employed to spread the semantic information throughout the generator by modulating the activations with a spatially adaptive learned transformation. We follow the same generator and discriminator architectures and loss functions used in (Park et al.. 2019), where the generator contains a series of SPADE residual blocks with upsampling layers. The loss functions to train SPADE are summarized as:
D.Do
- B (mnn(1, -1 + DSPD (v, x]]] B, [m][mi(0 -1 SPp (y, (spp(y)))
L.Gsro - F.y PSeD (y: (seD + X T, + A2
where GSPD. DSPD stand for the SPADE generator and discriminator, and T, and T. represent the VGG and discriminator feature matching L loss functions, respectively (Park al., 2019; Wang et al., 2018). We pre-train this network using pairs of real RGB images, and their corresponding real segmentation masks, y, from the target scene data set.
In the next section, we will describe how to employ the synthesized segmentation masks in an end- to-end manner to improve the performance of both the semantic bottleneck and the semantic image synthesis sub-networks.
3.3 END-TO-END FRAMEWORK
After training semantic bottleneck synthesis model to synthesize segmentation masks and the se- mantic image synthesis model to stochastically map segmentations to photo-realistic images, adversarially fine-tune the parameters of both networks in an end-to-end approach by introducing an unconditional discriminator network on top of the SPADE generator (see Figure 2).
This second discriminator, D2, has the same architecture as the SPADE discriminator, but is de- signed to distinguish between real RGB images and the fake ones generated from the synthesized semantic layouts. Unlike the SPADE conditional GAN loss, which examines pairs of input segmen- tations and output images, (y, æ) in equation 2, the GAN loss on D22, L D2 is unconditional and only compares real images to synthesized ones, as shown in equation 3:
LDz = - E, [min(0, -1 (2())) - E. [min(0, -1 D2(G(=)))
(3)
LG = - E. (D2(G=)))) + LGggo + ALGGs G(=) = GSPD (GsP(G)s(=))
where GSR represents the semantic bottleneck synthesis generator, and Lass is the improved WGAN loss to pretrain Gsn described in Section 1. In contrast to the conditional discriminator in SPADE, which enforces consistency between the input semantic map and the output image, D2 is primarily concerned with the overall quality of the final output. The hyper parameter À determines the ratio between the two generators during funing. The parameters of both generators, GSB and GSPD, as well as the corresponding diseriminators, Dsn and DsPD, updated in this end-to-end fine-tuning.
We illustrate our final end-to-end network in Figure 2. Jointly fine-tuning the two networks in an end-to-end fashion allows the two networks to reinforce each other, leading to improved perfor- mance. The gradients with respect to RGB images synthesized by SPADE are back-propagated to the segmentation synthesis model, thereby encouraging it to synthesize segmentation layouts that lead to higher quality final images. Hence, SPADE plays the role of a loss function for synthesiz- ing segmentations. but in the RGB space, hence providing a goal that was absent from the initial training. Similarly, fine-tuning SPADE with synthesized segmentations allows it to adapt to more diverse set of scene layouts, which improves the quality of generated samples.
4 EXPERIMENTS AND RESULTS
We evaluate the performance of the proposed approach on two datasets containing images with complex scenes, where the ground truth segmentation masks are available during training (possibly only for subset of the images). We also study the role of the two network components, semantic bottleneck and semantic image synthesis, on the final result. We compare the performance of SB- GAN against the of-te-oftth-aan art BigGAN model (Brock et al., 2019) as well as ProGAN (Karras et al.. 2017) baseline that has been trained on the RGB images directly. We evaluate our method using Fréchet Inception Distance (FID) as well as a perceptual evaluation.
5

---

Under review as a conference paper at ICLR 2021
SB-GAN
Figure 3: Images synthesized on Cityscapes-5K, Best viewed on screen; zoom in for more detail. Although both models capture the general scene layout, SB-GAN (Ist generates more convine- ing objects, e.g. buildings and cars.
SB-GAN
PoGAN
BigGAN
Figure 4: Images synthesized on Cityscapes-25K. Best viewed on screen; zoom in for more detail.
Images synthesized by BigGAN (3rd row) blurry and sometimes defective in local structures.
Datasets We study the performance of our model on the Cityscapes and ADE-indoor datasets as the two domains with complex scene images.
Cityscapes-55 (Cordts al., 2016) contains street scene images in German cities with training and validation set sizes of 3,000 and 500 images, respectively. Ground truth segmentation masks with 33 semantic classes are available for all images in this dataset.
Cityscapes-2551 (Cordts et al., 2016) contains street scene images in German cities with train- ing and validation set sizes of 23,000 and 500 images, respectively with 19 semantic classes. Cityscapes-5K is subset of this dataset, providing 3,000 images in the training set here as well as the entire validation set. Fine ground truth annotations are only provided for this subset, with the remaining 20,000 training images containing only coarse annotations. We extract the corre- sponding fine annotations for the rest of training images using the state-of-the-art segmentation model (Yu et al., 2017a) trained on the training annotated samples from Cityscapes-5KK
ADB-Indoor is subset of the ADE20K dataset (Zhou al., 2017) containing 4,377 challenging training images from indoor scenes and 433 validation images with 95 semantic categories.
Evaluation We use the Fréchet Inception Distance (FID) (Heusel et al., 2017) as well as a percep- tual evaluation of the quality of the generated samples. To compute FID, the real data and generated samples are embedded in a specific layer of pre-trained Inception network. Then, a multivariate Gaussian is fit to the data, and the distance is computed FID(r,g) ||m My 1 Tr(E., +
Σn - 2(E, ) where μ and denote the empirical mean and covariance, and subscripts x and g denote the real and generated data respectively. FID is sensitive both the addition of spurious modes and to mode dropping (Sajjadi et al., 2018; Lucic et al., 2018). On the Cityscapes dataset, ran five trials where computed FID 500 random synthetic images and 500 real validation images, and report the average score. On ADE-Indoor, this is repeated on batches of 433 images.
Implementation details In all our experiments, we set A = Ag 10, and X = 10. The initial generator and discriminator learning rates for training SPADE both in the pretraining and end-to-end steps are 10 4 and 4 10 respectively. The learning rate for the semantic bottleneck synthesis nub-network is set to 10-3 in the pretraining step and 10-5 in the end-to-end fine-tuning on

---

Under review as a conference paper at ICLR 2021
SI-CIAN
PoGAN
BiGGAN
Figure 5: Images synthesized on ADE-Indoor This dataset is very challenging, causing mode collapse for the BigGAN model (3rd row). In contrast, samples generated by SB-GAN (Ist row) are generally of higher quality and much more structured than those of ProGAN (2nd row).
Table I:
FID of the synthesized samples (lower is better), averaged over 5 random sets of samples. Images were synthesized resolution of X 2X on Cityscapes and x X on ADE-Indoor.
(a) X 256
(b) X = 128
ProGAN SB-GAN
SB-GAN
ProGAN BigGAN SB-GAN
W/O FT
CITYSCAPES-5K 92.57
83.20
65,49
178.19
57.48
CITYSCAPES-25K 63.87
71.13
62.97
56.7
64.82
54.92
ADE-INDOOR 104.83
91.80
85.27
85.94
156.65
81.39
Cityscapes, and to 10-4 for ADE-Indoor. The temperature hyperparameter, т. is always set to 1. For BigGAN, followed the setup by Lucic et al. (2019)', where modified the code to allow for non-square images of Cityscapes. We used one class label for all images to have an unconditional BigGAN model. For both datasets, we varied the batch size (using values in {128, 256, 512, 2048)), learning rate, and location of the self-attention block. We trained the final model for 50K iterations.
4.1 QUALITATIVE RESULTS
In Figures 3, 4, and 5, we provide qualitative comparisons of the competing methods on the three aforementioned datasets. We observe that both Cityscapes-5K and ADE-Indoor are very challeng- ing for the state-of-the the- ProGAN and BigGAN models, likely due to the complexity of the data and small number of training instances. Even at a resolution of 128 128 on the ADE-Indoor dataset, BigGAN suffers from mode collapse, illustrated in Figure 5. In contrast, SB-GAN sig- nificantly improves the structure of the scene distribution and provides samples of higher quality. On 2it5 the performance improvement of SB-GAN is more modest due the large number of training images available. is worth emphasizing that in this case only 3K ground truth segmentations available train SB-GAN. Compared BigGAN, images synthesized by SB-GAN are sharper and contain more structural details (e.g., one can zoom-in on the synthesized cars). Additional synthesized semantic layouts and images are illustrated in Figures 10 in the Appendix.
'Configuration as in https: htt.ttttbbb.//////////////...t com.b.c.o///ee http:ogg///opprraaaaa//allaeaeee exampl leroole_coofifio www.nggggggggggnnnnn./ iiagenimmaaaetttmm 128.gin
7

---

Under review as a conference paper at ICLR 2021
Table FID of the synthesized samples when conditioned on the ground truth labels. For SB-GAN, we train the entire model end-to- end and extract the trained SPADE.
Table 3: Average perceptual evaluation scores when each evaluators has selected a quality score in the range of (terrible quality) to (high quality) for each image.
SPADE
SB-GAN
ProGAN
BigGAN
SB-GAN
CITYSCAPES-5K
72.12
60.39
2,08
2.48
CUTYSCAPES-25K
60.83
54.13
2.53
2.27
2.61
ADE-INDOOR
50.30
48.15
2.35
1.96
2.49
4.2
QUANTITATIVE EVALUATION
To provide thorough empirical evaluation of the proposed approach, we generate samples for each dataset and report the FID scores of the resulting images (averaged across 5 sets of generated samples). We evaluate SB-GAN both before and after end-to-end fine-tuning, and compare our method to two strong baselines. ProGAN (Karras et al., 2017) and BigGAN (Brock et al., 2019).
The results are detailed in Tables Ia and lb. First, in the low-data regime, even without fine-tuning, our Semantic Bottleneck GAN produces higher quality samples and significantly outperforms the baselines on Cityscapes-5K and ADE-Indoor. The advantage our proposed method is even more striking smaller datasets. While competing methods are unable to learn high-quality model of underlying distribution without having access to a large number of samples, SB-GAN less sensitive to the number of training data points. Secondly, we observe that by jointly training the two components, SB-GAN produces stateoof-the-an results across all three datasets.
We were not able successfully train BigGAN resolution of 256 512 due to instability observed during training and mode collapse. Table lb shows the results for a lower-resolution setting, for which training collapses. BigGAN is, to a certain extent, able to capture the distribution of Cityscapes- 25K, but fails completely ADE-Indoor. Interestingly, BigGAN fails to capture the distribution of Cityscapes-5K even at 128 x 128 resolution.
able successfully train BigGAN. We report the results before the
Generating by conditioning on real segmentations To independently assess the impact of end- to-end training on the conditional image synthesis sub-network, we evaluate the quality of generated samples when conditioning isons to the baseline network SPADE (Park et al., 2019) are provided in Table and Figures 13 and 14 in the Appendix. We observe that the image synthesis component of SB-GAN consistently outperforms SPADE across all three datasets, indicating that fine-tuning on data sampled from the segmentation generator improves the conditional image generator.
ground truth validation segmentations from each dataset. Compar-
Fine-tuning ablation study different components of SB-GAN in the Appendix. In particular, we consider three settings: (I) SB- GAN before end-to-end fine-tuning, (2) fine-tuning only the semantic bottleneck synthesis compo- nent, (3) fine-tuning only the conditional image synthesis component, and (4) fine-tuning all jointly.
To dissect the effect of end-to-end training, we perform a study on
4.3 PERCEPTUAL EVALUATION
We used Amazon Mechanical Turk (AMT) to assess the performance each method on each dataset using ~600 pairs of (synthesized images, human evaluators) with total of 200 unique synthesized images. For each image, evaluators were asked to assign a score between to 4 to each image, indicating low-to-high quality images, respectively. The results summarized in Table 3 and are consistent with our FID-based evaluations.
CONCLUSION
We proposed an end-to-end Semantic Bottleneck GAN model that synthesizes semantic layouts from scratch, and then generates photo-realistic scenes conditioned the synthesized layouts. Through extensive quantitative and qualitative evaluations, we showed that this novel end-to-end training pipeline significantly outperforms the state-of-the-art models in unconditional synthesis of complex
8

---

Under review as a conference paper at ICLR 2021
scenes. In addition, Semantic Bottleneck GAN strongly improves the performance of the state-of- the-art semantic image synthesis model in synthesizing photo-realistic images from ground truth segmentations. As future work, one could explore novel ways to train GANs with discrete outputs, especially to deal with the non-differentiable nature of the generated outputs.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. 2017. 2
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In ICCV, 2019. 3
Samaneh Azadi, Matthew Fisher, Vladimir Kim, Zhaowen Wang, Eli Shechtman, and Trevor Dar- rell. Multi-content gan for few-shot font style transfer. CVPR, 2018.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrim- inator Rejection Sampling. arXiv preprint arXiv: 181000678, 2019a. 3
Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, and Trevor Darrell. Compositional gan: Learning image-conditional binary composition. arXiv preprint arXiv:1807.0756/ 2019b. 3
Aleksandar Bojchevski, Oleksandr Shehur, Daniel Zügner, and Stephan Günnemann. Netgan: Gen- erating graphs via random walks. arXiv preprint 3xXiv:880300816: 2018. 3
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. 2019. 3,5, 3, 8
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adver- sarial networks. In ICLR, 2019a. 2
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In CVPR, 2019b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS, 2016.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks, IEEE Transactions Pattern Analysis and Machine Intelligence, 2016.
2
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014. 1,2
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. In NeurlPS, 2017. 2, 4
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to Nash equilibrium. In NewrlPS, 2017. 6
Seunghoon Hong, Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In CVPR, 2018. 3, 11,
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imet to-iiageee translation with conditional adversarial networks. In CVPR, 2017. 2,

---

Under review as a conference paper at ICLR 2021
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In ICLR, 2017.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. CVPR, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2017. 2, 3, 4,
Matt J Kusner and José Miguel Hernández-Lobate Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.0405: 2016. 3
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. CVPR, 2017. 2
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019. 3, 11, 12
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. Adversarial ranking for language generation. In NeurlPS, 2017. 3
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NeurlPS, 2017. 3
Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. Neural text generation: past, present and beyond. 2018. 3
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs
Created Equal? A Large-scale Study. In NeurIPS, 2018. 6
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. ICML, 2019. 7
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2016. 4
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In JCML, 2017.
Taesung Park, Ming-Yr Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019. 1, 3, 4, 5,
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context en- coders: Feature learning by inpainting. In CVPR, 2016. 2
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In CVPR, 2019a.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In CVPR, 2019b.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 2
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where draw. In NewrIPS, 2016. 3
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. In NewrIPS, 2018. 6
10

---

Under review as a conference paper at ICLR 2021
Anna Volokitin, Ender Konukoglu, and Luc Van Gool. Decomposing image generation into layout prediction and conditional synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 372 -73, 2020. 3
Ting- Chuu Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018. 1, 3,5
Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adver- sarial networks. In European conference on computer vision, pp. 318-335. Springer, 2016. 2,
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang. Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine Fine-grained text to image generation with attentional generative adversarial net- works. In CVPR, 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016. 12
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017a. 6, 12
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017b. 3
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. 2.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019. 2
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 6
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- cycle-consistent adversarial networks. In /CCV, 2017. 2.
A APPENDIX
A.1 DESIGN DIFFERENCES OF SB-GAN AND TEXT-TO-IMAGE MODELS INFERRING SEMANTIC LAYOUT
Here, we clarify differences between our proposed model and the conditional text-to-image synthesis models inferring a semantic layout. As illustrated in Figure 6-(b), the text-to-image synthesis models proposed by Hong et al. (2018) and Li et al. (2019) decompose the synthesis task into multiple steps. Given the text description, semantic layout is constructed by generating object bounding boxes and refining each box by estimating object shapes. Then, an image is synthesized conditionally on the generated semantic layout from the first step. We note that the above approaches, opposed to ours, are conditional and require supervision in the form of textual descriptions. Secondly, they sequential in nature and synthesize masks of few objects (e.g. person, elephant), but not fully fine- tne-grained semantic map (e.g. missing sky, grass, etc.). In stark contrast, our approach unconditionally and progressively synthesizes the full semantie layout of the entire scene from noise input in an end-to-ent network design as shown in Figure 6-(a).
A.2 ADDITIONAL RESULTS
In Figures 7, 8, 9, and 10, we show additional synthetic results from our proposed SB-GAN model including both the synthesized segmentations and their corresponding synthesized images from the Cityscapes-2545 and ADE-Indoor datasets. As mentioned in the paper, on the Cityscapes-255 dataset, fine ground truth annotations only provided for the Cityscapes-5k subset. We extract
11

---

Under review as a conference paper at ICLR 2021
451
Input Sentence
Sequential Bouncing Box Generator
Biriary Mask Synthesis
2
2568012
Segmentation Leyout Synthesis
Conditional Image Synthesis
Conditional Image Syntheels
Hierarchical text-to-image synthesis in Hong et al, 2018, Li 2019.
unconditional synthesis of complex scenes
Figure 6: Architectural differences between our unconditional semantic bottleneck synthesis net- work and the conditional semantic layout synthesis network in Hong et al. (2018) and Li et al. (2019). (a) Schematic of our unconditional Semantic Bottleneck GAN. We progressively train an adversarial segmentation synthesis network generate realistic segmentation maps from scratch, then synthesize photo-realistic image using conditional image synthesis network. End-to-end coupling of these two components results state-of-the-art unconditional synthesis of complex scenes. For more detail about our conditional image synthesis network, one can refer to Section 3.2. (b) Schematic of the hierarchical text-to- toxt-to-iman toatiaggigeee synthesis models inferring semantic layout (Hong et al., 2018; Li et al., 2019). From an encoding of the input sentence, object bounding boxes generated sequentially using an auto-regressive decoder, and are refined by a synthesized binary shape mask in the next step. The final image is synthesized given the constructed semantic layout and the text description. Note that whereas (b) conditionally generates masks only for objects, our model (a) unconditionally generates segmentation maps for the entire scene.
the corresponding fine annotations for the rest of training images using the state-of-the-art segmen- tation model Yu al. (2017a); Yu & Koltun (2016) trained the training annotated samples from Cityscapes-5K.
Fine-tuning ablation study To further dissect the effect end-to-end training, we perform a study on different components of SB-GAN. In particular, consider three settings: (1) SB-GAN before end-to-end fine-tuning, (2) fine-tuning only the semantic bottleneck synthesis component, (3) fine-tuning only the conditional image synthesis component, and (4) fine-tuning all components jointly. The results on the Cityscapes-5K dataset (resolution 128 x 256) are reported Table 4. Finally, the impact of fine-tuning the quality of samples can be observed in Figures 11 and 12.
Table 4: Ablation study of various components of SB-GAN. We report FID scores of SB-GAN before fine-tuning, fine-tuning only the semantic bottleneck synthesis component, fine-tuning only the image synthesis component, and full end-to-end fine-tuning. Experiments performed on the Cityscapes-5K dataset at a resolution of 128 x 256.
No FT
FT SB
FT SPADE
FT Both
70.15
66.22
63.04
58.67
Generating by conditioning real segmentations Figures 13 and 14 present additional exam- ples illustrating the impact SB-GAN on improving the performance of SPADE (Park al., 2019), the state-of-the-an semantic image synthesis model ground truth segmentations. The third row
12

---

Under review as a conference paper at ICLR 2021
Synthesized Segmentations
Synthesized Images
Figure 7: Segmentations and their corresponding images synthesized by SB-GAN trained on the Cityscapes-255 25K dataset.
these two figures show examples the synthesized images conditioned on ground truth labels when the SPADE sub-network is extracted from trained SB-GAN model.
13

---

Under review as a conference paper at ICLR 2021
Synthesized Segmentations
Synthesized Images
Figure 8: Segmentations and their corresponding images synthesized by SB-GAN trained on the Cityscapes-255 dataset.
14

---

Under review as a conference paper at ICLR 2021
Synthesized Segmentations
Synthesized Images
Synthesizod Segmentations
Synthesized Images
Figure 9: Segmentations and their corresponding images synthesized by SB-GAN trained on the ADE-Indoor dataset.
15

---

Under review as a conference paper at ICLR 2021
Synthesized Segmentations
Synthesized Images
Synthesized Segmentations
Synthesized Images
Figure 10: Segmentations and their corresponding images synthesized by SB-GAN trained on the ADE-Indoor dataset.
16

---

Under review as a conference paper at ICLR 2021
Synthesized Sogmentations
Synthesized Images
SB-GAN wio FT
SB-GAN
SB-GAN wi FT
SR-GAN
Figure 11: The effect of tne-ttnnng on the baseline setup for the Cityscapes-25K dataset. We observe improvements in both the global structure of the segmentations and the performance of semantic image synthesis, resulting in images of higher quality.
Synthesized Segmentations
Synthesized Images
SB-GAN w/o FT
SB-GAN
SB-GAN w/o FT
SB-GAN
Figure 12: The effect of fine-tuning (FT) on the baseline setup for ADE-Indoor dataset. Analo- gously the results 255, observe improvements in both the global structure of the segmentations and the performance of semantic image synthesis.
17

---

Under review as a conference paper at ICLR 2021
Ground Truth Segmentation
SPADE
SB-GAN
Ground Truth Segmentation
SPADE
SB-GAN
Figure 13: The effect of SB-GAN improving the performance of the ffte-offtheear semantic image synthesis model (SPADE) on ground truth segmentations of Cityscapes-25K validation set. For SB-GAN, train the entire model end-to-end, extract the trained SPADE sub-network, and synthesize samples conditioned on the ground truth labels.
18

---

Under review as a conference paper at ICLR 2021
Ground Truth Segmentation
SPADE
SB-GAN
Ground Truth Segmentation
SPADE
SB-GAN
Figure 14: The effect of SB-GAN improving the performance of the tfte-of-the-an semantic image synthesis model (SPADE) on ground truth segmentations ADE-Indoor validation set. For SB-GAN, we train the entire model end-to-end, extract the trained SPADE sub-network, and syn- thesize samples conditioned on the ground truth labels.
19


Reviews:
[{"review_id": "aF-dixAE0H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests a new approach for unconditional generation of complex scenes. The approach performs generation in two steps: first a semantic map is generated from noise using a conventional generator architecture, then the semantic image is turner into an RGB image by SPADE translator.\n\nThe paper has several strengths. First, the idea is clear and may make sense (though this has not been shown convincingly). Furthermore, the paper is well written and has detailed related work review (though some important papers are missed). The results are also interesting.\n\nDespite the strenghts, I think that the paper may not be suitable for ICLR in the current form for the following reasons:\n\n1) Novelty. The idea of two stage generation of complex images with GANs has been proposed before in a well-known paper [Wang & Gupta, Generative Image Modeling Using Style and Structure Adversarial Networks, ECCV16] . There, the image of normals rather than semantic segmentation served as an intermediate (\"bottleneck\") representation, otherwise the idea is very similar. It is likely that the normal map may be a better intermediate representation since it is continuous-valued and does not need to deal with discretization issues. Overall, a comparison and a proper positioning w.r.t. [Wang&Gupta] is needed.\n\n2) Deficient comparisons with 1-step GANs. The authors for some reason chose Progressive GANs and BigGAN as the reference 1-stage GANs. This choice is totally unclear to me. StyleGAN v1 and v2 are improved versions of ProGAN and should have been tried instead. Given the use of SPADE (i.e. style based generator) in the authors' architecture, the comparison to StyleGAN would be all the more natural. I find it very likely that the result of StyleGAN can be very similar or better than the authors after similar amount of tuning, especially on Cityscapes 25K. I am therefore not convinced that the proposed idea is actually working.\n\n3) [Minor] There is a published CVPR workshop paper with a very similar idea https://openaccess.thecvf.com/content_CVPRW_2020/html/w23/Volokitin_Decomposing_Image_Generation_Into_Layout_Prediction_and_Conditional_Synthesis_CVPRW_2020_paper.html . The results are worse and some important differences exist. However, it does undermine the novelty. Still a CVPR-workshop paper may be missed by the community, so I weigh this issue as minor.\n\n4) The results are interesting, but they are not terrific. I wonder if the authors can scale their method to higher resolutions (which would be very useful for complex scenes), or if it would break down in some way. \n\n5) [Minor; suggestion not a criticism] There is an obvious use for what the authors are doing. The data they generate can be used to train semantic segmentation networks (essentially serving as dataset augmentation). I think having the evaluation of this aspect would be useful and would make the paper stronger.\n\n6) [Minor] The phrase \"In fact, due to the missing ordering relationships, generating smooth\nsegmentation maps cannot be enforced by smoothness among values of neighboring pixels.\" should be reformulated. A large body of work exist (generally associated with MRF/CRFs in computer vision; and also in statistical physics) on enforcing smoothness (in different senses) in discrete-valued maps, e.g. via Potts prior. \n\nTo sum up, the idea is clear and well described, but the paper does not convince me that the idea is working (improves over StyleGAN; can synthesize high-res images) and in particular that semantic bottleneck is working better than other bottlenecks (e.g. proposed by [Wang&Gupta]).", "extra_scores": {}, "extra_text": {}}, {"review_id": "QHz75c_RBj5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a novel approach for unconditioned image synthesis of complex scenes by intelligently coupling two major tasks; unconditional label generation and label-conditioned image synthesis. To overcome the limitation of failing to generate high-fidelity complex scenes using current GAN-based approaches, this method proposes to divide this into two parts: unconditional segmentation map synthesis network and conditional segmentation-to-image synthesis network. The former is based on the ProGAN with some modifications in losses to deal with discrete semantic labels, and the latter leverages the existing method (Park et al. 2019) based on SPADE residual blocks. Experimental results demonstrate superior performance on the complex scene synthesis. Additionally, the latter part for segmentation-to-image synthesis task also outperforms the existing method (Park et al. 2019) thanks to joint end-to-end training with the former using ProGAN.\n\n* Pros\n1) Decomposing the complex scene synthesis into two sub-tasks (segmentation map generation and segmentation-to-image synthesis) looks novel, also validating outstanding performance over SOTA.\n2) Segmentation-to-image synthesis is also boosted, thanks to joint end-to-end training with the unconditional segmentation map synthesis network.\n\n* Cons\n1) Though this paper is well-written, some parts need more details.\n- In Section 3.1, it would be nice to explain how to generate semantic segmentation maps progressively by referring to Figure 1.\n- Eq (3) uses loss functions from two sub-networks (semantic bottleneck synthesis network and semantic image synthesis network). \nWhy did you not use L_D_SPD and L_G_SPD in (2) for training the whole network with a pair of real RGB images and real segmentation maps. \nIt seems that L_1^VGG and L_1^Feat can also be used in (3).\n\n- Semantic bottleneck synthesis in Section 3.1 needs more explanations, e.g., how to convert real segmentation maps into probability maps, using argmax and soft argmax in forward and backward passes.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OdQgoGEcBw1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new paradigm for unconditional image synthesis with semantic layouts as the bottleneck. The presented approach is straightforward: we can first sample a semantic layout from a latent variable, and then perform image synthesis from this semantic layout. The proposed method is able to synthesize images that look more realistic than unsupervised image synthesis methods such as BigGAN.\n\nStrengths: - The idea of having some predefined intermediate representation (such as semantic layout) can help improve unconditional image synthesis. \n- The approach works well on the datasets with semantic layouts, including Cityscapes and ADE20K datasets.\n- The experiments are comprehensive and convincing.\n- The proposed end-to-end fine-tuning can improve the visual results.\n\nWeaknesses: \n- Apparently, there is a limitation of the proposed approach: it can only work on datasets with semantic layouts.\n- This overall idea of having semantic layouts as a bottleneck is simple. Technically, semantic bottle synthesis and semantic image synthesis are based on well-known models (ProGAN+WGAN, SPADE). The technical contributions may appear limited, but I appreciate the overall idea for unconditional image synthesis.\n- Why the perceptual evaluation of BigGAN in Table 3 for Cityscapes-5K is not available?", "extra_scores": {}, "extra_text": {}}, {"review_id": "6cSN0JQweg8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary. This paper considers a two-stage procedure of generation complex scenes (cityscapes or living rooms, i.e. without a central object): Firstly, a noise vector is mapped to a discrete semantic map (Gumbel softmax and straight-through estimator are applied). Secondly, the obtained segmentation map is translated to an RGB image using the SPADE architecture. Both networks are pretrained separately and then finetuned in an end-to-end manner.\n\n1. Decision. What I really like about this work is the attempt to dissect the scene generation into several steps. This may be of great interest in the case of datasets of limited size, as the authors showcase. What is more, this work tackles the problem of the generation of discrete semantic labels and has success.  However, the chosen intermediate representation is, to my mind, the main concern of the presented approach as ground-true semantic labels are not so easy to crowdsource. This makes the considered pipeline a bit limited in practical usage. A possible solution is to employ the off-the-shelf pretrained segmentation net (as in your Cityscapes-25k experiment): I believe the paper would be stronger with an additional experiment on more common datasets for scene generation (e.g., LSUN bedrooms), labeled with such a network. One more thing, I would recommend choosing a more recent and appealing baseline like StyleGAN instead of ProGAN. Summarizing, I tend to vote for accepting, though believe that the experiments could be more solid.\n\n1. Questions\n    1. Is the sampling step with the Gumbel-softmax trick crucial for the generation of semantic maps? Why the combination of simple softmax scores and straight-through estimator does not suffice?\n    1. How exactly do you downscale the ground-true segmentation maps to feed the discriminator at coarse scales of progressive learning?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper receives a mixed rating, with R3 rates the paper above the bar, R1 and R2 rates marginally above the bar, and R4 recommends rejection. The cited positive points include 1) decomposing image generation into first synthesizing segmentation masks and then converting segmentation masks to images, and 2) good results comparing to Progressive GAN and BigGAN. R4 raises several concerns, including the novelty concern and unconvincing experimental validation. After analyzing the papers, the reviews, and the rebuttal, the AC finds the arguments made by R4 more convincing. Decomposing image generation to a two-step approach has been illustrated in the prior work [Wang & Gupta ECCV 2016, Hong, Yang, Choi, Lee CVPR 2018]. The proposed method does not provide additional insights. The provided experimental results are not very convincing, either. As the proposed setting assuming the availability of segmentation masks, it is not surprising that it outperforms the unconditional baselines. Overall, the AC believes the paper does not have enough novelty to justify its acceptance and would recommend rejection of the paper.
Decision: Reject