Unconditional Synthesis of Complex Scenes Using a Semantic Bottleneck

1 INTRODUCTION . Significant strides have been made on generative models for image synthesis , with a variety of methods based on Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) achieving stateof-the-art performance . At lower resolutions or in specialized domains , GAN-based methods are able to synthesize samples which are near-indistinguishable from real samples ( Brock et al. , 2019 ) . However , generating complex , high-resolution scenes from scratch remains a challenging problem , as shown in Figure 1- ( a ) and ( b ) . As image resolution and complexity increase , the coherence of synthesized images decreases — samples lack consistent local or global structures . Stochastic decoder-based models , such as conditional GANs , were recently proposed to alleviate some of these issues . In particular , both Pix2PixHD ( Wang et al. , 2018 ) and SPADE ( Park et al. , 2019 ) are able to synthesize high-quality scenes using a strong conditioning mechanism based on semantic segmentation labels during the scene generation process . Global structure encoded in the segmentation layout of the scene is what allows these models to focus primarily on generating convincing local content consistent with that structure . A key practical drawback of such conditional models is that they require full segmentation layouts as input . Thus , unlike unconditional generative approaches which synthesize images from randomly sampled noise , these models are limited to generating images from a set of scenes that is prescribed in advance , typically either through segmentation labels from an existing dataset , or scenes that are hand-crafted by experts . Contributions To overcome these limitations , we propose a new model , the Semantic Bottleneck GAN ( SB-GAN ) , which couples high-fidelity generation capabilities of label-conditional models with the flexibility of unconditional image generation . This in turn enables our model to synthesize an unlimited number of novel complex scenes , while still maintaining high-fidelity output characteristic of image-conditional models . Our SB-GAN first unconditionally generates a pixel-wise semantic label map of a scene ( i.e . for each spatial location it outputs a class label ) , and then generates a realistic scene image by conditioning on that semantic map , Figure 1- ( d ) . By factorizing the task into these two steps , we are able to separately tackle the problems of producing convincing segmentation layouts ( i.e . a useful global structure ) and filling these layouts with convincing appearances ( i.e . local structure ) . When trained end-to-end , the model yields samples which have a coherent global structure as well as fine local details , e.g. , Figure 1- ( c ) . Empirical evaluation shows that our Semantic Bottleneck GAN achieves a new state-of-the-art on two complex datasets with relatively small number of training images , Cityscapes and ADE-Indoor , as measured both by the Fréchet Inception Distance ( FID ) and by perceptual evaluations . Additionally , we observe that the conditional segmentation-to-image synthesis component of our SB-GAN jointly trained with segmentation layout synthesis significantly improves the state-of-the-art semantic image synthesis network ( Park et al. , 2019 ) , resulting in higher-quality outputs when conditioning on ground truth segmentation layouts . Key Challenges While both unconditional generation and image-to-image translation are wellexplored learning problems , fully unconditional generation of the segmentation maps is a notoriously hard task : ( i ) Semantic categories do not respect any ordering relationships and the network is therefore required to capture the intricate relationship between segmentation classes , their shapes , and their spatial dependencies . ( ii ) As opposed to RGB values , semantic categories are discrete , hence non-differentiable which poses a challenge for end-to-end training ( Sec . 3.2 ) ( iii ) Naively combining state-of-the-art unconditional generation and image-to-image translation models leads to poor performance . However , by carefully designing an additional discriminator component and a corresponding training protocol , we not only manage to improve the performance of the end-to-end model , but also the performance of each component separately ( Sec . 3.3 ) . We emphasize that despite these challenges our approach scales to 256 × 256 resolution and 95 semantic categories , whereas existing state-of-the-art GAN models directly generating RGB images at that resolution already suffer from considerable instability ( Sec . 4 ) . 2 RELATED WORK . Generative Adversarial Networks ( GANs ) GANs ( Goodfellow et al. , 2014 ) are a powerful class of generative models successfully applied to various image synthesis tasks such as image style transfer ( Isola et al. , 2017 ; Zhu et al. , 2017 ) , unsupervised representation learning ( Chen et al. , 2016 ; Pathak et al. , 2016 ; Radford et al. , 2016 ) , image super-resolution ( Ledig et al. , 2017 ; Dong et al. , 2016 ) , and text-to-image synthesis ( Zhang et al. , 2017 ; Xu et al. , 2018 ; Qiao et al. , 2019b ) . Training GANs is notoriously hard and recent efforts focused on improving neural architectures ( Wang & Gupta , 2016 ; Karras et al. , 2017 ; Zhang et al. , 2019 ; Chen et al. , 2019a ) , loss functions ( Arjovsky et al. , 2017 ) , regularization ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , large-scale training ( Brock et al. , 2019 ) , self-supervision ( Chen et al. , 2019b ) , and sampling ( Brock et al. , 2019 ; Azadi et al. , 2019a ) . Improving the performance of GANs by disentangling structure and style has been studied by Wang & Gupta ( 2016 ) where structure is represented by a surface normal map and style is the texture mapped onto the structure . Another compelling approach which enables generation of high-resolution images is based on progressive training : a model is trained to first synthesize lowerresolution images ( e.g . 8× 8 ) , then the resolution is gradually increased until the desired resolution is achieved ( Karras et al. , 2017 ) . Recently , Brock et al . ( 2019 ) showed that GANs significantly benefit from large-scale training , both in terms of model size and batch size . We note that these models are able to synthesize high-quality images in settings where objects are very prominent and centrally placed or follow some well-defined structure , as the corresponding distribution is easier to capture . In contrast , when the scenes are more complex and the amount of data is limited , the task becomes extremely challenging for these state-of-the-art models . We aim to improve the performance in the context of complex scenes and a small number of training examples by disentangling the image generation problem into learning the structure represented by semantic layouts and filling in the RGB details using a semantic image synthesis model . A similar idea was proposed by a concurrent work ( Volokitin et al. , 2020 ) with substantial differences in the model and results . GANs on discrete domains GANs for discrete domains have been investigated in several works ( Yu et al. , 2017b ; Lin et al. , 2017 ; Bojchevski et al. , 2018 ; Lu et al. , 2018 ) . Training in this domain is even more challenging as the samples from discrete distributions are not differentiable with respect to the network parameters . This problem can be somewhat alleviated by using the Gumbel-softmax distribution , which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function ( Kusner & Hernández-Lobato , 2016 ) . We will show how to apply a similar principle to learn the distribution of discrete segmentation masks . Conditional image synthesis In conditional image synthesis one aims to generate images by conditioning on an input which can be provided in the form of an image ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Azadi et al. , 2018 ; 2019b ; Liu et al. , 2017 ) , a text phrase ( Reed et al. , 2016 ; Zhang et al. , 2017 ; Qiao et al. , 2019a ; Ashual & Wolf , 2019 ; Hong et al. , 2018 ) , a scene graph ( Johnson et al. , 2018 ; Ashual & Wolf , 2019 ) , a class label , or a semantic layout ( Odena et al. , 2017 ; Chen & Koltun , 2017 ; Wang et al. , 2018 ; Park et al. , 2019 ) . These conditional GAN methods learn a mapping that translates samples from the source distribution into samples from the target domain . The text-to-image synthesis models proposed in ( Hong et al. , 2018 ; Li et al. , 2019 ) decompose the synthesis task into multiple steps . As illustrated in the Appendix , given the text description , a semantic layout is constructed by generating object bounding boxes and refining each box by estimating object shapes . Then , an image is synthesized conditionally on the generated semantic layout from the first step . Our work shares the same high-level idea of decomposing the image generation problem into the semantic layout synthesis and the conditional semantic-layout-to-image synthesis . However , we note that the above approaches , as opposed to ours , are conditional and require supervision in the form of textual descriptions . Secondly , they are sequential in nature and synthesize masks of a few different objects ( e.g . person , elephant ) , but not a fully fine-grained semantic map ( e.g . missing sky , grass , etc. ) . In stark contrast , our approach unconditionally synthesizes the full semantic layout of the entire scene from a noise input in an end-to-end network design . Due to the above distinctions , their segmentation synthesis models differ significantly from ours in terms of architecture and design as shown in Figure 6 in the Appendix . 3 SEMANTIC BOTTLENECK GAN ( SB-GAN ) . We propose an unconditional Semantic Bottleneck GAN architecture to learn the distribution of complex scenes . To tackle the problems of learning both the global layout and the local structure , we divide this synthesis problem into two parts : an unconditional segmentation map synthesis network and a conditional segmentation-to-image synthesis model . Our first network is designed to coarsely learn the scene distribution by synthesizing semantic layouts . It generates per-pixel semantic categories following the progressive GAN model architecture ( ProGAN ) ( Karras et al. , 2017 ) . This fully unconditional generation of the segmentation maps is novel , very challenging , and a careful design is crucial , as described in Section 3.1 . The second network populates the synthesized semantic layouts with texture by predicting RGB pixel values using Spatially-Adaptive Normalization ( SPADE ) , following the architecture of the state-of-the-art semantic synthesis network in ( Park et al. , 2019 ) . We assume the ground truth segmentation masks are available for all or part of the target scene dataset . In the following sections , we will first discuss our semantic bottleneck synthesis pipeline and summarize the SPADE network for image synthesis . We will then couple these two networks in an end-to-end design which we refer to as Semantic Bottleneck GAN ( SB-GAN ) . 3.1 SEMANTIC BOTTLENECK SYNTHESIS . Our goal is to learn a ( coarse ) estimate of the scene distribution from samples corresponding to real segmentation maps with K semantic categories . Starting from random noise , we generate a tensor Y ∈ J1 , KKN×1×H×W which represents a per-pixel segmentation class , with H and W indicating the height and width , respectively , of the generated map and N the batch size . In practice , we progressively train from a low to a high resolution using the ProGAN architecture ( Karras et al. , 2017 ) coupled with the Improved WGAN loss function ( Gulrajani et al. , 2017 ) on the ground truth discrete-valued segmentation maps , illustrated in Figure 1- ( d ) . Similar to ProGAN , to increase the spatial resolution of the generated segmentation maps during training , we incrementally add layers to the generator and the discriminator . In contrast to ProGAN , in which the generator outputs continuous RGB values , we predict per-pixel discrete semantic class labels . This task is extremely challenging as it requires the network to capture the intricate relationship between segmentation classes and their spatial dependencies . To this end , we apply the Gumbel-softmax trick ( Jang et al. , 2017 ; Maddison et al. , 2016 ) coupled with a straight-through estimator ( Jang et al. , 2017 ) , described in detail below . We synthesize segmentation layouts by first generating per-pixel probability scores of belonging to each of the K semantic classes and then sampling a semantic class per pixel . The per-pixel probability scores are computed by applying a softmax function to the last layer of the generator ( i.e . logits ) which results in probability maps P ij ∈ [ 0 , 1 ] K , with ∑K k=1 P ij k = 1 for each spatial location ( i , j ) ∈ J1 , HK×J1 , W K. To sample a semantic class from this multinomial distribution , we would ideally apply the following well-known procedure at each spatial location : ( 1 ) sample k i.i.d . samples , Gk , from the standard Gumbel distribution , ( 2 ) add these samples to each logit , and ( 3 ) take the index of the maximal value . This reparametrization indeed allows for an efficient forward-pass , but is not differentiable . Nevertheless , the max can be replaced with the softmax function and the quality of the approximation can be controlled by varying the temperature hyperparameter τ — the smaller the τ , the closer the approximation is to the categorical distribution ( Jang et al. , 2017 ) : Sijk = exp { ( logP ijk +Gk ) /τ } ∑K i=1 exp { ( logP ij i +Gi ) /τ } . ( 1 ) Similar to the real samples , the synthesized samples fed to the GAN discriminator should still contain discrete category labels . As a result , for the forward pass , we compute arg maxk Sk , while for the backward pass , we use the soft predicted scores Sk directly , a strategy known as straight-through estimation ( Jang et al. , 2017 ) .

Summary: 1. Summary. This paper considers a two-stage procedure of generation complex scenes (cityscapes or living rooms, i.e. without a central object): Firstly, a noise vector is mapped to a discrete semantic map (Gumbel softmax and straight-through estimator are applied). Secondly, the obtained segmentation map is translated to an RGB image using the SPADE architecture. Both networks are pretrained separately and then finetuned in an end-to-end manner.