Unconditional Synthesis of Complex Scenes Using a Semantic Bottleneck

1 INTRODUCTION . Significant strides have been made on generative models for image synthesis , with a variety of methods based on Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) achieving stateof-the-art performance . At lower resolutions or in specialized domains , GAN-based methods are able to synthesize samples which are near-indistinguishable from real samples ( Brock et al. , 2019 ) . However , generating complex , high-resolution scenes from scratch remains a challenging problem , as shown in Figure 1- ( a ) and ( b ) . As image resolution and complexity increase , the coherence of synthesized images decreases — samples lack consistent local or global structures . Stochastic decoder-based models , such as conditional GANs , were recently proposed to alleviate some of these issues . In particular , both Pix2PixHD ( Wang et al. , 2018 ) and SPADE ( Park et al. , 2019 ) are able to synthesize high-quality scenes using a strong conditioning mechanism based on semantic segmentation labels during the scene generation process . Global structure encoded in the segmentation layout of the scene is what allows these models to focus primarily on generating convincing local content consistent with that structure . A key practical drawback of such conditional models is that they require full segmentation layouts as input . Thus , unlike unconditional generative approaches which synthesize images from randomly sampled noise , these models are limited to generating images from a set of scenes that is prescribed in advance , typically either through segmentation labels from an existing dataset , or scenes that are hand-crafted by experts . Contributions To overcome these limitations , we propose a new model , the Semantic Bottleneck GAN ( SB-GAN ) , which couples high-fidelity generation capabilities of label-conditional models with the flexibility of unconditional image generation . This in turn enables our model to synthesize an unlimited number of novel complex scenes , while still maintaining high-fidelity output characteristic of image-conditional models . Our SB-GAN first unconditionally generates a pixel-wise semantic label map of a scene ( i.e . for each spatial location it outputs a class label ) , and then generates a realistic scene image by conditioning on that semantic map , Figure 1- ( d ) . By factorizing the task into these two steps , we are able to separately tackle the problems of producing convincing segmentation layouts ( i.e . a useful global structure ) and filling these layouts with convincing appearances ( i.e . local structure ) . When trained end-to-end , the model yields samples which have a coherent global structure as well as fine local details , e.g. , Figure 1- ( c ) . Empirical evaluation shows that our Semantic Bottleneck GAN achieves a new state-of-the-art on two complex datasets with relatively small number of training images , Cityscapes and ADE-Indoor , as measured both by the Fréchet Inception Distance ( FID ) and by perceptual evaluations . Additionally , we observe that the conditional segmentation-to-image synthesis component of our SB-GAN jointly trained with segmentation layout synthesis significantly improves the state-of-the-art semantic image synthesis network ( Park et al. , 2019 ) , resulting in higher-quality outputs when conditioning on ground truth segmentation layouts . Key Challenges While both unconditional generation and image-to-image translation are wellexplored learning problems , fully unconditional generation of the segmentation maps is a notoriously hard task : ( i ) Semantic categories do not respect any ordering relationships and the network is therefore required to capture the intricate relationship between segmentation classes , their shapes , and their spatial dependencies . ( ii ) As opposed to RGB values , semantic categories are discrete , hence non-differentiable which poses a challenge for end-to-end training ( Sec . 3.2 ) ( iii ) Naively combining state-of-the-art unconditional generation and image-to-image translation models leads to poor performance . However , by carefully designing an additional discriminator component and a corresponding training protocol , we not only manage to improve the performance of the end-to-end model , but also the performance of each component separately ( Sec . 3.3 ) . We emphasize that despite these challenges our approach scales to 256 × 256 resolution and 95 semantic categories , whereas existing state-of-the-art GAN models directly generating RGB images at that resolution already suffer from considerable instability ( Sec . 4 ) . 2 RELATED WORK . Generative Adversarial Networks ( GANs ) GANs ( Goodfellow et al. , 2014 ) are a powerful class of generative models successfully applied to various image synthesis tasks such as image style transfer ( Isola et al. , 2017 ; Zhu et al. , 2017 ) , unsupervised representation learning ( Chen et al. , 2016 ; Pathak et al. , 2016 ; Radford et al. , 2016 ) , image super-resolution ( Ledig et al. , 2017 ; Dong et al. , 2016 ) , and text-to-image synthesis ( Zhang et al. , 2017 ; Xu et al. , 2018 ; Qiao et al. , 2019b ) . Training GANs is notoriously hard and recent efforts focused on improving neural architectures ( Wang & Gupta , 2016 ; Karras et al. , 2017 ; Zhang et al. , 2019 ; Chen et al. , 2019a ) , loss functions ( Arjovsky et al. , 2017 ) , regularization ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , large-scale training ( Brock et al. , 2019 ) , self-supervision ( Chen et al. , 2019b ) , and sampling ( Brock et al. , 2019 ; Azadi et al. , 2019a ) . Improving the performance of GANs by disentangling structure and style has been studied by Wang & Gupta ( 2016 ) where structure is represented by a surface normal map and style is the texture mapped onto the structure . Another compelling approach which enables generation of high-resolution images is based on progressive training : a model is trained to first synthesize lowerresolution images ( e.g . 8× 8 ) , then the resolution is gradually increased until the desired resolution is achieved ( Karras et al. , 2017 ) . Recently , Brock et al . ( 2019 ) showed that GANs significantly benefit from large-scale training , both in terms of model size and batch size . We note that these models are able to synthesize high-quality images in settings where objects are very prominent and centrally placed or follow some well-defined structure , as the corresponding distribution is easier to capture . In contrast , when the scenes are more complex and the amount of data is limited , the task becomes extremely challenging for these state-of-the-art models . We aim to improve the performance in the context of complex scenes and a small number of training examples by disentangling the image generation problem into learning the structure represented by semantic layouts and filling in the RGB details using a semantic image synthesis model . A similar idea was proposed by a concurrent work ( Volokitin et al. , 2020 ) with substantial differences in the model and results . GANs on discrete domains GANs for discrete domains have been investigated in several works ( Yu et al. , 2017b ; Lin et al. , 2017 ; Bojchevski et al. , 2018 ; Lu et al. , 2018 ) . Training in this domain is even more challenging as the samples from discrete distributions are not differentiable with respect to the network parameters . This problem can be somewhat alleviated by using the Gumbel-softmax distribution , which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function ( Kusner & Hernández-Lobato , 2016 ) . We will show how to apply a similar principle to learn the distribution of discrete segmentation masks . Conditional image synthesis In conditional image synthesis one aims to generate images by conditioning on an input which can be provided in the form of an image ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Azadi et al. , 2018 ; 2019b ; Liu et al. , 2017 ) , a text phrase ( Reed et al. , 2016 ; Zhang et al. , 2017 ; Qiao et al. , 2019a ; Ashual & Wolf , 2019 ; Hong et al. , 2018 ) , a scene graph ( Johnson et al. , 2018 ; Ashual & Wolf , 2019 ) , a class label , or a semantic layout ( Odena et al. , 2017 ; Chen & Koltun , 2017 ; Wang et al. , 2018 ; Park et al. , 2019 ) . These conditional GAN methods learn a mapping that translates samples from the source distribution into samples from the target domain . The text-to-image synthesis models proposed in ( Hong et al. , 2018 ; Li et al. , 2019 ) decompose the synthesis task into multiple steps . As illustrated in the Appendix , given the text description , a semantic layout is constructed by generating object bounding boxes and refining each box by estimating object shapes . Then , an image is synthesized conditionally on the generated semantic layout from the first step . Our work shares the same high-level idea of decomposing the image generation problem into the semantic layout synthesis and the conditional semantic-layout-to-image synthesis . However , we note that the above approaches , as opposed to ours , are conditional and require supervision in the form of textual descriptions . Secondly , they are sequential in nature and synthesize masks of a few different objects ( e.g . person , elephant ) , but not a fully fine-grained semantic map ( e.g . missing sky , grass , etc. ) . In stark contrast , our approach unconditionally synthesizes the full semantic layout of the entire scene from a noise input in an end-to-end network design . Due to the above distinctions , their segmentation synthesis models differ significantly from ours in terms of architecture and design as shown in Figure 6 in the Appendix . 3 SEMANTIC BOTTLENECK GAN ( SB-GAN ) . We propose an unconditional Semantic Bottleneck GAN architecture to learn the distribution of complex scenes . To tackle the problems of learning both the global layout and the local structure , we divide this synthesis problem into two parts : an unconditional segmentation map synthesis network and a conditional segmentation-to-image synthesis model . Our first network is designed to coarsely learn the scene distribution by synthesizing semantic layouts . It generates per-pixel semantic categories following the progressive GAN model architecture ( ProGAN ) ( Karras et al. , 2017 ) . This fully unconditional generation of the segmentation maps is novel , very challenging , and a careful design is crucial , as described in Section 3.1 . The second network populates the synthesized semantic layouts with texture by predicting RGB pixel values using Spatially-Adaptive Normalization ( SPADE ) , following the architecture of the state-of-the-art semantic synthesis network in ( Park et al. , 2019 ) . We assume the ground truth segmentation masks are available for all or part of the target scene dataset . In the following sections , we will first discuss our semantic bottleneck synthesis pipeline and summarize the SPADE network for image synthesis . We will then couple these two networks in an end-to-end design which we refer to as Semantic Bottleneck GAN ( SB-GAN ) . 3.1 SEMANTIC BOTTLENECK SYNTHESIS . Our goal is to learn a ( coarse ) estimate of the scene distribution from samples corresponding to real segmentation maps with K semantic categories . Starting from random noise , we generate a tensor Y ∈ J1 , KKN×1×H×W which represents a per-pixel segmentation class , with H and W indicating the height and width , respectively , of the generated map and N the batch size . In practice , we progressively train from a low to a high resolution using the ProGAN architecture ( Karras et al. , 2017 ) coupled with the Improved WGAN loss function ( Gulrajani et al. , 2017 ) on the ground truth discrete-valued segmentation maps , illustrated in Figure 1- ( d ) . Similar to ProGAN , to increase the spatial resolution of the generated segmentation maps during training , we incrementally add layers to the generator and the discriminator . In contrast to ProGAN , in which the generator outputs continuous RGB values , we predict per-pixel discrete semantic class labels . This task is extremely challenging as it requires the network to capture the intricate relationship between segmentation classes and their spatial dependencies . To this end , we apply the Gumbel-softmax trick ( Jang et al. , 2017 ; Maddison et al. , 2016 ) coupled with a straight-through estimator ( Jang et al. , 2017 ) , described in detail below . We synthesize segmentation layouts by first generating per-pixel probability scores of belonging to each of the K semantic classes and then sampling a semantic class per pixel . The per-pixel probability scores are computed by applying a softmax function to the last layer of the generator ( i.e . logits ) which results in probability maps P ij ∈ [ 0 , 1 ] K , with ∑K k=1 P ij k = 1 for each spatial location ( i , j ) ∈ J1 , HK×J1 , W K. To sample a semantic class from this multinomial distribution , we would ideally apply the following well-known procedure at each spatial location : ( 1 ) sample k i.i.d . samples , Gk , from the standard Gumbel distribution , ( 2 ) add these samples to each logit , and ( 3 ) take the index of the maximal value . This reparametrization indeed allows for an efficient forward-pass , but is not differentiable . Nevertheless , the max can be replaced with the softmax function and the quality of the approximation can be controlled by varying the temperature hyperparameter τ — the smaller the τ , the closer the approximation is to the categorical distribution ( Jang et al. , 2017 ) : Sijk = exp { ( logP ijk +Gk ) /τ } ∑K i=1 exp { ( logP ij i +Gi ) /τ } . ( 1 ) Similar to the real samples , the synthesized samples fed to the GAN discriminator should still contain discrete category labels . As a result , for the forward pass , we compute arg maxk Sk , while for the backward pass , we use the soft predicted scores Sk directly , a strategy known as straight-through estimation ( Jang et al. , 2017 ) .

Summary: This work proposes a novel approach for unconditioned image synthesis of complex scenes by intelligently coupling two major tasks; unconditional label generation and label-conditioned image synthesis. To overcome the limitation of failing to generate high-fidelity complex scenes using current GAN-based approaches, this method proposes to divide this into two parts: unconditional segmentation map synthesis network and conditional segmentation-to-image synthesis network. The former is based on the ProGAN with some modifications in losses to deal with discrete semantic labels, and the latter leverages the existing method (Park et al. 2019) based on SPADE residual blocks. Experimental results demonstrate superior performance on the complex scene synthesis. Additionally, the latter part for segmentation-to-image synthesis task also outperforms the existing method (Park et al. 2019) thanks to joint end-to-end training with the former using ProGAN.