You Only Need Adversarial Supervision for Semantic Image Synthesis

∗Equal contribution . Correspondence to { edgar.schoenfeld , vadim.sushko } @ bosch.com . 1 INTRODUCTION . Conditional generative adversarial networks ( GANs ) ( Mirza & Osindero , 2014 ) synthesize images conditioned on class labels ( Zhang et al. , 2019 ; Brock et al. , 2019 ) , text ( Reed et al. , 2016 ; Zhang et al. , 2018a ) , other images ( Isola et al. , 2017 ; Huang et al. , 2018 ) , or semantic label maps ( Wang et al. , 2018 ; Park et al. , 2019 ) . In this work , we focus on the latter , addressing semantic image synthesis . Semantic image synthesis enables rendering of realistic images from user-specified layouts , without the use of an intricate graphic engine . Therefore , its applications range widely from content creation and image editing to generating training data that needs to adhere to specific semantic requirements ( Wang et al. , 2018 ; Chen & Koltun , 2017 ) . Despite the recent progress on stabilizing GANs ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ; Zhang & Khoreva , 2019 ) and developing their architectures ( Zhang et al. , 2019 ; Karras et al. , 2019 ) , state-of-the-art GAN-based semantic image synthesis models ( Park et al. , 2019 ; Liu et al. , 2019 ) still greatly suffer from training instabilities and poor image quality when trained only with adversarial supervision ( see Fig . 1 ) . An established practice to overcome this issue is to employ a perceptual loss ( Wang et al. , 2018 ) to train the generator , in addition to the discriminator loss . The perceptual loss aims to match intermediate features of synthetic and real images , that are estimated via an external perception network . A popular choice for such a network is VGG ( Simonyan & Zisserman , 2015 ) , pre-trained on ImageNet ( Deng et al. , 2009 ) . Although the perceptual loss substantially improves the accuracy of previous methods , it comes with the computational overhead introduced by utilizing an extra network for training . Moreover , it usually dominates over the adversarial loss during training , which can have a negative impact on the diversity and quality of generated images , as we show in our experiments . Therefore , in this work we propose a novel , simplified model that achieves state-of-the-art results without requiring a perceptual loss . A fundamental question for GAN-based semantic image synthesis models is how to design the discriminator to efficiently utilize information from the given semantic label maps . Conventional methods ( Park et al. , 2019 ; Wang et al. , 2018 ; Liu et al. , 2019 ; Isola et al. , 2017 ) adopt a multi-scale classification network , taking the label map as input along with the image , and making a global image-level real/fake decision . Such a discriminator has limited representation power , as it is not incentivized to learn high-fidelity pixel-level details of the images and their precise alignment with the input semantic label maps . To mitigate this issue , we propose an alternative architecture for the discriminator , re-designing it as an encoder-decoder semantic segmentation network ( Ronneberger et al. , 2015 ) , and directly exploiting the given semantic label maps as ground truth via a ( N+1 ) -class cross-entropy loss ( see Fig . 3 ) . This new discriminator provides semantically-aware pixel-level feedback to the generator , partitioning the image into segments belonging to one of the N real semantic classes or the fake class . Enabled by the discriminator per-pixel response , we further introduce a LabelMix regularization , which fosters the discriminator to focus more on the semantic and structural differences of real and synthetic images . The proposed changes lead to a much stronger discriminator , that maintains a powerful semantic representation of objects , giving more meaningful feedback to the generator , and thus making the perceptual loss supervision superfluous ( see Fig . 1 ) . Next , we propose to enable multi-modal synthesis of the generator via 3D noise sampling . Previously , directly using 1D noise as input was not successful for semantic image synthesis , as the generator tended to mostly ignore it or synthesized images of poor quality ( Isola et al. , 2017 ; Wang et al. , 2018 ) . Thus , prior work ( Wang et al. , 2018 ; Park et al. , 2019 ) resorted to using an image encoder to produce multi-modal outputs . In this work , we propose a lighter solution . Empowered by our stronger discriminator , the generator can effectively synthesize different images by simply re-sampling a 3D noise tensor , which is used not only as the input but also combined with intermediate features via conditional normalization at every layer . Such noise is spatially sensitive , so we can re-sample it both globally ( channel-wise ) and locally ( pixel-wise ) , allowing to change not only the appearance of the whole scene , but also of specific semantic classes or any chosen areas ( see Fig . 2 ) . We call our model OASIS , as it needs only adversarial supervision for semantic image synthesis . In summary , our main contributions are : ( 1 ) We propose a novel segmentation-based discriminator architecture , that gives more powerful feedback to the generator and eliminates the necessity of the perceptual loss supervision . ( 2 ) We present a simple 3D noise sampling scheme , notably increasing the diversity of multi-modal synthesis and enabling complete or partial change of the generated image . ( 3 ) With the OASIS model , we achieve high quality results on the ADE20K , Cityscapes and COCO-stuff datasets , on average improving the state of the art by 6 FID and 5 mIoU points , while relying only on adversarial supervision . We show that images synthesized by OASIS exhibit much higher diversity and more closely follow the color and texture distributions of real images . Our code and pretrained models are available at https : //github.com/boschresearch/OASIS . 2 RELATED WORK . Semantic image synthesis . Pix2pix ( Isola et al. , 2017 ) first proposed to use conditional GANs ( Mirza & Osindero , 2014 ) for semantic image synthesis , adopting an encoder-decoder generator which takes semantic label maps as input , and employing a PatchGAN discriminator . Since then , various generator and discriminator modifications have been introduced ( Wang et al. , 2018 ; Park et al. , 2019 ; Liu et al. , 2019 ; Tang et al. , 2020c ; b ; Ntavelis et al. , 2020 ) . Besides GANs , Chen & Koltun ( 2017 ) proposed to use a cascaded refinement network ( CRN ) for high-resolution semantic image synthesis , and SIMS ( Qi et al. , 2018 ) extended it with a non-parametric component , serving as a memory bank of source material to assist the synthesis . Further , Li et al . ( 2019 ) employed implicit maximum likelihood estimation ( Li & Malik , 2018 ) to increase the variety of the CRN model . However , these approaches still underperform in comparison to state-of-the-art GAN models . Therefore , next we focus on the recent GAN architectures for semantic image synthesis . Discriminator architectures . Pix2pix ( Isola et al. , 2017 ) , Pix2pixHD ( Wang et al. , 2018 ) and SPADE ( Park et al. , 2019 ) all employed a multi-scale PatchGAN discriminator , that takes an image and its semantic label map as input . CC-FPSE ( Liu et al. , 2019 ) proposed a feature-pyramid discriminator , embedding both images and label maps into a joint feature map , and then consecutively upsampling it in order to classify it as real/fake at multiple scales . LGGAN ( Tang et al. , 2020c ) introduced a classification-based feature learning module to learn more discriminative and class-specific features . In this work , we propose to use a pixel-wise semantic segmentation network as a discriminator instead of multi-scale image classifiers as in the above approaches , and to directly exploit the semantic label maps for its supervision . Segmentation-based discriminators have been shown to improve semantic segmentation ( Souly et al. , 2017 ) and unconditional image synthesis ( Schönfeld et al. , 2020 ) , but to the best of our knowledge have not been explored for semantic image synthesis and our work is the first to apply adversarial semantic segmentation loss for this task . Generator architectures . Conventionally , the semantic label map is provided to the image generation pipeline via an encoder ( Isola et al. , 2017 ; Wang et al. , 2018 ; Tang et al. , 2020c ; b ; Ntavelis et al. , 2020 ) . However , it is shown to be suboptimal at preserving the semantic information until the later stages of image generation . Therefore , SPADE introduced a spatially-adaptive normalization layer that directly modulates the label map onto the generator ’ s hidden layer outputs at various scales . Alternatively , CC-FPSE proposed to use spatially-varying convolution kernels conditioned on the label map . Struggling with generating diverse images from noise , both Pix2pixHD and SPADE resorted to having an image encoder in the generator design to enable multi-modal synthesis . The generator then combines the extracted image style with the label map to reconstruct the original image . By alternating the style vector , one can generate multiple outputs conditioned on the same label map . However , using an image encoder is a resource demanding solution . In this work , we enable multi-modal synthesis directly through sampling of a 3D noise tensor injected at every layer of the network . Differently from structured noise injection of Alharbi & Wonka ( 2020 ) and class-specific latent codes of Zhu et al . ( 2020 ) , we inject the 3D noise along with label maps and adjust it to image resolution , also enabling re-sampling of selected semantic segments ( see Fig . 2 ) . Perceptual losses . Gatys et al . ( 2015 ) ; Gatys et al . ( 2016 ) ; Johnson et al . ( 2016 ) and Bruna et al . ( 2016 ) were pioneers at exploiting perceptual losses to produce high-quality images for superresolution and style transfer using convolutional networks . For semantic image synthesis , the VGGbased perceptual loss was first introduced by CRN , and later adopted by Pix2pixHD . Since then , it has become a default for training the generator ( Park et al. , 2019 ; Liu et al. , 2019 ; Tan et al. , 2020 ; Tang et al. , 2020a ) . As the perceptual loss is based on a VGG network pre-trained on ImageNet ( Deng et al. , 2009 ) , methods relying on it are constrained by the ImageNet domain and the representational power of VGG . With the recent progress on GAN training , e.g . by architecture designs and regularization techniques , the actual necessity of the perceptual loss requires a reassessment . We experimentally show that such loss imposes unnecessary constraints on the generator , significantly limiting sample diversity . While our model , trained without the VGG loss , achieves improved image diversity while not compromising image quality . 3 OASIS MODEL . In this section , we present our OASIS model , which , in contrast to other semantic image synthesis methods , needs only adversarial supervision for generator training . Using SPADE as a starting point ( Sec . 3.1 ) , we first propose to re-design the discriminator as a semantic segmentation network , directly using the given semantic label maps as ground truth ( Sec . 3.2 ) . Empowered by spatiallyand semantically-aware feedback of the new discriminator , we next re-design the SPADE generator , enabling its effective multi-modal synthesis via 3D noise sampling ( Sec . 3.3 ) .

Summary: In this paper, the authors approach the problem of conditional image generation via generative adversarial networks. To this end, they propose an approach that utilizes only semantic segmentation annotations and adversarial loss. No perceptual loss is required. Their discriminator leverages semantic labels to improve the image generations. They evaluate their approach on a variety of datasets including ADE20K, COCO, and CityScapes. They demonstrate substantial quantitative and qualitative performance over baselines and perform an ablation analysis.